First Glance At PIT Mutation Testing Tool
Long time ago I got attracted to mutation testing (http://en.wikipedia.org/wiki/Mutation_testing). Alas, at that time all the Java mutation testing tools were useless: abandoned and forgotten long time ago. Lately I discovered a new tool - PIT Mutation Testing (http://pitest.org/) - which is actively developed. So I gave it a try. A very limited try (but that is only the beginning!).
The Idea
Does "mutation testing" rings any bells? No? I'm not surprised, so let me briefly describe the idea (there is much more to discuss, but let us keep it short).
Suppose you have some classes and a suite of tests. Now imagine you introduce a change into one of your classes, for example by reverting (negating) one of if conditional expressions. This way you created a mutant of your code. What should happen now if you run all tests once again? Provided that the suite of tests contains a test that examines this class thoroughly, then this test should fail. If no test fails, this means your test suite is not good enough. And that is exactly the idea of mutational testing.
As you can see mutation testing goes much further than code coverage (which is frequently used as a metric of tests quality, which, BTW, is a silly thing to do).
PIT Tryout
To see for myself what PIT can do, I created a toy-project (https://github.com/tomekkaczanowski/pit-testng). When you run it (see README on github), you get the results as presented on the picture below:
You will find more on mutation testing in my book
"Practical Unit Testing
with TestNG and Mockito"
As you can see the report, generated automatically by PIT, presents which mutants of the original class survived when run against my tests.
Conclusions
- PIT is still very young, but already impressive and actively developed
- I haven't tested it yet on any codebase of reasonable size - only on a toy-project
- PIT works with JUnit (its primary testing framework), but recently support for TestNG was added (Yupi-yey!)
Ok, that would be it for now. I'm really excited about PIT, and its further development. I contacted Henry Coles - author of PIT - and hope to somehow help him (or at least encourage him to keep going!). If you also see some value in mutation testing, give PIT a try and join the community.
Stay tuned, I will surely blog some more on PIT.
Mutants
Since this blog post is to large extents devoted to mutants, I couldn't help myself but to put these two pictures. :)
 |
 |
This used to be my blog. I moved to http://tomek.kaczanowscy.pl long time ago.
Codebase size
Hi Tomek,
I'm pleased so say that PIT is being used successfully on some codebases a little larger than this one - notably Apache Jackrabbit, as well as a number of commercial ones.
The next release (0.25) will be largely about supporting TestNG and improving the experience for maven users (the later thanks largely to input from the author of another project you might be interested in http://jumi.fi/ who is using PIT as part of his development).
After that there are lots of things to do . . .
http://pitest.org/roadmap/
expecting a lot!
Hi Henry,
it is great to hear that PIT is already able to work with normal-sized project. What I think is missing, is a good example that people could see and follow. For example, if you write a short information on how to check Apache Jackrabbit with PIT, then that could attract some attention (I guess this is only a matter of adding few things to pom.xml, isn't it?).
--
Cheers,
Tomek Kaczanowski
Hi Tomek, I've not tried it
Hi Tomek,
I've not tried it myself, but if you checkout the Jackrabbit code it should be already configured to use PIT.
I'm hoping I might be able to persuade Thomas Mueller (author of the H2 database and Jackrabbit commiter) to write up an account of how the Jackrabbit team have been using PIT at some point in the future.