PIT - development progress - interview February 2013

Mutation testing - once a academic toy and today a new trend in testing. Let us hear what Henry Coles, author of the PIT Mutation Testing tool, has to say about its development.

Basic Information

  • Name: PIT Mutation Testing
  • Website: http://pitest.org/
  • License: Apache License 2.0
  • Inception Year: 2011
  • Author: Henry Coles

Interview

Hi Henry, could you please introduce PIT in few words?

PIT is a mutation testing system, that is to say it's a program that deliberately inserts bugs into your code, then sees if your test suite can find them. In this way it is possible to find out if your test suite is any good at finding bugs or not.

Mutation testing has been around since decades but it has never gained a lot of attention. What made you invest your time into such (supposedly) academic area?

Mutation testing is generating a lot more attention than it was, both in and out of academia. Aside from PIT there have been new opensource tools released recently for Ruby (mutant https://github.com/mbj/mutant), .Net (Ninja Turtles http://www.mutation-testing.net/), PHP (mutagenisis https://github.com/padraic/mutagenesis), Python (MutPy https://bitbucket.org/khalas/mutpy/) and the number of papers being published has increased sharply (see https://www.cs.unm.edu/~forrest/classes/readings/tse-mutation-testing.pdf)

So I think it's an idea who's time has arrived.

As for why I invest my time in it, the short answer is because its interesting. It's a chance to solve a very different set of problems from the ones I work with in my day job, and use a different set of technology and techniques. It's also brought me in contact with people round the world I wouldn't otherwise have met.

The other short answer is because I wanted a tool I could use at work, and none of the existing ones looked to be fit for purpose. They were unworkably slow and didn't work with modern development stacks - fixing that proved very rewarding.

What are your development plans? What features is PIT missing?

I'm quite excited about a new feature that will ship with 0.30 - integrating PIT with version control. This will make it easy to mutation test only code that has changed - which provides an effective route to introduce mutation testing to any codebase, regardless of it's size and how poor the existing coverage is. You can commit new code and changes with a knowledge that you have effective test coverage for that change.

The incremental analysis feature introduced in 0.29 isn't yet complete - it's clunky to use and there are a number of optimizations still to implement that will make it much faster so that needs some work.
After that, a likely area of focus is the infinite loop detection. PIT currently deals with this in the same way as all the other systems, but I think it may be possible to change the approach and achieve some significant overall performance improvements.

A key missing feature is a stage to check that tests within a suite are fully repeatable/isolated before starting the mutation analysis - I'm not sure if this will end up being a separate test-lint tool, or being part of PIT itself.

A while ago I had a play with idea of delta execution. I managed to make things much slower rather than faster, but I haven't quite given up on the idea and might take a second look.

I'd also like to look at mutating other JVM languages. Although I'd like to tackle Scala - I think this would need to be mutated via a compiler plugin rather than via the byte code, which would require a huge change to PIT's design, so Groovy would be the most likely candidate. I don't actually speak Groovy however, so I'd have to fix that first.

Finally at some point I'm going to have to have a think about strategies for dealing with equivalent mutations. In my experience they are much less of an issue than the academic literature suggests, however they do happen and PIT doesn't currently provide any help to the user the track or assess them.

How does PIT differs from other mutation testing tools?

There are two major differences.
The first is performance. PIT is much faster than the earlier mutation testing systems and scales better. This is largely the result of the way PIT picks tests to run against each mutant. Before PIT runs a mutation analysis, it first collects line coverage data for each test. When it is assesses a mutant it will then only run tests that actually hit the affected line.
The earlier systems either ran all the tests all the time, or pushed work onto the programmer to get them to somehow indicate which tests should be run.

It's easy to see the difference this can make. In the extreme case, if a mutant was on a line not covered by any test, earlier systems would run *all* the tests, PIT would run *none* and instantly mark the mutant as surviving.

The other difference is that PIT's goal is to be a tool that is usable in real world software development, and it is actively maintained. Most of the other tools main focus is to provide a tool to support academic research - this means they have little interest in fixing issues or adding features that don't further academic research (e.g making them work Powermock, integrating them with maven supporting TestNG etc).

How would you encourage people to use PIT? Any success stories you could share?

I know from the mails I receive that PIT is being used all over the world in a variety of different companies, however most people seem to be a bit shy when it comes to talking about it.

I'm hoping to get a write up soon from a team at a large communications and media company who have been using PIT for some time, and Kyle and Sean a theladders.com have put however put together a nice writeup of their experience (see http://dev.theladders.com/2013/02/mutation-testing-with-pit-a-step-beyon...)

I think most usage of mutation testing is much more modest than the story above - individuals or teams within larger organizations using it as part of their feedback loop during development, or just bringing it out occasionally before to asses the risk of a particular refactoring or when taking delivery of code from a source they don't entirely trust.

Is PIT supported by development tools?

Out of the box, PIT provides a maven plugin, ant task and command line tool. There's also a growing eco-system building around PIT that provides integration with other development tools such as Gradle, Sonar, Eclipse and IntelliJ (http://pitest.org/links/)

Most of these are still immature, but will hopefully become more robust and feature rich over time.

What tools do you use when working with PIT?

Cloudbees kindly provide a CI server which runs the tests, findbugs and cobertura (not sure why as I prefer jacoco).

UCDetector has been deployed against the codebase many times after I've moved code around.

There was a period where I was using sonar but this was a hassle on a laptop and I haven't found a free sonar host (pit is on sonarsource's nemo instance, but it isn't much help as a feedback mechanism as it isn't clear what triggers a scan and I don't control the rules).

After the next release I'm going to apply a bit of structure to the packages using a tool I've been hacking out (see https://github.com/hcoles/highwheel)

Thank you Henry and good luck with mutants! ;)

Link to mutant has been moved

The link to the mutant project is actually pointing to the original one, but development has moved to https://github.com/mbj/mutant. There's a note in the README of the old project pointing to the new project too.

Thank you, fixed!

Thank you, fixed!

Please comment using