Editor's Note

When have we taken the art of testing too far? Perhaps when we start testing our tests? Or perhaps when we start testing the tests of our tests?

PIT, which has just released version 0.24, is running inside one of these time testing paradoxen (plural of paradox?), testing the tests by creating mutations, and in a fashion of violence that is somewhat unfamiliar in the Java community, attempting to kill those mutations. "PIT is a fast bytecode based mutation testing system for Java that makes it possible to test the effectiveness of your unit tests. You can think of mutation testing as either as an automated test of your tests, or as a much more in depth form of code coverage." -http://pitest.org/

Anyways, for those masochists that are into this type of thing, PIT .24 is out, and it now works with all of those popular mocking frameworks out there, such as Mockito, JMock, EasyMock, PowerMock, JMockit. Check out the release notes below:

End of Editor's Note

******************

PIT 0.24 has been released and is available now from google code and maven central. A major improvement made in this release is the addition of support for JMockit. This makes PIT the only mutation testing system to be compatible with all the main mocking frameworks (Mockito, JMock, EasyMock, PowerMock and JMockit),

See

http://pitest.org/posts/2011/12/05/release_0.24/

For further details of the release.

Edited by: Cameron McKenzie on Dec 5, 2011 4:00 PM Edited by: Cameron McKenzie on Dec 5, 2011 6:08 PM Edited by: Cameron McKenzie on Dec 5, 2011 6:09 PM

I have to say the edits and additions to my post aren't entirely in the spirit of the original submission.

Mutation testing isn't quite the madness it might at first seem - it is basically just another form of code coverage, but one that avoids the principle disadvantage of line and branch coverage which don't actually measure how much of your code is tested. They measure only how much is executed by your tests. i.e they can identify code that defintely isn't tested, but never give a measure of code that definitely is.

Mutation testing solves this, but with an issue of it's own - high computational cost. PIT is an attempt to solve this issue and other usability issues that have prevented the widespread use of earlier tools such as jester and jumble.

There's plenty of research into mutation testing - the idea has been around since the 70s. An overview of much of the literaure is available here

http://www.dcs.kcl.ac.uk/pg/jiayue/repository/

 

 

 

 

The title is wrong, it should be "Testing the tests using PIT".

I used PIT a few times and I think it is great. I use it to verify my utility methods (static methods) are really tested well enough, because code coverage only shows half of the picuture. I plan to use it more in the future.

About 15 years ago, most developers didn't know what "unit testing" is, and now it's very popular. In the last 5 years "code coverage" is getting more and more popular (at least where I work).

Mutation testing could very well be a small silver bullet ("The Mythical Man-Month"). Unlike writing the code and writing the tests, both of which can't be automated, mutation testing *can* be automated. It helps to understand what part of the code still probably contains bugs, and mutation testing is a lot better in this area than code coverage alone.

The guy who edited the article should remember the first articles about "steam locomotives", "automobiles", and "flying machines" were probably edited in a similar way. "This telephone has too many shortcomings to be seriously considered as a means of communication. The device is inherently of no value to us."

It's a big turn-off for me as a reader when the editor gets opinionated.

As far as the concept of testing the tests is concerned, in my opinion we desperately need to test the tests. I fear that the only way to do it right now is code coverage and code reviews. Mutation testing is a welcome new idea. I just hope we get a desirable implementation soon and that I start using it. I have not used PIT but I am glad it exists.

It took me a little while to get my head around this but once I understood it, I think this could have great value to my team.  This can be a great time saver over trying to review every test that is created.  It should, in theory, help us identify areas to focus on.

On a side note, I think a better description might help.  Maybe:

"PIT runs your unit tests against automatically modified versions of your application code.  When the application code changes, it should produce different results and cause the unit tests to fail.  If a unit test does not fail in this situation, it may indicate an issue with the test."

Thanks for all the comments everyone.

It looks like I need to do a bit of work on the site making it clearer what PIT and mutation testing are all about and the benefits. (James if you don't mind I might steal/adapt your snippet).

Thomas is right - it is embarassingly recently that I hadn't even heard of code coverage tools, let alone started using them. For me they are now indispensable. I think I might stop short of claiming mutation testing as the next automobile or telephone, but I think it certainly has the potential for widespread adoption.

Interestingly now that I'm googling around for mutation testing quite regularly I'm seeing a lot of new tools being produced - over the last six months new tools have been announced for PHP, C# and (I think) Ruby.

According to the paper below there has also been a marked increase in academic papers over the last few years

http://www.dcs.kcl.ac.uk/pg/jiayue/repository/TR-09-06.pdf

So I suspect mutation testing is already starting to gain some traction.

 

 

 

 

It looks like I need to do a bit of work on the site making it clearer what PIT and mutation testing are all about and the benefits. (James if you don't mind I might steal/adapt your snippet).

Absolutely.  I look forward to seeing how this tool works in practice.

 

 

 

 

It looks like I need to do a bit of work on the site making it clearer what PIT and mutation testing are all about and the benefits. (James if you don't mind I might steal/adapt your snippet).

Absolutely.  I look forward to seeing how this tool works in practice.

 

 

 

 

It looks like I need to do a bit of work on the site making it clearer what PIT and mutation testing are all about and the benefits. (James if you don't mind I might steal/adapt your snippet).

Absolutely.  I look forward to seeing how this tool works in practice.