News: Article: Find software bugs, defects using code coverage

  1. In an article published on SearchSoftwareQuality.com, Joe Ponczak and John Miller question common beliefs on code coverage. In the article they assert that even with 100% code coverage, one cannot be assured that their application is devoid of critical bugs. The meat of the article starts with a description of code coverage and then expands on this with a definition of statement and branch coverage. With this foundation, Joe and John explain how these two different types of coverage are inadequate.
    A branch is the outcome of a decision, so branch coverage simply measures which decision outcomes have been tested. This sounds great because it takes a more in-depth view of the source code than simple statement coverage, but branch coverage can also leave you wanting more.
    In the accompanying example, the authors demonstrate how to use branch testing to achieve 100% coverage. However, as complete as the tests are, they still miss a bug that is lurking in the code. To top it all off, this is just a small example and things start getting much more difficult very quickly as this size of the problem increases.
    So, achieving 100% statement and 100% branch coverage may not be adequate, and testing every possible path exhaustively is probably not feasible for a complex method either. What's the alternative? Enter basis path coverage.
    What is path coverage? To answer that question, the authors start by offering a definition of an execution path. The claim made is that path testing allows one to find the smallest subset of paths through the code that tests every decision outcome independently of each other. Once again the authors turn to their example to demonstrate the steps one needs to take to define test cases. The result is that testing now finds the bug that had been previously missed. The authors conclude that although any type of coverage testing is a step in the right direction, there are advantages to moving beyond statement and branch coverage. Is coverage an important part of your QA process?

    Threaded Messages (8)

  2. See www.cijug.net for a presentation given at the Central Iowa Java Users group on this very subject.
  3. Presentation not found[ Go to top ]

    Where can I find the video or sheets?
  4. Not again.[ Go to top ]

    Why does this stuff keep comming up, time and time again. Who wants, who needs 100% code coverage? And who ever said code coverage leads to/equals quality? See www.testing.com/writings/coverage.pdf for a rather nice treatment of this subject and forget about the article.
  5. I find that those who are against the whole "code coverage" idea have either not seen the benefits first-hand or are just lazy (and probably think unit testing is a dumb idea). Code coverage analysis allows you to easily find classes or modules that are untested, which should then become a testing priority. You just need to be educated enough to understand that coverage doesn't equal quality, but instead helps achieve better quality. Everything you do as a developer should lead to better quality of a product. - Why do you use design patterns? Better architected software improves maintenance and thus quality. - Why do you write unit tests? Because they find bugs early which improves the product's quality. - Why do you use a code coverage tool? Because it shows you areas that needs focused on testing, which can expose bugs, improving the product's quality. I was also skeptical about the code coverage idea at first when I first started using Clover. After my team and I started writing tests for untested modules, we started exposing bugs that may not have been uncovered until a customer found them. That's when I realized the benefit.
  6. AFAICR from my old studies, SW testing is a NP-complete problem. Why should we bear this continuous warm-water discovery ? Just for curiosity, I have seen a 100% coverage C module that didn't work at all, in any case !! It was 17 year ago. Guido.
  7. I think this article is very interesting. It helps define coverage in terms of usefulness. The examples are a bit junior, but they get the point across.
  8. It is worth noting that refactoring can improve the testability of a piece of code. Sometimes you need to treat your code in a white box manner but your code just doesn't have the hooks or your methods are too complex to test all the possible combinations. In this case adding hooks and breaking complex methods into simpler ones can help readability and testability. It can also make your components easier to manage via JMX.
  9. If you want to have reasonably tested code, code coverage is just one of the forms of automated coverage you need. Sometimes you have code which does allow 100% of coverage. public class HelloWorld { private HelloWorld() { // don't create an instance of this class. } public static void main(Strings... args) { System.out.println("Hello world."); } } In this case, you could run main, but you might only get 60% line coverage because the constructor isn't accessible. You could write a test using reflection, but this has dubious value. The simplest example of a code coverage test failing to be sufficient is to use a single threaded test on a component which has to be thread safe. The single threaded test can exercise every combinational of path but the component can still fail when accessed in a multi-threaded way. In short, code coverage should be just one of the tests you do. I would claim good tests find about 80% of bugs. The rest can be found with other techniques such as code analysis, load testing, performance testing, stress testing, peer review, and manual testing. I have seen examples of applications which have been used for years with major bugs still in them. You can never really take it for granted that you have bug free code in a complex application just based on testing.