Discussions

News: The Software Inferno (how to convince management of the value of tests)

  1. Here’s an approach I’ve tried to convince management of the benefit of object level unit tests. I don’t know if it has worked yet, and I welcome suggestions to improve it. The Software Inferno It can be difficult to convince people the value of unit tests. As Paul Wheaton says in his article "Evil Unit Tests," people are both proud of the fact that they have them and bothered by the fact that there is so much work required to create and maintain them. As Paul states, a Unit Test is "The smallest amount of testable code. Often a single method/function, sans the use of other methods or classes." Methods with multiple paths of execution may require multiple unit tests. We don’t need that kind of test Often, opposition to tests at this level is based on the perceived return on investment: the level of effort to create and run them plus ongoing maintenance costs vs. what they add to delivering the finished application. The argument is often that the application as a whole is being delivered so the JUnit tests must test application functionality, not objects in isolation. In addition, since the functionality of the application will be verified by the Functional Tests performed by a test team and Acceptance Tests witnessed by the Customer, lower level tests are redundant and omitting them allows more value to be delivered in the form of more functionality. An Analogy In the 1974 production of Irwin Allen's The Towering Inferno, Paul Newman played an architect who designed a 138 story glass tower for a builder, played by William Holden. On the afternoon of the opening celebration the maintenance staff was testing the backup generators and a switch in the power room erupted in smoke. The architect was called and discovered an overload occurred due to the use of the wrong type of wire. At the same time on an upper floor, an overload in an electrical switch box started a fire in a utility room. A security guard, played by O.J. Simpson, read a computer warning from a heat sensor, but the automatic fire alarm system did not recognize it as a fire and did not notify the fire department. When he later saw the smoke on a video camera, he instructed another guard to call the fire department manually. When the fire department arrived, the fire chief, played by Steve McQueen, asked why the ventilation system did not automatically reverse to remove the smoke and whether the sprinkler system was working. The architect informed him that neither was working on the floor with the fire and he did not know why. The story revealed that the builder needed to cut $4 million from construction costs and had demanded that $2 million come from the cost of the electrical subsystem. In order to accomplish this, the electrical subcontractor, played by Richard Chamberlain, changed the architect’s specifications and reduced the robustness of the electrical system to the minimum that would meet the requirements of the building code. The other $2 million came from cost reductions in other subsystems with potentially similar results. The fire department was unable to successfully fight the fire and the architect ultimately had to blow up the water tanks on an upper floor to put it out. Many people lost their lives and the building was destroyed. Now how does this relate to software? The glass tower is our application. Paul Newman is the Software Architect who designed the system to provide the functionality desired by William Holden, the Customer. Richard Chamberlain is the Project Manager who must deliver his portion of the application within the Customer’s budget. O.J. Simpson is the Operations and Maintenance team. Steve McQueen is the new developer sent in to fix the problems with the application, but finds the problems are so pervasive that he cannot save it and the application has to be rebuilt practically from scratch at a cost that far exceeds what it would have cost to test that each component was sufficiently robust to support the whole. Not only that, but since there were insufficient unit tests, he cannot be sure that the rebuild will not break the portions of the application that did not crash and burn. But the high level tests passed... The building passed its Integration Tests: Electrical, telephone, elevator, air conditioning, security systems all work. The building passed its Functional Tests: Residents are living in the building, business tenants are operating there. The building passed its Acceptance Tests – The city inspectors passed all of the systems as meeting the building code. How could disaster have been prevented? Many things change once an application is released. Users perform actions and use values untested prior to release. Problems with algorithms and logic due to untested corner cases are revealed. New functionality breaks previously working functionality but no one knows this because the previous functionality had no low level test to reveal the breakage. Many of these problems can be tested for with low level unit tests. Examples from the analogy: assertEquals("highAmperageConduitWire", switchbox.getWireType); floor.setTemperature(140 degrees); assertTrue(floor.getSprinklerSystem().isWaterFlowing()); floor.setSmokePresent(true); assertEquals("reverse", ventilation.getDirection()); fireAlarmSystem.setHeatSensorStatus("alarm"); assertTrue(fireAlarmSystem.fireDepartmentNotified()); Another (shorter) Analogy Pharaoh wants a new pyramid built, but the straw harvest was poor so the ratio of straw to mud for the bricks is cut in half. No one tests whether the new bricks can stand the load. When the fourth level is reached, the bricks at the base crumble and the pyramid collapses. Why is software different? Do automobile manufacturers destroy a car in a crash test without first testing the characteristics of the bumper and other related components in isolation from the car? Does NASA send up the space shuttle to test the heat properties of the ceramic tiles that line its surface or are the tiles tested on Earth first? Post your convincing analogy or help me with mine. If you think it is less expensive to fix the defect after the user discovers it rather than have developers write tests that are as comprehensive as possible at the unit level, let me know that too.

    Threaded Messages (31)

  2. Not being a sponsor of unuseful unit tests, but I think that the examples does not dig into some details: WHAT integration tests related to security were successful ? In this respect, do you think that applying unit test could have surely saved the building ? The correct answer is: it depends. On the quality of unit tests, not their presence or not. That's why I think some QA methodologies are quite stupid. They assert that you must have unit tests. Stop. It does not says that you must have GOOD unit tests. It is up to the accepting personnel verify the wellness of a test plan. As usual, it is far more a matter of people than methods. For those who like readings, this could be useful http://www.cs.wustl.edu/~schmidt/editorial-5.html Guido.
  3. I vote for unit testing.[ Go to top ]

    Ok, good unit testing, but I think that it is easier to teach people how to write a good test than to write good code or refator for example.
  4. "Good" tests[ Go to top ]

    Hi, Probably the first step to having GOOD tests is actually considering to have unit tests in the first place. I think that the quality of tests are a second-level discussion. Cheers!, Carlos
  5. Just plain old common sense[ Go to top ]

    My standpoint about unit test was always quite simple to express - just use common sense. To me, unit-testing getters/setters and alike (adding item to an aggregated Collection) and making mocks 10x more complex that the class under test are equally senseless. Unit testing is about units, not about classes, right? So depending on the type of application it could be full of sense to split application into manageable units. Writing unit test for these units should not mean creating fixtures that are order of magnitude more complex that the classes they test. For example, let's consider my own pet project, JCR-based CMS. There are - JCR as persistence layer, - my own content abstraction layer on it, - some logics to process content requests and - some logics to render content with templates I could do it in a very orthodox manner by making unit tests for every layer separately, which would indeed take a lot of time. But hey, I didn't want to re-implement JCR API as a mock, so I just used Jackrabbit as it was. And I didn't want to mock my own abstraction layers either. So finally my unit tests are just talking to my own API - create content chunk 1, chunk 2, chunk 3, send request, retrieve rendered result, assert result matches to the expected snapshot, done. This allows me to deal with 20-30 lines code per test case (mostly copy-pasted) and still cover all complex logics of content retrieval and rendering. This is just what I call common sense applied to testing: achieve maximum test coverage and 100% test automation without investing 10x more in your tests than in your code. Test against your external API's. Start from the top and consider relatively loose-coupled large chunks of your application as potential units for testing, rather than from the bottom, from individual classes.
  6. Re: Just plain old common sense[ Go to top ]

    I think it boils down to the semantics of the term unit test. I have no issue with your definition, which seems to imply a unit of cohesive piece of logic. I am all for such tests, whether we call them unit tests or functional tests.
  7. Functional tests need Unit tests[ Go to top ]

    For those of you who think that you don't need unit test, just Functional Tests : What do you do if your Functional Test fails? For those of us used to TDD and unit tests - we write a Unit test to expose the failure at a low level (unit level). We can now fix the code that broke - it now passes the new unit test. We can run a regression test at the unit level by rerunning all unit tests at the click of a button. This shows that the fix has not broken anything else in that unit. Since we have fixed the bug found in the functional test - this exact bug will not arise again in the functional test. The functional test will pass (or at least break in a different part of the code). The point here is that Unit tests help functional tests by (a) Helping fix the bug (b) Providing a regression test so that the bug will be flagged if other code improvements introduce the same bug. (c) Allows to run test on on isolated portion of the functional test that fails - a lot handier than running a complex piece of functionality each time we make a change to the code, in order to see if the changes we made have fixed the bug. Regards, Fintan
  8. missing...[ Go to top ]

    code coverage measurements and unit testing has to go hand in hand. some people say that you need "good" unit tests... the only real way to measure that is to see how much of the code is being touched by your unit tests. if your team is doing unit testing then you run a measurement tool and find out your coverage is 50-60% then you've got problems. the teams i'm involved with do red/green lines with coverage percentages which are just as important as the red/green fail/pass numbers. i think it is 90 is green. (this applies to # of class, # of methods, and # of lines tested)
  9. Re: missing...[ Go to top ]

    i think it is 90 is green.

    (this applies to # of class, # of methods, and # of lines tested)
    Not much different from QA assertion like you "need unit tests". I have seen a 99% coverage totally broken C module many years ago. Again, it is primary a matter of people. Guido
  10. sure[ Go to top ]

    well, it should be given that if your "tests" don't actually test the requirements that your system isn't going to meet the requirements. i was referring to all the cases i've seen where people say "yes we're doing unit testing" but slapping one JUnit TestCase in a package with 20 classes is not "testing". same is true for your scenario, not writing tests based on requirements is not testing.
  11. Re: missing...[ Go to top ]

    some people say that you need "good" unit tests...
    You bet! I've seen unit tests that execute all the paths through the code but do not check the results properly. e.g., a test that calls a method that instantiates and returns a fully populated Customer object and the assertion just checks that the object is not null because the developer was too lazy to check each of the fields for correct values. Code reviews of unit tests will help here, but having 100% code coverage with bad unit tests only proves that the code does not throw an exception on that path, not that it tests correct functionality. Yes, the functional tests will catch some of these errors, but not all. e.g., the optional middle name of the customer should have been populated but was not. Now if you have a very, very thorough QA team that writes tests for every possible code failure, you will catch the defect before release. If not, some that could have been caught immediately by a "good" unit test will have to be fixed in a costly bug fix cycle. But wait! Having QA write all those extra functional tests will delay the release... better tell them to skip those and just make sure the minimum functionality is in place. Let the customer debug the app for us!
  12. waste[ Go to top ]

    Writing unit tests is a waste of time. Testing is for suckers.
  13. Address design first, testing second[ Go to top ]

    The biggest problem I have with many people's understanding of unit testing is that they don't get the 'unit' part. Most code I see is not written in a modular fashion. There are no components that are usable outside of parent subsystem. Asking developers that develop code this way to write unit tests is like asking a fish to climb a tree. They don't understand what it is they are being asked to do. It addresses the wrong problem. Unit testing is for testing components for correctness outside of the larger system. For example, if I am creating a special type of container for use in many applications, the users of this class should be able to rely on it's correctness e.g. when they successfully put an item in the collection, they can retrieve that item later. This is where unit testing comes in. I verfiy that my component works as designed and (hopefully) documented so that users of that component can focus on what they are doing with it. An analogy might be that a unit test is checking a bolt for it's various strength tolerances. A functional test is testing the final product, such as a car, made with those bolts. What many are really talking about when they discuss 'unit testing' (including some comments in this discussion) are really functional tests. Functional tests are as important as unit tests (if not more) but they are not unit tests and require a different approach. JUnit, by itself at least, is not a good tool for functional testing. I can't say whether TestNG is or not.
  14. Modularity and pluggability gives your software reusability.Goiod design can be fruitless if the implementation is vogus and vis-versa.Now even if we have a good design, well analysed and addressed and a good implementation, without testing we leave many hidden bugs to show up may be much after the final product is released.What i believe is address design and during the same time think over ur implementation whether it is modular, easily testable,how many test cases you can have for them. Unit testing renders your components confidence that these individual components will not break. After you come up with the finished product, integration testing is always there to test the integrity. cheers, http://javaicillusion.blogspot.com/
  15. Unit testing, yes but.....[ Go to top ]

    Blindly unit testing every every method of every class never made sense to me. In most projects, I could not justify the time and effort needed. I prfere to do unit testing at hot spots, i.e., code at the platform/ framework level where calls from functional use cases converge or funnel in. I prfer to be more thorough and rigorous for functional testing as far as coverage and corner cases.
  16. "Not only that, but since there were insufficient unit tests, he cannot be sure that the rebuild will not break the portions of the application that did not crash and burn." Unit tests test a class or method in isolation (with stubs or mocks), so how can a change in one portion ever break a unit test in another portion? In other words, I don't understand the quoted sentence... Johan.

  17. Unit tests test a class or method in isolation (with stubs or mocks), so how can a change in one portion ever break a unit test in another portion? In other words, I don't understand the quoted sentence...
    I think the distinction of Unit Tests to test a method or class in isolation, in other words, the distinction between Unit Tests and Funtional Tests is far too simplistic. Every Unit Test must either be a functional or a non functional test. Full stop. It might be a functional feature that is or is not verbose in the requirements, but it is still functional (or non functional). Also, the notion that unit tests test "in isolation" using "stubs and mocks" is more often than not simply not true. A unit test should always test some kind of contract. For example, if you create a large EJB application, you'd better be writing unit tests to test all your interfaces. Also, if the purpose of a component is, say, "to store a customer", a Unit test should test if it stores a customer, not if it *would* store a customer if one *would* connect a database that *would* contain a correct database schema and that *would* have a certain transactional behavior (Using stubs/mocks to test the algorithmical part of the storage, if any, is of course fine) Not doing so and leaving those errors to be discovered by the functional test team or the customer's acceptance tests is neglect! Doing this level of testing is probably as important as building the Unit Test on the byte shifting method. And as for your original question, things crash and burn all the time, because things tend to be interdependent and aggregated. Just because two aggregated things work in isolation does not necessarily mean that they work together, even if both are well encapsulated!
  18. In the linked article, the author says:
    The following two images demonstrate classes using classes that use classes. There are functional tests that exercise these classes working together. Fixing a bug in one class breaks many functional tests .... I have seen this happen many times. In one case, a small change broke 47 tests. Meetings were held to decide if the bug should just be left in the code! In the end, it was decided to turn off all tests until the time could be set aside to fix all of the tests. Months passed. Things got mighty stinky ....
    Unless I'm mistaken, this demonstrates a complete misunderstanding of what functional testing is. If a bug fix in one class breaks one or more functional tests then the functional tests were wrong (or at least their expected results were). A functional test proves that the system does what it should do, not what it "does" do as seems to be the implication here. In the example, if the functional tests were correct, valid against the requirements, then the bug in question would have been found early, that's what we are trying to achieve. You can live without unit tests, but not without functional tests. Unit tests are OK, until you have system, then focus should move to testing that systems functionality.
  19. In the example, if the functional tests were correct, valid against the requirements, then the bug in question would have been found early, that's what we are trying to achieve.
    I don't understand that. Functional tests demonstrate that application conforms to the specification. Or, more informally, that it does what the customer wanted. This means that they can't be run "early", as all functionality necessary for a functional test is not yet implemented in early phases. So the bug can't be found "early" with just functional tests.
  20. Bostjan,
    In the example, if the functional tests were correct, valid against the requirements, then the bug in question would have been found early, that's what we are trying to achieve.
    I don't understand that.

    Functional tests demonstrate that application conforms to the specification. Or, more informally, that it does what the customer wanted. This means that they can't be run "early", as all functionality necessary for a functional test is not yet implemented in early phases. So the bug can't be found "early" with just functional tests.
    The example describes how a bug breaks functional tests that 'exist'. If these tests were correct, they would have spotted the bug. By definition this would have happened earlier than the other mechanism that exposed the bug and consequently 'broke' the functional tests. As I say, as soon as you have a system (as opposed to a bunch of units) you can functionally test that system. Unit tests might help you get there. And, why not produce functional tests early, then when your system is ready, you can execute them ?
  21. Re: Software Inferno[ Go to top ]

    There is a simple question that should stress the importance of GOOD unit tests? How do you actually know you application works? Maybe you think it works but where is the proof? Aren't unit test part of the answer to this question. Don't they help solidify and prove the fact that the software you are building is actually working, maybe at component level but still the smaller parts make up the whole.
  22. Re: Software Inferno[ Go to top ]

    Unit testing all objects bottom up in a multi tier application is often not practical unless there is unlimited time and resource. Rather, the focus should be on functional testing at higher tier. In my opinion, it's not critical to reveal all possible corner cases in low level objects through rigorous unit tests. Not all of them may be exercised for the different functional test scenarios of interest. I would let the functional tests be the driver and expose whatever corner cases in the low level objects need to to be exposed and tested. Moreover in a real system there are other artifacts, beyond complied code in a low level object e.g., database, network, container logic etc.
  23. Bob, I like a lot your movie analogy, not only is it funny, it is often very positive to look cool and relax in important meetings, this always give more credit to one's speech ! I would add this to your "Towering Inferno" example, there might be another kind of acceptance tests : destructive tests. Of course, this is not applicable to concrete skyscrapers ! But from my ancient functionnal and load test background (no low level unit testing there), i always applied my tests gradually until i got to properly destroy the system by overloading one or many parameters : input field values, number of simultaneous connected clients... In the case of the movie, this kind of destructive test would have showed the problem... to the price of the building destruction which is of course not acceptable. Luckily in the software industry, we can simply reboot.. But still you can use this movie to show the necessity of destructive tests in order to find the limits of the system, can't you ? What do you think of "destructive tests" ? Cheers, Christian
  24. Destructive tests[ Go to top ]

    Also known as stress tests. You could stop short of the destruction if the limits are specified. For example, add clients to the application until its response time exceeded the maximum response time required. For the tower, the test would increase the current incrementally until the wire exceeded its maximum temperature rating. If the maximum temperature was achieved with a current below the required capacity, the test fails without destroying the building. Bob
  25. Re: Destructive tests[ Go to top ]

    Hi Bob, Additionnally, in a responsible company, the people who are responsible for stress or funtional testing are NOT the same who develop the system. I always manage to have a manager from a different team (and possibly from the customer and not a contractor) be responsible of validation via functionnal and stress tests. I even spend lots of time in meetings at project startup to ensure that this division of responsabilities is well acknowledged and understood. This way, some people are finally responsible for the quality of the delivery. And if things finally go wrong, at least my personal responsability as a developper is minimized, and somebody else will take the blame ! ;o) Cheers, Christian
  26. I like the direction that you are taking here, but I think the scenario does not fit unit testing. I do subscribe to the unit testing philosophy becuase I want to be the first to run my code :). This scenario is well suited to illustrate how Peer and Formal Review can be even more cost effective and that performance and system testing are really necessary. The key fact that allowed the electrical disaster was that the design was changed - something that developers do all the time for various reasons during implementation or bug fixing. This is why a project manager should schedule code reviews and also system and performance tests as well. Unit tests created by the same coder are not going to reveal performance/robustness issues although they should traverse all the exceptions paths to at least prove they work. Bottom line is someone got away changes, and yes code review the unit tests too. We always find something in code review. Rob
  27. Unit Tests Encourage Better Code[ Go to top ]

    Unit tests are not a panacea for all ills. However, I find that the discipline of having to write them forces me (and others) to produce code which is modular, loosely-coupled and with clearly-defined dependencies so that they can be stubbed out/mocked up for unit tests. My code is better because I know I'll have to test it. Like any practice, unit testing can be taken to the nth degree and become a burden rather than an asset. Nevertheless, used sensibly it encourages good coding practices and, in combination with wider-scope functional tests, reduces the incidence of bugs and finds them earlier. I'm a firm believer in the maxim that if the code is hard to test, it's probably because it's badly written.
  28. "Right, Bob, I understand your concerns. In case our application breaks, we will indeed have to close the company and all lose our jobs. BUT: If we are slower than our competitor XY, we will lose the company and our jobs anyway. So let's take the risk of not unit-testing, get into the market ASAP and maybe we can add unit-testing later". I am so tired of hearing that "unit-testing takes time"! It is just not true if you look at the complete picture. Writing good tests for your non-trivial code, you will find bugs immediately - either when you code it, or when someone else breaks it by that "little performance optimization". And "immediate" is good, because you are still close to the code and will be able to fix even complex problems VERY QUICKLY. And: You will fix it before you commit your code to the vcs, before some other developer trips over it, before the QA-release is built and sent to QA. You don't have to read QA's bug report, you don't have to discuss what the underlying problem for "general-problem.html" really is. You don't have to read nasty stacktraces, you don't have to search huge logfiles, you don't have to comment on the bugreport nor assign or close it. All this time is saved if you find and fix the problem immediatley - which is very likely with good and many unit-tests. Also don't forget the time each bugfix costs you when you are already working on the next new complex feature: Having to switch to the requested bugfix (while you were in a thinktank-meeting about that new supercool algorithm) dramatically slows you down. Same for your QA-people, who originally wanted to design that new elaborate JMeter-Test, or install this cool-but-complicated browser-simulator - but now have to look a rather boring bugfix, that could have been found by unit-tests months before. Fixing bugs in complex environment is only fast when you do it immediately. Any delay here makes simple 5-minute fixes cost hours or even days. Your tower doesn't have to crash until unit-testing becomes valuable. Tell this to your boss, but tell the story about the tower to your fellow developers instead; the story is for them! :-) Good luck! Per
  29. "Right, Bob, I understand your concerns. In case our application breaks, we will indeed have to close the company and all lose our jobs. BUT: If we are slower than our competitor XY, we will lose the company and our jobs anyway. So let's take the risk of not unit-testing, get into the market ASAP and maybe we can add unit-testing later".
    I completely agree with this statement. The problem (at least in my situation, but I think it's quite common) is that the company first addresses time and budget limits when selling software, therefore quality is going to suffer badly. The logic I see is : first sell it, then plan&build it. When the company sells the software, it is thinking about the immediate revenues, but no one calculates the future expenditures for regression bugs and/or software which becomes so tangled that a simple modification requires several days of analysis. This happens especially:
    • if you are in a dominant market position (you will not lose customers because of your bugs)
    • you have locked in your customers (they can't switch to another vendor)
    • the company knows that the sofware you are building is not planned to last long
    Writing good tests for your non-trivial code, you will find bugs immediately - either when you code it, or when someone else breaks it by that "little performance optimization". And "immediate" is good, because you are still close to the code and will be able to fix even complex problems VERY QUICKLY. And: You will fix it before you commit your code to the vcs, before some other developer trips over it ... All this time is saved if you find and fix the problem immediatley - which is very likely with good and many unit-tests.
    This is absolutely true imho, but (this is the sad thing) in a company's mind 5 days to be spent in hacking&bugfixing three months after the software is gone in production are a lot better that 5 minutes spent now on a unit test. just my 2 cents, Riccardo Govoni.
  30. This is absolutely true imho, but (this is the sad thing) in a company's mind 5 days to be spent in hacking&bugfixing three months after the software is gone in production are a lot better that 5 minutes spent now on a unit test.
    Well, if all bugs would pop up only in three months, then I might even agree with "let's save those minutes now to squeeze in another nice feature". But in my experience, the majority of bugs resurface within two or three weeks (assuming you have collective-code-ownership, at least some reviews, and a QA-team). So if your release-date is more than 3 weeks ahead, you are likely to have to spend those 5 days before the release, not after. And I guess it will rather be nights than days, since on the days you still have the normal tasks which you promised to deliver. Regards Per
  31. Convincing management is easy[ Go to top ]

    Small quibble with the title - convincing the management is the easy part. It is convincing the developers that is harder. Issues with testing: 1. Defining "unit" is the hard part. Books tend to use simplistic examples. When classes are coupled "what is the unit being tested". 2. When code is refactored, how many people re-factor the tests? Do the tests really test "units" after code has been refactored?
  32. I like the pyramid analogy, it seems more accurate to the purpose of a unit test. The Inferno is more about how the management handle any project, it would go great on a topic on dealing with budget cuts. Namely, before your budget gets cut ensure that there are excellent testing procedures already in-place. Or watch it burn and crumble! Nicely written though and should make people think.