Cedric Beust on Untested Code

Discussions

News: Cedric Beust on Untested Code

  1. Cedric Beust on Untested Code (41 messages)

    In his blog, Cedric Beust cuts through the rhetoric to answer the question of how much unit testing is really needed. Though he calls untested code the "dark matter of software", he also recognizes that this dark matter is not going away any time soon. Unit testing is difficult and time consuming. This observation, made in a posting to the JUnit mailing list, was followed up by the question: Does everyone manage to unit test all of their stuff all of the time? In digging through the many questionable responses Cedric managed to find a few insightful gems. One response to the question suggests that there are only two types of code: Code that is tested and works and code that is not tested and doesn't work. Cedric's take on the subject is that there are ways of knowing that your code is working without actually testing it. So we should also include a third category: Code that is not tested and works.
    This kind of code is very common, in my experience. Almost prevalent. And this is also why I am convinced that if other circumstances warrant it, it's okay to write the code, ship it and write the tests later.
    Cedric concludes that at the end of the day you need to use your head and not follow one liners from a book. Is your code 100% covered with tests 100% of the time?

    Threaded Messages (41)

  2. Another option[ Go to top ]

    I certainly believe in the black and white choice; if you can't prove it, it doesn't work. But that doesn't mean that I test 100% of my code. That is my goal of course, but it isn't realistically achievable. Instead, I have to make a choice on what level of potentially not-working code I'm going to release. Looking at it from that perspective really drives you to do as much as you can because you don't have a false sense of complacency from working-but-untested code. The view of potentially-not-working also reenforces the fact that it's the first place that you need to look for bugs, since there is no guarantee that it is working in the first place.
  3. Another option (part 2)[ Go to top ]

    Despite what my previous comment sounded like, I do agree that there are other aspects of software that are as or more important than test coverage. It is simply an attempt to find a balance that works for you and your customers. The problem with saying that 'since it's always worked we don't need to have tests for it', is that you doesn't take into account what happens when something that your customers have always seen working breaks suddenly. If you don't have a comprehensive test suite, both at the unit and system level, the bug goes to the customers who then have to deal with it. That late in the game it becomes much more expensive both monetarily and reputation-wise to fix. The strive for 100% isn't to eliminate all bugs (which unit testing can never do on its own) or to satisfy some personal professionalism need. It's to give yourself the *best chance* to deliver quality code and to be able to manage large scale change in the future.
  4. Re: Another option[ Go to top ]

    "Beware of bugs in the above code; I have only proved it correct, not tried it." -- Don Knuth. Before you write unit tests for a piece of code, you don't know if it works or not. After you've written unit tests for a piece of code... you know that it works well enough to pass the tests. Unless your unit tests assert the correctness of every possible value that could be passed into every method (at which point the tests would simply be a parallel implementation of the method being tested), you don't _know_ that a piece of code works, to the degree of black and white certainty that seems to be implied by 100% code coverage advocates. It's a value proposition. How much testing effort do I need in order to be confident with a piece of code? At some point the cost will outweigh the benefit, and that's where you stop. As such, it's not too far-fetched to believe that in some cases, for some fragments of code, you won't find anything worth testing.
  5. Re: Another option[ Go to top ]

    Indeed! There is anther kind of code: the one that passes the unit tests, but doesn't really work. After all, is it possible enum all the possible asserts? And who is testing that the unit tests are correctly reflecting the software requirements - as the developer herself writes the test cases? To believe that test cases garantee working code is totally missing the point. The value of test and TDD is not, and can not provide that kind of insurance. However, it does and can help produce better quality code. Ric
  6. Re: Another option[ Go to top ]

    I certainly believe in the black and white choice; if you can't prove it, it doesn't work.
    I think that this belief is a good guide for development. Of course it isn't true, but its worth believing in. I suppose the same thing could be said for documentation. If it isn't documented then how do you know it works even with tests? (Of course there are those who consider the tests documentation) Both of these sound a little like that lonely tree falling in the woods.
  7. The unit test should confirm to the reviewed requirement. If not, although, unit test is done, the code may not confirm to the requirement and won't cover all the scenarios.
  8. Manual testing[ Go to top ]

    It sounds like the only form of testing is automated unit tests. What about good old manual testing? According to my experience, writing automated test cases takes at least the same time as actually writing the code. But for manual testing, it use to be enough with one test person per four programmers. And even if you are using automated tests, you will still need manual tests. According to these metrics, automated testing is at least four times as expensive as manual testing. It is also very hard to write automated test cases that tests the GUI code. Simulating mouse movements etc needs very sophisticated software. Automated test cases for database access is also rather hard to write. The are almost an infinite number of states that need to be tested. Fredrik Bertilsson http://frebe.php0h.com
  9. Re: Manual testing[ Go to top ]

    It sounds like the only form of testing is automated unit tests. What about good old manual testing? According to my experience, writing automated test cases takes at least the same time as actually writing the code.
    Agree with you. It seems like automated unit tests was introduced not because it is effective, but because programmers dislike testing (and like programming) :-) Many companies release software with known bugs today and feel ok with it. Why spent time on unit testing instead of fixing problems, that your customers _already_ have ? If your customers have no problems - why you should worry about testing ? Maxim Kramarenko, project manager and software tester :-) http://www.trackstudio.com - hierarchical issue tracking software
  10. Re: Manual testing[ Go to top ]

    It seems like automated unit tests was introduced not because it is effective, but because programmers dislike testing (and like programming) :-)
    Unfortunately most programmers also dislike writing test cases... I think that automated unit tests can benefit a lot, especially for base components that are used in many different contexts/applications. But the (TDD) evangelists talks very little about testing the GUI and testing together with the database. They never talk about manual (integration) testing, like it is not needed if automated unit test is introduced. Fredrik Bertilsson http://frebe.php0h.com
  11. Re: Manual testing[ Go to top ]

    I think that automated unit tests can benefit a lot, especially for base components that are used in many different contexts/applications. But the (TDD) evangelists talks very little about testing the GUI and testing together with the database. They never talk about manual (integration) testing, like it is not needed if automated unit test is introduced.
    Agree. The main problem with unit tests - they can check that software works fine just from developer viewpoint, not from product manager or customer viewpoint. Customers can even use known bugs in software to implement required functionality, and bugfix from developer viewpoint can lead to major problems for the customer. If you have no detailed and stable specs (from customer?), you probably cannot implement good tests. Maxim Kramarenko http://www.trackstudio.com - hierarchical issue tracking software.
  12. Re: Manual testing[ Go to top ]

    Unit test is meant for the programmer use ... to ensure that the program is performing what the programmer expect to do. It is especially useful if the programmer decide to do refactoring to a large and complex program. For your concern on the Product Manager and Customer, we have the Integration Tests and Acceptance Test to cover their concerns. In this aspect, the Unit Test has not value to them, only has value to the programmer
    I think that automated unit tests can benefit a lot, especially for base components that are used in many different contexts/applications. But the (TDD) evangelists talks very little about testing the GUI and testing together with the database. They never talk about manual (integration) testing, like it is not needed if automated unit test is introduced.


    Agree.

    The main problem with unit tests - they can check that software works fine just from developer viewpoint, not from product manager or customer viewpoint. Customers can even use known bugs in software to implement required functionality, and bugfix from developer viewpoint can lead to major problems for the customer. If you have no detailed and stable specs (from customer?), you probably cannot implement good tests.

    Maxim Kramarenko
    http://www.trackstudio.com - hierarchical issue tracking software.
  13. Building the Code right[ Go to top ]

    +1 Unit tests show if the programmer is 'building the Code right', not if the programmer is 'building the right Code' ... (if we want to ensure that we build the right code, we could use some kind of automated, repeatable Acceptance Test Frameworks like FIT) Beside that, a Unit Test has some more valuable Aspects, like (1) having an 'Insurance Base' for Refacoring (as said), (2) delivering a good documantation of how to use / interact with the API / Model (3) giving a good feedback of the complexity of the design (i.e. violating law of demeter).
  14. Re: Manual testing[ Go to top ]

    But for manual testing, it use to be enough with one test person per four programmers.
    Is that so? Let us say you spend two days every week testing stuff I develop today. After a year, I will have produced a substantial code base. Are you saying that it still takes the same amount of time to test my stuff after I have added a years worth of features to it? I think that is totally unrealistic. If you are aiming for something close to a constant test coverage of existing functionality, you are going to have testing costs that will grow. The more likely outcome is that you will be shipping code that is in an unknown state. That is all well and good, but I hope you tell your customers that you are using them for testing. Adding tests retroactively after manual testing costs have grown beyond what your sponsors are willing to finance is an almost impossible proposition. You will have several months worth of downtime. These months are a godsend to your competitors - they will be able to steal market share. Geir Hedemark
  15. Re: Manual testing[ Go to top ]

    But for manual testing, it use to be enough with one test person per four programmers.


    Is that so?

    Let us say you spend two days every week testing stuff I develop today.

    After a year, I will have produced a substantial code base. Are you saying that it still takes the same amount of time to test my stuff after I have added a years worth of features to it?

    I think that is totally unrealistic. If you are aiming for something close to a constant test coverage of existing functionality, you are going to have testing costs that will grow.

    The more likely outcome is that you will be shipping code that is in an unknown state. That is all well and good, but I hope you tell your customers that you are using them for testing.
    Realistic or not, in the organizations I have been working, this have been the fact and I don't consider the code being in an unknown state. If I write automated unit tests, I have to spend at least the same amount of time writing tests as actually writing the code. I also sounds like you think that automated tests can fully replace manual tests. I don't think that is realistic. Fredrik Bertilsson http://frebe.php0h.com
  16. Is your code 100% covered with tests 100% of the time?
    Should we cover with tests unit tests code too ? :-) If no, how we can ensure that it works ? If we can test unit tests manually, why we cannot test original software the same way ? Maxim Kramarenko http://www.trackstudio.com - hierarchical issue tracking
  17. Re: Cedric Beust on Untested Code[ Go to top ]

    Is your code 100% covered with tests 100% of the time?


    Should we cover with tests unit tests code too ? :-)

    If no, how we can ensure that it works ? If we can test unit tests manually, why we cannot test original software the same way ?

    Maxim Kramarenko
    http://www.trackstudio.com - hierarchical issue tracking
    Automatic unit tests can not ensure working software (an impression most of advocates seem to impose to their audiance). Those who believe in that is mistaken. TDD does help improve the coding quality when used properly. e.g. those "base components" mentioned in previous post. IMHO that is value of tests, not that it can ensure you of something. ric
  18. Is your code 100% covered with tests 100% of the time?


    Should we cover with tests unit tests code too ? :-)

    If no, how we can ensure that it works ? If we can test unit tests manually, why we cannot test original software the same way ?

    Maxim Kramarenko
    http://www.trackstudio.com - hierarchical issue tracking
    Is your name Gödel?
  19. Is your code 100% covered with tests 100% of the time?


    Should we cover with tests unit tests code too ? :-)

    If no, how we can ensure that it works ? If we can test unit tests manually, why we cannot test original software the same way ?

    Maxim Kramarenko
    http://www.trackstudio.com - hierarchical issue tracking
    That would be Jester (the JUnit test tester)...
  20. Theory vs reality[ Go to top ]

    I think he raises a good point. Automated unit testing is good in theory but often not in reality when money is tied to deliverables. I had a developer working for me and he insisted on writing unit tests for everything. I had no problem with that so long as we made our deadlines which were set more by financial needs than by the actual amount of work. He slowly but surely, stopped writing unit tests. So, I think the question is whether or not unit testing is the right solution? I personally think there has to be a better option...I just don't know what it is.
  21. Re: Theory vs reality[ Go to top ]

    I think he raises a good point. Automated unit testing is good in theory but often not in reality when money is tied to deliverables.

    I had a developer working for me and he insisted on writing unit tests for everything. I had no problem with that so long as we made our deadlines which were set more by financial needs than by the actual amount of work. He slowly but surely, stopped writing unit tests.

    So, I think the question is whether or not unit testing is the right solution? I personally think there has to be a better option...I just don't know what it is.
    You've presented a good example, Aaron. I suspect what you've described is common. I've experienced that myself, as the developer who gradually wrote less tests because of time/budget constraints. In my case, I think the problem was that the time/budget constraints were tied *only* to shipping a first version of the software. There was more time and budget allocated to "testing and debugging", which was *after* completing a first version. I felt this was unfair, because if you take more time developing because unit tests are part of your development, you should be allocated part of that testing budget. Managers didn't see it that way; they only saw that time to ship would be longer. Has anyone else experience this type of scenario? I think it's the same thing with documentation. That always comes at the end, and rarely gets done properly. Developers don't feel like doing it, and often other tasks arise (bug fixes, new features) which have both a higher priority from the manager *and* higher interest from the developer. Documentation doesn't stand a chance! Documentation would have more of a chance of getting done if "test-first" became "document-first". You want to implement a feature? Document it, write a test for it, and then develop it. That would take even more discipline, and more time/buget to ship, but if you want it done, it would have more of a chance of getting done. Writing docs a bit at a time is worth it in order to avoid the daunting task of writing *all* the docs at the end. I know, it's a tough sell for everyone. Frederic
  22. Re: Theory vs reality[ Go to top ]

    In my case, I think the problem was that the time/budget constraints were tied *only* to shipping a first version of the software. There was more time and budget allocated to "testing and debugging", which was *after* completing a first version. I felt this was unfair, because if you take more time developing because unit tests are part of your development, you should be allocated part of that testing budget. Managers didn't see it that way; they only saw that time to ship would be longer.

    Has anyone else experience this type of scenario?
    In corporate business only the current or next quarter is important, so your scenario is of course common. But I have alse seen many decisions in the software industry based on the assumption "it will reduce the cost for maintenace". After all, the use of OO is mainly motivaded by the assumtion that future maintenace cost will be reduced. Most people in the business seem to agree that the cost for the first shipment are higher using OO, assuming the cost for future shipments will be lower. But the big question regarding automated unit tests is whether they will reduce the overall cost for the entire lifecycle, or not. Personally I think that is only the case for (some) base components and only if major refactoring is done. Otherwise manual system/integration testing is less expensive. Fredrik Bertilsson http://frebe.php0h.com
  23. Re: Theory vs reality[ Go to top ]

    This is a weird outcome, weird meaning different from my direct experience. Writing Unit Tests shouldn't slow your productivity down. They've saved my butt enough times that I would feel naked without a solid set of automated tests for any substantial code base. I'm not sure how you can do a major refactoring without them and not lose sleep at night? As far as productivity goes in a development cycle, I spend maybe 60-70% designing and only 40-30% writing code and tests. But, getting back to the thread, I think you do have to compromise what you can test automatically and where writing those tests doesn't make sense. However, if I had to guess I'd say we could be doing much more automatically than many believe. Testing is a craft like programming and it has to be done at several levels to ensure some level of quality. I'd be surprised if a TDD leader said anything against integration, UI, and other user tests. Many say that a separate QA team is junk but that isn't the same thing. ______________ George Coller DevilElephant
  24. Re: Theory vs reality[ Go to top ]

    Writing Unit Tests shouldn't slow your productivity down.
    How much time do you spend writing test cases compared to the time you spend writing the actual code? Personally I estimate the ratio to 1:1.
    I'm not sure how you can do a major refactoring without them and not lose sleep at night?
    But a major refactoring demands a lot of manual testing too? Or do you say that automated tests fully replaces manual testing? One other strategy is to try to avoid major refactoring. Fredrik Bertilsson http://frebe.php0h.com
  25. Re: Theory vs reality[ Go to top ]

    Even if the ratio is 1:1 overall productivity goes up because I don't chase the same bugs again and again. Integration and user testing go much smoother because they don't get stuck trying to test buggy code. It may be tempting to say that if I can program X LOC an hour that I can only code X/2 LOC if I write tests. It doesn't work out that way in practice. I'd still say that overall I crank out X LOC. There is the thought that if you are working too hard at something you are doing it wrong. I'm thinking of the mentioned programmer who had to stop writing unit-tests to get his/her code completed on time. Effective unit test writing is itself a craft that has to be learned. I'm no master but I'm good enough to know and have experienced different quality unit-tests over several projects. Don't paint me as a unit-test only goof ball. Unit tests aren't everything, I think people have already made that point elegantly on this post (including myself). I'm only posting because there seems to be some backlash against automated testing altogether which I find a bit, surprising. I won't comment on the "try to avoid major refactoring" line other to say that if you can do this successfully on all your projects and still keep your design tight and future-proof then you are a much better developer/designer than I'll ever be. ______________ George Coller DevilElephant
  26. Re: Theory vs reality[ Go to top ]

    Even if the ratio is 1:1 overall productivity goes up because I don't chase the same bugs again and again.
    But if you need one tester per four programmer without automated tests, and using automated test needs the double amount of programmers, the overall productivity goes down, right?
    I'm only posting because there seems to be some backlash against automated testing altogether which I find a bit, surprising.
    The reason is that automated tests in most cases makes the overall productivity lower. It may very well improve quality, but in real life quality gains has to be compared to productivty losses.
    I won't comment on the "try to avoid major refactoring" line other to say that if you can do this successfully on all your projects and still keep your design tight and future-proof then you are a much better developer/designer than I'll ever be.
    I don't claim that it is possible to avoid major refactoring and still keep the design tight and future-proof. But mostly the management thinks productivity is more important than a tight and future-proof design. After all, the user doesn't know if the software design is tight or not. Fredrik Bertilsson http://frebe.php0h.com
  27. Re: Theory vs reality[ Go to top ]

    The productivity over quality line is pretty familiar. It bookends nicely with users not caring about what's under the covers as long as the UI is pretty and appears to work. I don't buy it for a second. We may have to agree to disagree on this one. It sounds like you are pushing against not only unit testing but automated testing in general. If I'm wrong let me know - maybe be more specific about your team's software process and why it is working for your environment. My experience is that automated tests provide a huge gain toward delivering quality software on time. My reality is that as great as we programmers are, often times we inject defects while fixing others or adding new functionality. If I find a defect and fix it I want to write a test to make sure it stays fixed. That 5-10 minutes spent writing the test may save hours of time during future test/release cycles and days of time in maintenance costs. I'm not just talking unit tests either. A good smoke-test harness (think black box testing) written at the beginning of a project will help ensure you are ready for integration on a daily basis. I guess I base some of my strong reaction to the view that eyeball-only testing will save time and increase productivty is that it presumes that all important defects can be identified by those not intimate with the code base. Sure the user won't care if the design is tight. To me that is a silly argument. The user of an automobile may not care about the under-the-hood design, that is untill it stops working and costs too much to maintain. The user of software doesn't know enough to care about many of the things we so called software engineers are being paid to care about. Eyeball testing won't find memory leaks, volume performance issues, swallowed exceptions, etc. ______________ George Coller DevilElephant
  28. Re: Theory vs reality[ Go to top ]

    It sounds like you are pushing against not only unit testing but automated testing in general. If I'm wrong let me know
    I am not pushing against automated testing in general. The problem I see is that because of the lack of (cheap) tools for automated integrations testing, people seem to focus on automated unit testing because it is a lot easier. My point there is a fault safty in thinking automated unit test will solve all quality problems.
    My experience is that automated tests provide a huge gain toward delivering quality software on time.
    I think you are correct, but how much time do you spend writing the tests?
    That 5-10 minutes spent writing the test
    How long did the actual code change take?
    may save hours of time during future test/release cycles and days of time in maintenance costs.
    Maybe, maybe not.
    Sure the user won't care if the design is tight. To me that is a silly argument.
    So, why did you make it?
    Eyeball testing won't find memory leaks, volume performance issues, swallowed exceptions, etc.
    I use manual testing (not sure about your definition of eyeball testing) for finding memory leaks and volume performance issues, using OptimizeIT. I think it would be rather difficult to integrate OptimizeIT into a automated test. Fredrik Bertilsson http://frebe.php0h.com
  29. Re: Theory vs reality[ Go to top ]

    Even if the ratio is 1:1 overall productivity goes up because I don't chase the same bugs again and again.
    This is one problem with the automated testing concept. Not everyone has this same need. So, not everyone will see the benefit. I think the true benefit of automated testing comes in during refactoring (significant) but if your product never makes it out the door, then it wasn't worthwhile. But even in refactoring, I personally believe a developer should have a thorough understanding of the code and the business domain before making code changes. I don't like developers doing things with the expectation that if they miss something that the unit test will catch it. I've see unit tests used as a crutch more than they should be. These are some reasons why there has to be a better alternative or at least different alternatives based on the environment of your company and the type of developer.
  30. Re: Theory vs reality[ Go to top ]

    This is one problem with the automated testing concept. Not everyone has this same need. So, not everyone will see the benefit.

    I think the true benefit of automated testing comes in during refactoring (significant) but if your product never makes it out the door, then it wasn't worthwhile.
    I guess I can respect that some projects are small enough or disposable enough where automatic testing may not be a plus. What's not coming across to me is this ballooned up concept that automated testing has to be such a difficult and timeline-busting concept. I hate saying this because it sounds arrogant but are the development teams you are talking about pretty green or maybe just don't understand the available tools/frameworks? For Java, JUnit is about as easy a framework as you'll encounter - easier to learn and understand than log4j I'd argue. That along with all the IDE and Ant support it just doesn't make sense to me how writing a decent batch of unit and integration tests would cut your productivity in half. I'd need to see some real world proof since my experience is directly opposite. I could maybe see some loss in productivity if a team made the mistake of waiting until the end of a development to write the tests - but nobody subscribes to that being a good methodology.
    But even in refactoring, I personally believe a developer should have a thorough understanding of the code and the business domain before making code changes. I don't like developers doing things with the expectation that if they miss something that the unit test will catch it.
    I'm not a logic expert but this seems like a straw-man argument. You've set up a mythical idiot-developer to help prove that automated testing doesn't work as well as it is promoted in making refactoring possible. If you replace "unit test" with "manual test" or "banana pie" the statement still holds true, which means it really argues for nothing. I'm arguing that major refactoring probably isn't even feasible without unit tests.
    I've seen unit tests used as a crutch more than they should be. These are some reasons why there has to be a better alternative or at least different alternatives based on the environment of your company and the type of developer.
    Alternatives to what? A reasonable amount of automated test-coverage supplemented with user acceptance testing? To me that is a winning combination most of the time. From project to project you need to scale the amounts but taking one out in the name of productivity is a mistake in my opinion. Again, it would be interesting to hear stories of large projects that were completed on time and had no major defects in integration and production using manual testing only. It would be interesting to know the project domain, the methodologies used, and the team size and make up. I'd be scared stiff to be on that project with my opinions as they are so it would be educational for me to find out why for some teams this works. ______________ George Coller DevilElephant
  31. Re: Theory vs reality[ Go to top ]

    For Java, JUnit is about as easy a framework as you'll encounter - easier to learn and understand than log4j I'd argue. That along with all the IDE and Ant support it just doesn't make sense to me how writing a decent batch of unit and integration tests would cut your productivity in half.
    This depends on what you consider "decent". And under a tight timeframe, there is always a time cost regardless of ease.
    I'm arguing that major refactoring probably isn't even feasible without unit tests.
    I think anyone with over 15 yrs of experience would be able to say they've done this type of major refactoring without automated unit tests.
    Again, it would be interesting to hear stories of large projects that were completed on time and had no major defects in integration and production using manual testing only. It would be interesting to know the project domain, the methodologies used, and the team size and make up. I'd be scared stiff to be on that project with my opinions as they are so it would be educational for me to find out why for some teams this works.
    1. If a large project is needed to prove the value of automated unit testing, it gets back to my point that perhaps it isn't applicable in all cases. 2. I'm suprised by "scared" stiff. Automated unit testing to the extent it is positioned today is relatively new. Programs were successfully written before JUnit and other frameworks. 3. I think testing is good. I'd just like to see an alternative that works in all sizes and projects, all timelines. I guess conceptually I think it would be nice if I could do my manual tests and have the state of my program recorded and played back...in general, if I don't think I should need to type my tests.
  32. Re: Theory vs reality[ Go to top ]

    Alright, I'm tired of being the only poster here who believes that a certain amount of automated testing can help most projects achieve better quality. I feel weirdly alone here on this side of the argument considering how much success automated testing has seen in recent years. Oh well, this isn't about being right or wrong but only about posting educated opinions. I think we've accomplished that much and I'm out of ideas. Thanks for the lively discussion. ______________ George Coller DevilElephant
  33. Re: Theory vs reality[ Go to top ]

    Alright, I'm tired of being the only poster here who believes that a certain amount of automated testing can help most projects achieve better quality.
    I don't think this is the matter of disagreement. Anyway I think that automated testing can belp projects achieve better quality. But the matter of disagreement is the size of the cost for this quality improvement.
    I feel weirdly alone here on this side of the argument considering how much success automated testing has seen in recent years.
    I think it is debatable if automated testing has been successful or not. Once again, I think there are many areas (like base components) there automated testing does improve both productivity and quality, but there are also many other areas there this is not the case.
    Oh well, this isn't about being right or wrong but only about posting educated opinions. I think we've accomplished that much and I'm out of ideas.
    If find it questionable if not weighting costs towards benefits is an educated opinion. Fredrik Bertilsson http://frebe.php0h.com
  34. You are not alone[ Go to top ]

    George, You are most definetely not alone. I assure you, the silence on your side of the argument proves more the words - we are busy writing tests while the other side speaks :) Jokes aside, I personally believe in the power of automated testing. One of the best environments I've worked was where QA/test team was development-oriented and daring to embrace JUnit as the tools of their trade. A lot of manual testing scenarious morphed into test code - this has raised productivity on both sides of the wall. We as developers could run this ultimate test suite before we gave the program to them. In fact it was very rewarding to develop the middle-layer "framework" of utility classes for them. As for development side, I find myself much more comfortable with "green" color and high 90ies numbers for test coverage then without. It doesn't make make code better, but it provides confidence in certain activities like re-factoring mentioned here. I've used JUnit to express all level of tests, from unit ot system/integration ones. In fact, in some projects I went as far as to cover the 3rd party interface I had to use with tests and it saved me a lot of problems several times when library versions had to be changed, etc. In one of my current teams we practice a weird form of pair-programming when one person writes tests while the other codes implementation. As once coined by someone "XP approaches only work well together, pulling one out of the context usually brings little benefit". Same is true for tests, if one looks only at one side - spending same amount of time on tests as the actual code, it seems not to bring a lot of advantage. However one may simply choose to look at a bigger picture to see the value. Sincerely, Ed Letifov
  35. Re: Theory vs reality[ Go to top ]

    2. I'm suprised by "scared" stiff. Automated unit testing to the extent it is positioned today is relatively new. Programs were successfully written before JUnit and other frameworks.
    Frameworks are not necessary for automated unit testing. When I first started reading the hype about unit testing it seemed kind of shocking. On one level it made complete sense, on another no sense, and finally I wondered how people developed quality software without it...but that was an uneasy feeling because I thought I hadn't been doing unit testing. Then I thought about it some more. Automated unit testing is really about taking chunk of your code, writing some automated tests for it, and using those tests often to make sure the "unit" of code works. Saying "a unit is a class" is stupid. Saying "a unit test cannot read files, talk to a db, make network connections, etc" is stupid. A unit is a chunk of code, usually tightly coupled enough that the parts can't easily be separated. (yes, if there are too many pieces in there, then you have a problem) A test harness can be anything that exercises the code and produces readily understandable results. Under that definition, I cannot imagine forgoing automated unit testing on all but the most trivial pieces of code. No one taught me to do that...it just made sense. You ask yourself "Hmmm...I wonder if this piece of code works?" so you carve it out and test it by itself, because testing it with all the other code is too complicated.
  36. Re: Theory vs reality[ Go to top ]

    Saying "a unit test cannot read files, talk to a db, make network connections, etc" is stupid.
    Resetting the files, db or network before every execution of the test scenario to the initial state, is not always simple.
    A test harness can be anything that exercises the code and produces readily understandable results.
    Frameworks like junit does not support testing a swing GUI. Evalutating a report program producing a PDF file is not easy. Fredrik Bertilsson http://frebe.php0h.com
  37. Re: Theory vs reality[ Go to top ]

    This is a weird outcome, weird meaning different from my direct experience.

    Writing Unit Tests shouldn't slow your productivity down. They've saved my butt enough times that I would feel naked without a solid set of automated tests for any substantial code base. I'm not sure how you can do a major refactoring without them and not lose sleep at night?
    We are generally start development of the next major product release with major refactoring. Then we test it a little (manually). Then developers working on new features, each of them tested by original developer, other developers and myself. Our developers have no "own" code, very often developer A implement the feature, developer B test it/report bugs and developer C fix bugs. All our developers know (almost) all source code and every of them can fix (almost) every bug. As the side effect, every new feature is are reviewed and tested by (almost) all team members many times before release, so I can sleep fine :-) Actually, this way allow us perform VERY major refactoring - we shouldn't worry that we should rewrite all our unit tests if we change UI, internal architecture, etc. Maxim Kramarenko http://www.trackstudio.com - hierarchical issue tracking
  38. Possible avenues of solutions: Use some AI methods like Genetic Algorithms to automate unit-testing and some integration testing. The pressure to deliver code often compromises testing, particularly unit testing. One can’t just pretend that there is no pressure to deliver. When a programmer has been worker 50 to 60 hours per week and he is coding something at 11:30 PM at the office, does he spend a few more hours to unit-test or just assume all is OK and check the code in? I think that a possible path of solution is to us some AI algorithms to assist the automation of unit-testing and even small system testing. I work in that area. I have read some research papers that applied Genetic Algorithms to search for bugs and report bugs. The good thing about Genetic Algorithms is that it works alone without much information about the problem. The only thing we need to give is the range of each field in an object model. It works like this: You have a population of agents (possibly several thousands). Each agent represents a test case for the test. The population reproduces and evolves as in Darwinian evolution. Many different combinations of test cases are fed into the application. Sooner or later some bugs will be found: a winning agent. A winning agent will reproduce more than others and its gene will be remembered for a very long time in the population. All bugs can be found this way. This type of approach has been looked into and documented but it’s in the world of AI.
  39. A few comments: * UT is not only for the developers. It brings a much more stable code to the QA team, which can then find the "real" problems in the S/W. It is good for the presale/postsale guys who get better feed back from the customers. It is no magic, it won't make the product better if it sucks. But it is a major part in the chain, IMHO. * UT is exepensive. In my experience too, it takes around 50% of the development time. I think that if we want to treat our profession with some respect, than it's a price we need to pay (and yes, I'll mention the automobile metaphore here again. Not all software is installed on rockets and spacecrafts, but it's supposed to work, I believe) * UT is somthing that needs to be acknowledged by senior management. It's a matter of resource allocation, if it's not understood that "you need to go slow in order to go fast" - then the scenarios that were mentioned above are doomed to happen, and perhaps there's no point in starting such an effort. * The whole idea of automation is that UT are repeatedly changed. If a not-previously-tested value causes a bug to be seen - then you add a test case. And now it's there. Yes, it's time consuming. But after doing that, I can tell you that it's totally worth it. The amount of problems that were discoveded at early stages (often because of ill or complex , but legal inputs) was undoubtedly worth it.
  40. Well agreed that Unit Test is expensive ... especially you are dealing with legacy code ... At times, I feel brave to give ahead to do with without UT, at other times I have not much choice as I am dealing with legacy code... but i tried to put in UT as soon as I can. Make me feel safer.... well ... I am not as "brave" as most other developers ... but it is really cheaper without UT? in short term, it may seen so ... but in longer term ... say months later ... it is really a blessing to see your UT there ... to tell you exactly how your code is "suppose" to work .... it becomes a great help ... and you and modify without "FEAR" ... or much less fear without UT ... you might be "lost" in your own code, and may take several hours to understand it again for those in my same situation ... of modifying legacy code ... I would suggest to read the book by Michael Feathers http://www.objectmentor.com/resources/bookstore/books/welc/
    A few comments:

    * UT is not only for the developers. It brings a much more stable code to the QA team, which can then find the "real" problems in the S/W. It is good for the presale/postsale guys who get better feed back from the customers. It is no magic, it won't make the product better if it sucks. But it is a major part in the chain, IMHO.

    * UT is exepensive. In my experience too, it takes around 50% of the development time. I think that if we want to treat our profession with some respect, than it's a price we need to pay (and yes, I'll mention the automobile metaphore here again. Not all software is installed on rockets and spacecrafts, but it's supposed to work, I believe)

    * UT is somthing that needs to be acknowledged by senior management. It's a matter of resource allocation, if it's not understood that "you need to go slow in order to go fast" - then the scenarios that were mentioned above are doomed to happen, and perhaps there's no point in starting such an effort.

    * The whole idea of automation is that UT are repeatedly changed. If a not-previously-tested value causes a bug to be seen - then you add a test case. And now it's there. Yes, it's time consuming. But after doing that, I can tell you that it's totally worth it. The amount of problems that were discoveded at early stages (often because of ill or complex , but legal inputs) was undoubtedly worth it.
  41. without UT ... you might be "lost" in your own code, and may take several hours to understand it again
    I think that this is a key to the good codebase - no "own" code. Complex (and buggy) code cannot survive reading, fixing and testing by all team members in random order. Finally they are refactor it, fix it, and will know in detail how it works and how to test it. You can consider this as "adaptive" pair programming - complex and bad code get attention from many team members (not just 2 developers) and no additional developers require just to see how one write good and simple code. Maxim Kramarenko http://www.trackstudio.com - hierarchical issue tracking
  42. Testing is for suckers.