Should Developers Test the Product First?

Discussions

News: Should Developers Test the Product First?

  1. In "Should Developers Test the Product First?," James Bach suggests that developer testing, while important, might not be quite as important as some think it is. It makes sense, considering that developers know how the code should work, so they tend to test the 'happy paths' and not the real ones.
    Many testers would advise the programmer to test the product himself, first. I have a different answer. My answer is: send me the product the moment it exists. I want avoid creating barriers between testing and programming. I worry that anything that may cause the programmers to avoid working with me is toxic to rapid, excellent testing. Of course, it's possible to test the product without waiting to send it to the testers. For instance, a good set of automated unit tests as part of the build process would make the whole issue moot. Also, I wouldn't mind if the programmer tested the product in parallel with me, if he wants to. But I don’t demand either of those things. They are a lot of work. As a tester I understand that I am providing a service to a customer. One of my customers is the programmer. I try to present a customer service interface that makes the programmers happy I'm on the project.
    Later, he adds this:
    Sometimes a product is so buggy that I can't make much progress testing it. Even then, I want to have it. Every look I get at it helps me get better ideas for testing it, later on. Sometimes the programmer already knows about the bugs that I find. Even then, I want to have it. I just make a deal with the programmers that I will report bugs informally until we reach an agreed upon milestone. Any bugs not fixed by that time get formally reported and tracked. Sometimes the product is completely inoperable. Even then, I want to have it. Just by looking at its files and structures I might begin to get better ideas for testing it. My basic heuristic is: if it exists, I want to test it. (The only exception is if I have something more important to do.)
    Most agile proponents would suggest that there is no possible exception to such a rule, if sending code to testers is a regular process. He also doesn't quite address code that doesn't face users directly (i.e., library code that has no UI component.) What do you think?

    Threaded Messages (49)

  2. Basic unit testing and verfication is part of and developer's job. Beyond that, it's at worst a painful experience that the developer won't take all that seriously and at best a complete waste of time.
  3. Basic unit testing and verfication is part of and developer's job. Beyond that, it's at worst a painful experience that the developer won't take all that seriously and at best a complete waste of time.
    Which is exactly why most software is crap. Peace, Cameron Purdy Tangosol Coherence: Clustered Shared Memory for Java
  4. LOL[ Go to top ]

    Can't disagree with Cameron. In general, I think that testing in software is no different from testing in other industries. We do as little as possible. In some cases, testing had to be mandated by law ... in some surprising cases. For example, the FDA was created in part because the food and drug industry had no concern for QA, including testing. This despite the fact that some of those products killed people ... So it is hardly surprising that the software industry suffers from this problem. I've heard people blame this on a lack of management commitment or poor processes. I've heard it said that software developers, by nature, are a lot more interested in solving problems than in checking their solutions. I would argue that this is true of human beings in general, but then how could I explain accountants ? I don't think any of that is true. I think that the real core here is that software is really complicated and that testing it is really complicated. When I look at the lowest defect rate stuff, like that NASA produced (produces ? I'm out of touch), the unusual things I notice are the big budgets for cost and time. It seems that, given time and money, the software developers there did (do ?) a great job finding and removing defect. So it's not a software developer thing. Is it a management thing ? Well, in a sense, I suppose it is, but not in a realistic sense. What I mean is that if management could arrange for the time , money, and morale of the NASA team, they could expect better results. Of course that's just not possible most places most of the time. So it's not a management thing either. No, it's partly that testing software is amazingly complicated. It takes a surprising amount of time and money. Most businesses can't afford to pay. This seems like a good place for a foolish observation. I am amused nowadays when a developer or manager beats his chest and says : 'It's not rocket science.' That's right. What we do is not rocket science. It does not require specialized math skills or the astounding imaginative feats of physics geniuses. Nope, all it requires is tracking thousands of logical pathways in millions of data contexts, just your average everyday impossible task for any human being. It's much more complicated than rocket science. Anyway, so I think that 'it's partly that testing software is amazingly complicated'. What's the other partly part in my view ? Well, simply that the software we make is, in fact, good enough. We (rightly) observe its poor quality. But we also run the country with it. Perhaps we can make an analogy with one of Henry Ford's (the first one) cars ? No, sir, it is not the best possible, but it is usable and affordable. And, you know, that's the human way. We don't usually care to wait for perfection ....
  5. Which is exactly why most software is crap.
    I would say that the biggest reason is very simple: people do not care about results; there is no sense of connectedness to the results. Or worse: people hate the end results.
  6. Which is exactly why most software is crap.


    I would say that the biggest reason is very simple: people do not care about results; there is no sense of connectedness to the results. Or worse: people hate the end results.
    I full-heartedly agree with this statement. If developers don't test well, it's because of apathy. I hate getting bug reports from QA with a passion. I hate having a flaw in something I spend so much time working on. I attempt to test as many aspects of my code as possible before I hand it to QA (within reason). As another poster stated, I know the code best and I know what I'm "unsure/uncomfortable" with on the sites I develop. I make sure to test those because my QA team rarely finds them. Another reason I test so thoroughly is because our QA team is understaffed. We've actually had instances where we pulled developers off of other projects to help test another site's release. (such a waste)
  7. I have to disagree with this approach, mostly because it seems to excuse laziness on the part of developers. First, if a developer does not test his or her own work, how does one know when to turn it over for the next level of testing? My fear is that this leads to a "throw it over the wall" mentality. If the developer can inflict the pain of utterly pointless retesting on someone else, they are more likely to just throw hacks together and how they somehow work. It is so much better to test it yourself while the problem and solution are still fresh in your mind. There is always overhead with a release to testing, even if it is nothing more than sending an email and having someone switch contexts. I am a fan of releasing early and often to get feedback and new direction. However, that assumes that what you are releasing will allow someone to give that feedback. If all you can get is "It still crashes when I click 'Start'", there is no real value here except being able to claim the problem is in someone else's hands for a few minutes. Yes, I have seen developers turn "fixes" over to QA right before a status meeting just so they can say they made progress. It is dishonest a waste of valuable testing time. Finally, I think the author is missing the positive side developers knowing the code so well. That should allow them to test the exceptional conditions, the branches in the code that may not be apparent to someone who does not know the implementation, and the parts they themselves suspect could be wrong. If a developer is only testing the "happy paths", it is nothing but negligence. That is the sort of unprofessional behavior that makes all of us look bad and, as Cameron pointed out, is one reason most software is garbage. If a tester, in his or her professional judgment, sees a value in testing as soon as the code compiles, so be it. However, that does not absolve the developers from doing their own jobs. Vince Frisina http://www.jroller.com/page/vfrisina
  8. I have to disagree with this approach, mostly because it seems to excuse laziness on the part of developers.

    First, if a developer does not test his or her own work, how does one know when to turn it over for the next level of testing?
    I don't think anyone is saying that the developer shouldn't do any testing. I argue that it's not worthwhile to make developers do functional testing. In general, developers: 1. don't like testing 2. aren't good at testing (because of 1) 3. Are harder to find and cost more than testers 4. Have trouble seeing the problems they don't see. In other words, if I make a major error, it's not reasonable to expect that upon putting on a tester hat, I will suddenly become aware that I am making am major error.
    My fear is that this leads to a "throw it over the wall" mentality. If the developer can inflict the pain of utterly pointless retesting on someone else, they are more likely to just throw hacks together and how they somehow work.
    That seems pretty absurd. Have the testers had their tongues cut out? Te reality is, having someone else check your work is likely to make you do a better job. People don't look behind my stove so I don't really clean back there. I've seen it over and over that if you let developers (escpecially contractors) check their own work, they deliver unmaintainable crap because no one is there to stop them.
    There is always overhead with a release to testing, even if it is nothing more than sending an email and having someone switch contexts.
    You seem to be ignoring that this is true if the developer is doing the testing too. It's actually a greater context switch than putting another task in a tester's queue.
    I am a fan of releasing early and often to get feedback and new direction. However, that assumes that what you are releasing will allow someone to give that feedback. If all you can get is "It still crashes when I click 'Start'", there is no real value here except being able to claim the problem is in someone else's hands for a few minutes. Yes, I have seen developers turn "fixes" over to QA right before a status meeting just so they can say they made progress. It is dishonest a waste of valuable testing time.
    And where are these developers' managers when this is happening? If this is what you see then you have a management problem. Why would such behavior be acceptable?
    ... That is the sort of unprofessional behavior that makes all of us look bad and, as Cameron pointed out, is one reason most software is garbage.

    If a tester, in his or her professional judgment, sees a value in testing as soon as the code compiles, so be it. However, that does not absolve the developers from doing their own jobs.
    Why do you think a functional testing should be a developer's job? What other profession has such an inefficient organization? Why shouldn't we do the same in software engineering?
  9. One controversy at a time ;-)[ Go to top ]

    Why shouldn't we do the same in software engineering?
    Well, mainly because there is no such thing as 'software engineering' in most organizations. There may be in yours, and you may be an engineer - I'm not making any personal attacks here. The people there would have to have MS degress in something called - and recognized in your state - as 'software engineering'. Right ? Because for a guy to call himself an engineer when he's just a software developer would be pointless and destructive to the software developement community in general, right ? I'm not just flaming here. The software development community in general has not yet earned the right to be called engineers. I think that one of the issues we face is acquiring the discipline and experience required to be engineers ... but that is some way in the future, as far as I can tell. It's a big deal.
  10. Why shouldn't we do the same in software engineering?


    Well, mainly because there is no such thing as 'software engineering' in most organizations. There may be in yours, and you may be an engineer - I'm not making any personal attacks here. The people there would have to have MS degress in something called - and recognized in your state - as 'software engineering'. Right ? Because for a guy to call himself an engineer when he's just a software developer would be pointless and destructive to the software developement community in general, right ?

    I'm not just flaming here. The software development community in general has not yet earned the right to be called engineers. I think that one of the issues we face is acquiring the discipline and experience required to be engineers ... but that is some way in the future, as far as I can tell. It's a big deal.
    So would you say that there was no such thing as an engineer before such certifications existed? What would you say makes someone an engineer?
  11. So would you say that there was no such thing as an engineer before such certifications existed?
    I would say exactly that assuming that a degree is a certification in your sense) ... because 'engineering' does not depend upon the genius of an inidividual. 'Engineering' in my understanding has more to do with an accumulated body of knowledge and generally accepted practice and standards than it has to do with an individual. And this has real value ... that Leonardo Da Vinci was a genius did not help humanity as much as that there are generally accepted standards for building bridges.
    What would you say makes someone an engineer?
    In the United States, I would say that an engineer is someone who has earned an engineering degree recognized by his state. No more, but certainly no less. I hold this view because this approach seems to have worked well for recognized engineering disciplines. I take the time to post because I believe that software development may become an engineering profession and that we do need to make strides in that direction. We depend upon the genius of individuals at this point ... early in our history; but that is not a reliable basis for progess. The first step, as always, is to recognize the need, which is made more difficult if we hide it under dubious titles.
  12. hijacking this thread[ Go to top ]

    gents..you're hijacking this thread. lets talk testing
  13. What would you say makes someone an engineer?


    In the United States, I would say that an engineer is someone who has earned an engineering degree recognized by his state. No more, but certainly no less. I hold this view because this approach seems to have worked well for recognized engineering disciplines.
    I guess I disagree with that. Here's an example of someone I consider an engineer whom I think does not meet the muster you describe: http://en.wikipedia.org/wiki/Isambard_Kingdom_Brunel
  14. ?[ Go to top ]

    Brunel was appointed as an engineer ... so I would not argue that he was not an engineer. I don't know enough about it, although mechanical engineering was a recognized trade at that time. In any event, I don't know what the standards were at the time, but I would not expect them to be the standards that prevail in the United States today. Of course, many people disagree with my view. I try to maintain it as an effort, however miniscule, to spur software developers to reach the standards set by universally recognized engineering disciplines. That's not an easy challenge, and I do not think we have met it yet. Hijacking the thread .. well, really, it needs to be highjacked. Whether or not developers are required to test at a certain level (obviously they test at some level) is not up to them.
  15. Re: ?[ Go to top ]

    Brunel was appointed as an engineer ... so I would not argue that he was not an engineer. I don't know enough about it, although mechanical engineering was a recognized trade at that time. In any event, I don't know what the standards were at the time, but I would not expect them to be the standards that prevail in the United States today....
    Here's my thing. I think software engineering is at the same stage as mechanical engieering was in Brunel's day. It is still being worked out. We can't standardize on who is a software engineer because hardly anyone agrees! The techologies are still so new and the techniques changing so rapidly that we are still in the primordial stage of our field. I agree that at some point in the future it will become more like a mechanical or structural engineer is today and we can make steps towards it but I don't see that happening for a while. In any event, software engieers make as much as $10K more on average than do software developers. Put software engineer on your resume because the distinction is almost completely arbitrary IMO ;)
  16. In the United States, I would say that an engineer is someone who has earned an engineering degree recognized by his state. No more, but certainly no less. I hold this view because this approach seems to have worked well for recognized engineering disciplines.
    Guess I'm not an engineer, then ... my degree is in Business, not engineering :-). Craig McClanahan
  17. Just so[ Go to top ]

    Well, that's right, Craig. You are not an engineer. I'm not sure why you think your post makes a difference ?
  18. LOL...[ Go to top ]

    Well, that's right, Craig. You are not an engineer.

    I'm not sure why you think your post makes a difference ?
    ROFL.. If I'm not mistaken Craig wrote Struts and Tomcat.
  19. Re: LOL...[ Go to top ]

    Well, that's right, Craig. You are not an engineer.

    I'm not sure why you think your post makes a difference ?


    ROFL.. If I'm not mistaken Craig wrote Struts and Tomcat.
    ROFL ? You are not completely mistaken, although of course many people 'wrote Tomcat', and Craig still is not an engineer. I don't think producing code makes a person an engineer, even if the code is popular. In some ways these two tools are exactly the kind of thing I think we would stop using if we had a true engineering focus ... Ah, I guess I don't want to go there. I have nothing against Craig personally and I don't want this to turn into a pointless criticism of some individual.
  20. As in any profession, the degree is what gives the individual a certain title, whether it be a doctor, lawyer, engineer, accountant, etc. and not an indicator of truly how smart (or not) that person is. In the software field you have two primary choices of study (depending on which university you attend): computer science or computer/software engineering. The CompSci path provides more of a programming one versus the engineering path which students must take classes covering programming in addition to electrical/mechanical engineering, physics, chemistry, as well as higher-level mathematics (differential equations, numerical methods, advanced algebra, etc.) as well as hardware-related courses which teach you how to actually build computer chips (adders/subtracters/multipliers/etc.) and clocking techniques. There is a difference between the two fields, regardless of the fact that most of us end up architecting and developing "software". Personally, I have a bachelor's degree in computer engineering as well as a master's degree in software engineering. I have worked with programmers who have degrees in business, psychology, etc. but decided they liked programming. It is my observation that those who have obtained more advanced degrees seem to be (in general) more structured in their approach to software, including builds, packaging, testing, etc. and deliver a better product. As was pointed out earlier, this is rare and therefore why most software is "crap".
  21. As in any profession, the degree is what gives the individual a certain title, whether it be a doctor, lawyer, engineer, accountant, etc. and not an indicator of truly how smart (or not) that person is.

    In the software field you have two primary choices of study (depending on which university you attend): computer science or computer/software engineering. The CompSci path provides more of a programming one versus the engineering path which students must take classes covering programming in addition to electrical/mechanical engineering, physics, chemistry, as well as higher-level mathematics (differential equations, numerical methods, advanced algebra, etc.) as well as hardware-related courses which teach you how to actually build computer chips (adders/subtracters/multipliers/etc.) and clocking techniques. There is a difference between the two fields, regardless of the fact that most of us end up architecting and developing "software".

    Personally, I have a bachelor's degree in computer engineering as well as a master's degree in software engineering. I have worked with programmers who have degrees in business, psychology, etc. but decided they liked programming. It is my observation that those who have obtained more advanced degrees seem to be (in general) more structured in their approach to software, including builds, packaging, testing, etc. and deliver a better product. As was pointed out earlier, this is rare and therefore why most software is "crap".
    Here, finally, is a true software engineer. Kudos to you, Shaun ! (May I ask where you obtained your software engineering degree ? No system in my state offers one. I would like to find a program.) I quite agree with your post. I should restate that in my view being an engineer does not make a person smarter or even better at building software. It just makes a person an engineer. In the long run engineers may produce software that is better on the average because of their structured approach and the shared knowledge of techniques and standards. I presume that's the goal of developing an engineering discipline in software development.
  22. (May I ask where you obtained your software engineering degree ? No system in my state offers one. I would like to find a program.)
    University of Arkansas (Fayetteville, AR) for BS in Computer Engineering. Southern Methodist University (Dallas, TX) for MS in Software Engineering.
    I quite agree with your post. I should restate that in my view being an engineer does not make a person smarter or even better at building software. It just makes a person an engineer.
    Exactly...an engineering degree is not symbolic of brains - I see this proven often.
  23. Thanks man !!!
  24. As in any profession, the degree is what gives the individual a certain title, whether it be a doctor, lawyer, engineer, accountant, etc. and not an indicator of truly how smart (or not) that person is.

    In the software field you have two primary choices of study (depending on which university you attend): computer science or computer/software engineering. The CompSci path provides more of a programming one versus the engineering path which students must take classes covering programming in addition to electrical/mechanical engineering, physics, chemistry, as well as higher-level mathematics (differential equations, numerical methods, advanced algebra, etc.) as well as hardware-related courses which teach you how to actually build computer chips (adders/subtracters/multipliers/etc.) and clocking techniques. There is a difference between the two fields, regardless of the fact that most of us end up architecting and developing "software".

    Personally, I have a bachelor's degree in computer engineering as well as a master's degree in software engineering. I have worked with programmers who have degrees in business, psychology, etc. but decided they liked programming. It is my observation that those who have obtained more advanced degrees seem to be (in general) more structured in their approach to software, including builds, packaging, testing, etc. and deliver a better product. As was pointed out earlier, this is rare and therefore why most software is "crap".
    Computer science is not programming. A computer scientist should know basic electrical engineering and have a good understanding of digital design. If anything, a computer scientist should have a stronger math background because really CS is just a step away from applied computational mathematics. I'm not saying there aren't plenty of CS programs that don't teach any CS, but that's another problem entirely. Outside of disputing what CS is, I agree entirely with you that people with solid CS/CE/EE (and maybe some mathematical disciplines) educational foundations tend to be much more structured in their problem solving, ultimately resulting in a better product. Most software developers are technicians. They're more like mechanics than engineers. They understand the techologies like Java, some frameworks, DBMSes, etc; but they neither understand how those technologies work (engineering) or the underlying theoretical constructs (science, and obviously engineering requires a good scientific background). Don't get me wrong, I think you can fill an entire highly intelligent brain solely with information regarding Oracle (or fill-in-the-blank product/technology), and thereby be a very skilled technician - but not an engineer.
  25. Well, mainly because there is no such thing as 'software engineering' in most organizations.
    In any event, this is pretty much a sematical argument. I really don't see how "not 'real' engineer" -> "do your own testing"
  26. Why shouldn't we do the same in software engineering?


    Well, mainly because there is no such thing as 'software engineering' in most organizations. There may be in yours, and you may be an engineer - I'm not making any personal attacks here. The people there would have to have MS degress in something called - and recognized in your state - as 'software engineering'. Right ?
    I don't think that MS can grant SW engineering degrees better that an university. At all.
    Because for a guy to call himself an engineer when he's just a software developer would be pointless and destructive to the software developement community in general, right ?

    I'm not just flaming here. The software development community in general has not yet earned the right to be called engineers. I think that one of the issues we face is acquiring the discipline and experience required to be engineers ... but that is some way in the future, as far as I can tell. It's a big deal.
    I lived in the times when analysis was structural and programming language was C. At that time a distinction of roles in analyst, designer, developer made sense. When I first touched OO way of thinking I soon realised that the distinction is totally absurd. A Java (or C++) developer, as a role, is really a waste of time. OO implementations are, IMO, a continuous design and architectural decisions process. There is no room for plain writing of programming language statements (assuming that a - Java - SW developer knows pretty well the API and the syntax and have little or no knowledge of design patterns). Guido.
  27. Re: One controversy at a time ;-)[ Go to top ]

    When I first touched OO way of thinking I soon realised that the distinction is totally absurd.
    A Java (or C++) developer, as a role, is really a waste of time.
    OO implementations are, IMO, a continuous design and architectural decisions process. There is no room for plain writing of programming language statements (assuming that a - Java - SW developer knows pretty well the API and the syntax and have little or no knowledge of design patterns).

    Guido.
    and OO also makes coffee, cleans the table after you, among other great things ...
  28. Re: One controversy at a time ;-)[ Go to top ]

    When I first touched OO way of thinking I soon realised that the distinction is totally absurd.
    A Java (or C++) developer, as a role, is really a waste of time.
    OO implementations are, IMO, a continuous design and architectural decisions process. There is no room for plain writing of programming language statements (assuming that a - Java - SW developer knows pretty well the API and the syntax and have little or no knowledge of design patterns).

    Guido.


    and OO also makes coffee, cleans the table after you, among other great things ...
    Maybe we have different understanding of the role of a SW developer, that could be the cause of your, I will not qualify further, odd post. What I mean is that I think it is impossible (really hard) to think at coding in OO way (even in C - take a look at X11R6 Xt Intrinsics Toolkit source code - ) without really strong design, engineering and refactoring background in people that write statements. Unless you decide to: 1. Accept a certain amount of mess that someone with the above background must fix later 2. Someone with the above backgroud provides a rather detailed design of the system that leaves about no choice to the implementor 3. Leave the mess where it is until you have to change those pieces of code Well, I don't like to work this way. Simply. I try to prevent problem as early as I can. I don't like to discover design problems at testing time. Guido.
  29. Re: One controversy at a time ;-)[ Go to top ]

    What I mean is that I think it is impossible (really hard) to think at coding in OO way (even in C - take a look at X11R6 Xt Intrinsics Toolkit source code - ) without really strong design, engineering and refactoring background in people that write statements.
    Thanks for explaining, I understand your PoV better now. Reading your statement from earlier post, I thought you felt OO was the panacea to all the software development problems, including the need to test. The testing role will remain in projects, who plays that role, and how much of testing skills specialization one project has should be dependent on the problem the s/w is intended to solve. Space shuttle s/w testing needs/time/budget will vary from simple DB CRUD/reporting website s/w. My opinion: it definitely helps to have another person look over my work, if only to see problems I may not have foreseen. Whether that other person is another developer, pair developer, a tester, BA, or the end-customer has varied for me from project to project [not necessarily commensurate with value the projects were bringing to the organization; there has been gap between how many resources of what kinds *should* have been allocted to projects v/s how many were *actually* allocated; and I have experienced both +ve and -ve gap].
  30. I have to disagree with this approach, mostly because it seems to excuse laziness on the part of developers.

    First, if a developer does not test his or her own work, how does one know when to turn it over for the next level of testing?


    I don't think anyone is saying that the developer shouldn't do any testing.
    I understand that, and I suspect that the difference between my position and yours is where we draw the line between the testing that should be expected of developers and the testing that should be performed by a testing organization alone at different points. The original author spoke about getting software that barely ran. In my opinion, the developer has not done his or her job at that point. One has to have a good faith belief that the software is ready--whatever that means in a given situation--before handing it off to someone else to verify. My concern is that there are always pressures to show progress, to move on to the next task, and this approach may encourage people to cut corners. I cannot recall having a problem where developers spent too much time checking their work. Rather, the problem I have observed has always been at the other extreme, in which we treat quality as the concern of some outside entity like QA.
    I argue that it's not worthwhile to make developers do functional testing.

    In general, developers:

    1. don't like testing
    2. aren't good at testing (because of 1)
    3. Are harder to find and cost more than testers
    4. Have trouble seeing the problems they don't see. In other words, if I make a major error, it's not reasonable to expect that upon putting on a tester hat, I will suddenly become aware that I am making am major error.
    I agree that when I make an error in code, I am not likely to realize it, especially when it is a misunderstanding on my own part rather than I simple typo. That is why having someone else read your code catches more errors than just reviewing it yourself. However, when I make a mistake and then see a test fail, I am more likely to recognize the problem and know exactly where I made it because I just wrote the faulty code. I would say it is much easier to find and fix the problem when the error is fresh than hours or days later when I get an error report from a tester.
    I've seen it over and over that if you let developers (escpecially contractors) check their own work, they deliver unmaintainable crap because no one is there to stop them.

    Absolutely. I would not want to do away with testers, but rather ensure that they are used more efficiently. Software should not be turned over to testing unless the developer has a good faith belief that it is ready.

    And where are these developers' managers when this is happening? If this is what you see then you have a management problem. Why would such behavior be acceptable?
    We are talking about development process, and process cannot be separated from management. Where the process allows or even encourages a developer to turn over work that he or she does not believe is ready, you get some odd behaviors. In this case, if they were found out and the fact that they were just gaming the system is apparent, they got their hands slapped. However, this is just a symptom of the belief that their job is just to churn out code but making sure it works is someone else's problem. That is the very attitude that I believe contributes to the general lack of quality in software. Unfortunately, when management starts believing it as well, the behavior I mentioned becomes acceptable and the entire organization suffers.


    ...

    That is the sort of unprofessional behavior that makes all of us look bad and, as Cameron pointed out, is one reason most software is garbage.

    If a tester, in his or her professional judgment, sees a value in testing as soon as the code compiles, so be it. However, that does not absolve the developers from doing their own jobs.


    Why do you think a functional testing should be a developer's job? What other profession has such an inefficient organization? Why shouldn't we do the same in software engineering?
    In which professions is one not responsible for checking his or her own work before handing it off? Authors proofread before sending copy to an editor. Accountants make sure the numbers balance before handing the books to an auditor. Doing otherwise brands you as sloppy in the eyes of your peers. The question is how much testing is required, and I think that our field as a whole does far too little, not too much. Vince Frisina http://www.jroller.com/page/vfrisina
  31. 4. Have trouble seeing the problems they don't see. In other words, if I make a major error, it's not reasonable to expect that upon putting on a tester hat, I will suddenly become aware that I am making am major error.


    I agree that when I make an error in code, I am not likely to realize it, especially when it is a misunderstanding on my own part rather than I simple typo. That is why having someone else read your code catches more errors than just reviewing it yourself. However, when I make a mistake and then see a test fail, I am more likely to recognize the problem and know exactly where I made it because I just wrote the faulty code.
    I think you are missing my point. The developer will not test for something that the developer doesn't think is an issue. If the developer misunderstands the requirement to any degree, that developers testing will also but under that incorrect understanding. I have been in the past and current am solely responsible for the testing of my code. Everyone on my team is likewise responsible for their own testing. It's an inefficient and ineffective way to approach development. I have worked on teams where the developers did the development did some basic testing (i.e. it doesn't blow up and looks fairly correct) and move on to more development. All the tester does is fail the test if it doesn't meet the reqirements. It's not a huge burden and it produced much better results and much happier customers. I can't even count the number of times I was completely misunderstanding a requirement. Having me not only waste time developing the wrong thing but then wasting a lot of time testing and fixing the wrong thing isn't helping anyone, it just compounds the problems.
  32. Have to agree[ Go to top ]

    I think you are missing my point. The developer will not test for something that the developer doesn't think is an issue. If the developer misunderstands the requirement to any degree, that developers testing will also but under that incorrect understanding.

    I have been in the past and current am solely responsible for the testing of my code. Everyone on my team is likewise responsible for their own testing. It's an inefficient and ineffective way to approach development.
    I think we have all had this experience and I personally cnnot see a reasonable argument against it. It's a problem.
  33. 4. Have trouble seeing the problems they don't see. In other words, if I make a major error, it's not reasonable to expect that upon putting on a tester hat, I will suddenly become aware that I am making am major error.


    I agree that when I make an error in code, I am not likely to realize it, especially when it is a misunderstanding on my own part rather than I simple typo. That is why having someone else read your code catches more errors than just reviewing it yourself. However, when I make a mistake and then see a test fail, I am more likely to recognize the problem and know exactly where I made it because I just wrote the faulty code.


    I think you are missing my point. The developer will not test for something that the developer doesn't think is an issue. If the developer misunderstands the requirement to any degree, that developers testing will also but under that incorrect understanding.

    I have been in the past and current am solely responsible for the testing of my code. Everyone on my team is likewise responsible for their own testing. It's an inefficient and ineffective way to approach development. I have worked on teams where the developers did the development did some basic testing (i.e. it doesn't blow up and looks fairly correct) and move on to more development. All the tester does is fail the test if it doesn't meet the reqirements. It's not a huge burden and it produced much better results and much happier customers. I can't even count the number of times I was completely misunderstanding a requirement. Having me not only waste time developing the wrong thing but then wasting a lot of time testing and fixing the wrong thing isn't helping anyone, it just compounds the problems.
    Yes, I did miss your point. Having faced different problems, you and I are discussing different solutions. Having developers be solely responsible for testing is not a good idea for all the reasons you listed. What I was advocating is that developers should do enough testing to have a good faith belief that the software does what they expect it to do before handing it off to someone else to verify. At the very least, that means running unit tests and a smoke test. If the software does not run, no tester can tell you that you misunderstood the requirements. I also believe in releasing early and often to customers or their proxies who can then give feedback and new direction. That addresses the problem of developers misinterpreting the requirements or their own misstating of their needs. However, it only works if the customer/proxy can actually evaluate the work product. The problem I have faced is that there are a large number of developers who do not know what they are doing. Some are just green and need to learn how software development works outside of a classroom. Others have no urge to learn. On top of that, all of us are frequently under pressure to deliver something with quality being treated as a problem to be solved later. My concern is that the process described by the original author would lead to the quality of a developer's work being treated as someone else's problem. That cuts out a valuable feedback loop in which the rate at which code is turned over is throttled to the rate at which a certain level of quality can be maintained. Vince Frisina http://www.jroller.com/page/vfrisina
  34. The problem I have faced is that there are a large number of developers who do not know what they are doing. Some are just green and need to learn how software development works outside of a classroom. Others have no urge to learn. On top of that, all of us are frequently under pressure to deliver something with quality being treated as a problem to be solved later. My concern is that the process described by the original author would lead to the quality of a developer's work being treated as someone else's problem.
    I guess it's hard for me to imagine not doing some basic validation of my work before marking it done and moving on to the next task. The problem you are running would require some creative management such as carrying a baseball bat when you talk to these developers about their work. In all seriousness though, I would consider this a managment issue. I can see how this would be a waste of time for the testers. I have had cases where I had something working and someone checked in some code that broke it completely or my test cata was not hitting the bad situations. As long as these are the exception and not the rule, I don't think it should a huge problem for the testers.
  35. In which professions is one not responsible for checking his or her own work before handing it off? Authors proofread before sending copy to an editor.
    Are you sure about that? The whole point of the editor is that he is skilled in editing. The author's skill is writing. It's ineffiecient for the author to attempt to do the editor's job before the editor does it. Skim over the work for obvious errors? Of course. But to polish it to great degree is just a duplication of effort.
    Accountants make sure the numbers balance before handing the books to an auditor.
    That's a different type of situation because often there are penalties if an auditor finds an error. It's an adversational relationship.
    Doing otherwise brands you as sloppy in the eyes of your peers. The question is how much testing is required, and I think that our field as a whole does far too little, not too much
    This is the point that the author (a tester) is making. The developer wants to polish program so that he doesn't appear sloppy. The author (the tester) wants to see what the developer is doing before he puts a lot of work into something of no value. Not wanting to be seen as sloppy is an admirable trait but really has nothing to do with producing a good result in this case.
  36. goals of testing?[ Go to top ]

    seems like there should be some goals of testing that drive the testing process... goals which are tightly coorelated to the eventual business goals of the app. as an example: how important is it to streamline a particular chunk of code if that code is invoked very infrequently? shouldn't testing focus on ways to balance achieving "perfect code" with acceptable performance/scalability needed to deliver the business benefits for which said app was developed? as i am new to the testing world, i'd appreciate your feedback on this line of thinking.
  37. ugh![ Go to top ]

    what uh horrible discussion. ya'll gots uh lot uh time on yer hands! You wascals you!
  38. Test and be Better[ Go to top ]

    I guess that my point of view here is that the more the developers test the better the developer becomes. Some of you guys might have been born superstar programmers since your bottle-feeding days; some of you may have been God-graced with the innate will to properly design and test your code, but I wasn't. I learned to spend more time on design, use more object concepts, think about boundary conditions, build my code with unit testing in mind, and spend the time to automate the testing and build through a lot of failure, a lot of frustration, and a lot of shitty time wasted testing it myself. You're right-spending your time trying to figure out some stupid bug like "button didn't work" is a huge waste of time-but it's still my bug, and the huge amount of time I wasted made me a little more careful designing in the future. I also learned a lot about building a better product for my customer by doing a lot of the functional testing myself and by spending a lot of time sitting with them. I like to think that development is more than just code monkeying, and that I have some kind of creative solution to offer. And the more that I interact with the client, the better that creative solution will be. Plus, I don't know what kind of magical functional specs you guys get, but mine never cut the mustard. When I spend time trying to do what my customers do, I better understand their requirements and can anticipate application problems before they occur, and develop what they want, not what some half-ass spec says they do. Those are my two cents. Joe
  39. Re: Test and be Better[ Go to top ]

    You're right-spending your time trying to figure out some stupid bug like "button didn't work" is a huge waste of time-but it's still my bug, and the huge amount of time I wasted made me a little more careful designing in the future.
    That's bug-fixing. No one is saying you wouldn't fix your own bugs. I don't know how you became confused about this. Testers don't fix bugs.
  40. When looking at the question of who does the testing, I have a hypothetical situation that just about every developer had encountered. You have a screen that has 15 possible input fields. The number of required fields is 4. There are also 2 sets of fields that are "or" required. My definintion of an "or" required field set is that within the set one of the fields is required. The rest of the fields are optional. 5 of the fields are multiple selection lists. What I have found is that there is a wide variance within organiztions of the quality of testers within their QA group. I have seen QA groups that wouldn't test all possible combinations. We can all agree that this is an infamnia. I am taking into assumption that an independant QA group WILL do the test of every possible combination. Let us also assume that the cost per hour of a QA person is half that of a programmer. My question for the community is this: Should we require a programmer/developer to run through every single permutation before handing off the code to QA? If you are running in a Agile development shop, do you have the people writing the test cases the same people that write the actual operating code? Why / why not? John Murray Sobetech
  41. My question for the community is this: Should we require a programmer/developer to run through every single permutation before handing off the code to QA? If you are running in a Agile development shop, do you have the people writing the test cases the same people that write the actual operating code? Why / why not?
    Straw man ;-) Doing tedious labor such as what you have described is not worth paying any person for, no matter how cheap they are. That type of effort is what automated testing systems are designed for. In the case of software libraries and APIs, developers need to write the code for automating those coverage and combinatorial tests. Testing isn't about keyboard banging. It's a process that is begun well before the first line of code is punched in to a keyboard. Testing is about architecting software in such a way that you can prove through its very description that it is indeed correctly implementable; designing software in such a way that it will tend to always fail or always work; and implementing software so that its quality is actually provable by testing. If an organization cannot adopt those things as basic principles, then applying Q/A to the end result will be of no value. Peace, Cameron Purdy Tangosol Coherence: Clustered Shared Memory for Java
  42. Testing isn't about keyboard banging. It's a process that is begun well before the first line of code is punched in to a keyboard.
    Yup. The Deming circle is always valid.
    Testing is about architecting software in such a way that you can prove through its very description that it is indeed correctly implementable; designing software in such a way that it will tend to always fail or always work; and implementing software so that its quality is actually provable by testing.
    I think 'architecting' is about architecting software, etc., 'designing' is about designing, etc., and 'implementing' is about implementing, etc.. Testing is not about any of these. Testing is a separate activity designed to examine whether the above have been completed. True, the people performing these activities should consider testability as one requirement ... All of those activities are QA activities, and as always, testing is only a part of QA ... yes, I know this is common knowledge. Splitting hairs ? No. Different people should have architect and tester roles, and all people have the QA role.
  43. It's a process that is begun well before the first line of code is punched in to a keyboard. Testing is about architecting software in such a way that you can prove through its very description that it is indeed correctly implementable; designing software in such a way that it will tend to always fail or always work; and implementing software so that its quality is actually provable by testing.
    I wholeheartedly agree. That is why I put in the ending part of my question.
    If you are running in a Agile development shop, do you have the people writing the test cases the same people that write the actual operating code? Why / why not?
    In the instances where I have experienced JUnit within the development process, the same person was writing the code to be tested and the test cases. I think my overall exposure is limited. But I am curious to see if this is truly common. Good design mandiates the definition of objects and the classes/methods that operate on them far before the actual physical development of the code. At that point test cases and JUnit (pick you flavor) classes can be developed independantly from the end developer. Therefore, QA/Testing/Verification/WhateverYouCallIt is effective since the test developer is detached from the solution. My contention is that all solution developers (of anything, not just software) are myopic. They only see the direct path to the solution with a couple of alternatives. Human nature percludes them from seeing all of the possibilities since their mind is set on making the black box operate. A dedicated test developer doesn't care about how the box solves the problem just the capabilites of the box. Just like code reviewers should be totally independant, test developers should be the same. John Murray Sobetech
  44. I think it is interesting to read so many replies, comments etc. without a baseline for the discussion. I think most of the opinions differ in this thread simply because you all mean different things when you say "test". If "test" is an end-to-end functional test, testing a system that does not execute or compile (another type of test...) does not make sense. If test includes code quality, maintainability, code convention conformance, etc. it definitely make sense to "test" even code that does not compile. If you are going to argue about "test", at least explain what "test" means in your context. Just my humble opinion :-) Daniel
  45. Pipe dream[ Go to top ]

    Developers that don't test first tend to let things slide to QA on purpose. Instead of spending 2 days on a job they feel should be 1, they'll push that to QA thinking "When it comes back as a bug I'll have more time to fix it". This is especially true of contractors. As for JUnit, most of the applications I build are database driven web apps which I inhert with no JUnit testing in them. So it's virtually impossible to take time to put this testing in the system, and even if time permitted it, we'd have to use DBUnit structures on the backend and something else on the front end. Automated testing IMO is more of a pipe dream. The tools just don't exist to use to its full potential mainly due to inconsistent application starting states, such as a database. It's faster and almost as reliable to put that time into the application instead of trying to make all the unit testing work.
  46. Re: Pipe dream[ Go to top ]

    Developers that don't test first tend to let things slide to QA on purpose. ... This is especially true of contractors.
    This might be your experience, however in my experience employees are many times more likely to "turn something over to QA" incomplete versus a contractor. A quality contractor is more likely to tell the project manager that they need more time as they don't have to fear losing their job or promotion due to not meeting a delivery schedule. Contractors tell the truth whereas employees tell "the boss" what they want to hear. Again, this is my experience.
    Automated testing IMO is more of a pipe dream. The tools just don't exist to use to its full potential mainly due to inconsistent application starting states, such as a database. It's faster and almost as reliable to put that time into the application instead of trying to make all the unit testing work.
    You might want to look into various other tools that allow you to create datasources for your DB and test against the DB or mock objects. Your automated unit tests should create the data they will use and then remove the data from the DB when finished. Spring will provide you with plenty of help in this regard if you are using it in your project. Automated unit testing can be put in place for any project, whether you are using Maven or just Ant for your builds and can be designed to be kicked off each time a new version of a source file is checked into source control.
  47. Y-E-S Coding up unit tests prior to the actual code helps in figuring out what you want your implementation to actually do. Some may disagree, but there have been countless times where when jotting down test method names, I say "Oh... and I need a test to cover this obscure, overlooked, and critical relationship over here." Unit tests help track down subtle bugs that may otherwise be overlooked. I do agree that Software Engineers are NOT QA, and people who QA are best at what they do. They find the problems and hitches that the actual implementors would otherwise overlook. My $0.02... write tests, then code, then QA.
  48. Many testers would advise the programmer to test the product himself, first. I have a different answer. My answer is: send me the product the moment it exists.
    I've worked in a company where involving the test team at the earliest possible minute is considered a good thing but have personally found it to be highly counter-productive. First, it gives programmers an excuse for not thoroughly testing their code (how could they be expected to if it must be relased to the test team on the same day it is completed?). Second, it deprives the test team of time they could spend planning better tests. Finally, it results in a lot of redundant and unnecessary defect reports that then consume the programmer's and others time. I think the key to success is to define the functional tests as the last iteration of requirements analysis. Ideally these tests can be automated, thus making them an activity a programmer can enjoy. Regardless, they need to be defined prior to the start of implementation as that will minimize the "happy path" problem -- the assumption being that one can better focus on defining comprehensive tests before one's mind is focused on the code that was just written.
  49. what a great tester![ Go to top ]

    Unfortunately, most testers have no clue as to what this tester sees as valuable which is constant feedback. The short circuit agile approach is frequently viewed by mgmt as uneccessary churn and developer delinquency. When you get "too" agile, everyone is coding with you. Not many people have the patience for it or the ability to understand the difference between unfinished and negligent. As a developer, I would never want to provide an easy facility for someone to make those judgements. No matter how much of a hurry we are in, what me and my team produce has to be stellar upon deployment to qc. Open minds are in the minority. Agile ones are even fewer. Great idea. Not gonna work in most shops.
  50. Best process to increase quality in mission critical software may be: Developer must test his software first, before sending to SQA. He must write reasonable justification for each bug slipped and caught later. The reasonable justification means, no excuses and rationalizations. This is actually instituted in one of the Japanese companies. But I didn’t know what were the results later. I love to know what happened, if any one knows it or tried this?