Discussions

News: A New Approach to Unit Tests

  1. A New Approach to Unit Tests (20 messages)

    In this blog post I discuss a new approach to unit tests putting class under test at the centre. It's also a discussion about a kick starting open source project (TestedBy) which aims to provide tools to run tests starting from class under test. Some feature we would provide with this software are: 1. The first one maybe the most trivial, but one of the most useful: within your IDE you can navigate sources starting from the class/method under test. 2. You will find this annotation in you javadoc. 3. Ant and/or maven and/or task/plugin to run tests starting from class under test. 4.A design by interface and contracts defining tests on interface and running them on all concrete implementations. 5. Test class generation 6. Tools to manage also round tripping (it’s important to ensure test classes and annotations don’t go quickly out of sync) For more information on TestedBy open source project visit it's homepage and for a complete discussion of this kick start or to contribute read this blog post Please give us your feedbacks.

    Threaded Messages (20)

  2. I'm going to try this[ Go to top ]

    It's so cool. "Unit test case" is the most important specification about interface, precondition, and assuming usage. The explicit link between implementation and specification must be very helpful.
  3. Re: I'm going to try this[ Go to top ]

    This is actually how we do testing in-house, though with less general home grown tools. Thanks a lot for starting an open source project around this idea - it's so much more useful than the JUnit way of doing things.
  4. Re: I'm going to try this[ Go to top ]

    Hi Stefano, There are a few problems with your idea: * The relationship between a business method and a test method is not a bijection. Sometimes, a business method is tested by several test methods, and sometimes, a test method will exercise more than one business method. * This approach will break in the face of refactoring since it describes class and method names as strings. Change the name of one and no IDE will be able to help you make sure you didn't just break the build. * This creates a dependency cycle between production and test code, which is a recipe for disaster in your infrastructure as your code base grows. * This will only work for test method that don't accept any parameters, but any functional tests that uses @DataProviders (TestNG) or parameterized test cases (JUnit 4) cannot be covered by TestedBy. You can already see an example of this problem with the "shouldAddTwoAndThree" in the example: why limit the verification to adding 2 and 3? -- Cedric
  5. Re: I'm going to try this[ Go to top ]

    Hi Stefano,

    There are a few problems with your idea: Cedric
    Agree. I can also understand the upside of this new approach and it's pretty interesting, but the downside seems like too much of a problem, or needs to be explained further. I find a lot of value in driving some use-cases through with unit tests, so this approach just isn't going to work. I typically have a BaseDaoTest, and BaseServiceTest which help boot up the DAO and Service layers in Spring, as well as a lot of standard JUnit tests. It would be useful to understand how this new proposed approach fitted in with tools like Cobertura, since unit testing is more or less pointless unless you're measuring coverage -- as well as ensuring that you're making useful assertions about test results.
  6. Re: I'm going to try this[ Go to top ]

    I find a lot of value in driving some use-cases through with unit tests, so this approach just isn't going to work. I typically have a BaseDaoTest, and BaseServiceTest which help boot up the DAO and Service layers in Spring, as well as a lot of standard JUnit tests.
    Ok, so you are saying this tests doesn't strictly relate to a class, but to a use case. Ok, I call it integration tests, and I have also a lot of this tests, written using Junit or TestNG. Keep it there and run with JUnit or TestNG. It's other type of tests, very useful, but not defining any contract. BTW having 2 tools for different test purposes isn't necessary bad, for example I prefer JUnit for pure unit tests and TestNG for integration ones.
    It would be useful to understand how this new proposed approach fitted in with tools like Cobertura
    Coverage is one of the most importat aspect of test. I don't see problem in running a coverage tools against TestedBy projects since test classes are pure Junit/TestNG classes. Anyway we will keep in mind coverage in all our addition.
  7. Re: I'm going to try this[ Go to top ]

    I call it integration tests, and I have also a lot of this tests, written using Junit or TestNG. Keep it there and run with JUnit or TestNG
    I don't bother differentiating between integration and unit tests. I know some people do and think its important, so probably not important what we call these. From a technical point of view, I believe (?) most coverage tools work best if a coverage report is generated from a single run of the tests. Clover lets you merge coverage metrics into a single report, but not sure if Cobertura does. Having 2 coverage reports for my 2c is pretty useless -- its important to have a unified coverage report. You might want to make sure that if you're recommending different testing approaches for 'unit' and 'integration' tests that a unified coverage report can still be produced.
  8. Re: I'm going to try this[ Go to top ]

    Thanks for the interest and your comments. You are probably right on some points, even if I think I've misunderstood on some others
    * The relationship between a business method and a test method is not a bijection. Sometimes, a business method is tested by several test methods, and sometimes, a test method will exercise more than one business method.
    I agree, even if it's not so usual in a correct BDD o TDD design.Anyway I don't see any problem here since you will have more annotation pointing same test. What I'm saying is not to create a bijection from test and methods but to run test insisting on class. I know it will drive to run some test more than one time, but let me remark that TestedBy test IS JUnt/TestNG tests and you can use TestedBy just t exercise test on changed classes during normal day work and standard JUnit/TestNG suite to run all tests if you have time consuming tests and would avoid multiple execution.
    * This approach will break in the face of refactoring since it describes class and method names as strings. Change the name of one and no IDE will be able to help you make sure you didn't just break the build.
    Not totally true, at least eclipse 4.4 can search in string during refactoring and tools may help here much more. Anyway in my previous answer I already said that IMHO that decision to rename of test classes and methods would start from class under test and not as standard refactory in a BDD aprroach since your are changing the behaviour/contracts you are verifying.
    * This creates a dependency cycle between production and test code, which is a recipe for disaster in your infrastructure as your code base grows.
    Maybe you are right, even if I think a logical dependency already exist, and it's a strong dependency. Anyway as said TestedBy tests are JUnit/TestNG tests, leave TestedBy if growing code is driving you to any problems. Or much better keep TestedBy only for it's added value (test defined on interface and run on all implementations) and do all other tests with standard approach. The beauty here is that the two approach are not mutually exclusive if you really need both.
    * This will only work for test method that don't accept any parameters, but any functional tests that uses @DataProviders (TestNG) or parameterized test cases (JUnit 4) cannot be covered by TestedBy. You can already see an example of this problem with the "shouldAddTwoAndThree" in the example: why limit the verification to adding 2 and 3?
    -- No, we don't limit anything here. My example has been simple just to favour reading by anyone, but as said TestedBy test classes are JUnit/TestNG plain classes, so you can use DataProvider and all other features. Thanks a lot for your interest. I'm glad to have opened a discussion around.
  9. Re: I'm going to try this[ Go to top ]

    Hi Stefano,

    There are a few problems with your idea:

    * The relationship between a business method and a test method is not a bijection. Sometimes, a business method is tested by several test methods, and sometimes, a test method will exercise more than one business method.
    When tests are properly written, this is a rare demand. However, I agree it IS a problem when the demand is there.


    * This creates a dependency cycle between production and test code, which is a recipe for disaster in your infrastructure as your code base grows.
    That's the case anyway.


    * This will only work for test method that don't accept any parameters, but any functional tests that uses @DataProviders (TestNG) or parameterized test cases (JUnit 4) cannot be covered by TestedBy. You can already see an example of this problem with the "shouldAddTwoAndThree" in the example: why limit the verification to adding 2 and
    Yup, that's my biggest problem with this approach. However, that should be fixable, making this a viable alternative.
  10. Re: I'm going to try this[ Go to top ]

    * This will only work for test method that don't accept any parameters, but any functional tests that uses @DataProviders (TestNG) or parameterized test cases (JUnit 4) cannot be covered by TestedBy. You can already see an example of this problem with the "shouldAddTwoAndThree" in the example: why limit the verification to adding 2 and


    Yup, that's my biggest problem with this approach. However, that should be fixable, making this a viable alternative.
    Let me remark that isn't a limit with TestedBy, test class can have all JUnit/TestNG features. My example doesn't include just to keep it easy to read and understand, but it's definitively possible.
  11. Test injection with great syntax[ Go to top ]

    Hi Stefano: Great blog entry. I've long admired the test injection pattern. The stynax and use of BDD makes your unit tests very readable. That's a great thing for test definition and also for test maintenance. Especially when I'm the developer picking up a unit test from someone else that created it. Frank Cohen http://www.pushtotest.com
  12. Coverage will clutter code....[ Go to top ]

    So the guys who came up with BDD make the point that it is a good idea to test the actual behavior of a class (or a method). And name the test method by what it is actually testing for! Great stuff! Wouldn't have thought about it. Now really, if this has not been common practice for the last 5 years or so, I wonder where I have been. Anyway, back to the topic. I would disagree, because I'd think this kind of annotation will create enormous code clutter. This is one of the things I would rather try and tackle with convention: A class that tests a particular class is placed in the same package than the original class (or in one derived, whatever the convention) and called Test. Its methods are called something like . That does the job, as far as I am concerned.
  13. Clutter code and Coupling[ Go to top ]

    I am intrigued by the idea to "loosely" link the test code with actual code; but I am not sure I will like the clutter it brings to the code; and I certainly don't like the strong coupling between the test code and actually code. The code should read like an story, too much decorations make it hard to read. In case of tests, unless the method is extremely simply, there are usually many test cases against one method (just think about a business method on a EJB session bean remote interface), piling them up on top the method with annotations is not pretty. Current format of the "link" between test and source code strongly couple of the two codes. Every time I changed the test name, or add new test, I have to change the source code's annotation as well. This is surely undesirable. Chester
  14. Re: Clutter code and Coupling[ Go to top ]

    First of all, thanks to give us your opinions.
    The code should read like an story, too much decorations make it hard to read.
    I'm more than agree, but I think code readability isn't so much influenced from method decorations. You should keep single block easy to read not the whole program, and maybe some decorations could help identifying real purpose of single method...before start to read the prose :) like a quick resume.
    In case of tests, unless the method is extremely simply, there are usually many test cases against one method (just think about a business method on a EJB session bean remote interface), piling them up on top the method with annotations is not pretty.
    But we are speaking of unit tests. A lot of unit tests on an EJB remote interface looks at me as poor design. IOW you should have underlying code already tested (as smaller iece of code as single methods and classes used by your EJB), so EJB methods need few or not tests at all.
    Current format of the "link" between test and source code strongly couple of the two codes. Every time I changed the test name, or add new test, I have to change the source code's annotation as well. This is surely undesirable.

    Chester
    Let me describe a different point of view: speaking of BDD tests when would you change a method name? When the behaviour of class under tests is changing. And changing of behaviour is something that is driven by class under test not by test. I'm putting one more time class under test at centre of development: programmer have to think class under test and its behaviour, eventually change or add tests annotation and then changes tests (maybe with tools help). Make programmer think and design class under test with a design by contract approach and leave test to verify behaviour are respected. How does it sound to you? Anyway a lot of works and thought have to be done to refine the idea and provide easy and effective tools and this kind of discussion and different point of view are more than welcome. I'll open asap a forum for the project to open this kind of discussion. Thanks for the interests
  15. Re: Clutter code and Coupling[ Go to top ]

    In case of tests, unless the method is extremely simply, there are usually many test cases against one method (just think about a business method on a EJB session bean remote interface), piling them up on top the method with annotations is not pretty.

    But we are speaking of unit tests. A lot of unit tests on an EJB remote interface looks at me as poor design. IOW you should have underlying code already tested (as smaller iece of code as single methods and classes used by your EJB), so EJB methods need few or not tests at all.
    I think I will disagree with that statement. EJB remote interface is true business contract. That's where the tests should be placed on. Called unit test or functional test, we use unit test tools to perform these tests. I agree more "unit test" should be placed on individual methods, but all methods put together needs to be test as well. In other words, we can't simply test parts and without testing the system.

    Current format of the "link" between test and source code strongly couple of the two codes. Every time I changed the test name, or add new test, I have to change the source code's annotation as well. This is surely undesirable.

    Chester

    Let me describe a different point of view: speaking of BDD tests when would you change a method name? When the behaviour of class under tests is changing. And changing of behaviour is something that is driven by class under test not by test. I'm putting one more time class under test at centre of development: programmer have to think class under test and its behaviour, eventually change or add tests annotation and then changes tests (maybe with tools help).
    Make programmer think and design class under test with a design by contract approach and leave test to verify behaviour are respected.

    How does it sound to you?
    Here is what I usually do: when I design an interface or class, I am more focus on what the methods in the class suppose to do; or what the concept that class suppose to represent rather than how do I test this. I will think of testing later when I finish my design. I will code the unit tests while I am implementing the class/methods. If the method is authorize(), I might write an test called testAuthorize(); later I might write another one; testAuthorizeWithInvalidRole(); or more one simply to capture an new found test case with testAuthorize2(). Put Class center or not, I would not like to change the source code, recompile and build and redeploy my EJB .ear file and wait for Application to fully deploy simply because I add a new test case, -- The coupling is way to strong. And I agree with others, production code shouldn't mixed with test code.
  16. Re: Clutter code and Coupling[ Go to top ]

    But we are speaking of unit tests. A lot of unit tests on an EJB remote interface looks at me as poor design. IOW you should have underlying code already tested (as smaller iece of code as single methods and classes used by your EJB), so EJB methods need few or not tests at all
    This kind of approach will usually create rather poor code quality. The one thing that is actually important to test is the contract that your code exposes. It is helpful to test individual building blocks of course but the only "unit" or "contract" that is really interesting for the customer or the actual user of the code is the interface he uses. When testing underlying components, you can be somewhat more confident that your overall service works properly - but you cannot be really certain. For this you must test a lot of business scenarions on your EJB and in particular a lot of conditions with boundary condition data across the whole interface.
  17. Re: Clutter code and Coupling[ Go to top ]

    For this you must test a lot of business scenarions on your EJB and in particular a lot of conditions with boundary condition data across the whole interface.
    Ok you are right here. But test boundary conditions and "contract" on a complex method is also an interesting documentation of this method. Metadata documenting would be useful, isn't it? I know a lot of annotation may clutter the code, maybe our annotations have to be designed better to reduce cluttering as much as possible. My general question is: is better to document contract defined and verified on an interface or totally avoid it because too much worried by cluttering code? As already said, I like an open discussion on these arguments, and I think a different point of view on tests may be interesting if it can open discussion about contract of interfaces and how java community are managing them. My described approach could provide a possible, maybe not perfect, solution to keep track and documentation of contracts defined and verified on interface, that is the main aims of my purpose, not to change the best practice on unit tests.
  18. Re: Clutter code and Coupling[ Go to top ]

    >Ok you are right here. But test boundary conditions and "contract" on a complex method is also an interesting documentation of this method. Metadata documenting would be useful, isn't it?
    Sometimes it might be. In which case I would like to see annotations on the names of the contract aspects in the original class only. It is up to some glue code or convention in the test framework to pick these up and verify if the proposed tests cover all defined contracts. How about calling the annotation @behavior :-)
  19. Nice idea[ Go to top ]

    I think the idea is very interesting, Although I too am concerned about the coupling between test and production code. A suggestion would be to also be able to configure test through xml rather than annotations! I know sounds worse, buts for some folks who don't like the coupling might prefer to throw that stuff in xml.
  20. I've just created a google group for the project to continue this discussion and brain storming around this idea. I hope some (or ideally all) the nice brains have commented here would join us and provide their point of view contributing to provide to the community the opportunity to have a different approach to tests. Maybe it will not be perfect or better than current one, but choice between multiple alternatives means freedom. For more details have a look there: http://www.javalinux.it/wordpress/?p=184
  21. In the new spring 2.5 testing framework there is a annotation called @Autowired that you can use to automatically wire the implementation in the Test Class. This way I can write my test to the interface as many as I want and the implementer can simply inject through spring their own implementation. Would that not be the similar functionality that TestedBy is trying to achieve. In addition the production code has no coupling to test code. As a designer I can ship my interface api and unit test separately for the implementers.