Parasoft last week released JTest 7, the latest version of their automated testing and error prevention tool.
The big new feature in v7 is "Sniffer", an add-on that monitors a running application and generates complete JUnit test suites based on actual application useage.
While JTest has always been able to generate unit tests via an analysis of source code, it couldn't generate functional tests as couldn't know acceptable inputs and outputs. Sniffer monitors the running of an application and generates unit tests, pulling expected input and output values from the real live application and use them in the test harness.
Classes or methods in different tiers of the Java application can be declaratively selected for test case generation, such as as the services tier, domain model, DAO layer. Sniffer will then monitor:
- how you interact with it - what functions are performed
- all the data that gets input and output
- the external dependencies - ie: a DB
The generated test suite will then both test the selected methods but also mock
any dependencies the classes had (such as JDBC) and return the same data recorded during the sniffing process, allowing the test suite to function outside of the container.
JTest 7 is not meant to replace the tried and true practice of writing tests as you write your code, but is more meant to reach companies interested in quickly building regression test suites but who may not have the time or resources to do so.
The tool is not meant to be used by QA staff but by the development team.
Of course, the proof will be in the implementation, which I currently can't try - I installed the Eclipse plugin, but every JTest operation, including looking at the JTest preferences throws an error. Maybe it's because I don't have a demo license. We'll see once I get one.
Still, what a great idea.
How does this release of JTest compare to Agitator 2.5? I am currently evaluating Agitator and JTest and our team is leaning towards Agitator. Agitator plays well with JUnit test cases but does not generate JUnit test cases which we find to be a weakness of the product. On the other hand it has fairly sophisticated unit testing (code and logic unit testing as defined on Cactus web site) capability and is fairly smart in its coverage. It truly does agitate the code and it can be taught to ignore valid test cases. It also has good support for mock objects, object factories, mocks and factories for J2EE resources (such as EJB, JDBC, JNDI, HttpSession, HttpRequest, HttpResponse etc.).
Anybody has tried Agitator? From my first look at JTest , it doesn't look as sophisticated as Agitator. It has very nice coding standard rules stuff which I am currently not interested in at this time.
We have used Agitator for about 18 months now, and it covers the same ground in a very different way. The issue I have with the concept of JTest is that I would think it does mostly constructive testing by testing what is intended to work.
Destructive testing on the other hand is much harder for a developer to get into the mindset of thinking about. What did they forget? What are the edge cases? What is the unexpected? What are all possible outcomes of my code? What happens when assumptions are wrong?
Those are all questions that Agitator addresses by analyzing our code, finding patterns of behavior, finding ways through the different paths, and determining what it is capable of, not just what you "told it to do." JTest might be biased by how you already exercise the code, if it is only watching the running application for its observations. Don't get me wrong, it is nice to have those constructive unit tests created, but I'm just as worried about code forks we haven't used yet as those we already have.
Agitator does both sides with running our unit tests (JUnit), assertion macros as Agitator native tests, assertions for auto-discovered conditions, etc.
We looked for a tool to fill the gap in our thinking, and Agitator does that well.
What about those of us who write the tests first? What we need is a tool that takes our tests and generates the application code for us:)
Surfing their website, but didn't find it.
(It must be negotiable, I guess...)
... "if you need to ask the price, you cannot afford it." :)
I have looked at Parasoft's testing software before - they already had a "black box" tester of web applications, and the whitebox code testing mentioned in the article, and they sure seemed powerful and complete in their testing coverage.
For development of "mission-critical" systems such tools can be a godsend.
I saw the demo and actually It promisses to bem a complete error preventing system with a rule engine atached.
There is a static analises module with a quick fix tool (I think it can be configured to automatically fix) and extensible rules to find simple, but hard to find bugs.
It also can save a lot of money applaying regression tests in legacy system (which is correcting and not preventing).
I just wonder if this tool can suport a TDD process, mocking my interfaces and as the development grows I change the mocks for real classes and in a continuous build enviroment It maintain my expectations in the real classes.
In this case I could use the test case generator as a tool to suport my design prolices. Otherwise I should still apply the code first process and fix my errors, not really preventing the erros.
Anyway, it seems to be a great tool and really woth applying it. If anybody has tried, please put your impressions in the list.
I have used JTest 6, though not yet version 7, and am pretty happy that it works as advertised.
We started looking at it with a view to it being a quality 'gate' through which all externally developed code had to pass, and therefore focused on the static analysis side of things as part of a nightly build process. This worked well.
Regarding the generation of test cases, this theoretically can be run server-side as well, but in practice it generates quite a few redundant or invalid test cases, and you need someone with knowledge of the code in order to validate whether the tests that fail should be ignored or not. Version 6 is able to generate 'brute-force' test cases only, that are based on inspection of the logical paths through the code. As such, it's not really much use for the functional tests - i.e. 'is it doing the right thing', rather than 'does it throw any unplanned exceptions'.
Version 7 sounds like a big step forward into the functional testing area, and I look forward to getting my hands on it.
So, as I said, you have used it just for static analysis of the code, haven't you?
I don’t think there would be any testing tool that would read developer’s mind. A developer needs a 'testing framework tool', that we already have (jUnit). Generating automated unit test cases based on the boundary conditions wouldn’t identity all your bugs in the code.
So these kind of tools is a good selling point for non technical manager or leads. As ‘Bruno Wolf’ said, you can’t sell this to a good developer.
The tool will not find all your bugs and a good developer knows that, but it will eliminate the task of writing simple standard JUnit tests. I can as a developer therefore concentrate on writing the more interesting unit tests and integration tests. Furthermore, when I have refactored code I only need to verify that the non-autogenerated tests are still valid before I start running the tests. I remember something about a 2000$ price tag for an earlier version but I am not sure.
we have really seriously tried to use this tool on two big J2EE projects. One was already running for some time, one was a totally new one.
When you read about this tool and play with it, you love it. Such a nice toy. But, then, in practice, it is NOT usable. The design by contract approach is too complicated and the developers are not able to use it reasonably. Even if you force them or try to make them to love the tool, you will not be successful. You will finish using it just for checking the static rules for the code.
I would be really interested in whether someone has used it for a big project successfully.
As for the price - it is HUGE. Just big companies can buy it and after the experience, I would say that it is NOT worth the money. Simple unit tests will do much better work.
Sorry for the pseudonyme - I was in touch with Parasoft and do not want them to know who I am ;-)