DDSteps is a JUnit extension for building data driven test cases. It separates the input and expected output test data from the test code. The data is kept in an Excel file for easy editing. The test code is divided into test steps, which makes for good reuse. Function testing can now easily be extended with new combinations of input by just adding rows in Excel and new test cases can be created using existing test steps.

An article called DDSteps - Data Driven Sanity describes how we turned a manual testing nightmare into an automated bliss. From the article's description of how to solve data-driven test problems:

Testing your solution for every type of input and making sure it will pass function tests is difficult at best. Usually, this is done "by hand" and requires real people. And, yes, it is boring and time consuming. So you have to automate it, just like you did with JUnit tests for unit testing.

By Function tests we mean, "Does it work like the use case says it should". We mean full, system, end-to-end tests, using a production-like environment. We mean using a browser, surfing the web application and then checking the real database for results.

Automating function tests is hard, mostly since they suffer from the same problem as the manual testing. Many of the tests are the same; just the data is slightly different in each test.
Data driven testing on the other hand separates the testing code from its input. This and the reusable test steps makes it easy to maintain and evolve the test cases, you get reuse instead of copy-paste. Need another field? Just add new column in your input and handle it in your test case and voila. There will be no explosion of test methods, no massive round of changes!

There's also a video (6MB, WMV format) you can watch to see a simple example of data driven testing. (Editor's note: this is very similar to FIT, another testing mechanism based on datasets, which is based on HTML tables rather than Excel, but definitely represents another approach, with DDSteps being focused more on JUnit integration.)

Good tool. I will try it!

Good tool. I will try it!

Please do. And let us know what you think at http://jira.ddsteps.org

/Björn

Generally speaking, this is yet another attempt at addressing the shortcomings of unit testing and it's fitness for purpose in the real world when it comes to proving that a system actually does what it should. If this is what you want to achieve then I would argue strongly that you should drop the pre-occupation with Junit completely.

Specifically, in the case of DDSteps ...

Testing your solution for every type of input and making sure it will pass function tests is difficult at best. Usually, this is done "by hand" and requires real people. And, yes, it is boring and time consuming. So you have to automate it, just like you did with JUnit tests for unit testing.By Function tests we mean, "Does it work like the use case says it should". We mean full, system, end-to-end tests, using a production-like environment. We mean using a browser, surfing the web application and then checking the real database for results

...this is exactly what it doesn't provide !

For functional testing, you'd be much better off using something like this TestMaker and then you can really simulate "using a browser" and doing "end-to-end" tests and drive it all from data files and whatever other logic you may fancy, limited only by your imagination and familiarity with Jython.

TestMaker looks like a good tool.

However, we needed something that we could integrate not only into our programming tools but also into our development process.

We needed clean separation of test data and test code, massive test data, really reusable test code, the ability to drive our web sites etc. And we needed all of this right under our fingertips where do our normal coding in Java and right beside our normal unit tests. JUnit might be in need of an update, but it can be found most everywhere and this is important to our customers.

We couldn't find anything that fit our bill.
This is what we came up with.

Cheers,
Björn

For the testing I need to do, I have input and output that can be larger that several dozen megs. Potentially more than 100. I don't really see that being put into an Excel spreadsheet.

When you have data that large, you probably use some kind of recorded data? If you have it on file, you can use Excel to decide combinations of input and output files. Maybe several inputs will yield the same output? Spring will turn a file name in you Excel data into a java.io.File or InputStream, which you can use.

We invented DDSteps to do web site testing (form input and database validation), and we feel it work nicely in that context, so we wanted to give it back to the FOSS community, that has given us so much. I hope you try it for your testing, and please give us feedback on how we could improve if needed. But one tool rarely meets every demand, and maybe yours isn't what DDSteps is targeted at.

Best regards
/Adam Skogman, DDSteps project lead

I find that there are really two kinds of use cases for function testing tools. I believe both are equally needed and worthy.

One is for a recording tool, like TestMaker or Rational Robot. The user group is typically testers and business people. The test is usually built "after the fact", when a piece of functionality exists.

But as a developer, I want write tests at the same time, or even before I code. Test driven is what I want to be. And I want the full power of Java, and to use my IDE, and I want an API that does not depend on recording. And since JUnit already is integrated and has fancy reporting, we wrote DDSteps to build on that legacy.

Regards
/Adam Skogman, DDSteps lead

FYI, here is how data-driven testing works in TestNG.

1) You supply a method that will supply the various parameters to your test method (called a "Data Provider")

2) You hook up your test method with this Data Provider

Here is an example:

    // This method will provide data to any
    // test method that declares that its Data Provider
    // is named "test1"

    @DataProvider(name = "test1")
    public Object[][] createData1() {
     return new Object[][] {
       new Object[] { "Cedric", new Integer(36) },
       new Object[] { "Anne", new Integer(37)},
     };
    }

    // This test method declares that its data
    // should be supplied by the Data Provider
    // named "test1"

    @Test(dataProvider = "test1")
    public void verifyData1(String n1, Integer n2) {
     System.out.println(n1 + " " + n2);
    }

will print:

  Cedric 36
  Anne 37

The advantage of this approach is that the data provider can get its information from anywhere: Java code, a text file, an Excel spreadsheet, an XML file, a database or even a JMS connection. Your test method never changes, only the data does. DRY bliss.

It also shows that test methods become much more intuitive when they can receive parameters...

--
Cedric
http://testng.org

FYI, here is how data-driven testing works in TestNG.1) You supply a method that will supply the various parameters to your test method (called a "Data Provider")2) You hook up your test method with this Data ProviderHere is an example:    // This method will provide data to any    // test method that declares that its Data Provider    // is named "test1"    @DataProvider(name = "test1")    public Object[][] createData1() {     return new Object[][] {       new Object[] { "Cedric", new Integer(36) },       new Object[] { "Anne", new Integer(37)},      };    }    // This test method declares that its data     // should be supplied by the Data Provider    // named "test1"    @Test(dataProvider = "test1")    public void verifyData1(String n1, Integer n2) {     System.out.println(n1 + " " + n2);    } will print:  Cedric 36  Anne 37The advantage of this approach is that the data provider can get its information from anywhere: Java code, a text file, an Excel spreadsheet, an XML file, a database or even a JMS connection. Your test method never changes, only the data does. DRY bliss.It also shows that test methods become much more intuitive when they can receive parameters...-- Cedrichttp://testng.org

Nice!

Hi Cedric,

DDSteps is not in any way locked into Excel - rather it is very open. In your test case you specify a DataLoader (a convenience class chooses Excel for you) which may load data from any source (database, XML, CSV etc) or provide it in code as an array. So we share the same design!

The approch we use to get data into the test is "injection". We felt it was a good thing to not have to write a list of arguments, and then manually apply the argument to the (often many) places in your test where you wanted to use that piece of data. Rather, data is injected into not only the test case, but also any object accessible by getter/setter from your test case. This, in our oppinion, means less code in the test case, with the trade-off of having to put in JSP EL-style expressions in your data file to show where you want your data injected.

I agree totally with you that DRY bliss comes from not changing the test method, and we never do change the method, only data.

TestNG is the most important test framework to emerge lately, am I would love to continue this discussion as we want to move to TestNG in a not to distant future!

/Adam Skogman, DDSteps project lead.

Hi, I was wondering how DDSteps compares to Fit. I noticed the note regarding the similarity between the two but I was hoping if you could elaborate on specifics regarding this.

Part of the power of fit comes from the fact that those involved in a project other than the developers can specify the "requirements" of the system by discussing the inputs and expected outputs in the fit tables. The tables used in Fit lend to this type of collaboration. Does DDSteps concentrate on a more developer centric approach to data driven testing?

DDSteps and Fit are similar in principle, that's true, but the implementation is quite different.

Fit/Fitnesse is more geared towards a tester using html tables/Wiki pages as input and combining these with "actions" you can end up with programming your tests in those pages and with calling code in Java.
We wanted clean separation of test data and test code to enable better maintenance and reuse of both data and code. Also, this enables large scale combinations of input (test data). It was very important to us that it would be in the development cycle, right alongside our code in our everyday tools (ide, version handling, continuous integration etc). When we refactor our code, then our function tests are refactored too. We wanted CruiseControl to handle our integrations. We wanted row numbers from test data to be used when reporting problems in our IDE.
And we wanted all of this just by adding some jar-files to our project, not another separate tool.

DDSteps other part, the test steps, makes for really good reuse of test code. It enables developers and testers to talk about tests in distinct steps - NavigateToOwner, NavigateToAddNewPet, ValidateAddPet and so on (not too far away from the english used in the test documents:). New tests cases are then easily created using existing steps.

There are more differences. Fit is a good tool even though it wasn't what we needed.

Part of the power of fit comes from the fact that those involved in a project other than the developers can specify the "requirements" of the system by discussing the inputs and expected outputs in the fit tables. The tables used in Fit lend to this type of collaboration.

In both cases testers and developer can cooperate on the input using html/wiki in Fit/Fitnesse, or as in our case spreadsheets.
We find it easier to manage and cross-reference data in Excel as compared to other input formats (html, xml etc). But, then again, each has its advantages.

DDSteps is very much in the development process, but it currently lacks the visibility as seen from the tester when it comes to the test code.
I would like for DDSteps to be able to generate a test report based on the test code. Then we hopefully could throw away the Word document that is currently used to define a test :-)
This is still on a discussion level though.

Does DDSteps concentrate on a more developer centric approach to data driven testing?

Yes. Every time a byte or a line is changed in the code, should the system be verified - from unit tests up to function tests.
Waiting for someone to hit a button or a let out a release candidate to QA, is too late.

Btw, being no expert on Fit, please correct if I got something wrong.

Cheers,
Björn Granvik
Jayway

Testing your solution for every type of input and making sure it will pass function tests is difficult at best. Usually, this is done "by hand" and requires real people. And, yes, it is boring and time consuming. So you have to automate it, just like you did with JUnit tests for unit testing.

I don't see how automated testing can ever completely replace manual testing done by real people. Yes it's boring, but it is the only way you can detect for example serious layout broblems in web application. Furthermore, automated testing does not detect errors you don't think of when you plan the tests.

I'm a bit sceptic about separating data from the test code. In my opinion, such separation makes the test code less readable. Readability is important when you run your tests, run into errors, and then get back to test code to see what is actually wrong. Considering the testing scripts I've written, I can't see the advantages.

I don't see how automated testing can ever completely replace manual testing done by real people. Yes it's boring, but it is the only way you can detect for example serious layout broblems in web

application.

I think basically you're right. Automated testing cannot totally remove the need for manual testing. And it shouldn't. However, it certainly lifts it to a new level, where it should be.
Programs should do the "time consuming and boring" testing. The testing that requires thought and insight should still be done manually.
I just don't want to do the boring stuff :-)

Furthermore, automated testing does not detect errors you don't think of when you plan the tests.

Which applies to any testing really, whether it's based on carbon or silicone or Word. Changing your test specification, scripts, code as you go along is normal work.

I'm a bit sceptic about separating data from the test code. In my opinion, such separation makes the test code less readable. Readability is important when you run your tests, run into errors,

and then get back to test code to see what is actually wrong. Considering the testing scripts I've written, I can't see the advantages.

Well, our test code typically has variables like "customerType" instead of "3" (or CUSTOMER_TYPE) since the test data is injected through set/get accessors. To me the first is readable and enables us to do lots of combinations of test data. Imagine testing combinations of customer type, date formats, currency, checking for I18N support, cross site scripting holes, etc...

Using straight values or constants quickly becomes unwieldy.

Regards,
Björn
Jayway

I totally agree. Automated testing is a great way to ease the burden of regression testing. I just criticized the way the original posting made it sound like automated testing completely liberates one from manual testing.

Well, our test code typically has variables like "customerType" instead of "3" (or CUSTOMER_TYPE) since the test data is injected through set/get accessors. To me the first is readable and enables us to do lots of combinations of test data. Imagine testing combinations of customer type, date formats, currency, checking for I18N support, cross site scripting holes, etc...

Now I got your point. But still, consider the following piece of jWebUnit (http://jwebunit.sourceforge.net) script:

clickLinkWithText("Search Customers");
setFormElement("customerName", "Fictional");
selectOption("customerType", "Medium sized enterprises");
submit("search");
assertLinkPresentWithText("Fictional Enterprise, Inc.");

Here the test data is embedded in the test code and (to me atleast) this is quite readable. I admit that if you wan't variation in the test data, this approach doesn't work that well. Anyway, for my purposes it has been adequate.

clickLinkWithText("Search Customers");
setFormElement("customerName", "Fictional");
selectOption("customerType", "Medium sized enterprises");
submit("search");
assertLinkPresentWithText("Fictional Enterprise, Inc.");

Here the test data is embedded in the test code and (to me atleast) this is quite readable.
I admit that if you wan't variation in the test data, this approach doesn't work that well. Anyway, for my purposes it has been adequate.

True. The actual value isn't present in a testcase based on DDSteps. It is more visible what data is used in the code snippet above. As Adam wrote in his reply, in DDSteps you would have to check the trail (stored snapshots of your website's apperance at defined points of the execution of the test case) or debug the code.

In each instance you would have to decide what type of test is the most appropriate - unit, integration, function etc.
DDSteps doesn't do away with this, it offers a better way to do function testing using multiple test data. Use it wherever you feel it is appropriate.
We've found it to be very useful :-)

/Björn
Jayway

I don't see how automated testing can ever completely replace manual testing done by real people.

You are quite right - we'll never completely replace manual testing. However, that is no reason not to replace the 99% of testing that can be automated. A full function test cycle can be cut down to minutes, not weeks as when manually testing. And think of the man-hours that you can use for development, not testing!

it is the only way you can detect for example serious layout broblems in web application.

Again, you are right. Layout problems are, in our experience, caught by the developers when they test drive the app, and in some rare cases by the acceptance tests (the only manual tests we have retained).

Also, for just this reason, DDSteps writes a trail of HTML pages that it sees, so that you may go back into any test that was run, and see what the automated test saw.

I'm a bit sceptic about separating data from the test code. In my opinion, such separation makes the test code less readable.

Actually, data in Java code is usually quite messy. We felt that our code got more readable. And when something does go wrong, DDSteps will tell you which data caused the failure, and you can go back and re-run (in debug mode if yoiu like) with only that data.

I have been in a project where we used Excel to initialize the test database. It sucked for the following reasons:

- it was slow (loaded only one table per 2 seconds), but maybe ddsteps is better
- there is no convenient way to create and fill the excel file
- there is no way to check for referential integrity in the file
- there is no type checking of entered data in excel (emtpy column, is it empty string or null)
- each table was in a tab, the tabs had to be in order for the insert to succeed
- some excel files contained 70 tables (well, this was maybe more a wrong of the project itself)
- because of the previous points, it was hell to find errors and fix these (and no way to look into the sheets or change them by using sql queries)
- we had to fix these problems everytime the schema changed

XML files potentially have the same problem, but you can work around that with clever text tools and XSL.

Anyway, my advice is to use a read-only databases for storing test data. The database can have any form you like, a hsql/h2 file could already be sufficient. Database copies can be very quick.

I feel your concern for the speed of Excel parsing - but I can put you to ease.

We use the very nice Java Excel API (http://www.andykhan.com/jexcelapi/) and performance has been good. We do make sure that we cache the parsed Excel file, but even the first load is really snappy.

Why don't you give it a try, and if you have any problems, just send me an email, and we'll look into it?

Regards
/Adam Skogman, DDSteps lead
Jayway AB

I don't see how automated testing can ever completely replace manual testing done by real people. Yes it's boring, but it is the only way you can detect for example serious layout broblems in web

A proper functional test, driven from a simulated end-user perspective can go a long way towards achieving this. By treating the application as a black-box and firing external requests at it, then using a pattern based approach to comparing actual responses with expected results, you can detect formatting problems as well as verify the end results at a more granular level. In addition, you are putting every tier in the application through its paces and proving it in its actual deployment setting, not just testing a notional composition of its units to satisfy a developers understanding of those units.

I'm a bit sceptic about separating data from the test code. In my opinion, such separation makes the test code less readable. Readability is important when you run your tests, run into errors, and then get back to test code to see what is actually wrong. Considering the testing scripts I've written, I can't see the advantages.

Test data absolutely has to be seperated from your test code, otherwise you are writing legacy tests with a high maintenance overhead. You probably don't write (proper) code with hard coded values in it so why would you write the test that way. Infact this statement points at the need to test the test code and that has a bad smell about it. Proper end-to-end testing will seperate your testing from your code completely and the same scenarios that you use for functional testing can be used to load test, a large part of decent load testing being based on realistic end-user scenarios.

If you insist on relying on a unit based approach to functional testing then DDSteps is probably useful, as a set of shears is to someone building a house of straw.

If you insist on relying on a unit based approach to functional testing then DDSteps is probably useful, as a set of shears is to someone building a house of straw.

DDSteps builds on the desire to code function test using Java. One major component of DDSteps is the TestStep design pattern, where tests are composed from large-scale, functional steps (e.g. "log in", "enter data into new person form"). This maps pretty well 1:1 to how test cases are written as a high-level spec, and the steps are re-used in multiple scenarios.

The implementation of such a step uses API:s like JWebUnit an Spring JDBC - you never use any of the application code to do database access etc. when doing functional testing. So, we have achieved both high-level abstraction and low-level power through Java.

DDSteps is a lot of things: data driven unit testing, Spring based integration testing and platform for end-to-end functional testing, and they are sometimes confused.

/Adam Skogman, DDSteps lead
Jayway AB

I don't see how automated testing can ever completely replace manual testing done by real people. Yes it's boring, but it is the only way you can detect for example serious layout broblems in web

A proper functional test, driven from a simulated end-user perspective can go a long way towards achieving this. By treating the application as a black-box and firing external requests at it, then using a pattern based approach to comparing actual responses with expected results, you can detect formatting problems as well as verify the end results at a more granular level. In addition, you are putting every tier in the application through its paces and proving it in its actual deployment setting, not just testing a notional composition of its units to satisfy a developers understanding of those units.

I'm a bit sceptic about separating data from the test code. In my opinion, such separation makes the test code less readable. Readability is important when you run your tests, run into errors, and then get back to test code to see what is actually wrong. Considering the testing scripts I've written, I can't see the advantages.

Test data absolutely has to be seperated from your test code, otherwise you are writing legacy tests with a high maintenance overhead. You probably don't write (proper) code with hard coded values in it so why would you write the test that way. Infact this statement points at the need to test the test code and that has a bad smell about it. Proper end-to-end testing will seperate your testing from your code completely and the same scenarios that you use for functional testing can be used to load test, a large part of decent load testing being based on realistic end-user scenarios.If you insist on relying on a unit based approach to functional testing then DDSteps is probably useful, as a set of shears is to someone building a house of straw.

DDSteps sounds very promising and practical to me. I think your point about load testing is especially important.

However, testing by humans has such important benefits that I tend to agree that we should not expect to replace it. For example, the automated test will not suggest a new feature or a workflow improvement, nor will it incorporate knowledge of specific customer preferences in its examination of the software. Human testers are a significant source of product improvement IMO.

Hope that's not too far off-topic.