You may have heard of a new term circulating amongst developers, magazines, and online developer haunts: test-driven development. What is it? Test-driven development is a methodology that emphasizes creating tests as an integral part of the development process. If you care about the quality of the applications you write, you test before deploying. Just about everybody tests in one way or another, so you might ask: isn’t testing already a part of the development process? Test-driven development makes testing an integral part of the development process.
Many people in the early days of software development tested an application by executing the application and manually interacting with the application, specifying inputs and observing outputs. Much software development continues to happen in this fashion. The manual approach has one key merit: it is extremely easy to learn and to understand how it works. If you can operate a keyboard and mouse and understand how to navigate a windowing system, you can test any desktop application today. If you can operate a web browser, you can test any web application today. These are such trivial skills today that it is safe to assume everyone who does software development passes muster. One nice thing about the manual approach is that you see the application exactly as the end user sees the application, although this is not really an advantage over other testing methods because other testing methods also have this benefit.
The drawbacks of the manual approach are numerous:
- Manual testing is repetitive. Every time you make a change, whether it be a new feature addition, alteration of existing behavior, or a bug fix, you need to re-test the affected sections in order to ensure that you have not broken them with the code you have added, changed, or removed.
- Manual testing is error-prone. People are terrible at repetitive tasks, especially when the tasks are boring. People overlook details, leading to shipped code that is broken. Even more problematic, changes to one feature may influence the behavior of other pieces of code that are not immediately obvious as being related. Thus, even if you are conscientious about testing code that you think you have affected by your changes to the application, you may still let slip bugs or broken functionality, not because of carelessness on your part, but because software development is complex and software has many interlocking parts and it is impossible for anyone to keep track of every possible antecedent and dependent of a given section of code on modern software projects.
- With manual testing, you lack the ability to test non-visible components in isolation. If you are testing only what you see, you are not testing what you do not see. That seems like an obvious statement, but its importance is high. Software is complex. A large number of the different types of bugs require “under the hood” testing in which you analyze the behavior of individual components in order to identify and resolve incorrect or broken code. This is especially the case with server-side software. Nearly all of the important code in server-side software such as web-based applications resides in the business logic layer. Testing the view layer is only an indirect way of testing the proper functioning of the business logic and misses many details. You want both the business logic and the presentation layer to work correctly, of course, but getting the business logic working properly is the foundation for getting the entire application working properly.
- With manual testing, others cannot verify proper functioning of your application. Other people (e.g. other developers or even your end users) must accept your word that you tested the application and that it conforms to expectations for all aspects of the application. Other than your written or verbal acknowledgement that you tested the application and that you found it to pass all visual inspection, others have no way of verifying proper functioning apart from laboriously testing the application themselves, something they may not be qualified to do because they do not understand the boundary conditions and logic of the application.
Automated testing addresses the inefficiencies of manual testing. So, to answer the question of how test-driven development actually means something other than what has been done for years and years with software development, test-driven development refers to the automated testing of software during the development process, and evolving those tests in conjunction with the evolution of the code base of the application. Note that there are two points here. One is that the testing is automated, implying that it is repeatable and portable. Repeatable means you can run the tests over and over again and get the same results with the same body of code being tested. Portable means someone else can run your tests and verify for himself that your application passes all tests. The second point about test-driven development is that it involves the evolution of the tests along with the evolution of the code base of the application. It does no good to have tests that become out of date with the code they test because then the tests do not really verify proper functioning of the application code. Thus, the test code needs to move in tandem with the application code.
There are two primary methods of performing automated testing of software. The first is via recorded macros that can be played back. This approach is used by tools such as WinRunner by Mercury Interactive (http://www.mercuryinteractive.com/products/winrunner/). While easy to set up, macros tend to be brittle and require frequent revisions because they usually rely on the physical location of buttons and widgets as opposed to structural location within a parsed document. As a developer, it can be frustrating if the testing framework generates spurious error messages and requires significant amounts of time in order to keep tests updated with the application code.
The second primary method of performing automated testing of web applications is via a programmatic API. This means that you have a test framework, a software library that provides functions for checking for conditions and failing if the conditions are not met and functions for reporting on the number and type of errors, and you make calls to the framework from your test code. Your test code will be a collection of standard Java classes that extend classes in the test framework. Your test code will instantiate classes from your application code and call methods to verify that the expected results are returned when specific inputs are supplied. The programmatic API approach is the approach taken by JUnit and HttpUnit, tools for unit testing and black box web testing, respectively. It is a flexible approach minimizing maintenance time of tests in the most common cases and making possible the testing of complicated functions in software applications, especially server-side applications. This type of testing is interchangeably called programmatic testing, API-driven testing, or programmatic API-driven testing.
One downside of API-based automated testing versus recorded macros is that the API-driven approach takes a little while longer to set up. It is hard to beat setup time when you are talking about pointing and clicking. A second downside is that most customers are unable to write programmatic tests. Customers understand the business processes, not the technology. Customers may find the point and click approach significantly easier, which is important to consider if you need to involve the customer in the development cycle, as recommended methodologies like extreme programming advocate.
Nevertheless, the API-driven approach is superior in many respects and should be used for most applications because applications tend to evolve significantly over time, so the time spent maintaining tests tends to be a larger factor than the setup time of tests. Furthermore, the recorded macro approach suffers from the serious limitation that you must have the application code written before you can build the test for it. If the methodology you develop under does not require testing before writing the application code, great. Otherwise, if you adhere to the test-first methodology, something the author recommends, you are out of luck with recorded macros because you must have a working application to interact with before you can record your macro for playback.
A good approach may be to have both point and click recorded macro tests and programmatic API-driven tests. The point and click tests allow you to involve the end user in testing that the application appropriately addresses business needs. The programmatic tests let you make sure that components of the application work as intended from a technical perspective.
Under programmatic testing, you can test your application in two ways: functional testing and unit testing. Functional testing, also called black box testing, means testing the application at a high level and without knowledge (or omitting knowledge) of the internal implementation of specific features. Functional testing verifies that the software application fulfills business needs, preferably by simulating what the end user would do in the same manner that he would interact with the application. The end user could be a business person using a web-based application or a developer making use of a software library you have developed by calling on the API you have supplied. If you want to split hairs, functional testing can actually be different from black box testing because you can technically do functional testing from within the container (if you are working with a web application), but in practice most functional testing is done to a black box that you assume no knowledge of, apart from the published interface. Unit testing, in contrast, involves verification of code at a lower level. Unit testing requires knowledge of the internals of the application in order to put the internal components through their paces. You need to know the classes and methods of the application, not just the form fields and buttons on the screen and the navigation of the application. You also need to know how those classes and methods are implemented. If the software application you work on is a software library intended for developer use, your unit tests should test all of the important classes and methods, even those that are not documented in the API you make available to your developer end users. Functional testing involves interacting with your software application by simulating clicks on buttons and entry of information into the visible forms of your application. Unit testing involves interacting with your application by making Java method calls to the methods of your application.
Automated testing provides good answers to the four drawbacks of manual testing mentioned earlier:
- Eliminate repetition. By automating tests, you transfer the burden of performing them to the computer, which will faithfully follow the exact same sequence of actions each time you invoke the tests.
- Reduce error. Computers perform repetitive tasks in the exact same way each time. Furthermore, because the tests are cumulative and are easy to invoke, even if a change in the application code results in broken code in some far-flung line of code remote to the directly affected application code, you will still catch the errors because the old tests you wrote to exercise that remote section of code will complain. The cumulative nature of automated tests is of great value and is one of the chief advantages of automated testing over manual testing. With manual testing it is impractical to test ever more portions of the application by hand.
- Allow individual testing of isolated components. Automated testing, at least of the programmatic API-driven variety, allows you to test even non-visible portions of the application such as the business logic, which if you think about it is the heart of the application. Programmatic testing involves calling your application code from another program (the test code). The granularity is up to you. You can simulate a web browser or a point and click desktop user, called black box or functional testing because you do not assume anything about the internals of the application. You can call the public API of a software library, which is also considered black box or functional testing because it assumes nothing about the implementation of the API. You can also call the internal methods of hidden or public classes, verifying that the implementation works the way you intended. This is called unit testing because it applies at the level of individual methods and classes and as a result is aware of implementation details such as algorithms used. You choose the level of detail that is appropriate for the test you want to perform.
- Enable end users of your software to verify for themselves that the software functions properly. Not only is this valuable from the perspective of preventing bugs that might corrupt data were the customer to use your software, but it also helps root out unforeseen bugs in your software that arise in new environments. A new environment can be a different operating system, new hardware, different network configuration, or myriad other differences. Automated testing lets you apply testing discipline to other environments and ensure that your application functions everywhere that matters.
There are several tools that are especially helpful for automated testing:
- JUnit. JUnit is the grand daddy of unit testing. Other tools often build on top of JUnit because it provides assertion checking and reporting of results, both of which you need whether you are doing unit testing or functional testing. JUnit is available at www.junit.org.
- HttpUnit. HttpUnit is a testing framework that builds on top of JUnit to allow both black box and in-container testing of web applications. It is a functional testing tool and allows you to verify that the software application meets business needs and conforms to expected behavior at the visible level. Interestingly, the HttpUnit codebase actually has very little to do with testing. HttpUnit is a library that facilitates HTTP access to web applications, including support for features such as state management (cookies), request submittal, response parsing (HTML parsing), and other features you might expect in a web spidering toolkit. HttpUnit also has a ServletUnit class that lets you write in-container test cases. Built on top of the JUnit assertion checking and reporting functionality, HttpUnit becomes a useful tool for testing web applications. HttpUnit is available at www.httpunit.org.
- jWebUnit. jWebUnit is a helper toolkit that builds on top of HttpUnit to reduce the amount of code you must write to test web applications. Roughly speaking, you can think of it as a macro library for HttpUnit with predefined shortcuts for HttpUnit code snippets that simplify the most common actions during testing of web applications. HttpUnit provides a fairly low level interface that lets you customize many things. You may or may not find jWebUnit to be a convenience, given that you can do anything with HttpUnit that you can with jWebUnit. It may take more code with HttpUnit, but you will have more control. jWebUnit is available at http://jwebunit.sourceforge.net/.
- StrutsTestCase. StrutsTestCase is a testing framework that builds on top of JUnit to allow testing of Struts applications. Struts is a Model-View-Controller (MVC) platform that has become quite popular among Java developers who work on web applications for the reason that it promotes the writing of maintainable, componentized code with separation of the application into data, presentation, and logic. Struts makes in-container functional testing and unit testing of web applications difficult because it sits between the servlet container and your application. That means the test framework needs to be aware of Struts and be able to work with Struts in order for you to do in-container testing. Black box testing of the sort that HttpUnit supports should continue to work just fine because you make no assumptions about the implementation of the web application. However, you would not be able to use HttpUnit for in-container testing of Struts applications because HttpUnit expects to sit between your application and the servlet container with nothing else in between. StrutsTestCase is built from the ground up to allow in-container testing of Struts applications. StrutsTestCase is available at http://strutstestcase.sourceforge.net/.
If you accept that testing is good, nay necessary, then you should begin to automate your testing or continue to use automated testing if you already use it. If you accept that automated testing has benefits that make it superior to manual testing, then it is reasonable to evolve the automated tests along with the application code base. The maintenance of the tests is a minor additional burden and yields peace of mind that your application is continuing to function as specified by the requirements of the tests. By adopting both automated testing and the tenet that tests should be maintained along with the application code base, you have adopted test-driven development. Your applications will improve in reliability and quality, and your customers, whether they be external retail customers or internal customers at other groups in your company, will thank you or bother you less about unstable software or both. Finally, writing programmatic tests is just more fun than repeatedly click-click-clicking to test software.
Stay tuned for the next article in the test-driven development series. The next article, entitled “Test-driven Development for the Business Tier,” will go into implementation details of how you test server-side components, both EJBs and regular Java classes.
About the Author
Wellie Chao has been interested in software development since 1984 and has been doing software development professionally since 1994. He has led software development projects to build enterprise-class software applications and has extensive experience with Java and Perl. He is the author of several Java books, including one called "Core Java Tools" (Prentice Hall) that covers extreme programming and test-driven development using open source Java tools such as Ant, CVS, and JUnit. His published articles on IBM developerWorks, DevX, TheServerSide, and elsewhere cover topics of interest to Java developers in enterprise software development settings. He graduated with honors from Harvard University, where he studied economics and computer science, and now lives in New York City.