Home

News: Distributed JUnit tests from GridGain 1.6

  1. Distributed JUnit tests from GridGain 1.6 (19 messages)

    JUnit tests (or unit tests in general) are an embarrassingly parallel problem that begs to be Grid-Enabled. Builds can take a long time just because hundreds of tests are running sequentially on a single box. By simply grid-enabling unit tests you can cut your build time by an order of magnitude and have a quicker turn around on bug fixes, making the whole process more dynamic and agile. GridGain is planning to ship Distributed Unit Tests with 1.6 release (Nikita Ivanov blogged more about it here). Here is what distributed JUnit integration will look like:
    1. Just start several grid nodes. Nodes can be anywhere... it could be your neighbor's desktop, a dedicated server some place, or it could be installed anywhere in the world as long as GridGain is properly configured.
    2. Use GridJunit3TestSuite (for JUnit3) instead of regular test suite and then use your favorite Java IDE and its JUnit integration as you usually do. Following example shows that tests A and B will be executed sequentially as a group and test C will be executed in parallel with A and B. Both A and B group and test C will be executed on remote nodes (i.e. on the grid) in parallel./** * Sample distributed test suite. */ public class JUnit3SampleDistributedTestSuite { /** * @return Test suite. */ public static TestSuite suite() { // Distributed test suite. GridJunit3TestSuite suite = new GridJunit3TestSuite("Grid JUnit3 Distributed Test Suite"); TestSuite abGrp = new TestSuite("A & B Sequence Group"); abGrp.addTest(new TestSuite(TestA.class)); abGrp.addTest(new TestSuite(TestB.class)); // Run tests A and B in one suite sequentially on remote node. suite.addTest(abGrp); // Run test C on remote node. suite.addTest(new TestSuite(TestC.class)); return suite; } }
    3. Once you have created your distributed test suite, just hit "Run" in any IDE of your choice and distributed test will be executed as if it was a local run - your JUnit tests will run on the grid completely transparently. The entire semantic of running JUnit tests from a Java IDE or from command line is fully preserved as well as all GUI tooling.

    Threaded Messages (19)

  2. Congrats with this functionality! BTW, Have you looked at http://testng.org It has distributed test cases execution for years already and quite mature. Some of the testNG features: # DK 5 Annotations (JDK 1.4 is also supported with JavaDoc annotations). # Flexible test configuration. # Support for data-driven testing (with @DataProvider). # Support for parameters. # Allows distribution of tests on slave machines. # Powerful execution model (no more TestSuite). # Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc...). # Embeds BeanShell for further flexibility. # Default JDK functions for runtime and logging (no dependencies). # Dependent methods for application server testing. Personally testNG is my favorite. Switching back to JUnit would be a painful loss in the needed functionality that offered right our of the box!
  3. Hi Ruslan, TestNG is certainly on the list. We just took a swipe on JUnit because it is still a primarily work horse for unit testing. I’m not sure I would say that TestNG has a good distributed execution support (it’s rather very rudimentary). I also think that distribution of the work load (in this case the distribution of test execution) is rather an orthogonal concern for the framework like TestNG. Anyways, if we see a good pick up on JUnit usage we’ll definitely integrate with TestNG. Best, GridGain - Grid Computing Made Simple
  4. Effective distributed testing[ Go to top ]

    Congrats with this functionality!

    BTW, Have you looked at http://testng.org
    It has distributed test cases execution for years already and quite mature. ...
    Hi everybody, surely I am not one of the greatest experts around about TestNG, but as far as I know its support for distributed testing in not so mature as you state really. As far as I can see there is not official documentation covering the topic but this blog entry http://beust.com/weblog/archives/000362.html that nevertheless highlights some lacks in the current implementation. On the other side, the JTR Project (http://jtrunner.sourceforge.net) provides an interesting support for distributed testing (both for JSE and JEE) that already covers issues such as zero-configuration of JTR passive-nodes in terms of runtime libraries, runners implementations (that must be dynamically moved from the active/master node) and so on. All the details are provided in the following section of the JTR Users' Guide http://jtrunner.sourceforge.net/JTR/UG%209%20Distributed%20Testing.html The most important thing is that distributed testing is achieved with no modification to your runner implementations, while this does not seem to me like being the approach that should be taken by the proposed GridGain: am I right? Francesco
  5. I would like to clarify that GridGain 1.6 is not out yet. This is the release that is being planned to be out hopefully within a couple of weeks. The original intention of this post was to find out who really needs Distributed Junit? Does your company really need it? Would you start using Distributed Junit right away or in the nearest future? Or is it acceptable for your business to wait 1+ hours for test results after a developer makes a commit? Please let us know. Best, Dmitriy Setrakyan GridGain - Grid Computing Made Simple
  6. Hi Dmitriy, There is some beta release? I would download and try it. Thanks
  7. Hi Fernando, Unfortunately there is no Beta release. But 1.6 is coming out very shortly (matter of 1 or 2 weeks), so hopefully you won't have to wait too long. This release will truly revolutionize how developers will run their JUnits on daily basis. Not only you can make your build server execute your tests faster, but imagine being able to press a Run button in Eclipse or IDEA and have your tests execute on the Grid by an order of magnitude faster than usual. It will still look like a local run to you, with Green successes and Red failures in your IDE with all the logging preserved as usual, but you would be able to find bugs and fix them much faster. The whole project will become much more dynamic and agile! Best, Dmitriy Setrakyan GridGain - Grid Computing Made Simple
  8. Great Idea[ Go to top ]

    Nikita, this is a great idea! In GigaSpaces we are using this approach for quite some time with Tigris. By creating awareness to testing parallelism,and by providing an integrated framework, you are doing a great job. Well done! Guy
  9. If the main reason for slow performance of the tests is the database, then there cannot be gained speed by the grid. Our tests do a lot of database operations (storing and loading for example). So they are slow. You will now object, that it is a bad idea to use database so heavily. That may be right, but I don't think anyone will rewrite the tests.
  10. Well, you can still get a significant performance jump by running your test in parallel. RDBMS are extremely good at parallel access (caching, etc.) and I’m absolutely positive you will see dramatic performance increase comparing to sequential run of the tests. It won’t be linier, of course, in this case but it will significant nonetheless. Best, Nikita Ivanov. GridGain - Grid Computing Made Simple
  11. Maybe so for database grinding applications, but for web apps only a small % of the tests should hit the database. Everything above the persistance layer can use mock objects, so there's plenty of gain to be had. Assuming GridGain can use a separate in-memory database per host and your app doesn't have any vendor specific SQL then even db tests could be made to fly. Lastly bit surprised TSS hasn't announced the launch of Selenium Grid. http://seleniumgrid.thoughtworks.com/ Same idea as GridGain but for Selenium RC
  12. Hi Steve, One important point regarding Selenium (which should be announced – totally agreed) is that it is not for grid enabling JUnits or TestNG. We targeted JUnit (and later TestNG) integration as it is a most used unit testing framework – something that most of us use by default in our daily Java development. Selenium is a web testing tool (runs in the browser), Grinder is a load generator and TestMaker is a heavy wrapper on top of unit testing (just look at its XML...). All are very different products. Best, Nikita Ivanov. GridGain - Grid Computing Made Simple
  13. Setting up test environment[ Go to top ]

    How would I set up remote test environment? Setting up classpath and loading all necessary classes isn't enough. What if I need to deploy my application prior to test it? Kind of integration tests. Are you planing any special Maven2 support for testing? I know that Globus Toolkit allows file staging in GRAM which means that you may set up environment before you begin the job (running tests) and clean the environment after that job. What GridGain can offer for this purposes?
  14. Re: Setting up test environment[ Go to top ]

    Setting up classpath and loading all necessary classes isn't enough. What if I need to deploy my application prior to test it? Kind of integration tests.

    Are you planing any special Maven2 support for testing?
    The beauty of the approach chosen by GridGain is that all your native tooling is at your hands. Just use your native build tool, Maven or Ant, and start your application on as many computers as you like (just make sure to start GridGain with it). Then you should be able to start running the tests from any computer, and when grid-enabled, your tests will travel to other computers with fully initialized environment. For example, let's say that you need to deploy into JBoss prior to test. Just start GridGain inside of JBoss (fully supported) on as many nodes as you need, and you should be good to go. Best, Dmitriy Setrakyan GridGain - Grid Computing Made Simple
  15. Distributed unit tests[ Go to top ]

    Hi GridGain: Congratulations on the announcement. I'm eager to see 1.6 and do a comparison of it to PushToTest TestMaker. To my knowledge TestMaker and the Grinder are the two open source projects that operate unit tests on a grid of test machines. Have you looked at TestMaker and the Grinder and what do you think? In TestMaker's case we implemented the distributed test environment using TestNodes. One defines a TestScenario document to identify which TestNodes to operate the unit test and what order and frequency to run the unit tests. For example, here is a unit test that uses our HTTP protocol handler in a unit test: package com.pushtotest; import com.pushtotest.tool.protocolhandler.ProtocolHandler; import com.pushtotest.tool.protocolhandler.HTTPProtocol; import com.pushtotest.tool.response.Response; import junit.framework.TestCase; public class WebAppTest extends TestCase { private HTTPProtocol http = null; public WebAppTest(String testName) { super(testName); } protected void setUp() throws Exception { String url = "http://examples.pushtotest.com"; http = ProtocolHandler.getProtocol("http"); http.setUrl( url ); } protected void tearDown() throws Exception {} /** * Test of calcInvoiceAmount method of the service */ public void testCalcInvoiceAmount() { System.out.println("calcInvoiceAmount"); int PONum = 135; String expResult = "1000"; String result = http.connect(); assertEquals(expResult, result); } } The setUp() method instantiates a TestMaker HTTP protocol handler object and points it to the Web application URL. The testCalcInvoiceAmount uses the protocol handler to make a request to the application and asserts an exception if the result is not equal to 1000. TestMaker uses the unit test as a load test in the following TestScenario: DPL Test ./dpltest <!-- The count of sequences the test operates concurrently --> ... This TestScenario instructs TestMaker to distribute the WebAppTest.jar containing your unit test to the TestNode on localhost (the default TestNode installed with TestMaker on your development system.) The TestNode then instantiates 50 concurrently running Threads (crlevel value="50".) Each thread instantiates a WebAppTest object, calls the setUp() method at the beginning of the test, and then repeatidly calls the testCalcInvoiceAmount method during the duration of the test. At the end of the test the TestNode calls the tearDown() method, ends the 50 running Threads, and then repeats the process at 100 Threads. If I want to run the test on additional TestNodes then I add more definitions to the TestScenario. It seems like GridGain is doing something similar and I would like to hear your impressions. -Frank Cohen http://www.pushtotest.com
  16. Distributed Unit Test != Load Test[ Go to top ]

    Hi Frank,
    I'm eager to see 1.6 and do a comparison of it to PushToTest TestMaker.
    There is nothing to compare. PushToTest is a great Load-Testing product, but it has absolutely nothing to do with Distributed JUnit testing. You entirely missed the point. Distributed Junit support from GridGain is not trying to solve load testing problem. It is geared towards running your daily JUnit suites in parallel on remote nodes to speed up the overall execution. Best, Dmitriy Setrakyan GridGain - Grid Computing Made Simple
  17. Hi Dmitriy: I'm glad you wrote that GridGain "is geared towards running your daily JUnit suites in parallel on remote nodes to speed up the overall execution." I did not get that from either your announcement and your Web site. I did write "It seems like GridGain is doing something similar and I would like to hear your impressions." And I still would like to know how GridGain speeds execution of unit tests, and why this is even needed? Also, the TestMaker example above shows using a unit test as a load test. I could also show you a TestScenario using a unit test as a distributed functional test and as a service monitor periodically checking a service by running a unit test. TestMaker is not just for load tests. Please tell me more about how GridGain speeds execution of unit tests. -Frank Cohen http://www.pushtotest.com
  18. Please tell me more about how GridGain speeds execution of unit tests.
    The answer is simple. Assume that you have 3 unit tests that take 5 minutes each to complete. By running them sequentially on one computer it will take a total of 15 minutes. However, if you have 3 boxes running GridGain, then each test should execute in parallel on a separate box. Hence, the total execution time will become 5 minutes and not 15 minutes. This example alone gives you 10 more minutes to fix bugs or develop new code. But what if you can also cut your test execution time from 1 hour to 15 minutes by simply adding 3 boxes running GridGain? Extra 45 minutes to fix bugs is not bad at all in my view =) I think that benefits are obvious.
    and why this is even needed?
    I guess you haven't seen tests that take over an hour to run sequentially =) How long does a regression test suite for your product take to run? Best, Dmitriy Setrakyan GridGain - Grid Computing Made Simple
  19. Load balancer for grid?[ Go to top ]

    Hi Dimitry: Thanks for the answer. I think I get it now. "How long does a regression test suite for your product take to run?" On my MacBook Pro with 2.16 Ghz Intel Core 2 Duo at 3 Gbyte of memory it takes about 3 1/2 minutes to compile TestMaker. So I'm not in your demographic! :-) I'm really curious to know if TSS people are running unit tests that take 45 minutes or longer? In the example you give of having 3 unit tests taking 5 minutes each, is there a software load-balancer to let GridGain figure out which box to run the unit tests? So, I don't have to manage that part of the grid. -Frank
  20. Re: Load balancer for grid?[ Go to top ]

    Hi Frank, With all due respect I have yet to come across any serious enterprise system that would take less than, say, half an hour to run all unit and regression tests. Notice I'm not talking about build – the build itself, for example for GridGain, takes less than 1 minute – and then ~40 minutes for running all tests. I think demographics here are quite substantial. Furthermore, this demographic will grow as more and more people will understand the value of unit testing and agile development in general. Best, Nikita Ivanov. GridGain - Grid Computing Made Simple