Discussions

News: WebTest vs Selenium: WebTest wins 13 - 5

Bookmark and Share Get thread feed

  1. WebTest vs Selenium: WebTest wins 13 - 5 (14 messages)

    • Posted by: Tomi Schuetz
    • Posted on: October 29 2007 08:14 EDT
    The maintainability of automated tests depends primarily on the skills of the test authors, but different tools have different features that impact their efficiency. This blog post compares the features of two open source automated web testing tools: Canoo WebTest and Selenium.
    Automated functional tests of web application should become as natural as unit tests. Some tests are better than no tests, no matter which tool is used. Selenium does a good job to introduce newcomers and has many advantages (besides the price) over its commercial model QTP. Nevertheless at least when the size of your suite grows, you should pay attention to your efficiency if you want to last and "the ROI on WebTest is many orders of magnitude higher than any other tool I’ve used" (Lisa Crispin, author of Testing Extreme Programming). When comparing Selenium and WebTest, 3 categories of web application can be considered. First the applications that are supported both by WebTest and Selenium. This contains most of the applications. Then the ones that use browser features (mostly javascript) that are not yet supported by HtmlUnit. Due to HtmlUnit's awesome progress in JavaScript support, the size of this category continuously shrinks. The last category concerns applications that use for instance PDF documents or applets and for which Selenium has no support. In all cases where WebTest can be used it is far more efficient than Selenium to ensure the quality of web applications because it provides more feedback and takes less time (both to execute and to analyse results). When I started writing this post I didn't expect that Selenium would get such a bad score. Comments are welcome to show which advantages of Selenium over WebTest I've missed!
    [Editor's note: the author of the blog post is a WebTest and HtmlUnit committer, which he offers early in the blog posting.]
    Reply to this

    Threaded Messages (14)

  2. Re: WebTest vs Selenium: WebTest wins 13 - 5[ Go to top ]

    The big thing is that Selenium gives record/playback. This approach allows replacement of much of repeated, manual UI testing with record once, maintain, and replay -- and allows it by lower-skilled QA personnel. Unless/until something like WebTest allows record/playback it isn't addressing this issue.
    Reply to this Reply to original

  3. Record / Playback[ Go to top ]

    The big thing is that Selenium gives record/playback.

    This approach allows replacement of much of repeated, manual UI testing with record once, maintain, and replay -- and allows it by lower-skilled QA personnel.

    Unless/until something like WebTest allows record/playback it isn't addressing this issue.
    I've been using Selenium extensively for over a year and don't think the record / playback feature adds much. The major obstacle with Selenium IDE is the high level test maintenance due to duplication. Record / playback is going to make this worse if anything. As for using 'lower-skilled QA personnel' - Good QAs are essential to a project's success, and it's the developers that should make most use of tools like Selenium
    Reply to this Reply to original

  4. Selenium Test Maintainability[ Go to top ]

    I agree with this, the biggest problem with Selenium (and all other web based testing frameworks for that matter) is not whether or not they can be recorded or not - recording only works for the simplest possible cases and not at all for AJAX stuff - but rather how you maintain these tests as the application that they test is changed. The technique that I have used, reasonably successfully, is to not use Record/Playback at all (since it doesn't work) and instead build a simple DSL that models the app that you are actually testing (and that translates things back to the more generic web testing framework that you are using) and construct your tests in that. This makes test writing easier and gives you a layer of indirection to help with maintainability. Much more here: http://citechnology.blogspot.com/2007/09/custominks-domain-specific-language-for.html
    Reply to this Reply to original

  5. Sahi web test tool[ Go to top ]

    It is true that web test is harder to maintain and longer to develop. However, there is value testing all the application layers including the GUI. We have use few year ago httpunit, we also use jmeter to perform fonctionnality testing (I realize it was not a very good idea). About 8 months ago we found Sahi. It is not a perfect tool but it is the best we have found to test j2ee web application. As already discuss in this discussion thread, Record/Playback is not the best for maintainability. however, the recorder does a great job. The best is obviously to build reusable fonctions. It is easy to do this using Sahi since Sahi tests are build using java and javascript code. For each test, our stragegy is mostly to initialize the application using java code and then perform the test using the javascript that drive the web application. Our application use a lot of javascript and we didn't found much limitation using this tool. sahi web site: http://sahi.co.in/ Phong Do solabs.com
    Reply to this Reply to original

  6. Tellurium is a different way to write Selenium tests[ Go to top ]

    The "Record and replay" style really does not work for our company. Even the UI is not changed, there are still data dependence in the testing flow and the script generated from Selenium IDE is really difficult to refactor. That is why we threw this testing method away in the first place and developed the Tellurium framework on top of Selenium. Tellurium is not a "record and reply" style testing framework, but UI module centric. That is to say, you should define your UI module first. Based on your Ui module, you can write as many tests as you like just like writing JUnit tests. Tellurium also support pure DSL scripts and Data Driven Testing. Please check http://code.google.com/p/aost/ for details. Thanks.
    Reply to this Reply to original

  7. Re: WebTest vs Selenium: WebTest wins 13 - 5[ Go to top ]

    The big thing is that Selenium gives record/playback.

    This approach allows replacement of much of repeated, manual UI testing with record once, maintain, and replay -- and allows it by lower-skilled QA personnel.

    Unless/until something like WebTest allows record/playback it isn't addressing this issue.
    see here http://webtestrecorder.canoo.com/
    Reply to this Reply to original

  8. How about 'Element' from PragmaticQA[ Go to top ]

    We were looking for a web-based functional testing tool that can do all the basic functional testing and management of test-cases & test-scripts. Also we didn't want to use any proprietary scripts or complex xmls. We found Watir very impressive in that front. It does the basic functional testing in a simple & elegant way, using Ruby script. But we wanted more than the scripts to drive the browser, we wanted an whole ecosystem, something sits in between Confluence & JIRA and eases our life. We built a functional testing suite, that wraps around Watir as the functional testing tool. This also uses a web-recorder which was adapted from http://www.hanselman.com/blog/. The whole functional testing suite is completely web based, so it doesn't need any installation on the client side except Ruby & Watir, which are standard modules. At a high level it does all these - Manages test-cases - Manages test-scripts. It can even autogenerate the test-scripts. Also the test scripts can be tested immediately over the web as we generate / hand-edit them.It supports versioning of test-scripts. All these scripts are pure Ruby code. - Association of Pre-conditions to test-cases.. So you need not write the same 'login' script every time for each script. You can include the condition name and the script will included dynamically during execution. - Create 'test suite' by selecting these test-cases & run them. - Compare test results for previously executed test-cases. The Niche features in this product that i haven't seen in others are - Completly web based & no install on the client side except Ruby & Watir - It is not just a functional testing tool, it is web based suite that offers all the essential features. - The test-scripts are written in pure Ruby, so it is easy for anybody to maintain them. - All the data is stored on server, so easy access for all the team members. - All the parameters in the script can be externalized and inserted dynamically when the scripts are executed. - It allows creation of profiles, where we can override all the externalized parameters for these test-cases, so it can be run on multiple environment. We can have separate set of parameters for QA & Prod environments where the username, password, urls etc. Check it out @ www.pragmaticqa.com It has a demo functional test written for Google. Try it out & drop us an email. Thanks Murali Varadarajan
    Reply to this Reply to original

  9. www.pragmaticqa.com[ Go to top ]

    The website is http://www.pragmaticqa.com
    Reply to this Reply to original

  10. Re: WebTest vs Selenium: WebTest wins 13 - 5[ Go to top ]

    Record/Playback is becoming a minor help when using Ajax. That is, recording Ajax pages requires recording and reproducing javascript actions and events. Is this supported in the Selenium/WebTest? William
    Reply to this Reply to original

  11. Selenium + Ajax[ Go to top ]

    Selenium copes with Ajax, but you need to use waitForText rather than verifyText etc. e.g. Command: click Target: link=OK Command: waitForText Target: xpath=/descendant::div[@id='ajaxResponse'] Value: Success
    Reply to this Reply to original

  12. More ideas:[ Go to top ]

    Cool, i'd suggest some more subjets for those guys: WebTest vs DBUnit JUnit vs Cactus Database vs LDAP ... Selenium, WebTest(HttpUnit), DBUnit, JUnit and others are compimentary. There are things that you can do with one and cannot do with another.
    Reply to this Reply to original

  13. Running JUnits faster on the grid[ Go to top ]

    If you want to run your JUnit faster by putting them on the grid with a simple annotation take a look at GridGain. Detailed instructions can be found on Wiki. Best, Nikita.
    Reply to this Reply to original

  14. WebTest *and* Selenium?[ Go to top ]

    Possible to develop a middle layer that allows one to write one set of tests that can be run on both WebTest and Selenium? This way people can frequently run WebTest and may run Selenium nightly? It should automatically skip tests that obviously don't run in one environment (possibly with the help of an annotation if it is required).
    Reply to this Reply to original

  15. Does anyone else use badboy?[ Go to top ]

    A bit out of context based on this discussion, but I was wondering if anyone else has experience with Badboy - http://www.badboy.com.au ? It's got a stupid name but is pretty powerful and has a friendly license. Downsides are that Badboy is a windows only product; pluses are that it uses the IE activex control and can test a "rendered" page complete with ajax calls and javascript. We've set up an automated web test suite that runs on a nightly basis. Our only issue is maintainability. With an active project, the testing script really needs an individual to keep it up. Just wondering about other peoples experiences doing automated web testing. Thanks, Will Ezell dotcms.org
    Reply to this Reply to original