Discussions

News: Article: A Practitioner's Approach to Performance Testing

  1. In A Practitioner's Approach to Performance Testing," Alok Mahajan and Nikhil Sharma offer a very workable overview of the activities involved in performance testing, offering concrete definitions and filling in potential gaps, such as how to describe the performance of a system, how to plan performance testing, preparing performance reports, and some best practices.
    The application is horribly slow.", "I don't get the response even after I get my coffee.", "This application is useless". Sounds familiar? How many times have we heard these quotes or or felt like that ourselves? The common thread between these statements is that the performance of the application is not good. Performance - the (in)famous buzzword. What is it? What does it mean? In this article, we'll touch upon what is involved in testing an application for performance. With every passing day, organizations are becoming more and more conscious about the performance of their Enterprise Solutions. As the IT industry matures and the technology evolves, so does the awareness about expectations from an Enterprise Application. Focusing just on the design / implementation and Zero-functional-defect solutions are things of the past. With increasing maturity in technology and IT staff, the 'Non-functional' aspects of the system are fast becoming focus-areas. So what exactly are the non-functional aspects and/or requirements? Non-functional requirements (NFRs) tell the IT team, about the kinds of usage and load the application will be subjected to, and the expected response time. We'll go into the details of this "response time" shortly. NFRs define the Service Level Agreements (SLAs) for the system and hence the overall Performance of the Enterprise Application. Besides performance SLAs, NFRs also cover several other aspects, such as security, but for this article we are concerned with performance related objectives only. Managing and ensuring the NFRs (SLAs) for an Enterprise Application is called Performance Engineering. Performance engineering is a vast discipline in itself which includes Performance Modeling, Performance Prototyping, Performance Testing, different types of analyses, Performance Tuning, etc. This article will not explain Performance Engineering, Queuing Theory and the science behind the various laws. This article just covers the basics about the Performance Engineering and key activities in Performance Testing.
  2. It would have been nice to have seen the tasks outlined in the article aligned to the industry standard for engineering in performance into a software product - Software Performance Engineering (SPE). Even before the software is built never mind tested. http://www.spe-ed.com I also think the article should make it more explicit that performance testing as described is a very small part of software performance engineering and that the primary goal of performance testing (and engineering) should be to learn more about the software and system execution (performance) models rather than validating that the software passes some artificial (and possibly incorrect) test script. Knowledge acquisition is paramount and a key benefit in introducing software performance engineering across the complete application lifecycle - from development straight through to production. Performance Engineering != Performance Testing http://blog.jinspired.com/?p=175 William
  3. Performance Engineering != Performance Testing
    Agree completely with you on that William. In fact if you cautiously skim thru the second section of the article itself you will find the following text -
    Performance engineering is a vast discipline in itself which includes Performance Modeling, Performance Prototyping, Performance Testing, different types of analyses, Performance Tuning, etc. This article will not explain Performance Engineering, Queuing Theory and the science behind the various laws. This article just covers the basics about the Performance Engineering and key activities in Performance Testing.
    And we believe thats pretty explicit :) !!! Looking forward to more inputs and feedback from TSS readers on this and we will address SPE in future articles (hopefully!). thanks, Nikhil.
  4. Performance engineering is a vast discipline in itself....
    I am being fastidious but the usage of ** discipline ** instead of process in my opinion gave too much weight to testing. But maybe this is a bias I have after witnessing so many performance issues in production that were never picked up during performance testing because the test team focused too much on the what (the articles key graphs) rather than the how and why and as soon as things changes (which they do) they had very little to offer operations during problem management other than "it passed our tests". I suppose disconnecting performance testing from the process and limiting the requirement on teams to fully understand the software execution behavior (models) across the lifecycle makes it easier to offload this work (testing) to external suppliers. Unfortunately without a body of knowledge that is continually being archived and updated you might as well be hiring a witch doctor for a diagnosis of a production problem (see following blog entry). Performance Models: Software versus System http://blog.jinspired.com/?p=219 William
  5. http://blog.jinspired.com/?p=175
    The diagram on "what is SPE" shown in your blog seems to be centered around modeling and analysis.Correct me if I missed something but isn't performance engineering beyond modeling and analysis? I think the representation of SPE there misses several core engineering activities like architecting,designing and coding for performance and ensuring that performance best practices are followed.. These are very essential activities since they help avoid the performance problems in the first place.. I think a good PE strategy is one that is proactive and ensures that performance problems do not happen in the first place instead of reacting to them after they occur.. Ofcourse, if a system is already built that there may not be too many options than to make your performance testing robust so that you don't have to fix problems in production.. http://msdn2.microsoft.com/en-us/library/ms998534.aspx has a good overview of the foundations of performance engineering IMO... Infact the several chapters there at http://msdn2.microsoft.com/en-us/library/ms998530.aspx are a good reference for someone interested in PE though they have a .NET flavor(you can draw parallels for any other technology..) -Shyam
  6. This diagram depicts the 12 main activities (hint: clock) I see within any software performance engineering activity. These are derived from the following two books: High-Performance Client/Server http://www.amazon.com/High-Performance-Client-Server-Chris-Loosley/dp/0471162698/ Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software http://www.amazon.com/Performance-Solutions-Responsive-Addison-Wesley-Technology/dp/0201722291 I am not sure what you mean "centered around modeling and analysis" as I always assume people design and development products (architectures included) with some degree of modeling and analysis beforehand whether this is formal or not. There are 12 activities listed and only 3 relate to models and only one of these related to simulation which is what I think it throwing you off. If you looked carefully you would see the activities span the complete cycle. The activity chart operates very like a normal clock so it is cyclic and that activities will be repeated over and over again as knowledge acquisition increases and changes are applied to both the system or software itself. Software performance engineering (the process) does not limit its application to actual software already constructed or under construction so the omission of architecting, designing, and coding for performance is deliberate but this is well understood within the SPE community. Many performance problems can be engineered out of a software product long before the first line of code is written. At the same time SPE can and is used to derive a system/software performance model of a existing system in test or in production. Here the model ** accurately ** reflects execution abstractions (workflows, business process, client-server interaction,...). Such models are extracted from or mapped to actual measurement models collected by application performance management and problem diagnostics tools such as JXInsight. There are documented design patterns for performance in both SPE related books listed. These are kept apart from the process because of the rapid rate of change in technology landscape. The process is likely to outlast any design pattern. I do not think you can get any more proactive than SPE. SPE does not assume it owns or directs the other software development activities you listed. SPE might not even be used on a project which is clearly pointed out in the first activity "Assess Performance Risk". SPE is there to guide the teams (dev, test, ops, capacity planning) if performance is critical for the application or particular use cases. A benefit of SPE models collected during development and testing, which have varying levels of execution details, is that one has a reference point to start the problem resolution. With testing you get "it passed our tests with x=y, and z=w but other than that I have not got a clue". William
  7. Just write your own article[ Go to top ]

    Just write your own article and submit it if you want to promote your products and services. I'm even more sick of people who try to hijack an article for their own marketing purposes than I am sick of articles that are just thinly disguised press releases. This isn't a specific attack against William Louth, but just because his posts today exhibit all the typical behavior that for some reason I feel a need to vent against, including: pitching his own product pointing to his own blog (to pitch his own product) not associating himself with his own product that he is pitching not reading the article picking a specious criticism that is typically answered in the article, but is irrelevant anyway defending his untenantable position even after being corrected repeatedly pitching the product he has already pitched
  8. Re: Just write your own article[ Go to top ]

    Hi Aaron, Please read my posts again after clearing away any prejudices/preconceptions. You might this time around see that in fact I was pitching a standard methodology to performance engineering (SPE) which is independent of any product and in theory could be done with a spreadsheet if one had the measurement data. I pointed to my blog because I had already commented on this before in a different context. I did not associate myself with a product via a signature because I thought everyone on TSS already knew and this seemed too promotional. You just cannot please everyone. I read the article and was not impressed with its simplistic approach and view of performance engineering through the eyes of a tester. This is my field of expertise so I feel obliged to comment and criticize. Can you point out content which you found particular interesting? Corrected? When did that happen? Discipline and Process are sufficiently different in the context of the article. The article ironically reflected the major problem I see with performance testing done outside the context of a process and model. Pitching is for sales people or VC's promoting alternatives to Java EE on TSS posts without stating their vested interests in alternative platforms. I referenced my WORK, which by the way is free for development usage, once explicitly which I will continue to do whilst it demonstrates how I have solved the problems that people are still learning about. If there is something being pitched it is ME. Aaron you are free to have and express you own opinion but I think you are off slightly on this one but that is my own steadfast opinion irrespective of commentary.
  9. Re: Just write your own article[ Go to top ]

    ...I read the article and was not impressed with its simplistic approach and view of performance engineering through the eyes of a tester. This is my field of expertise so I feel obliged to comment and criticize. ...
    Just to clear some concerns here (again on SPE and Performance Testing ) we specially intended to cover Performance Testing FIRST, since by inundating the first part itself with SPE theories and processes, we felt we could have lost our audiences who are specially concerned (interested in knowing) about the ground parameters and approaches for testing, rather than the entire SPE process (as 'some' say or discipline as 'we' have mentioned :) ). In fact, I believe these theoretical process aspects can be well covered once the audience is clear about the basics like, what is NFR, what all is involved in Performance Testing and how to go about it... Moreover, I tend to believe that Performance Testing (in the form of some basic load testing & modelling) first came into existence and which later evolved into a more formal 'discipline' which is now called as SPE. So it makes more sense to cover the basics first and follow (KISS:Keep-it-simple-stupid) so as to reach wider audience. thanks, Nikhil.
  10. Re: Just write your own article[ Go to top ]

    Moreover, I tend to believe that Performance Testing (in the form of some basic load testing & modelling) first came into existence and which later evolved into a more formal 'discipline' which is now called as SPE.
    I think my commentary is now to substantiated with the above statement. Maybe you take this view because your became a performance tester first and then became aware of the bigger picture which is addressed in SPE. There is no evolution here. Performance analysis is focused on measurement, interpretation and communication. There is no explicit mention of testing here. Testing is a means to an end at a very late stage in the process. Testing on its own is like building on top of a deck of cards. SPE came into existence because performance testing by itself failed to recognize the parameters of the problem itself. Admittedly performance testing is much more prevalent in the the industry but that was because the skill levels for a professional performance engineer are very much higher and that for a long time the techniques and tools were only know to a few "wizards". Lets be honest here the majority testers focus in the tool (HP LoadRunner) and less so on the process or application software execution model. I have met many load runner users (performance testers by your definition?) who have very little understanding of the runtime or software stack they are testing. How can one say this is engineering? I like to believe that SPE has addressed this to some degree by demystifying the approach and techniques these engineering "wizards" commonly used to achieve their status. SPE focused attention away from activities that are less effective but still relevant at one stage in the life cycle. I believe Witch Doctors predate Medical Doctors but I know which one I would go to if I had a problem and I did not want just a confirmation that x + y = 2 when x = 1 and y = 1. William
  11. Re: Just write your own article[ Go to top ]

    There is no evolution here....
    I think you are not getting the point here. There is always an evolution in all branches and discipline. There HAS TO BE ! You have also metioned -
    SPE came into existence because performance testing by itself failed to recognize the parameters of the problem itself.
    So thats what essentially I mean by evolution of SPE ...and I agree completely with you on that. I believe, what you understood by my statement is that when working on a project ... we should first work on Performance Testing and then move on to SPE ... which is INVALID. So let me clear your concern again, that SPE is the overall methodology, but we intentionally covered Performance Testing first. Please do not take this to be any indication of our ignorance on SPE. -Nikhil.
  12. Wow!!![ Go to top ]

    Wow. What a long discussion... So people did read our article and liked it too. :-) I'll try to keep this to the point, not discussing any specific sentence/comment. The more you know, clearer you can see. Clearer you can see issues/gaps in industry knowledge sharing. Being formally trained in Performance Engineering (Huh - that Queueing theory) and having practised it for a while, we felt a gap we can address with this article. This is our attempt in giving back to IT community and to clear the clutter for an IT person. As the subject of PE goes, this is not the forum as the article specifically calls it out. (viz. PT is just a tiny bit of PE). In this forum, we can talk about if something is not clear/incorrect/some relevant reads/additional comments & so on... Thoughts? May be when somebody publishes his PE article on TSS, we can discuss it there. ;-) So long then...