Discussions

News: Continuous Performance: The Next Advance in Software Development

  1. A new TSS article, by Cody Menard, proposes a best practice known as Continuous Performance. According to Cody's theory, testing performance several times a day makes performance tuning vastly more efficient, and can result in software that performs better overall, because code is cleaner throughout the development process. What does the community think about this theory?

    Read 'Continuous Performance: The Next Advance in Software Development'

    Threaded Messages (40)

  2. I have a bunch of problems with this article....

    First, it's hard to work out the concrete steps the author is recommending. I can imagine using JUnit or a similar tool to measure the performance of my builds, and I can imagine collecting those metrics so we can identify the exact time those problems were introduced. Is there anything else we can do ? How do you trade performance improvement over the other development priorities - features, bugs, documentation etc. ?

    Second, I think there's a risk of premature optimization. Often, making an application go faster means making it harder to understand, or introducing caching or other technologies. In both cases, you introduce both cost and risk into the project - code that's hard to read is expensive to maintain, and is more likely to include bugs. Caching requires additional cache management code, which again raises maintenace costs and can be the source of fiendish heisenbugs.

    When dealing with performance issues, I think you have to pick the right battles to fight - concentrate on areas which cost you the most. Is there not a risk with Continuous Performance practices that there will be several knee-jerk "optimizations" concentrating on relatively minor issues, rather than a single major optimization effort ?

    Nonetheless - I am already thinking about the exact metrics I need to collect, and how to integrate that into my build process.
  3. Neville,
    Donald Knuth wrote "Premature optimization is the root of all evil". You eluded to that and we certainly agree. However we are not recommending that you optimize your code with every build but simply to include performance testing along with functionality testing in your development process. In many cases, performance issues are the result of bad design or poor coding at the component level. Some of them could be caught and corrected early. Left unchanged until the massive performance optimization project they will be buried deep into the code, causing painstaking debugging and will be much more costly to correct.

    Many customers that we've talked to agreed that they have to test for performance early but lack the discipline and the tools to do it. Our customer surveys have shown that with the right set of tools, developers are willing to measure the performance of their code and correct any bottleneck early in the process. Most performance consultants look first for faulty SQL calls or try to isolate the component consuming the most memory or CPU. Developers could detect some of the same issues during unit testing or system testing.

    We have integrated our product IronEye SQL (that measures the performance of SQL statements), with Ant and JUnit. With this trio, you have a simple, low-cost solution to include automatic performance testing of your SQL traffic with every build. You could easily catch a long running SQL and correct it early in the cycle. With our IronEye Cache, you can introduce caching and simulate various caching conditions without changing one line of code. The goal is not so much early optimization but rather early identification and correction of possible bottlenecks that will hurt in the long run.

    cody
  4. We've begun to address this exact issue where I work. I'm a big believer in Knuth's early-optimization-is-evil idea. One thing I struggle with is how to establish the metrics for "good" performance. Developers here use Wintel laptops for their continuous integration building. So what's the acceptable throughput, response time, transactions/sec, etc. in that case? How does that compare to the "real" hardware solution/implementation, which typically has many machines, each with potentially multiple CPU's, running a different O/S, deployed on a production network topolgy? The performance of these two implementations would be very different and might show significantly different performance bottlenecks.

    Maybe that SQL code that a developer is optimizing has a trivial impact compared to some JMS pub/sub issues upstream of the database calls? So isn't that wasted time? XP says something to the effect of "code the simplest thing that works" - no more, no less. You may need to optimize that simple (i.e. slow) code later - but chances are, you may not!

    I agree in principle with the "Continuous Performance" approach. You can do well by having knowledgeable developers (!) and conducting code reviews that include performance. That's what we're striving for here. We're also setting up some performance mini-labs (a few Linux boxes) to try to ferret any obvious issues early on.

    But I still see performance tuning as something iterative. It's just too hard to predict where the bottlenecks will be. We've done some large scale performance tests here and only then were we able to discover some wicked little problems that I don't think we would have seen until we did the test on that scale. Then we try to fix the bottleneck, retest, move on to the next one if necessary.

    Paul Carter
  5. Establishing performance metrics[ Go to top ]

    Paul,

    I'm interested in your comment about establishing the right performance metrics... Would this be something that a tool could help you work through as you set up your performance testing suite?

    At IronGrid (I am the CEO), we are focused on building tools that support Continuous Performance. With each point-solution oriented tool, we are focusing on a specific aspect of performance testing and/or optimization. With all of the tools, however, we are working to facilitate the integration of performance into the regular build cycle.

    Acknowledging the risks of optimizing too early in a development cycle, we feel that collecting, storing, and being alerted to performance related issues throughout the development cycle is key to improving overall code quality, and the resulting application performance. Our customers have been telling us that they want to be able to be alerted when problems arise, as close to when the problem is created in the code base... thus, we have added Ant tasks to identify code with performance problems as part of the build process.

    For instance, with our SQL monitoring tool, IronEye SQL, our aim is to provide straightforward monitoring of SQL transactions in a Java application by identifying excessive SQL statements or long running SQL statements being generated by EJBs, JDBC calls, or O/R mapping tool. While individual snapshots can be useful, the integration into the regular build process enables a developer or development team to see trend information, and provide a comparison to both a baseline as well as a set of specified target performance metrics.

    Thus, back to my question at the beginning of this post, are there other ways that tool vendors could support either the set up of performance testing metrics, or the integration of continuous performance into a continuous integration strategy?

    Cliff Sharples, CEO IronGrid
    cliff@irongrid.com
  6. Cliff,

    It's often difficult, if not impossible, to use response time in a development environment as an indicator of performance. In fact, in my experience, I've found that examining response time is only a small part of the analysis necessary to identify, understand, and resolve performance issues.

    Most often, the most insidious performance issues center around seemingly-harmless loops or other repeated access of expensive resources, such as the database, JNDI, or remote EJB's. The only way to catch this behavior is to either a) experience it in a performance test or b) quantify and understand the cost of the component operations within a particular development unit. The latter option can be exceedingly difficult as layers of architecture sometimes contain hidden costs.

    One set of tools that I've begun to explore is JavaCC and JJTree for static analysis. It's conceivable to analyze the code as part of the build process and gather metrics for each class in an attempt to quantify potential performance costs. As each class is analyzed, the metrics for that class are stored; thus, if we encounter another class that invokes it, we can roll up the total costs.

    I'm simplifying the process, obviously, since such analysis could quickly evolve to a simulation engine, to allow for conditionals, loops (and loop boundaries), polymorphism, caching, etc. But a simple tool to quantify basic costs (not in terms of actual service times, but simple frequencies) would help to highlight potential problems. (For instance, WebLogic 6.x and beyond introduced the additional cost of serialization into every JNDI lookup. Armed with this knowledge and an indicator of the frequency of JNDI accesses, a developer would be able to assess potential JNDI performance problems in their particular components.)

    Incidentally, this is essentially an analysis-driven variation on the methodology that Smith/Williams propose in "Performance Solutions".

    The other problem I've frequently observed is inadequate performance conditions in unit testing. Most developers probably understand that as data volumes grow (millions of records versus thousands of records), performance will likely degrade. However, I've seen case after case of designs which fail to address what I usually refer to as "use case complexity". An example would be an ecommerce application that can accept an unlimited number of items in the shopping cart, but is only tested with a maximum of ten or so. In production, what will happen if a customer suddenly orders 100 items? Another example would be a page that presents a list of query results, but fails to anticipate large result sets (1000 or more rows). As obvious as these seem, I've seen repeated instances of these designs in production systems.

    I don't think I've provided a clear answer to your question, but I think that's partly because there isn't a clear answer. However, the old maxim is probably a good indicator: "Either do less-expensive things, or do expensive things less." Either way, you need to operate with an understanding of the costs of your designs and the frequency with which those costs will come to bear.

    Hope that helps,
    cramer
  7. Establishing performance metrics[ Go to top ]

    This article quotes another article mentioning more than 1 million programmers use Java. I thought the more recent figures were upward of 3 to 4 million?

    Anyway, good article which at least highlights the need to consider performance early in the project life cycle.

    A few comments:

    * earlier consideration of performance will at least make sure the tools and environment for performance testing are available early in the project lifecycle. This will make it easier to at least benchmark different design/implementation alternatives and allow the team to familiarize themselves with these tools.

    * if done well, should reduce performance surprises (just like how continuous integration reduces integration surprises)

    * "...addressing performance with every build." is too often and will lead to some developers focusing too much on performance instead of other attributes (like maintainability).

    * much care is required to ensure the environment used for performance testing matches the environment product will be deployed, including effects like physical location of different machines, loading from other applications, realistic data sets, etc. If you don't get these right, you may waste time optimizing in the wrong areas.

    * I would tend to have only a few individuals focus on this kind of testing, who would analyse the results, and then provide feedback to the rest of the team only if there was something significant. This would mitigate time lost to unnecessary optimizations.

    Cheers,

    Don Morgan
    DeveloperAdvantage
    www.developeradvantage.com
  8. Frankly, I'm rather amazed. Your quote:

    "We've done some large scale performance tests here and only then were we able to discover some wicked little problems that I don't think we would have seen until we did the test on that scale. Then we try to fix the bottleneck, retest, move on to the next one if necessary. "

    ...has been echoed here on theserverside in various forms but with the same general concept. The message that I hear repeatedly here is (to paraphrase and aggregate many posters) "Hey, we run performance tests, find the problems, fix 'em and move on. What's the big deal?".

    My amazement comes in because this does not match the enterprise projects I've seen over the past several years. It seems everyone here solves performance problems by tweaking SQL, throwing in a cache, or something equally simple in a couple of days. Whereas I've seen a number of projects have serious performance problems that took some major engineering and pain to fix, and most often in environments where performance testing was left until last.

    Is my experience atypical? Does everyone on the server side just plain rule, and fix perf problems almost effortlessly at the end of their project lifecycle? That seems to be the message here....

    \Carter\
    But I still see performance tuning as something iterative. It's just too hard to predict where the bottlenecks will be. We've done some large scale performance tests here and only then were we able to discover some wicked little problems that I don't think we would have seen until we did the test on that scale. Then we try to fix the bottleneck, retest, move on to the next one if necessary.
    \Carter\

    Just like there are many forms of functionality testing (unit tests, regression tests, etc), I'd say you should have multiple levels of performance tests as well, from the simple ones that can be run by anyone at anytime, and ramping up to large stress tests run only every few weeks. And in my experience, you rarely need massive tests to show bottlenecks - usually pretty simple tests with simple server setups can catch most problems. But they often go uncaught because developers seem to rarely run multiple-user tests on anything.

    On predicting where bottlenecks are - that's the point of having performance tests. Certainly coders can and should keep performance in mind while developing, but you need to run regular tests as well to make sure their mental picture is matching reality. And if you want to have any chance of being realistic, you also need realistic performance requirements up front.

         -Mike
  9. <Mike Spille>
    My amazement comes in because this does not match the enterprise projects I've seen over the past several years. It seems everyone here solves performance problems by tweaking SQL, throwing in a cache, or something equally simple in a couple of days. Whereas I've seen a number of projects have serious performance problems that took some major engineering and pain to fix, and most often in environments where performance testing was left until last.
    </Mike Spille>

    I completely agree with this assessment. It's been my experience that there is very little "low-hanging fruit" in a complex enterprise application. Even things as straight-forward-sounding as tweaking SQL or throwing in a cache often require serious redesign to implement effectively. For instance, the most effective caching strategies require a detailed understanding of the lifecycle of the objects/data to be cached, and often a simple read/write-through cache (if even feasible in the current architecture) won't yield enough of an improvement. (For evidence, ask Cameron Purdy whether the Tangosol folks find it more difficult to plug their product in to an existing application or to rework that application to make effective use of the cache.)

    While I agree in principle with Knuth's observation, I find that it's become something of a truism, and people use it far too often either to simply avoid the cost of working performance into already-tight development schedules (the management perspective) or to simplify a design or implementation choice (much like developers sometimes use YAGNI and "Use the simplest thing that will work" themes from XP to justify a hack-ish or band-aid approach). I won't deny that I've found myself guilty of these sins many times in the past.

    Performance, though, is a problem best addressed with a) as complete an understanding as possible of the system design and your specific component's place within it, and b) a solid methodology for identifying and resolving performance issues as early in the lifecycle as possible. The Continuous Performance approach can address both of these points. More frequent performance testing yields a more complete picture of your system/component and its potential flaws. This information is also derived earlier in the development cycle (before it ships to QA or production), so the cost of resolving any issues is likely to be lower than if these issues were shipped to QA or production.

    There are limits to the effectiveness of every performance strategy. No, there is no guarantee that consistent attention to Continuous Performance will catch all performance problems. For instance, it's difficult to understand the impact of garbage collection on an enterprise application under load without actually load-testing, and even then, there may be specific production use cases that result in unanticipated behavior. The important benefit to the Continuous Performance approach (or any such methodology) is risk-reduction.

    One additional point: Cody included references to some papers by Connie Smith and Lloyd Williams. They have collaborated on a book, as well, entitled "Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software". In it, they present an approach to extending performance engineering even earlier in the development lifecycle, so that performance issues can be identified even before a substantial amount of coding has occurred. It does rely heavily on modelling, especially sequence or activity diagramming, so it's not going to be for everybody. However, it's definitely a worthwhile read.

    Cramer
  10. \Cramer\
    While I agree in principle with Knuth's observation, I find that it's become something of a truism, and people use it far too often either to simply avoid the cost of working performance into already-tight development schedules (the management perspective) or to simplify a design or implementation choice (much like developers sometimes use YAGNI and "Use the simplest thing that will work" themes from XP to justify a hack-ish or band-aid approach). I won't deny that I've found myself guilty of these sins many times in the past.
    \Cramer\

    Well said. I've also seen a lot of people quote famous technologists more as an excuse to get out of something, or to avoid something they don't feel like doing. And it makes it easier to succumb (and we all do sometimes) if someone "famous" seems to back it up.

    \Cramer\
    There are limits to the effectiveness of every performance strategy. No, there is no guarantee that consistent attention to Continuous Performance will catch all performance problems. For instance, it's difficult to understand the impact of garbage collection on an enterprise application under load without actually load-testing, and even then, there may be specific production use cases that result in unanticipated behavior. The important benefit to the Continuous Performance approach (or any such methodology) is risk-reduction.
    \Cramer\

    I used to have the same viewpoint, until very recently. I've been involved in tuning a very large enterprise application using multiple websphere servers, fairly aggressive use of JMS, and a fairly complex architecture overall due to some interesting business requirements. What I've found after a lot of measurement, and alot of tuning, is that you can often take simple measurements under light load, and then extrapolate from there to get a "best case" performance measure. With best case meaning everything scales perfectly.
    Measurements are broken up into wall-clock time, and resource usage. On wall clock, I look at things like measuring disk-forcing time of transaction logs, EJB look up times, times of SQL statements, etc. On the resource side, you look at CPU utilization, disk utilization, network use, etc.

    Using all that with a very light load (5-10 users), you can _very_ quickly determine your best case scenarios and your minimum response times, with a minimum of effort.

    The key to it all is knowing the baseline response times of underlying resources, and gauging resource utilization for a small but varying user load. And this can be very easily automated for regular checks of the code base.

    Unfortunately, very few developers and testers I know ever even bother to baseline critical numbers in their system, so they never know what the wall is. They haven't a clue how many Objects of a given size can be pumped over their network, how much sustained I/O their disks will take, what the average JNDI lookup time is, the minimal cost of a database. They either don't care, or worse, they read an article somewhere quoting times and take it as gospel.

    Anyway, where this comes in handy (beyond the obvious) is as a rough scalability gauge that can let you test performance reasonably on varying hardware. Gauge the baselines on your dev PCs and environment, gauge the baselines on your production environment, and you can run perf tests on your PC with a coarse idea of what that will translate into in production.

    Obviously this isn't a comprehensive solution. There are flaws in this approach, which is why tests in the production setup with significant loads still need to be done regularly (at least every couple of weeks, IMHO). But by taking steps like these early in a project, you can have developers run regular performance tests of their own, and be able to make some sense out of them agianst requirements and the target deployment system.
  11. Mike/Cramer,

    I'm replying to a few posts at once here . . .

    I didn't mean to mislead you - the performance fixes in my limited experience have not always been trivial. In a few cases here, they've been significant, but localized. In other cases, though, there were sweeping API changes that had a larger, system-wide impact.

    My group is starting to do something like Continuous Performance by addressing performance thoughout the software development lifecycle. This is definitely new for us. In the past, the development team wrote the code and then threw it over the wall to the performance group. The perforance group tested it, freaked out when it didn't perform well, and then threw it back. So we're trying to do better.

    Maybe we would violently agree, but I still think that there can always be performance issues that won't show up until you do a larger, full-scale test. Even if you do get developers to do performance testing of their own code, they typically do just that - test their own components in isolation. Many performance issues don't appear until you get all components of the system working together under load. We found one issue with our JMS subsystem that didn't show itself until around 350 concurrent users. Before that, it seemed to scale well. Part of the problem is that developers (myself included) may not always grasp the ways in which their component will be used - or *how much* it will be used. I realize this may be part of a larger problem, but it happens.

    Moving on, and with all due respect, I quoted Knuth/XP because I firmly believe in those principles. I think it's just too hard to predict where *all* of the critical bottlenecks will be. Plus, developers often get excited about this cool little cache thing they could write to improve performance (to what degree, we don't know), rather than working on feature completion.

    But, I think you can also make some very good guesses and I really like your idea (Mike) of "best-casing" performance. I think that makes sense and that's something I'd like to explore further. As I mentioned before, we are taking some of the other steps you mentioned - developers will be asked to performance test their code for more than just one user. We've got a few mini-labs in progress where we can do small/medium-scale tests. We're also handing out a certain commercial package to help developers profile their code.

    Thanks for the fruitful discussion,

    Paul Carter
  12. \Carter\
    My group is starting to do something like Continuous Performance by addressing performance thoughout the software development lifecycle. This is definitely new for us. In the past, the development team wrote the code and then threw it over the wall to the performance group. The perforance group tested it, freaked out when it didn't perform well, and then threw it back. So we're trying to do better.
    \Carter\

    Yeah, this is pretty common in the industry. But it sounds like you're trying to improve on the situation, which probably puts you ahead of the game compared to many :-)

    \Carter\
    Maybe we would violently agree, but I still think that there can always be performance issues that won't show up until you do a larger, full-scale test. Even if you do get developers to do performance testing of their own code, they typically do just that - test their own components in isolation. Many performance issues don't appear until you get all components of the system working together under load. We found one issue with our JMS subsystem that didn't show itself until around 350 concurrent users. Before that, it seemed to scale well. Part of the problem is that developers (myself included) may not always grasp the ways in which their component will be used - or *how much* it will be used. I realize this may be part of a larger problem, but it happens.
    \Carter\

    Absolutely, and I'm sorry if I came across otherwise. Sometimes a performance problem creeps in and catches you completely by surprise. Often, it's a simple bug that was just plain missed, and doesn't become apparent until a really big load is thrown at the code. Or sometimes it's a design flaw that was just plain missed for any number of reasons. As you said - it happens.

    That's why I like a layered approach - testing at various levels. It works for funtionality testing, and works just as well for performance testing. Some of the challenges are of course different, but the payoff can be very large.

    \Carter\
    Moving on, and with all due respect, I quoted Knuth/XP because I firmly believe in those principles. I think it's just too hard to predict where *all* of the critical bottlenecks will be. Plus, developers often get excited about this cool little cache thing they could write to improve performance (to what degree, we don't know), rather than working on feature completion.
    \Carter\

    I've seen so often it makes me want to cry. But this is where baselining costs can make a big difference. If real perf testing is done regularly by everyone (or at least by many people), you build a culture where you _know_ what the costs are in the system. And this is solid, concrete knowledge that can be used to dissuade people from going crazy with useless optimization. If someone wants to put in their pet cache, for example, you have a baseline to show what the basic costs of the framework and hardware already are. And many times you can show that their concepts of relative costs are flawed.

    \Carter\
    But, I think you can also make some very good guesses and I really like your idea (Mike) of "best-casing" performance. I think that makes sense and that's something I'd like to explore further. As I mentioned before, we are taking some of the other steps you mentioned - developers will be asked to performance test their code for more than just one user. We've got a few mini-labs in progress where we can do small/medium-scale tests. We're also handing out a certain commercial package to help developers profile their code.
    \Carter\

    It works well for me. The real benefit is that best-casing eliminates a huge amount of uncertainty, and the less uncertain a developer is about the system they're working on, the more confident they'll be - and they'll create more code of higher quality.

        -Mike
  13. CARTER>
    My group is starting to do something like Continuous Performance by addressing performance thoughout the software development lifecycle. This is definitely new for us. In the past, the development team wrote the code and then threw it over the wall to the performance group. The perforance group tested it, freaked out when it didn't perform well, and then threw it back. So we're trying to do better.
    CARTER>

    Paul,

    How is your group going about addressing performance throughout the software development cycle? From what we've seen with companies we are talking to, I agree with Mike that you're probably doing better than most by trying to address problems of the wall that often seems to be between Development and QA/Test.

    I agree with a lot of the discussion that has gone on about the benefits and dangers of a concept like continuous performance, however the ability to measure early and often, establish a base line, uncover appropriate performance issues early, and have performance trend metrics from the development process later when more substantial/robust load testing is being performed, all seem to be positives which can contribute to overall project successes in terms of meeting deadlines with systems that meet their specified performance levels.

    One question I have is whether getting developers to be at least aware of performance issues throughout the development process is a little like the dentist telling his patient to floss his teeth every day... of course everyone agrees that it is a good idea, but actually getting that person to floss each day is a huge challenge!! How have you guys addressed this both from a cultural / organizational standpoint, as well as from a tools / practices standpoint?

    Thanks,
    Cliff
  14. Early Warning Tools[ Go to top ]

    Cliff,

    In the last 3 years I have been retained regularly by a major application server to help resolve their customers performance problems. Before arriving at site the blame would tend to be placed on the vendor but in 95% of the cases it was the customers development teams fault. Some of the common reasons ranked by importance:

    1. Excessive Database Interaction (75%)
    2. Excessive Memory Consumption (10%)
    3. Partition & Clustering Setup (5%)
    4. VM Choice and Configuration (5%)
    5. Poor Web Code (3%) (excluding persistence & jdbc code)
    6. Poor Bean Code (2%) (excluding persistence & jdbc code)

    Within (1) this can be broken down into transaction execution behavior and SQL performance. The big winner here is transaction behavior. Its not the cost of a simple SQL statement that kills an application, it is the cost of a simple SQL statement executed multiple times within a business transaction and across multiple resource transactions.

    You do not need a real work load environment to detect transactional issues. What developers need is a tool that provides early detection such issues. If you look at all the recent major press releases by performance management vendors, which offer products in the 60,000-100,000 bracket, you will learn the majority of the time its this decribed problem.

    Whats shocking about this is that its so late in the day, the customer has bought more functionality than he really needs, and most likely paid over the odds.

    I am not knocking these products in terms of coverage but it sometimes feels like the Big J2EE appserver vs Little Web Container choice. Some vendors shout about tracing across Component VM's (which JDBInsight supports for CORBA systems) but is this not already the issue (Read SD Magazine Online - Errant Architectures).

    <plug>
    Press Release - http://www.jinspired.com/jdi12deved.html

    Jinspired Provides Free JDBInsight 1.2 Developer Edition License

    DUBLIN, IRELAND - 14 April , 2003 - Jinspired (www.jinspired.com), a leader in J2EE™ transaction analysis, today announced that it is providing a free license for the Developer Edition of its innovative performance tuning product, JDBInsight 1.2.

    J2EE™ developers using JDBInsight 1.2, can now acquire a greater understanding of the transactional behaviour of their J2EE™ components from day one. With the early adoption of JDBInsight in the development life cycle, many of the common performance problems can be prevented. JDBInsight's unique and powerful information visualizations of SQL and JDBC™ transactions, make it an ideal tool for those development teams looking to reduce the risks of incurring future performance and scalability problems with their J2EE™ applications.

    How to get a free developer license?
    Visit http://www.jinspired.com/products/jdbinsight/download.html

    </plug>

    Regards,

    Wiliam Louth
    Product Architect
    Jinspired
    www.jinspired.com
  15. Cliff,

    Thanks for the note. The short answer to your question is: I'm not sure how it will work. We've recently done quite a bit of work to establish a continuous integration environment. We do nightly builds with unit tests and regression tests (you break the nightly, you buy treats!). Our build process is such that any developer can get any version of the software up and running in about 1/2 hour, usually less. It took some convincing and some cultural changes, but once developers saw the power of being able to easily build/rebuild from scratch, it was an easier transition.

    Introducing a continuous performance environment will be trickier. It's really a discipline issue for the developers. Good developers must do certain mundane tasks as part of developing software. This will be another such task. I try to be aware of performance as I write code - but I don't usually write performance tests to verify what I've done - so it'll be new for me as well.

    As I mentioned, we'll try to raise awareness with code reviews. But getting people to run their own performance tests will be tough to do. We have some tools to help, but I'm not sure how we'll get people to actually use them. I've found that you have to make it easy to use, or it won't get done.

    Thanks,

    Paul Carter
  16. Paul,

    Thanks for all the information on what you and your team are doing to integrate performance testing/tuning into the application development project. If possible, I would love to be able to check in with you once in a while to see how the process is going. I am very interested in the cultural / organizational challenges as much as the tools employed to power a continuous performance strategy. While I am a big believer that there is significant value in improving the way performance testng and optimization is approached, I also recognize that there are challenges associated with implementing this strategy.

    Good Luck, and if you're ammenable, send me an email at cliff at irongrid dot com

    Thanks,
    Cliff Sharples
    www.irongrid.com
  17. A clever saying proves nothing. -Voltaire

    I tend to chalk up such 'truisms' to intellectual laziness. Some years ago I started collecting counter-quotes to use as retorts against such tactics. My favorites are the Voltaire quote above, and an unattributed counter to "If it ain't broke, don't fix it.", which is, "Just because it works, doesn't mean it isn't broken."

    I have a long rant on the topic of optimization, that's fairly in line with a number of the positive feedbacks to this article. I'll spare you all but a few key bits here, in the interests of brevity. I'm sure that if someone bothered to ask Knuth, he'd tell us, but I somehow doubt that he really intended that little phrase to be wielded like the club that is has become. Everyone's definition of 'premature optimization' varies. Wildly. For many, it essentially is a polite way of telling someone to stop trying to be as clever as they mistakenly believe they are, and to just write the damned code. For others, it means "Anything that I can laud over this coworker that I don't like," and for others still, it means, "anything that I perceive as making more work for me this week, and next week be damned."

    These days, especially when people talk of SLAs for software, some of the 'performance' issues have morphed into product requirements. You WILL do thus and such in no more than this much time. You WILL scale to this many transactions per second. Is that still 'performance tuning', or is it 'Meeting my requirements'?

    Personally, I don't use the phrase 'premature optimization', because it just doesn't make sense anymore, IMO. And the concept of 'low-hanging fruit' is, to me, just baffling. Fruit growers care nothing of low-hanging fruit. They show up with a machine and shake ALL of the ripe fruit out of the tree, at once, and cart it all off. Typically, code with a performance problem will be off by a couple orders of magnitude. In a situation where you have to make a piece of code 20 times faster (ie, remove 95% of the execution time from the code), ANY code that accounts for more than half a percent of the cumulative time is a target for optimization. You get no points for only making the code twice as fast, or four times faster. This isn't college. You aren't being graded on a curve. However, my decision criteria for fixing a performance problem tends to leave some people either scratching their heads, or thinking that I just fix things at random. And they would be wrong.

    If I have a module that is not performant, and I'm going to be making heavy changes in it, I tend to make ALL of the reasonable changes, not just the most spectacular ones. Why? Because I'm going to have to regression test this code. It's going to hurt, whether I make five changes, or seven. I'm not going to get another opportunity to touch the code for some time. Certainly, I'll never be able to justify swinging back through the code to mop up that last 5% that I missed. Many projects with performance problems throw their hands up and claim that they've done all they can, precisely because they've swept through the code getting all the 'low-hanging fruit', and now they're left with all of the hard-to-reach problems, being nickeled and dimed to death by minor performance issues that they'll never be able to justify fixing, even though their program is sluggish.

    Instead, I talk of unwise optimizations, which are, to me, those that provide questionable gains, those that expose themselves across module boundaries (and hence complicate maintenance), and those that are difficult to test/verify, thereby complicating testing. Gains are questionable when there are simpler alternatives that achieve similar goals, such as reducing query counts instead of implementing caching (see also, verification), or those that will not be noticeable, such as taking an infrequently executed transaction from .5 seconds to .2 seconds, when the human at the end won't be able to tell the difference.

    If you want to use a clever trick with StringBuffers in your toString() method, that's fine, as long as your code is testable, my code is oblivious to your changes, and as long your peculiar code is well- or self- documented. However, if you want me to start passing StringBuffers around, you'd better have a performance problem that can't possibly be solved any other way, after a great deal of time spent considering the issue.
  18. Yes, I disagree with a lot of points made in this article as well...

    <quote>
    By then, however, the root cause of the performance problem often is extraordinarily difficult to pinpoint.
    </quote>

    If your application suffers from a severe performance problem, an optimizing tool will point it out right away, regardless of where you are in the development cycle. Optimizing tools do not care about the amount of lines of code that have been written, they merely analyze runtime behavior, so you don't gain much by applying them early in the cycle.

    There is definitely a huge risk of premature optimization with the technique depicted in this article. For example, the early stages of your implementation might use mock objects to simulate database access, and running an optimizing tool at this point will probably reveal that you are spending an awful lot of time in String and StringBuffer. So you will be tempted to optimize these parts.

    Then you introduce a real database in your application and all your carefully crafted String optimizations become irrelevant because they are now dwarfed by database accesses.

    All that being said, it certainly doesn't hurt to monitor the performance of your software as it is being developed, but you should only push the red button when you notice a severe performance regression at a point where it wasn't supposed to happen.

    --
    Cedric
    http://beust.com/weblog
  19. We do this in our group on a nightly basis. By spotting regressions early, it is much easier to tell what change introduced the problem. This saved us on several occasions in the last release of WebLogic Server. It is important to capture trend information, and comparison to a base line of the prior release.
  20. \Beust\
    If your application suffers from a severe performance problem, an optimizing tool will point it out right away, regardless of where you are in the development cycle. Optimizing tools do not care about the amount of lines of code that have been written, they merely analyze runtime behavior, so you don't gain much by applying them early in the cycle.
    \Beust\

    I completely disagree - in fact I had to re-read your statement a few times to be sure I was reading it correctly.

    Here's a few salient points that were underscored by a recent project I worked on with a very large, very well established code base:

      - Finding a performance problem early often means few lines of code need to be affected if you need to put in a performance fix. Sadly, some performance optimizations mean changing an aspect of an API that's widely used. If test for performance late, and discover an API change with other accompanying changes would be best, then that change is going to be very, very painful.

      - One or more developers may have poor "performance sense". Often it doesn't matter, but poorly written code can pile up over time. Regular performance tests starting early in the cycle can catch this so developers are educated on the performance consequences of their design and implementation choices.

      - Performance problems tend to pile up, and mask each other, over time. In the project I alluded to in the beginning of this post, performance monitoring on this established code base found 5 seperate performance problems. They were somewhat interrlated to some extent, but some also masked others or created a mountain of data that was difficult to sort out. The end result was several days spent pinpointing one problem, fixing it - then re-running the tests and finding that the fix did indeed work, but now a new problem had been unmasked.

    \Beust\
    There is definitely a huge risk of premature optimization with the technique depicted in this article. For example, the early stages of your implementation might use mock objects to simulate database access, and running an optimizing tool at this point will probably reveal that you are spending an awful lot of time in String and StringBuffer. So you will be tempted to optimize these parts.
    \Beust\
     
    Agreed, there is a potential for abuse here. But only if your project doesn't have established performance guidelines. A well run, large project will state right up front what the perf requirements are - how many users, average/max response times per request, that sort of thing. When you have well defined requirements like this, pre-mature optimization isn't going to happen. Using your mock-object example early in a project's lifecycle, this sort of thing would likely meet and greatly exceed the perf requirements, and no tuning would be indicated.

    However - imagine if your simple mock object framework _didn't_ meet the requirements. This is an early indicator that your architecture and/or implementation is flawed right from the get-go, and a sign that the developers should re-think things. I'd rather know that, with concrete numbers, early on.

    Many developers don't seem to think performance is a big deal anymore (look how fast this machine is!). Alot of developers, including many people who've posted on this site, seem to discount load testing code under multiple user loads altogether ("that's QA's job after we finish"). And alot of developers are just plain ignorant on the subject, through no fault of their own.

    Personally, I think every developer should keep performance in the back of their mind. It doesn't need to be an obsession, just something to take into consideration when they're designing and coding. And constant performance tests is a perfect tool and provides a nice balance - it's a safety net, and an indicator that the project isn't (or is!) heading for trouble.

    \Beust\
    All that being said, it certainly doesn't hurt to monitor the performance of your software as it is being developed, but you should only push the red button when you notice a severe performance regression at a point where it wasn't supposed to happen.
    \Beust\

    Well, as I said, if your code can't meet the performance requirements in early iterations (assuming you have such requirements), it's in everyone's best interests to catch this early before more time and money is wasted on an architecture that can't even support the target audience when it's still in skeletal form. You may scoff at this, but I've seen it happen time and again, even when it was obvious to some people early on.

    And the best thing about integrated performance tests is that they're generally objective.

        -Mike
  21. Cédric,

    I agree with you. But I think the good point of this article is to pinpoint that performance should be tested early, as design is the main lever on performance and design is usually an early stage, even in continuous integration. Testing at the tuning phase is usually too late as you usually won't gain more than a x2 whereas design can secure you a x10. Therefore, as long as a developer stays in the 20% range of Pareto's principle, it's fine. SQL is a good case as a developer often have several ways to implement a statement. A performance tool fits very nicely to choose which syntax will bring the best performance and that can be set up very quickly. The problem with continuous performance is that performance needs to be tested against precise metrics and satisfactory numbers (normally agreed upon by the client), but those metrics usually only apply to a functionality that is fully implemented, and very seldom to a function. A developer applying such principles should always refer to the SMART approach (Specific - Measurable - Achievable - Realistic and Traceable) otherwise such testing is dangerous.

    Also, using Jack Shirazi's book as a reference for continuous performance is a bit dangerous because many times, he joily walks over the border of code readability that comes at the cost of maintainability. And he acknowledges it. Most of the times, these improvements won't get you much performance anyway.

    I too think an optimiser should do the trick most of the times... but I would concede that it can be used as soon as a functionality is fully implemented. That's usually early enough.

                    Yann
  22. understatement[ Go to top ]

    Yann: Testing at the tuning phase is usually too late as you usually won't gain more than a x2 whereas design can secure you a x10.

    Yann, you are too humble ;-)

    Design decisions can make such an enormous difference that saying 10x performance difference is an understatement. In many cases, it makes the difference between an app being able to even run or not. A couple of years ago, I worked on a system in which one little design decision in one central class caused it to run like a dog on two e10k servers ... supporting only about 200 users total ... that's not a "10x" problem, that's a 1000x or 10000x problem!

    The importance of a good design cannot be overstated. With a good design, it becomes more difficult for an individual engineer to introduce system-wide performance problems; in other words, a good design can compartmentalize later bad implementations, allowing them to be fixed incrementally at a later point in the cycle.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  23. understatement Tangosol??[ Go to top ]

    Cameron,

    Off topic, but I'm evaluating Coherence for a project right now and am having a couple of small problems with something. Can I ask you a question about a couple of things?

    my email is mckerrj at nacse dot org

    Jason McKerr
    Northwest Alliance for Computational Science and Engineering
  24. understatement Tangosol??[ Go to top ]

    Off topic, but I'm evaluating Coherence for a project right now and am having a couple of small problems with something. Can I ask you a question about a couple of things?

    Yes, of course. See email. Is this the "sim" project that I heard about?

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  25. Coherence - help[ Go to top ]

    Apologies to everyone for the bad post

    Cameron,

    I've registered to download Coherence for evaluation but I'm not able to log in. I'd hoped to evl this weekend, if you're still around can you give me a hand.

    Thanks
    Aaron

    AaronRobinson67@hotmail.com
  26. Coherence - help[ Go to top ]

    No problem - see email.

    It's the week-end; are you giving new meaning to "Continuous Performance Improvement"? ;-)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  27. I would say that a good design doesn't make the system faster, but it does facilitate refactoring to make it faster, once bottlenecks are identified. The example you gave, where one slow class was crippling performance of the whole system, might actually be an example of a *good* design (but a poor implementation of that class). If the code is all in one place, it's easy to fix.

    Here's a good article by Martin Fowler on performance, with a quote that reiterates this point:

    (from http://martinfowler.com/ieeeSoftware/yetOptimization.pdf):
    "All of this reinforces the key rule that first you need to make you program
    clear, well factored, and nicely modular. Only when you’ve done that should you optimize."

    Since Fowler coined the prhase 'Continous Integration', I thought it would be interesting to see what he says about performance. Fowler seems to have a slightly different slant than the server side article. Here's the link to Fowler's famous 'Continous Integration' article:
    http://martinfowler.com/articles/continuousIntegration.html

    I'm also concerned that the phrase 'continous performance' might encourage developers to think about performance continously as they develop, which I think is exactly wrong. That can lead to developers obsessing about temp object creation, optimizing loops, etc., when the real bottleneck might be with the database, network, io, etc. Here's another quote from Fowler that addresses this issue:

    "For many programmers, performance is something you pay continuous attention to
    as you program. Every time you write a fragment of code, you consider the performance implications and code the program to maximize performance. This is an obvious technique -- pity it works so badly.

    Performance is not something you can work on in this way. It involves specific discipline. Some performance work comes from architectural decisions, some from a more tactical optimization activity. But what both share is the fact that it is difficult to make decisions about performance from just looking at the design. Rather, you have to actually run the code and measure performance."

    Another caution: While the idea of running automated performance tests is great, you would need to make sure that they simulate actual load. Just timing a bunch of unit tests might not be an accurate test. Here's another quote:

    "Never optimize a multi-user system with single user tests."

    Finally, the article kind of implies that performance problems add up linearly. I.e. that Jeff writes some code that takes 1 second, and Sally writes some code that takes 1.5 seconds, those will add up to a 2.5 second delay. The more code that's written, the poorer the performance. In practice, it doesn't always work out that way. Often 20% of the code takes 80% of the time. Rather than continously optimizing 100% of the code, it makes more sense to just optimize the 20% of the code that's the bottleneck. Realistically, things might work like this: Jeff writes something that takes 30 milliseconds. Sally writes something that takes 12 milliseconds. And then Fred writes some data access code that takes 14 seconds. It makes sense to spend *all* your time optimizing Fred's code, instead of splitting your time 3 ways.

    The key then is to use a profiler, to identify the real bottlenecks. The article doesn't mention this. However, having an automated set of realistic performance tests to run through the profiler is great.

    Automated performance testing is great; I'm not so sure whether *continious* performance tuning is so great. It could be abused and conflict with the 'make it work, make it clean, then, if there's a peformance problem, make it fast' idea.
  28. Stephen,

    I agree that a solid design, with clear, well-factored code are (or should be) the top priorities for all developers, whether the project is a one-off utility for your own personal use or an enterprise system for a Fortune 500 company. But I've stopped espousing the "Make it work, then make it fast" motto in my own projects, simply because I see too many people take it too far to the extreme of actually ignoring the performance impacts of their design and implementation choices.

    Certainly, we don't want developers to while away their hours squeezing out every extraneous bit of bytecode. But they should develop enough of an understanding of the overall design and their component's place in that structure to know or guess-timate the costs of certain choices. If performance engineering is delayed to post-integration or later, and enough poor design choices are locked down in the interim, the cost of identifying and resolving performance issues rises dramatically.

    I do think it's difficult to gather accurate response-time metrics as part of a build-triggered performance test, but there are certain unit-level metrics that can indicate the presence of potential problems, without leading to excessive early optimization. Obviously, there are expensive resources in any system, such as the database, remote objects, JNDI, etc. It's useful to try to quantify the frequency with which a particular unit will access these resources, given varying inputs.

    If, for example, a method takes a List as an argument, and interacts with a database based on the contents of that List. It's rather obvious that a design that iterates over the List and makes an individual query for each contained object will scale poorly, but a well-designed unit test would be able to identify this as well, even early in the development lifecycle--especially with the use of mock objects.

    Call it Continuous Performance or Software Performance Engineering or whatever. The real name of the game is risk-reduction because performance problems often resist optimization and tuning, and only give way before a full-scale re-architecting assault.

    cramer
  29. Guestimating performance[ Go to top ]

    Cramer,

    Thanks for the thoughtful reply.

    In general, I think that expecting developers to know or guestimate the performance cost of an implementation doesn't work so well. Developers (myself included) are often bad at estimating real performance costs; you need to measure. Also, performance hacks are often just an excuse for bad code. (Performance was often the excuse C programmers gave for using pointers, long jumps, etc., in a C++ program.)

    But there are exceptions, as you mentioned. Mostly they have to do with remote access or io -- hitting the db, writing to disk, minimizing network round trips, etc. But for a business app, I wouldn't start optimizing in memory code loop code, for example, until I had proven that it was a bottle neck. On the other hand, I might go ahead and eliminate unnecessary db queries, without measuring.

    Steve
  30. Guestimating performance[ Go to top ]

    \Molitor\
    In general, I think that expecting developers to know or guestimate the performance cost of an implementation doesn't work so well. Developers (myself included) are often bad at estimating real performance costs; you need to measure. Also, performance hacks are often just an excuse for bad code. (Performance was often the excuse C programmers gave for using pointers, long jumps, etc., in a C++ program.)
    \Molitor\

    I hear where you're coming from. But at the same time, many developers tread the coding waters in complete ignorance of relative costs _in their environment_. Many people fling numbers out about GC costs, JNDI lookup costs, database access costs, and similar things - but much of this is of the nature of "I read it in an article", "according to this whitepaper", "this really smart guy told me". Very few individual developers seem to bother timing these things for themselves, and it's ashame because there's nothing stopping them, and the answers can be quite illuminating.

    Ultimately, I believe developers are responsible for the performance of their code. If you believe this (and I understand that not everyone does), then it's not acceptable to say that developers are bad at estimating performance costs. Once you know the baseline costs of your frameworks and languages, a decent developer should have a pretty good idea of how a chunk of their code is going to perform.

    When developers don't act this way, weird performance mythologies spring up and become accepted as gospel. People post here that Java GC is now so fast that they can safely ignore generating lots of short-lived temporary objects. Individual propogate ideas about network speeds that were accurate 5 years ago, but need to be re-evaluated when 100MBit LAN connections are common. People automatically assume their database is slower than a dead poodle, and create extravagant caching schemes without measuring a single SQL call.

    Ultimately, I agree, you have to measure. But this should not be an absolution for developers.

    \Molitor\
    But there are exceptions, as you mentioned. Mostly they have to do with remote access or io -- hitting the db, writing to disk, minimizing network round trips, etc. But for a business app, I wouldn't start optimizing in memory code loop code, for example, until I had proven that it was a bottle neck. On the other hand, I might go ahead and eliminate unnecessary db queries, without measuring.
    \Molitor\

    One thing to keep in mind, is that performance measuring only tests the current code base as it's used now, and only with the datasets the test is given. If you go with a pure approach of profiling only, and not considering performance while you code, you can measure great today but end up with huge problems down the road as the codebase expands and matures.

    As a trivial example - a developer may do a linear scan on a List today, thinking "well, it only holds a few items", and so long as it's a few items your performance tests will look fine. But overtime, this List may accumulate more and more stuff, until it becomes a performance problem. Meanwhile - the developer thinks "Lists aren't so bad", and may have gone on and used the same approach in many other code areas. And some of those areas will be performance time bombs...

    The above is obviously a somewhat contrived example, and can be easily fixed in localized locations, but the underlying idea I think holds. The advantage of considering performance while you code is that you can use your brain and anticipate likely growth in the future, and take small steps to try not to handicap later generations of the software.

    We really can't think of everything, and we can't perfectly anticipate the future. But that doesn't mean that we should turn the responsibility completely over to tools. And the irony is that performance-minded (minded - not obsessed!) developers don't take any longer to produce code than their breathren. In fact, a few moments of quick consideration about performance might stretch your development of a class by only a few minutes. But that consideration might save alot of heartache down the road.

         -Mike
  31. I would disaggree with the idea that all a good design does is facilitate refactoring. The difference between good and bad design also includes whether you solve the problem in an efficient way, or something which looks good on paper, but wastes abhominable amounts of time. Design includes "how do we solve this in the first place?", not only "how do we split this solution into classes?"

    As a random example: fine grained vs coarse grained access to remote EJB's _is_ a design issue: fine grained access with 100 getters and setters to a stateful EJB looks great on the drawing, but it's going to be a massive performance problem compared to packing all that in a data object and doing it all in one go. That's a fairly easy design issue to solve, but some variations of ignoring such issues aren't. It's too easy to let it go out of hand and end up with a situation where you must essentially redo the whole design to fix the problems.

    And IMHO the whole "premature optimization is the root of all evil" idea is as false as any blanket generalization, and yes, it has become more of a generic excuse than anything else. Yes, there are evil optimizations. Cramming 100 operators on one line because you know your C compiler generates 2 less bytes of code in that situation, that's evil. Don't ever do it, prematurely or otherwise. Using a better algorithm or a better class design is not evil, and won't make the program any harder to read or maintain. (E.g., in the above example, packing the data in a single call to a stateless session bean isn't any harder to read or maintain than 50 individual setter calls to a remote stateful bean.)

    And the excuse that computers nowadays are fast enough anyway, is just as false as a blanket generalization too. Yes, they're fast enough so you don't have to optimize your own StringBuffers manually any more. No, they're still not fast enough so you can use an exponential time algorithm instead of a linear one. Why? Because the quantity of data to process has also increased over time. The data sets we have to process nowadays are not the same data sets that were used 20 years ago. And because we've already found other ways to "waste" the extra CPU cycles. E.g., whereas 20 years ago a program would sit on the same machine as the database and read the file directly, now we have a network and an SQL parser between them. A program which generates an avalanche of SQL statements will choke on database access nowadays, just as quickly as 20 years ago it would have choked on disk access.

    It's also worth noting that not everyone is equal. Even an extremely experienced programmer doesn't have equal experience with every single framework or problem type. There are too many frameworks for _anyone_ to know all of them. (And I'm not even getting into the hordes of computer-illiterate people hired off the street on account of "hey, we'll just give him a couple of weeks to learn Java.") Some people don't need to worry about performance prematurely, because they already know the pitfalls of that particular framework or problem, and can do the right thing in the first place. Some people, on the other hand, really have no clue what they're doing there. The above example of using a stateful bean where a stateless bean would have been enough, is what happens in those situations. And I fail to see how getting some clue early about such design issues is an evil thing.
  32. RE Cedric's comments[ Go to top ]

    <cedric>
    There is definitely a huge risk of premature optimization with the technique depicted in this article. For example, the early stages of your implementation might use mock objects to simulate database access, and running an optimizing tool at this point will probably reveal that you are spending an awful lot of time in String and StringBuffer. So you will be tempted to optimize these parts.
    </cedric>

    Imho in most cases you don't need to integrate performance tests into your test/build cycle. But in some cases where you suspect it can be performance sensitive it's a good idea to test the performance as an integrated part of your test suite. By test I don't mean Unit Tests, I mean Acceptence Tests. In AT you run the whole story in a real context, no mock objects but real stuff. Testing performance in ATs can be more reliable than UTs, less chances of premature optimization because it tests a whole story in a real context, rather than a microscopic one.

    Ara.
  33. I have read this article on Performance and its importance. Its really true that the developers overlook the aspect of performance in the day to day programming.
    Performance should be addressed from the begining and during design itself, for ex. avoiding Vectors (when there is no simultaneous requests) and using ArrayList and avoiding creating String objects and other similar technical aspects.
  34. Some simple steps[ Go to top ]

    Having worked both as a build engineer and performance tester I cen certainly relate to this issue. My opinion on the issue is:
    a. Continuous performance testing is ime consuming but not beyond reach. Most of the people do have functional automation in place. That same tool (Winrunner for example) could also bring out 10 user performance report. We can additionally also throw on JProbe automated data collection to substantiate the report. What we must be clear about is that this form of testing is only checking the code impact through comparisons. It is not likely to tell about architecture worthiness.

    b. There is another easy way to measure the performance. If there is a common testing server put Topaz (a Mercury production environment measuring tool). This way you will not have to maintain the automated scripts that break every day and you don't know if it is due to the script, the data or the application.

    c. A good preventive measure for performance problem is developing coding standards and checklist. More importantly every developer should rate all his java file for the performance impact. Potential for serialization should be mentioned.


    With continuous performance testing we are more likely to be testing with light load. There is a pitfall in this. You may have good response time but I do want the app to be light on resources as well otherwise scalability will be affected. The idea at the development stage should be building code that is efficient on resources. The time should not be spent in configuring the environment such as no. of threads, no of connections etc which relate more to making the maximum use of the resources available. Performance apart at this stage I would also like to weed out the potential for functional errors seen only at higher load.
    regards,
    Vikram
  35. Early Performance Tuning[ Go to top ]

    I do share the view that performance testing (not neccessarily tuning) should be done early but I have a different view of what the type of work that should be done.

    The work I consider important (after the initial design has been constructed based on business inputs) in relation to performance testing is in determining the impact of various architectural design issues. A bad design can only be improved so much by tuning. When I built large systems I always kept a list of deferred design issues that was worked on by performance engineers. The engineers would go off and ** attempt ** to provide further insight into the possible implications of various approaches by creating some prototypes or do plain research. For example in the J2EE world one must look at decisions in relation to choice of component architecture (ejbs, jsp->jdbc, corba), component design (stateless, stateful), persistence mechanism (homegrown, cmp, bmp, jdo), messaging style, backend integration, database design. The hard part is making a decision that factors in so much when you know so little.

    Is tuning SQL so important at the early stage? I tend to think not. Its hard to generalize because so many developers are using J2EE in different contexts but looking at the cost of SQL performance on day one seems a bit wasteful if the goal is tuning. Persistence code or configuration will change during the early stages and in most cases you are still trying to get a proper database system up and running. What is important is the validity of the SQL with regard to the business transaction? Are the fields mapped correctly to columns? Is the update correct in terms of what happened at the component level? Does the set of SQL statements executed in a single transaction of multiple transaction (allowing for subtransactions) valid?

    Individual SQL statement tuning is low hanging fruit - assuming you have got a good JDBC profiler which has a small overhead. What is difficult is understanding the SQL in terms of the business transaction? This is the hard part and where large performance savings can be made, unfortunately this usually means changes to the component code, component interaction and component configuration.

    Is the use of Ant justification for the label of continous performance testing? No. Ant is a great tool but it is abused by many because of its popularity. In this case I cannot see the difference between setting up a event rule within a monitoring product and having Ant tell you this SQL is above average. The people running the Ant build task are probably not building and deploying onto a large scale system which means these little asserts are fluff - poor performing SQL shows itself under heavy underload.


    William Louth
    JDBInsight Product Architect
    www.jinspired.com
  36. I Agree[ Go to top ]

    I agree. But there are many catches:

    1)Very rarely the code is Integrated from the start, escpecially in Large projects.
    2)The performance may not dependent on one module. It may be dependent on other non-development external factors like tuning the servers, network congestion, development hardware etc...
    3)Appropriate tools may be required which may not be available at the early stages.
    4)Not all developers in Team may be that co-operative. As seen in most of the projects code cleaning and ap performance tuning is done at the last as nearly all are running after deadlines.
    5)Many leads are interetested in showing the results first and look into tuning at later stages.
    6)Performance testing scope document need to be defined very well, pointin out various stages to test performance.
    ...... and more.

    These are some of the very common issues seen in projects. We are talking of a very Ideal situation here in Cody's theory and I am not saying some companies may not provide ideal environment (But I have not seen any). But there are lot of constraints of which some are mentioned above can be stopping blocks.

    Cheers.

    H Singh
  37. Hi,
    I apologize, my post is unrelated to the theme of discussion but it's an excellent opportunity for me. Most of the people here I believe have rich experience of performance tuning. I am a budding performance test engineer. I really have desire to do well in this field. I would like to ask to all those writing or reading the forum, what would they expect from a performance tester if they had to recruit one for their organization. What kind of information you will look for in the resume and what questions would you ask in the interview?
    Currently I use Loadrunner tool. Performance testing is only limited part of my current assignment. I keep finishing my work fast enough to find time to browse performance discussions, browse performance docs and then study a bit about performance monitoring tools, general j2ee architecture e.t.c.

    I appreciate your altruistic efforts.
    regards,
    Vikram
    chandna@yahoo.com
  38. Vikram,

    1. You need to be able to understand architecture. Usually that means "reverse engineering" a mental architectural diagram from the source code of the application, so it also implies very good Java (or whatever platform/language) skills.

    2. You need to understand the standards / specifications, and the pit-falls of mis-application of specific technologies. This usually means that you personally screwed up and/or fixed these types of problems before.

    3. You need to understand the "platform", which in this case is a J2EE app server, the Java platform, possibly the underlying hardware and OS.

    4. You need to be able to apply meaningful load to an application. Meaningful implies both sufficient load, and load that will tend to highlight problems by stressing various parts, or doing a great job of emulating real users.

    5. You need to be able to profile, both at a low level (Java profilers) and at a high level (like William's tool does ... see also Precise, Wily, et al.)

    6. You typically need to understand the back end architecture. Hopefully that just means databases, but quite often it includes all sorts of antiquated and dillapidated systems.

    7. Probably most importantly, you need to be a good listener, a good communicator, and have desire and capability to learn while underwater. First, listen, because most problems are already known, even if the people that know them don't know that they know them -- 90% of the time you can get a good feel for the problem just by listening to the engineers, the QA people, the ops personnel, etc. Once you understand the problem, you need to be able to explain it clearly and concisely to the interested parties. Finally, no two problems are identical, and you have to keep an open mind and constantly learn from each experience, and it is hard because you get inundated with knowledge from the interested parties and from the systems that you are looking at (aka you are underwater from the get-go.)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  39. Thx Cameron[ Go to top ]

    For the detailed information. I will try to work on these lines.
    regards,
    Vikram
  40. "...testing performance several times a day..."

    dumb.
  41. Hi,

    As far as I know
    >Continuous Integration initially sprang up in "Extreme Programming"
    >environments, a byproduct of hurried Web application development whose
    >acceptance ascended in lockstep with Java.
    is not correct,
    as Steve McConnell describes "daily builds" already for Windows NT 3.0,
    see book "Rapid Development".

    * Stefan