Solving production performance problems is always a difficult challenge, especially in an age where a given product can incorporate dozens of different Apache projects, a variety of frameworks and -- given the distributed nature of modern architectures -- software components that span multiple geographies. So, it was with great enthusiasm that I broached the software performance problem topic with Java expert Jevgeni Kabanov.
JK, as he's affectionately known, is not only the CEO of ZeroTurnaround -- the company behind application performance monitoring (APM) tool XRebel -- but he also holds a Ph.D. in computer science, which he earned by defending a thesis about boosting Java productivity. With these types of credentials, one would anticipate some exceptionally insightful wisdom on the topic of how the average, everyday programmer could improve application production performance and avoid running into runtime bottlenecks.
Sadly, when asked about improving application performance, his advice had less to do with witchcraft and wizardry, and instead was more of a message back to readers to get back to the basics and stop repeating many of the common mistakes of the past.
Below are Kabanov's thoughts on three common software performance problems:
Overload of database queries
The most common performance issue stems from the volume of database queries, said Kabanov.
"Just having too many queries to the database, this is the problem 80% of Web applications are having today," he said.
It's actually not a surprising transgression. The fact is, modern object-relational mapping frameworks -- such as Hibernate and the Java Persistence API -- obfuscate what goes on behind the scenes when Java programs interact with back-end data stores. As a result, programmers are unaware of just how often their persistence framework is making a JDBC call.
The second most common performance problem JK encounters is developers putting far too much data into the HttpSession. "A ton of applications must be restarted on a regular basis because they consume too much memory," Kabanov said. "And the most common place where they consume too much memory is in the HttpSession."
So what can developers do to avoid this production performance problem? One of the reasons the HttpSession gets so bloated over time is that an application needs a piece of data at a few specific times in its lifecycle, and developers don't want a big, round trip to the relational database on the back end every time that piece of data is needed. Instead, they just read that piece of data as soon as the user logs in and push it into memory.
A simple way to avoid this type of problem is to use a NoSQL database on the Web tier, and instead of transient data loading into the HttpSession, it gets stored locally in a manner that is both easy to read and quick to access. The other benefit is that your applications will be able to scale easier because distributed memory management becomes less of an issue.
The third performance trap JK sees organizations fall into is one in which they architect SOA-based systems that use a variety of SOAP- and REST-based calls.
"Many people are transitioning to Web services, but don't have a good overview of how will it perform in production," Kabanov said. "It is so easy to do something stupid accidentally."
There's nothing inherently wrong with a SOA-based architecture, but the lag time encountered when Web services are invoked over a local network can really throw a wrench in the machine. And the latency problem doesn't typically arise until systems go into production.
A final piece of advice
Finally, Kabanov makes the point that when poking around the inside of an organization's data center, far too often his team encounters a bevy of exceptions that are never dealt with. "There are often exceptions thrown in the log that nobody ever sees," Kabanov said. "This is not always a performance problem per se, but it is indicative of some type of production issue."
There's always the hope, when you talk to someone like Kabanov, that he will bequeath some pearl of wisdom that changed the way you develop applications forever. But the reality is people are making the exact same mistakes today that they were making 10 years ago, and as such, today's advice isn't that much different from yesteryear.
So, if you want to avoid production performance problems, go back to the basics: Minimize your database calls, don't bloat your session, avoid network latency issues -- especially when it comes to Web services development -- and if you see an exception in the log files, deal with it before a bigger problem arises.
What are your most common performance challenges? Let us know.
You can follow Jevgeni Kabanov on Twitter: @ekabanov
You can follow Cameron McKenzie as well: @potemcam
Understanding performance problems
Load testing vs. software performance testing tools
App performance management tools