This story came in from Joseph – a performance engineer at a large fleet management service company. Their fleet management software runs on .NET, is developed in-house, is load tested with JMeter and monitored in Production. A usage and configuration change of their dependency injection library turned out to dramatically impact CPU and memory usage while not yet impacting end user experience. Lessons learned: resource usage monitoring is as important as response time and throughput!
On Wednesday, July 3rd Joseph’s ops team deployed the latest version into their production environment. Load (=throughput) and response time are two key application health measures the application owner team has on their production dashboards. Especially after a new deployment an unchanged behavior is what you are looking/ hoping for: Image of Production Load Dashboard
Joseph is an experienced performance engineer, so he knew to check more than just these two metrics. Looking at host and process monitoring data he identified a 6x increase in CPU and 40% increase in process memory usage which alarmed the folks responsible for the infrastructure that hosted the ASP.NET Services: Image of Memory Dashboard
Despite the fact that the app still shows the same response time and throughput this is very alarming. Why? Because it means that the application is much more expensive to run as compared to the previous build.
Whether it runs on physical, virtualized or in a cloud environment: more resource consumption means higher operational costs. And – who knows what happens during peak load: will CPU and memory be enough or will it eventually impact response time and throughput? That’s the reason why analyzing changes in resource consumption is as critical to performance engineering then optimizing response time and throughput.
If you are a performance engineer, application owner or developer I suggest you read the rest of the blog to learn from Joseph’s best practices on analyzing and dealing with situations like this and what they changed in their development and deployment process