Based on our recent discussions – 80% of the problems that cause War Room scenarios are caused by only about 20% of problem patterns. Most of them are related to performance or architectural issues in the application or in the infrastructure that supports your application. A recent study concluded that 80% of development time is wasted with triaging and fixing problems which sums up to an estimated $60B of annual costs in the US alone. Now – whether you believe these numbers or not – if you have ever spent your evenings or weekends in a war room trying to fix yet another production problem or getting the final tasks for the release out of the door you need to change the way you develop and deploy software.
In the last weeks we had many great conversations with the DevOps community at events such as JavaOne (download slides), Star West (download slides), Web & PHP, and our Compuware APM PERFORM User Conference. We have even spoken with folks like Kevin Behr – one of the authors of The Phoenix Project. They all confirmed that these war rooms consume most of their engineering time despite their efforts of a leaner and continuous delivery model. The consensus was that DevOps needs to include a focus on performance and user experience to ensure happy and productive end users of the delivered software. But what does this mean exactly and what needs to be changed?
Cultural Shift: “Connect” your Teams
Looking at the top reasons why applications are slow, crash, or why development teams spend so much time on fixing problems rather than pushing out more changes more frequently, it is obviously that there is a disconnect between teams. Performance requirements are not well understood by all teams involved and there is no automated way to measure the success or failure of work that has been done. To fix this we have a couple of ideas that already work well for our customers. It will help you to align the goals off all teams involved in order to not end up with epic failures like the following pictures