Continuous integration sounds very nice in theory. Instead of relying on a tedious waterfall process that can break down at any point leading to catastrophic project failure, organizations can develop software in a much more modular and resilient way. Once any piece of the program passes its testing requirements, it's ready for deployment. There's no grand unveiling of a completed project after months or years of anticipation. Rather, software is produced in a more organic way with a focus on flow and functionality.
However, the transition to continuous integration (CI) isn't something that happens naturally. Just as the code needs to be checked for bugs, the CI process itself should be monitored for effectiveness. John Smart, author of Jenkins: The Definitive Guide, highlights four areas to evaluate:
#1 Are there still lots of manual tasks?
True continuous integration isn't possible without a serious effort to automate as many processes as possible. Check for any areas where processes are still being done manually. If system administrators are being asked to run scripts during any portion of development, testing, or deployment, this is an indication that there is still room for improvement. 100% automation may not be possible for certain processes that are viewed as high risk, but getting close to complete automation should be a goal.
#2 What's the quality of the output?
Reports shouldn't simply be churned out to give the appearance that things are getting done. Rigorously examine reporting and metrics to spot areas where the process tends to get bogged down or where acceptance criteria are not being met. Also, compare reporting for CI projects to previous projects to identify areas that have improved. These may be the easiest areas to improve even more to capture additional value from continuous integration. At a more basic level, keep an eye on the information radiator. Are all systems go more often than not?
#3 Has acceptance criteria been realized?
When acceptance criteria are created at the outset, these criteria form the basis of design and coding for the entire project. But it's easy to stray from the mark if the criteria documentation isn't incorporated into the system. Acceptance criteria should be automated as tests and become living documentation. That way, it's updated throughout the CI process and is always accurate. Reports and living documentation serve as two sides of the same coin, demonstrating to stakeholders that code is meeting requirements. Keep in mind that unit tests shouldn't look like technical documentation that is difficult to understand.
#4 Can deployment take place immediately?
The whole purpose of continuous integration is to keep the process rolling. The time it takes to deploy successfully is a key indicator of the progress of an organization's transition to CI methodology. A fully automated deployment can often take place in minutes. Even a partially automated process should take no more than a few hours. If it takes days to deploy new functionality, there's a serious problem somewhere in the pipeline. The same holds true if rollbacks are common. There should, of course, be systems in place to revert to a previous working version if necessary. However, continuous integration should be done with confidence that the testing is more than adequate to assure a smooth deployment.
What continuous integration strategies do you use? Let us know.