Article: Best Practices for Risk-free Deployment

Discussions

News: Article: Best Practices for Risk-free Deployment

  1. John Birtley has written an article that addresses enterprise development as a complete lifecycle, pointing out best practices for configuration management, build processes, and deployment. His article brings up a series of ideas geared to yield "risk-free deployment," including software solutions and processes aimed at issues that J2EE developers face in the field.

    Read Best Practices for Risk-free Deployment.

    Threaded Messages (31)

  2. BuildMonkey or CruiseControl[ Go to top ]

    I have to admit though that I thought I was about to read an article on ControlControl (or BuildMonkey as it was previously known). The article did contain some interesting thoughts and well written but seemed to avoid the 'hows' and focused on the 'what'.

    Good to see that the process of deployments is getting some broader focus rather than just lumping it into the 'iterative development' basket.
  3. BuildMonkey or CruiseControl[ Go to top ]

    I definitely have to agree with Quinton that this article merely focuses on the 'what' and not on the 'how'. The author is mentioning the 80/20 somewhere in the article and that's exactly what it's like. Almost all teams have structured build scripts and while having your software built by Cruisecontrol using those build scripts is only the first step, it's probably the one that will make the largest impact.

    In my opinion it's about getting the software developers aware of the fact that a build/release management process is important. We frequently have developers do everything: from a simple support call to releasing the actual software that fixes a couple of bugs. Sometimes this goes wrong and then the developer sees the real value of a process the author of the article is aiming at.

    One advice to all people not having a formal release process. Start off with CruiseControl, have it build your software every 15 minutes (and don't let anybody go home if the build is broken) have your artefacts deployed to a internally available webserver (as long as you have archives of your latest builds available, deploying to staging/production environments is sooo easy), add CVS tagging to it (it's a matter of adding a couple of targets to your build file where the tagging process uses the label passed to you by CC), know what version has been put into production (use the same label, incorporate this into a manifest somewhere and include it in the comments of the HTML pages generated for example) and then slowly start to expand the build/release process with things like reports (generate them with Maven if you like), automatic server testing, etcetera. Especially with the latter you're going to run into configuration management issues, but if you get this far :)...

    Oh and buy yourself a build machine. It's going to save you hours and hours!

    cheers,
    Alef Arendsen
  4. BuildMonkey or CruiseControl[ Go to top ]

    have it build your software every 15 minutes

    Just curious ? why we have to build so frequently ? Why not simply once per day ? Is this depending on the size of the team ?

    Chester
  5. Build intervals[ Go to top ]

    Read the continuous integration article by Martin Fowler (you should be able to find it on his site or at the cruisecontrol site). The idea of integrating as much as possible reduces the amount of time between a developer committing a (faulty) piece of code and discovering the actual error. If you only discover it the day after chances are that other developers have already committing additional stuff resulting in more errors maybe.

    All continuous integration toolkits (that I know of) have support for schedules and modification checking. So even thought the interval is set to 15 minutes, the build process won't be triggered if there aren't any modifications in the version control system.

    It depends a the amount of activitiy in a project and also on the size of the team, although I can't live without it anymore even if I'm doing a pet project just on my own. If there is an environment that supports integration testing for larger scale projects (with multiple databases maybe or large schemas) I tend to see developers starting to rely on the build process. They don't (have to) do all the integration testing anymore by hand (even if parts of them are automated, it's still a hassle).

    And by the way, if you have machine dedicated to building your software, why not try to utilize it per the better part of the day :).

    regards,
    Alef Arendsen
  6. Build intervals[ Go to top ]

    We are following the continous integration process, but may not to the letter T.

    We have a rather small development team and we felt it's adequete to set build interval as 1 build per day. The complete build is at mid-night.

    This does not mean developer will not get the latest code from cvs, they will do. The build may break since one might have changed something. This can be easily resolved by inform one or another.

    Very frequently build does not add too much value, IMHO. Developer will say: yes, yes, I know I break the build, but I am working on this code, I will check in when I am done. Don't interrupt my concentration. In fact, if there is too frequently build, the developer will simply reduce the frequency of their check-in and wait for longer period because they afraid they break the build.

    I could totally wrong on this one that's way I am curious how other has done and way.

    Regards,

    Chester
  7. build breaking[ Go to top ]

    We are following the continous integration process, but may not to the letter T. We have a rather small development team and we felt it's adequete to set build interval as 1 build per day. The complete build is at mid-night. This does not mean developer will not get the latest code from cvs, they will do. The build may break since one might have changed something. This can be easily resolved by inform one or another. Very frequently build does not add too much value, IMHO. Developer will say: yes, yes, I know I break the build, but I am working on this code, I will check in when I am done. Don't interrupt my concentration. In fact, if there is too frequently build, the developer will simply reduce the frequency of their check-in and wait for longer period because they afraid they break the build.I could totally wrong on this one that's way I am curious how other has done and way. Regards,Chester

    This is tough. Nobody wants broken builds. The one thing you don't want is a broken build that gets ignored, as it gets worse and worse.


    I go for regular check-ins, with developers working in small bits that can be checked in (with updated tests) at least daily. If they are doing bigger stuff, they should be branching and remerging. Even then, no long-lived branches, as they are trouble at merge time.

    The other thing we do is comment out tests that we know are broken, until they are fixed. Naughty, I know, but what if you have a test for a problem that takes a major rework to fix. You need that test, but you dont want the continuous build to halt until that fix is in there.
  8. build breaking[ Go to top ]

    I am with you on this.

    We have build a JUnit Driver which allows us to exclude
    packages, classes, and methods, so that broken tests are temporarily move out of the way untill they fixed.
  9. build breaking[ Go to top ]

    Developers also need to understand that the source control management system is not intended to be a temporary storage space for their half-baked code. They absolutely can't check in code that doesn't compile, but really shouldn't even check in code that doesn't "work" (whatever that might mean) since that potentially screws up the whole development team. Sometimes developers will check in only part of a bug fix or feature which can also causes problems.

    John's article hints at it, but the point is that the SCM system contains one of the most important assets of any software development group: the code/config/data/etc used to build the product! As such, it should always be in a consistent, working state.
  10. if they break the build ...[ Go to top ]

    they haven't done proper testing, or have not checked in a complete set of changes. Either way they are wrong and must be punished :)

    Bring it to their attention so they will not do it again, and in the meantime, backoff their checkins that broke the build

    JohnH
  11. Build intervals[ Go to top ]

    [...]
    All continuous integration toolkits (that I know of) have support for schedules and modification checking. So even thought the interval is set to 15 minutes, the build process won't be triggered if there aren't any modifications in the version control system.
    [...]

    Well, that's not the problem. If you have a non-transactional version control system (like CVS, for example), the probability increases with decreasing build intervals, that developers are still checking in modified sources while the build process runs. Further, for large scale projects it is unlikely, that you can checkout, build, deploy, and perform integration tests for your application within 15 minutes.

    So, you have to be carefull with specifying those absolute values (15 minutes build interval, that is). Even if Martin Fowler says so :-)

    Regards,
        Dirk
  12. Build intervals[ Go to top ]

    CVS tags work on the sandbox you've checked out, so this is not really an issue. If somebody has committed something after you've updated the sandbox you're using the build, the tag won't include the latest revisions, just the ones the CI build has locally...

    Furthermore, you can specify a so-called quiet interval which indicated the source control system needs to be 'stable' (in other words, no modifications have been made) for the specified period.

    The only issue you could run into is that somebody is checking in code after the CI toolkit you're using has triggered the build process because of modifications and before the update process has finished. But I don't see this as a real issue. If the developer hasn't finished committing his code yet the build will probably fail. If he has finished, then the new code will be included in the build.

    regards,
    Alef Arendsen
  13. I've done dev with fast, automated deployment, and one lesson is that even if you can deploy in 5 minutes, you never want to do so.

    The problem is, that the moment you do it once, to fix a minor problem, you have set in your customer head the idea that any defect can be fixed in 5 minutes, if only they phone you up enough.

    Tell them that your automated process (you do have an automated process, don't you), only updates at night, after a clean build and retest. So there is no point bothering you until then. Acknowledge the defect, write the tests, fix it, test in staging, then finally roll out a stable, clean fix -not a "what can we do in five minutes" hack.

    This doesnt mean that your continuous integration server cant rebuild the staging stuff on demand, though we like a bit more of a co-ordinated upgrade in case people are testing against the staging server at upgrade time.
  14. Build intervals[ Go to top ]

    [...]All continuous integration toolkits (that I know of) have support for schedules and modification checking. So even thought the interval is set to 15 minutes, the build process won't be triggered if there aren't any modifications in the version control system.[...]
    Well, that's not the problem. If you have a non-transactional version control system (like CVS, for example), the probability increases with decreasing build intervals, that developers are still checking in modified sources while the build process runs. Further, for large scale projects it is unlikely, that you can checkout, build, deploy, and perform integration tests for your application within 15 minutes.So, you have to be carefull with specifying those absolute values (15 minutes build interval, that is). Even if Martin Fowler says so :-)Regards,    Dirk

    This is why there is a quietperiod configuration (with CruiseControl). Basically if CC detects a modification in the last 2 minutes for example, it waits a given period of time before rechecking; if it again detects a modification is will again wait. So there needs to be a full 2 minutes (or whatever you have configured) period of quiet in terms of modification.
  15. What about Luntbuild?[ Go to top ]

    Has anyone tried LuntBuild as a better CruiseControl? I am using it currently and it has great features and philosophy.


    http://luntbuild.sourceforge.net/
  16. I agree largely that this article sets out what we should be doing rather than prescribing exactly how we should be doing it and, given some of the comments, I felt that I should clarify this a bit.

    Firstly, and this may come as a surprise to a lot of us who have been doing this a while, it has been my experience that a lot of dev projects dont even know the "What" - blissfully unaware of the need for automating anything or having any kind of structured process around build and release.

    My main goal is to evangelise the "What", ie to define the problem clearly, without necessarily just trying to advertise software that takes care of the "How".

    If I had concentrated on the "How" - focusing on what we do to implement these best practices - then the article would have seemed more like an advert and less like technical evangelism.

    Out of respect for TSS, and its readership, I opted to stick with something that was sanitised as far as products go - there are enough smart people reading TSS to build or buy the tools that suit them.

    I absolutely could not agree more on the automated building and tagging - knowing what is where (and how it got there) is crucial.

    John Birtley.
  17. Why the "What" Rather Than The "How"[ Go to top ]

    John,

    Never fear. I am glad you are highlighting an often overlooked aspect of the production lifecycle. Even after wrapping your app in cruisecontrol or another auto-build process there is still a whole bunch of aspects related to deployment (many of which you touched upon in your article).

    Perhaps we should coin the new acronym DOP (Deployment Orientated Programming). Making an acronym out of things seems to work to increase visibility. I still remember when KFC was Kentucky Fried Chicken :)

    Q
  18. Why the "What" Rather Than The "How"[ Go to top ]

    Perhaps we should coin the new acronym DOP (Deployment Orientated Programming). Making an acronym out of things seems to work to increase visibility.

    Would that imply that we will see a prevalence of DOPE:s in the future? ;)
    (Deployment Orientated Programming Environment)
  19. Would that imply that we will see a prevalence of DOPE:s in the future? ;)(Deployment Orientated Programming Environment)

    Don't you mean Deployment Orientated Programming EngineerS :)

    I can tell you right now, that when you get into larger scale environments there is nothing more crucial then having something deploy right away.

    We have such a mixed environment and it can sometimes take a day or so for some of us just to get a new branch of code deployed onto our Unix port. Not having a smooth way of doing it has cost much time. Of course, you also have to have someone or someones assigned to the problem. Generally, if everyone owns the problem, nobody solves the problem :)
  20. As stated in the article, “this is just common sense” but well worth reiterating and I think John covered broad subject area well. Sticking to the "What" rather than the "How" was IMHO a good decision as there is plenty material available on implementing Build Management and focusing on issues and best practices will probably help the reader make more informed decisions.

    A quick mention of J2EE Application Deployment Specification, JSR 88

    http://java.sun.com/j2ee/tools/deployment/

    and its merits or failings would have been nice.

    Hoos
  21. Theory of Constraints[ Go to top ]

    The convoy of ships analogy is an example of the Theory of Constraints (TOC). Eliyahu Goldratt provides an example of it in a manufacturing setting in his book “The Goal”
     and explains it in further detail in his book a href="“Theory" rel="nofollow">http://www.amazon.com/exec/obidos/ASIN/0884270858/002-6402208-6548049">“Theory of Constraints”</a>. While these books are not directly related to the IT industry, the application still applies. Constraints apply to any process including the software development lifecycle. For example, it does not matter how fast or automated development is if requirements are the bottleneck. We need to learn determine where bottlenecks are and how to add capacity.
  22. Theory of Constraints[ Go to top ]

    Good that you're mentioning this since I think with modern agile methods (XP, Scrum), we're close to taking away the constraints and bottlenecks with respect to the development of functionality the customer wants. If done right (with the right people, the right tools and frameworks), the development team should be able to react quickly to changing requirements and be able to release software early and often. But this is (and I think John will agree) where I think one of the other constraints is: managing and releasing complex software can take quite some time; more than necessary...

    Of course, it's something developers should not be bothered with too much, but it's certainly a very important aspect of the software development lifecycle.

    By the way, simple things can change things for the better quite dramatically. Dependency Injection in combination with simple configuration management options (Spring's PropertyPlaceholderConfigurer mechanism for example) can be quite effective. But let's not turn this into a DI thread :).

    regards,
    Alef Arendsen
  23. Theory of Constraints[ Go to top ]

    For example, it does not matter how fast or automated development is if requirements are the bottleneck.

    Code generation is a big a boost if requirements are still changing. It means having to do less hand coding, and so having less hand code to throw away when requirements change. If the developer is the bottleneck, then code generation might be the solution.
  24. We use ANT as automatized build, and we are thinking about extending it to do the deployment too.
    If ANT access the repository in CVS, compiles, make the deployment, and restart the server... What are the main differences with what the article is proposing, especially with BuildMonkey?

    Joserra
    Nuevos Modelos de Negocio basados en Software Libre
  25. Using Ant as an Implementation[ Go to top ]

    Joserra,

    There is no difference if you use Ant - this is just one way of implementing what's in the article.

    One thing you may want to consider though is adding some post-deployment verification tests to make sure that the build and deployment was successful. For example:

         - use something like HappyApp to check the app from the inside out

         - HTTPUnit / EJBUnit or their ilk to test the app from the outside in

         - Custom Ant tasks to do other bits of checking (e.g. looking for patterns in log files, checking that daemons are running, user accounts created etc etc etc)

    The idea that I am trying to promote here is that the "automated build" is not just the single act of compiling the code and assembling the artifacts - it should also include deployment and post-deployment verification.

    If you incorporate deployment and post-deployment into your automated builds (regardless of how frequently those builds are done) then the number of problems you will encounter when your application goes live will be significantly reduced.

    Since puttingthe application live means that your customers can see it, you dont want it to look buggy.

    I have seen some of the highest quality (and thoroughly tested) code totally let down by poor deployment.

    Deployment is like christmas - it is going to happen. Dont leave it till christmas eve to do your shopping :)

    John.
  26. I don't think you should use Ant for deployment. You should use it (or Maven) to build, you should use it to commenc deployment, but you should not use it as the script language for deployment.

    The problem is that in deployment, things go wrong, and you need to back off carefully. Ant doesn't do failure outside the build process view of "absolute success" and "halt with a problem".

    The other issue is, as the article emphasises, configuration is central. Ant may be able to get binaries up with the ssh task, and trigger an app server reload, but it does nothing for making sure that the webapp is pointed at the database host, with the right database username and password.

    This is why I work by day on Smartfrog (http://smartfrog.org/ ) , which an HP Laboratories framework for automated distributed deployment, including
    -configuration
    -liveness (defined as the continuous verification of system health)
    -undeployment

    In SmartFrog, you provide "components", which are like Ant task implementations, but which have termination and liveness support too.

    I see BuildMonkey also do tools for deployment, and again, they should be choreographable by ant, but that does not mean that anyone should be using Ant do manage the nitty-gritty of deployment. I know I used to do that, and have also been known to advocate it in public, but we have moved on.

    Incidentally, I'll be covering this at ApacheCon 2005 in July, in Stuttgart. Hope to see people there :)

    -Steve Loughran

    steve dot loughran at gmail dot com.

    Ant developer, SmartFrog developer, co-author Java Development with Ant, etc etc.
  27. Deployment Scripts[ Go to top ]

    I wrote scripts in Java and beanshell to setup the app server, deploy the application and write the configuration files. Everything is automated, and reproducible. If you need to add a value to a configuration file you just add it to the config-appname.bsh script and then run it. The scripts are versioned with software so other developers get the new configuration along with the new code. System administrators just run the scripts when they upgrade.

    The server configuration is also fully automated using beanshell scripts that use JMX to configure the server.

    The whole application can be built and installed from scratch with these scripts. It really saves a lot of time and reduces errors.
  28. Read the book "Software Configuration Management Patterns" instead of this article.

    I didn't know that "3D" can be a trademark. 3D TM, wow! Yeah, we need another proprietary methodology. If it would be possible to patent it, they would do it.

    The author seems to fight against the developers. In his opinion they don't understand the big picture which he understands, as he claims. Don't fight developers needs, or you will have no success at all.

    The article is superficial. Its "best practices" are often vague and meaningless.

    Better read the book which I mentioned above. Beware of false prophets like John Birthley. You can use advisors, but you need enough know how to distinguish between good and bad advises. If you have that know how, you nearly don't need advisors like the author any longer who try to sell panacea.
  29. I have read the SMC patterns book - it is a GREAT book.

    I do not have a monopoly on knowledge, and I am not smarter than any of us poor souls who slave away pulling rabbits out of hats on development projects.

    Developers are What Its All About. My goal is to free them from the BS That they need to put up with on projects, and let them get on with what they do (and enjoy) best.

    Does a brain surgeon look for his own tools? No. He says "scalpel". Does a chef wash his own pots? No.

    Why, then, do we expect developers to get involved in the mundane mechanics of build and release? Let them create - they are absolute experts at it.

    If you interpreted my article as being hostile to, or derogatory to, developers then I apologise. I work with developers all the time and consider them amongst the most talented people I know.

    And my name is John Birtley - no 'H'.

    John.
  30. I was a little bit offensive in order to see whether I am right or wrong with my impression.

    We developers struggle with the integration people who force us to follow a beurocratic procedure. Every single file needs a CVS tag before it may be checked in and every CVS tag must be either a project number or an issue number of our issue tracking tool. We work against 4 week old API because we have no dayly built and are not allowed to check everything out. We are just allowed to check out defined versions (which may be 4 weeks old) and the components we need to work with. We don't practice "eat your own dog food". I am very angry on our integration process because it severly hinders me to refactor things or to fix bugs. Its very beurocratic. So I was offensive, although I didn't know if you are such a beurocrat too.
  31. Environment tracking[ Go to top ]

    Hi,

    I was curious about how to fit environment control into deployment process. Having seen different development and deployment context we found that it's often more complicated to impose control - or simply a check - over environments than on anything that can be automated.

    Inspection tools may look for IP addresses, availability of resources, versions of common libraries, but often a yes/no response is not a viable solution. "Yes, I didn't have the time to apply patch D45tg... So sue me!". In some other cases environments HAVE TO be different, simply because it's too costly to have a real replica. An environment comparison tool will have to deal with different topology and to decide whether they are comparable/compatible or not.

    Is there any suitable tool to do this job inside the deployment process, (or part of it) or simply we have to deal with custom solutions?

    Alberto Brandolini
  32. Environment tracking[ Go to top ]

    In some other cases environments HAVE TO be different, simply because it's too costly to have a real replica. An environment comparison tool will have to deal with different topology and to decide whether they are comparable/compatible or not.Is there any suitable tool to do this job inside the deployment process, (or part of it) or simply we have to deal with custom solutions?Alberto Brandolini

    Alberto,

    You may look at the free command line tool Devenv from Viewtier: http://www.viewtier.com/products/devenv/index.htm
    It can be used by developes to set up unified environments. Devenv also simplifies build automation if you use it with Parabuild http://www.viewtier.com/products/parabuild/index.htm