Discussions

News: Unbreakable builds

  1. Unbreakable builds (12 messages)

    Vincent Massol has been thinking about how we could have Unbreakable builds: "A thought struck me about a year back: what if we were able to prevent the continuous build from failing by design." He puts his thoughts out there, with a potential architecture for the implementation.
    Out of my last two development projects, one had a strong sense of quality and excellence in general and continuous build failures were the exceptions (about 3-4 per week for a 30 developers team) and the other one was quite the opposite and everyone was surprised when the continuous build was passing (there were about 5 build breaks a day as an average for a 40 developers team). I'm sure this is also pretty common to other projects. Obviously the best is to build (pun intended) a build awareness in the team. However, you'll need strong evangelists for this to happen who may not always be available and other circumstances may make this difficult.

    A thought struck me about a year back: what if we were able to prevent the continuous build from failing by design. There's a French saying that goes something like "it's better to cure than to heal". I think this is definitely a good idea to apply to continuous build failures. Why not make a continuous build system that cannot fail. At that time I thought it was a nice idea (I had meant to blog about it but I forgot) but I could not see very well how it could work. Now a year after, I really think it's a nice idea and I'd like to explore it.

    Read more: Unbreakable builds

    Cedric Beust also commented that developers should never build at the same time.

    Threaded Messages (12)

  2. Unbreakable builds[ Go to top ]

    A few years back I developed an enhanced version of Continuous Integration that has a similar effect to "unbreakable builds" as described above. In my case, the build could break, its just that the SCM would not commit the broken code. Therefore, the SCM never contained a broken build.

    I can't imagine implementing the technique in CVS as it lacks an important feature: atomicity of commit sets. However, it worked a treat with ClearCase and Perforce. ClearCase has a custom forms/dialog feature which was useful, in addition to email, to communicate immediately to the developer that their commit had failed and was being rolled back.

    There are caveats. It meant that only one developer could commit their work at a time, and during busy periods, a queue could form behind developers wanting to commit. For an experienced dynamic team I can only imagine this would get in the way.

    But in a scenario where the dictat is "the build shall never be broken" and whereby a team does not contain a strong personality to ensure the culture of never breaking a build, the technique can work well.
  3. breaking is communication[ Go to top ]

    Interesting concept....continuous integration provides the foundation for measurable progress as well as assuring that said progress is of a high enough quality.

    So I guess the logic behind making unbreakable builds is linked to something like the following statement;

    'if the continous integration process is stalled, e.g. broken build, it maybe percieved by management/development that progress has stalled as well...the collary....lets make unbreakable builds.'

    At the worst end of the spectrum, a high number of broken builds almost invariably means that there is too much volatility in requirements, data model, architecture, etc...or worst that the developers skills are not matched to the project. Productive work is not getting done, so reducing the number of broken builds is paramount!

    The opposite conclusion may be drawn, e.g. when having no broken builds or very few broken builds; usually in these rare scenarios a broken build is quickly sorted out. Work is getting done, spending more energy on reducing the occassional broken build, through design, prob has diminishing returns....not sure.

    A broken build is a sign that communication needs to happen, in the same sense I would worry if my compiler would never fail...never informing me of my various syntax errors...at worst masking fundamental arch design issues which show up at runtime...which makes problem so much more difficult to debug.

    in an Ant sense we need failure feedback to enable continous integration, not get rid of it through design...yes, yes reduce broken builds....but lets not throw the baby out with the bathwater! It is fundamental to allow Ant to fail during the build if something is unacceptable, doesnt compile, integrate, startup, install...this is *the* fundamental feedback loop...though perhaps as we get used to these techniques so we forget why we first used them.

    I maybe mistaken, I think perhaps the tail wags the dog with this concept, perhaps its because of the negative connatation of broken builds meaning 'inoperational'...'not working'...'cant demo today'....not something positive like 'great, glad the rule and development teams worked out that tricky set of interfaces which has continually escaped specification'...or 'cricky! one of our developers really doesnt know how JMS works'

    In my exp, there are lots one can do to reduce broken builds w/o ...for example new developers cause lots of broken builds because of inexperience or delaying integration of components stores up the potentiality of a broken build...early integration is paramount.

    I remain convinced that until computers write their own software programs that there will always be negativity from managers and even software developers from such things as broken builds and bugs.

    One last point: Measuring productivity through build stability might be a red herring, e.g. in the largest J2EE projects, managing component dependencies and database composition are the main barriers to productivity.

    Ultimately continuous integration, test driven, pragmatic development sets up strategies to take advantage of feedback not suppress/remove it.

    just my 2p

    Jim Fuller
  4. check code in client side[ Go to top ]

    Hi!

    If code is checked in client side, then it wont prevent other users to commit code. I can't see no reason why that could be done already with some ant/maven script's. Project leader just need to make clear that everyone using same method to commit code.
    Of course in CVS there is still window to break code, because there is no transactions, but it could still help situation.
  5. Uh[ Go to top ]

    Wouldn't it just be easier to properly train your staff, and increase team communication? This sort of thing is asking for trouble, and doesn't actually address the problem being encountered. It's the same as using the NullObject "pattern" to avoid NullPointerExceptions - you've fixed the symptom, not the cause.
  6. Two phase commit for SCMs[ Go to top ]

    I suppose that, in order to implement well the unbreakable continuous build, the SCMs should support something like two phase commit for change sets. First commit phase commits changes to the SCM, but these are only visible to the user that makes it and to the build process that this user initiated when he/she commited the changes. The second commit phase starts when build finishes successfully and it makes changes visible to all users.

    Otherwise, there are compromises that can be painfull.

    MC
  7. Two phase commit for SCMs[ Go to top ]

    I suppose that, in order to implement well the unbreakable continuous build, the SCMs should support something like two phase commit for change sets. First commit phase commits changes to the SCM, but these are only visible to the user that makes it and to the build process that this user initiated when he/she commited the changes. The second commit phase starts when build finishes successfully and it makes changes visible to all users.Otherwise, there are compromises that can be painfull.MC

    You can probably simulate this with two repositories. Code will be promoted to second repository on successful build. another option is to use a patch-driven approach.
  8. Two phase commit for SCMs[ Go to top ]

    Well, ClearCase just can do that. Using UCM, a delivery operation can be completed (committed) or cancelled (rolled back).
  9. If you consider that clearcase is using 'branches' to implement their UCM concpect of delivery, it becomes clear that you do not need clearcase, but can use any other scm that supports branches (i.e. almost all scms, including cvs, subversion etc. clearcase-ucm-enabled will give you a practical branching policy out of the box, but it is not hard to implement similar or better banching policies in other scms).

    The (here called) 'two phase commit for SCMs' is about the concern of protecting the main code line from 'unverified' checking in of code. This concern is actually documented in several scm branching patterns (for example http://www.cmcrossroads.com/bradapp/acme/branching documents the patterns 'docking-line' and 'staged integration lines' that try to solve just this problem using a mutiple phase check in procedure to protect the main code line).
  10. Stable build tags[ Go to top ]

    What about using tags on the source control for the stable versions? The latest version (head) is only for development. You can get a system the builds and passes all the unit tests. But this does not mean it is stable. It still has to go through some QA process that performs integration, performance and regression tests. Usually, such a process takes longer and you can't afford to wait that long until your coleagues see your chnages. Some of these types of testing can't be (easily) automated by scripts.
    I agree with the partial build ideea, as it helps the development process.
  11. Unbreakable Continuous Build[ Go to top ]

    The point is not to make the build unbreakable, it is to make the *continuous* build unbreakable!

    The main idea is to let the others continue working while you fix your own errors...

    What's important is that people commit continuously because this is what makes continuous integration working. We've hammered into the heads of develoeprs that they should "care about the build" because we know that if the build breaks, it will hamper others. Now, if we make the assumption that the build cannot break, why would developers need to worry about breaking the continuous build or not. They still need to make their code work.

    Anyway, my point here is simply that I am curious about implementing this and checking if it works or not...
  12. Unbreakable Continuous Build[ Go to top ]

    Interestingly it seems that you've gone full circle from a central "integration" build, back to the developer sandbox.

    In our process, if a developer follows procedure, he/she will rebase their local view (using clearcase), and peform a full build, install and automated Junit test run.

    When all that succeeds, then they deliver their code changes to the main integration branch. That's where we consider the CI to begin.

    1) A "build" machine gets the latest source, builds and publishes the output.
    2) A "deployment" machine grabs the new build, installs it, initializes data, starts services etc. and then re-runs the same Junit suites the developers have. Deployment outcome and test results are then published.

    To us, the most important thing was to *know* if/when the main branch:

    1) would not build
    2) would not deploy
    3) would not start up
    4) would not have 100% test pass

    While this process doesn't prevent a "broken" build in any sense, it identifies and alerts us to problems that normally fell on QA.
  13. geronimo[ Go to top ]

    cederic you very good and posting on wrong plce, please posting properly and remembering geronimo better beating wexlogix
    heh ,.. down boy