DevPinoy.org
A Filipino Developers Community
cruizer » Weighing in on pipelined continuous integration (CI)
   
Weighing in on pipelined continuous integration (CI)

This InfoQ link on pipelined CI got my attention because it was something that I can relate to. The project we're working on used to build on our CruiseControl.NET build server in only three minutes; now it's taking more than 18 minutes to complete. And it's a dual-core 2 GB machine at that!

Agile practices like test-driven development (TDD), continuous integration (CI) and the like are hinged on quick feedback.  The longer you put off the small tasks like testing and integration, the greater the risk of having things go wrong near the end of the project, the phase we dub as "crunchtime." There was one person who blogged that pipelined CI is a "pipeline of doom," meaning more things will go wrong if you pipeline or break your build into multiple but smaller builds.

I disagree. To me, the quicker one can do a check-in dance, the better. That doesn't mean one will have to do away with tests and go with half-hearted integration. It just means that you push them back so the more time-consuming ones are done later or less often. That way, the developers still get quick feedback even if the project code has grown considerably larger. For instance, in my current project, our architecture lead insisted on having just one solution (.sln) file for everything -- base code, server-side only code and UI code. I objected to this, since not everyone would be working on the server side code and it would only slow the build down. I proposed that we have two solutions (thus two builds) -- one for the server-side code and the other for the client-side code. But then, he's the lead so his opinion was the one taken by the whole team.

Now that our code has grown large and build times are five to six times longer, I realized that my opinion was correct. What I've noticed is that since the build takes longer, the other team members impatiently check in their code even if the current build is red. So bugs and broken behaviour pile up quickly. Other team members keep their code to themselves and only check in a few times a day. Our use of IBM/Rational ClearCase further aggravates this problem, since those devs who keep their code changes to themselves keep the source files exclusively checked out! Ah, it makes me pine for good old Subversion...

I'll be bringing this point up again in our next team meeting. I'll be suggesting that we break our solution down into server-side and client-side solutions so that one set of devs (those working on the server end of things) won't be slowing down the other (those working on the UI/client) and vice-versa. Also, I'll suggest that we move our integration tests out into a third solution, one that we can trigger anytime via a "force build" on the CCTray. So in case someone complains that the services aren't working properly, we'll just hit the "Force Build" button and see minutes later if things are green or red.

Posted 10-01-2007 12:59 PM by cruizer
Filed under: , , , ,

Comments

lamia wrote re: Weighing in on pipelined continuous integration (CI)
on 10-01-2007 2:58 AM

This is good to know. I was thinking of suggesting to my team how we could utilize CI Frameworks. I wanted to make a proof of concept but I can't think of any more ideas other than "it would send them an e-mail, or a gmail message informing of any problem that occured when they last checked-in a file".

Does Cruisecontrol come with a Visual Source Safe plugin? If we have several branches like

ProjectA-1.2.3

ProjectA-1.3.0

ProjectA-1.5.2

and we wanted to make a merging of those projects. Would a CI framework like CruiseControl help us in any way? Thanks!

cruizer wrote re: Weighing in on pipelined continuous integration (CI)
on 10-01-2007 3:06 AM

well CruiseControl.NET produces this web page that displays the results of the build, who checked in what, and what error messages appeared. so as long as code is checked in one developer at a time, it's easy to pinpoint who broke the build :P

jop wrote re: Weighing in on pipelined continuous integration (CI)
on 10-01-2007 3:37 AM

If you are using Visual SourceSafe and you need to do merging, you don't need continuous integration software. What you need is lots of patience, pots of coffee and an insane amount of good luck. :D

Seriously though, continuous integration is not really about software automation. Automation helps but in order for it to help, you must first have the correct thinking in place. Jim Shore explains it in this article much better than I can even dream of trying: www.jamesshore.com/.../Continuous-Integration-is-an-Attitude.html

jop wrote re: Weighing in on pipelined continuous integration (CI)
on 10-01-2007 4:17 AM

I believe the point of the "pipeline of doom" article is to use pipelines only as the *last* resort and only when the "team already has enough discipline to keep a continous integration build green".

Do you know why build has gone from 3 minutes to 18 minutes? Where is it spending most of its time? Have you reached the limits of your hardware (or sofware)? Can I improve the performance of my tests? What more can you do to improve the speed while still maintaining a single build?  Can I pipeline the build but still put all the tests in the primary build? How about splitting the project instead and have two fast independent full builds instead of one slow build?

Pipelining the build is not bad but hide it at the bottom of your toolbox. It might mean faster feedback for your team but the feedback that you are getting is no longer the complete picture.

cruizer wrote re: Weighing in on pipelined continuous integration (CI)
on 10-01-2007 6:44 PM

what I actually plan to do is to break the build into separate subsystems (one for the client software and the other for the server software) and to put the integration tests in a separate build. all unit tests should still be with their respective builds.

i hope i'm not compromising in this one. in my view i'm just trimming the "fat" :P

LaTtEX wrote re: Weighing in on pipelined continuous integration (CI)
on 10-02-2007 4:07 AM

I wonder how many projects are under your sln. 20? Hehehe.

Kelangan nga mabreakdwon na yan.

cruizer wrote re: Weighing in on pipelined continuous integration (CI)
on 10-02-2007 4:33 PM

yes, around that. anyway in the next few days we'll be breaking that down... :)

jong.net wrote re: Weighing in on pipelined continuous integration (CI)
on 11-11-2007 5:07 PM

In our current project, we already have about 20+ projects but we haven't integrated it with CruiseControl.NET. We'll be putting it in CC in the next few days too and I think, we may encounter that slow build. I'll be suggesting that solution to my team lead. Great post sir cruizer.

Copyright DevPinoy 2005-2008