This InfoQ link on pipelined CI got my attention because it was something that I can relate to. The project we're working on used to build on our CruiseControl.NET build server in only three minutes; now it's taking more than 18 minutes to complete. And it's a dual-core 2 GB machine at that!
Agile practices like test-driven development (TDD), continuous integration (CI) and the like are hinged on quick feedback. The longer you put off the small tasks like testing and integration, the greater the risk of having things go wrong near the end of the project, the phase we dub as "crunchtime." There was one person who blogged that pipelined CI is a "pipeline of doom," meaning more things will go wrong if you pipeline or break your build into multiple but smaller builds.
I disagree. To me, the quicker one can do a check-in dance, the better. That doesn't mean one will have to do away with tests and go with half-hearted integration. It just means that you push them back so the more time-consuming ones are done later or less often. That way, the developers still get quick feedback even if the project code has grown considerably larger. For instance, in my current project, our architecture lead insisted on having just one solution (.sln) file for everything -- base code, server-side only code and UI code. I objected to this, since not everyone would be working on the server side code and it would only slow the build down. I proposed that we have two solutions (thus two builds) -- one for the server-side code and the other for the client-side code. But then, he's the lead so his opinion was the one taken by the whole team.
Now that our code has grown large and build times are five to six times longer, I realized that my opinion was correct. What I've noticed is that since the build takes longer, the other team members impatiently check in their code even if the current build is red. So bugs and broken behaviour pile up quickly. Other team members keep their code to themselves and only check in a few times a day. Our use of IBM/Rational ClearCase further aggravates this problem, since those devs who keep their code changes to themselves keep the source files exclusively checked out! Ah, it makes me pine for good old Subversion...
I'll be bringing this point up again in our next team meeting. I'll be suggesting that we break our solution down into server-side and client-side solutions so that one set of devs (those working on the server end of things) won't be slowing down the other (those working on the UI/client) and vice-versa. Also, I'll suggest that we move our integration tests out into a third solution, one that we can trigger anytime via a "force build" on the CCTray. So in case someone complains that the services aren't working properly, we'll just hit the "Force Build" button and see minutes later if things are green or red.