Joe Arnold's Blog

Last year (31st, May 2007) I interviewed the Yahoo! Web Messenger team. They were one of the first teams I helped to coach at Yahoo!. During that interview we discussed their use of Continuous Integration. This is a transcript of that discussion. It didn’t take a lot to get them talking.

Me: So tell me about the continuous integration system.

Ben: We fought it for so long… and we’ve finally given in. And it’s the greatest thing ever.

Henri: We originally thought: “We build our stuff, it works. Why do we need something so complicated?” But it’s actually really really nice. We miss it a lot when we don’t have it. When we have check-ins that don’t get built right away, we don’t know if it’s working or not… it kind of hurts.

Mark: What’s really amazing is that every time the build machine process breaks for a day, it’s always the cause of trauma. A bug gets in! You would think it was coincidence if it wasn’t so regular. [laughs]

It’s unbelievable… The one day that the build machine breaks (and say there has been five check-ins) — now we don’t know which one broke this feature! “Ahhggg! if only the build machine were here, that way we would know if it was my check in, or your check in.” Every time [the continuous build system breaks] something like that happens!

It’s just proof that, without it, we would be constantly be saying, “how the hell did this bug show up? Whose fault is this? Is this my fault? I don’t even know?”

Ben: It has saved us an incredible amount of time — much more time than we have ever put in.

Mark: It [the continuous integration system] is a lot of heartache. We will say that. It’s great. We’ll agree. Anyone should know that they have to believe that there will be this big payoff because it consumes time every week to keep it going.

Jon: We had this bizarre bug where the emoticons were appearing off to the side. They were right on the contact list after you scroll. Nobody had this except Stephanie (a product manager). So we talked about it and I looked at it. And she was able to track it down through the build machine — where it did start to happen and where it didn’t start to happen. [She narrowed it down to] between two builds. And we had this one minor check-in where we added a drop shadow to the conversation list. We were like, there is no way that this could be the problem. But, low and behold, we removed the drop shadow and the problem was completely fixed. Because we had the [build] machine, we were able to track down where this bug was.

Mark: Here’s another big thing that people don’t appreciate until you get it: It means that everyone is always using the latest build — and the latest build changes. So Stephanie goes to lunch and logs back in. She’s using the latest build. And bugs are caught really fast. And features are checked in really fast. We knew the last part (features checked in fast), but we didn’t think of the first part (using the latest build).

Stephanie might say: “Oh! I’ve got icons now!” (Or later when getting a new build) “oh… the icons are not in the right spot.” [I would follow up] “Can you go back a build?” [Therefore,] I might find out that day that I broke something.

There isn’t such thing as a broken build. Because of the automatic continuous build integration system with email announcements, no build stays broken. That’s really a big deal. There isn’t such thing as a broken build. Other than the build broke for a moment, so [therefore] somebody is going to fix it. We all know something has happened because we all get the email. We can usually see from the change list who is responsible.

Word.

Tags: ci, continuous integration, XP