Presentation: Development at the Speed and Scale of Google

Since I’ve never had the good fortune of being able to afford QCon (one day this will change) I appreciate the fact that InfoQ post QCon videos online for free albeit late. Recently I watched ‘Development at the Speed and Scale of Google‘.

Prior to watching this presentation I knew only what I had encountered in the wider industry and really could not have foreseen any of what I was about to watch. The tools that I use on a daily basis and the difficulties that impede me now both seem primitive and outdated in comparison to the progress that Google has made. The key point on the subject matter of this presentation is that it is not about development but what makes it possible to develop at the speed and scale of google: in this case – build and release engineering.

Highlights from the talk that I found worthy of note are listed below.

  • Working on build and engineering tools requires strong computer science skills and as such the best people.
  • We cannot improve what we cannot measure. Measure everything. This, in my opinion, is a fantastic quote. This stops a team going off on open ended endeavours that yield either intangible or no results.
  • Compute intensive IDE functions have been migrated to the cloud such as generating and searching indexes for cross referencing types across a large codebase.
  • The codebase required for building and running tests is generally larger than that which is worked upon but delivering the entire codebase to every developer either in source or in binary form would kill the network. Here – a fuse daemon detects when surrounding code is required using a fuse (user space) filesystem and retrieves it incrementally on demand.
  • For similar reasons to the above point – they’ve developed a virtual filesystem under Eclipse and contributed it back. The obvious benefit is that directly importing a large code base into Eclipse kills it whereas incremental loads perform.
  • They build off source and not binaries and maintain an extremely stable trunk from which they release. If you imagine that all code is in a single repository (in fact the largest Perforce repository in the world) then it really puts into perspective the achievement of using only trunk.
  • The designated owners for a given project who review code have at their fingertips all the intelligence metadata on the code to assist them in the reviewing process. If you think about it that makes a lot of sense. To review you need more than just the code to spend your time effectively. You may want the output of introspection, test runs etc.
  • Compilations are distributed and parallelised in the cloud and output is aggressively cached. It’s fascinating to hear a case study where this has actually been implemented. I’ve often considered remote compilations but never come across a concrete implementation until now.

The importance of build and release engineering is often underestimated. It is often portrayed and perceived as an area of work that’s second class in nature and rather unglamorous. However, as this talk attests, it is very much the contrary. It can massively boost developer and organisational productivity and efficiency and requires the best people. I’ll end with a quote from the presenter: “Every developer worth their salt has worked on a build system at least once”.

Leave a Reply