Engineering is repeatability. If what you’re doing isn’t repeatable, it isn’t engineering, and may in fact just be performance art instead. (Not that there’s anything wrong with performance art per se, but if you bought a tin marked “Engineering”, and instead it contained performance art, you’d take it back to the shop.)
Part of being able to repeat something is knowing what it is that you’re trying to repeat. Or, now that we’re moving past the “fanciful opening paragraph” phase of this blog post and into specifics: part of making a software release is knowing exactly what source code it was built from, so that if there’s ever a problem with the release, the exactly-corresponding source code can be analysed to discover the cause. (Sometimes tools can make this hard; sometimes they also have an option that makes it easier again.)
In the days of CVS and Subversion, when there was a single repository for a project, repeatability was typically achieved by using the tagging mechanism of those version-control systems: anyone could later check out that same tag and obtain code identical to the release. (In the days before CVS and Subversion, we probably just wrote a code snapshot to a CD or a floppy and kept it somewhere safe. We were cowboys once and young.)
But now we live in the days of Git. It’s possible to use Git in a purely peer-to-peer fashion, every developer being their own island – but that could easily cause unrepeatability of releases, releases as performance art. To make releases as acts of engineering, you’re going to have to nominate one repository as the golden or master one, the one where the tags and SHAs of releases live.
The Linux kernel project, for which Git was originally designed, does this by having a release manager (Linus Torvalds himself) whose personal (but centrally-hosted and shared) repository is the golden one, and who makes releases from code he’s pulled from other developers’ repositories as they complete features.
There’s another way of doing it, though: one which is more familiar to developers who have used CVS or Subversion-era tools. And that is to have a central golden repository for the project, which everyone has write access to, and to use git push as if it were svn commit. But it may be inconvenient to build the releases on the machine hosting the Git repository, so release managers will still do that on their own machines in their own local Git repositories.
So, especially if the release manager is also a developer, the engineering question becomes: is this local commit, from which I’m building the release, also in the golden repository? Is my SHA upstream? (Or am I just pleased to see you?) This is an important question because, if you use bow-shaped branches – or any other rebase workflow – then the SHA of your commit may well be different by the time it has gone upstream.
What’s really needed is a short and pithy Git command that looks a bit like “git is-releasable HEAD” – but there isn’t one, so in order to answer the question “Is it OK to build a release from this commit?”, you need to build that facility from several Git commands.
First off, clearly it’s not OK to build the release if you have uncommitted changes:
git status # Sometimes needed to refresh the index (?)
git diff-index --quiet HEAD || echo Nope
Now you need to check whether your SHA exists upstream:
test "`git branch -r --contains HEAD`" != "" || echo Nope
That works by listing all the branches in the golden repository which contain your current commit. If there’s at least one such branch, it’s OK to build the release – otherwise, no branch in the golden repository contains your commit, so it isn’t currently repeatable, so it isn’t OK.
Some of this functionality is offered by git describe --dirty, but apart from being less fun than it sounds, it doesn’t answer the question about the golden repository. (Perhaps because, in its original Linux home, releases are always made from the golden repository.)
At Electric Imp, the version information that we incorporate into every build of the software, includes not just the Git SHA but also the “OK to release” information and a corresponding Git tag name (if any). The “OK to release” is marked by having a “+” in the version number when it’s not OK, i.e. when this must be an internal build only. (You can’t always force people to do the Right Thing, but you can at least make sure that they’re told about it when they’re doing the Wrong Thing.) Here’s the relevant part of our top-level SConstruct file:
gitRevision = subprocess.Popen(["git","rev-parse","--short","HEAD"],
stdout=subprocess.PIPE
).communicate()[0].strip()
subprocess.check_output(["git", "status"]) # Refresh index ready for diff-index
if subprocess.call(["git", "diff-index", "--quiet", "HEAD"]):
# Local diffs, give it a plus
gitRevision += "+"
print "Local diffs -- NOT FOR PRODUCTION USE"
else:
gitRemote = subprocess.Popen(["git", "branch", "-r", "--contains", "HEAD"],
stdout=subprocess.PIPE
).communicate()[0].strip()
if gitRemote == "":
# Doesn't exist in upstream branches (i.e., not pushed): give it a plus
gitRevision += "+"
print "Ahead of upstream -- NOT FOR PRODUCTION USE"
gitTag=''
if "+" not in gitRevision:
gitTagPipe = subprocess.Popen(["git", "describe", "--exact-match"],
stdout=subprocess.PIPE)
rawtag = gitTagPipe.communicate()
if gitTagPipe.returncode:
print "Not tagged -- NOT FOR EXTERNAL RELEASE"
else:
gitTag = " - " + rawtag[0].strip()
ei_version = gitRevision + gitTag + " - " + datetime.datetime.now().strftime("%c")
So a build from a known and stable SHA (i.e., one which exists in the golden repository) gets a version looking like this:
92a5ff6 - Fri Feb 7 18:25:04 2014
whereas a local build with a bunch of stuff I haven’t pushed upstream yet gets a version looking like this:
c47c6da+ - Sat Apr 12 20:51:25 2014
with the tell-tale “+” in the SHA. And a version that’s tagged in Git for external release has the tag added to the version string too:
af0f28a - release-27.10 - Fri Dec 13 11:08:38 2013
and that is, in fact, the current format of the string returned on the imp itself by imp.getsoftwareversion() – though we make deliberately very weak guarantees about the format of the string (it’s human-readable, it’s different for different releases) in case it ever needs to change in the future.
Feature branches are usually
short-lived, and indeed usually exist as named branches only in
developers’ local repositories. (With Git, if you merge a branch
locally into master and then push the result, the branching
structure is pushed to origin and becomes part of permanent
history, but the branch name isn’t pushed, and doesn’t appear
in the origin repository except perhaps in the commit comment of the
merge.)
Once a feature on a branch is complete
(and reviewed, and tested), the feature branch can be merged back to
master. This is done by rebasing the feature branch on top of
master, then doing a no-fast-forward (--no-ff)
merge; the thinking behind that style of merge, and full information
and walk-throughs of how to perform one, can be found at 
The heavyweight release process, which we use
for firmware releases, is based mainly on an abundance of caution.
The lightweight release process, which
we use for server releases, is based on responding with alacrity to
new requirements or to current events – for instance, unexpected load
on the servers might require new logging or instrumentation to be
added basically immediately.
Let’s start
by having a look at the desired result. On the right is a screenshot
from gitk, which shows the newest commits at the top. It
shows that the long-lived vendor branch “broadcom” has been merged
following a new delivery of the third-party code, and that the work of
integrating the new release itself happened on a 
At this stage,
history looks like the picture on the right: a bit like the “ready to
merge” stage of a normal bow-shaped branch, except of course with the long tail of the vendor branch heading off into the distant past.
Aha, see on the
right, someone’s beaten us to the punch again. We need to rebase our
work on top of theirs. But here’s the point at which a bit of caution
needs to be applied. Blithely using git rebase as if the
vendor branch weren’t there, will instruct git to rebase the
entire vendor branch on top of master, which will typically be
somewhere between undesirable and complete carnage.
So we’ve now got
a fully rebased merge branch, ready to merge back to master – as at
right. The merge itself is now just like a normal bow-shaped branch:
Let’s look at the desired end-result first. On the
right is a snapshot from gitk --all, showing a feature branch that
I’ve just merged, ready to push. As always with gitk, the
newest commits are shown at the top. Two different routes join the
same two endpoints, spanning the addition of my new feature: one
direct, and another taking many small steps, so that the overall
effect is a bow shape. This means that, though each individual commit
is readily accessible, so too (by comparing the top and bottom of the
bow) is the overall effect of the whole feature. Notice how the
commits are all bunched together, but with a limited, local amount of
branching: a run of N such branches consecutively would still, in some
sense, have O(1) complexity, not O(N). (At least, it would have O(1)
concurrently-active development branches, which is believably a
useful measure of the complexity, and the comprehensibility burden, of
history.)
So how to build history graphs that look like that? On the left is
gitk’s view of my local repository after doing some
work on a feature: in this case a feature made up of seven
commits. (
(the branch itself won’t be pushed upstream, but its name
appears in the merge message, so it ought to be fairly
descriptive):
Then I need to wind back my local master branch to match
origin/master:
Of course, git rebase master might
produce conflicts, which should be cleared up in the usual way. (And
if the conflicts are too hard to clear up, maybe a real merge is
called-for, so that history records that the project had a hard time
here.) But if all goes smoothly, I end up with the picture on the
right: a feature branch ready to merge, as above
– the only difference being that the branch is now on top
of a newer commit as the head of origin/master. So I can
carry on from that point.
