Previously on #rust:
|
First, set up Ubuntu 22.04 with USB boot and headless full-disk encryption in the usual way.
OK, perhaps doing so isn’t all that everyday (that remark was slightly a dig at Larousse Gastronomique, at least one recipe in which starts, “First, make a brioche in the usual way”). But this very blog has exactly the instructions you need. This time, instead of a 512GB Samsung USB SSD, I used a 32GB Sandisk USB flash drive — the test-runner won’t be needing tons of storage.
Alternatively, you could use any other reasonable computer you’ve got lying around — an Intel NUC, say, or a cast-off laptop. Most or all of the following steps will apply to any freshly-installed Ubuntu box, and many of them to other Linux distributions too. But you’ll have the easiest time of it if your CI test-runner has the same architecture and OS as your main CI server, which is another reason I’m pressing yet another Raspberry Pi into service here.
However you get there, what you need to proceed with the rest of this post (at least the way I proceeded) is:
- An existing Laminar CI server, on your local Ethernet.
- A Raspberry Pi to become the test-runner;
- with Ubuntu 22.04 freshly installed;
- plugged into your local Ethernet, able to DHCP (I gave mine a fixed address in OpenWRT’s DHCP server setup) — or, at least, somewhere where the CI server can SSH to it;
- with your own user, that you can SSH in as, and use sudo.
- A USB-to-Ethernet adaptor (I used the Amazon Basics one);
- an Ethernet switch, at least 100Mbit (I used this TP-Link LS1008);
- and an STM32F746-Nucleo development board.
In this blog post we’re going to:
- Connect everything together, making a separate test network
- Automate the remaining setup of the test-runner, using Ansible
- Arrange that the CI server can SSH to the test-runner autonomously
- Run a trivial CI job on the test-runner
- Package up the system-tests and run a real CI job on the test-runner
- Go and sit down and think about what we’ve done
1. Connect everything together, making a separate test network
For once, instead of being a whimsical stock photo of something tangentially related, the image at the top of this blog post is an actual photograph of the actual physical items being discussed herein (click to enlarge if need be). The only connections to the outside world are from my home network to the Raspberry Pi’s built-in Ethernet (lower left) and power to the Raspberry Pi and to the Ethernet switch (top left and centre bottom). The test network is otherwise self-contained: the STM32 development board is on a private Ethernet segment with just the Raspberry Pi’s USB Ethernet for company. This network has its own RFC1918 address range, 192.168.3.x, distinct from the rest of the home network. (The Wifi interface on the Raspberry Pi is not currently being used.)
The breadboard is attached to the Raspberry Pi’s GPIO connector, and at present is only used to provide a “testing in progress” LED attached to GPIO13 (glowing white in the photo, but therefore hard to see). GPIO usage could become more sophisticated in the future: for instance, if I was writing a HAL crate for a new embedded device, I could connect the Raspberry Pi’s GPIO inputs to the GPIO outputs of the embedded device (and vice-versa) and system-test my GPIO code.
The Raspberry Pi programs the STM32 over USB; apart from that and the Ethernet adaptor, the USB flash drive also takes up one of the four USB sockets, leaving just one spare for future enhancements. (But hubs are a thing, of course.)
2. Automate the remaining setup of the test-runner, using Ansible
The initial setup of a new Ubuntu installation is arguably best done manually, as you might need to react to things that the output of the commands is telling you. But once the basics are in place, the complexities of setting up everything else that the test-runner needs, are best automated — so they can be repeatable, documented, and explainable to others (in this case: you).
Automating server setup is not new news to cloud engineers, who often use tools such as Chef or Puppet to bring one or more new hosts or containers up-to-speed in a single command. Electric Imp used Chef, which I never really got on with, partly because of the twee yet mixed metaphors (“Let’s knife this cookbook — solo!”, which is a thing all chefs say), but mostly because it was inherently bound up with Ruby. Yet I felt I needed a bit more structure than just “copy on a shell script and run it”. So for #homelab purposes, I thought I’d try Ansible.
Ansible is configured using YAML files, which at least is one step up from Ruby as a configuration language. The main configuration file is called a “playbook”, which contains “plays” (think sports, not theatre), which are in turn made up of individual “tasks”. A task can be as simple as executing a single shell command, but the benefit of Ansible is that it comes with a huge variety of add-ons which allow tasks to be written in a more expressive way. For instance, instead of faffing about to automate cron, there’s a “cron” add-on which even knows about the @reboot directive and lets you write:
- name: init gpios on startup cron: name: init-gpios special_time: reboot job: /usr/local/bin/init-gpios
Tasks can be linked to the outcome of earlier tasks, so that for instance it can restart the DHCP server if, and only if, the DHCP configuration has been changed. With most types of task, the Ansible configuration is “declarative”: it describes what the situation ought to be, and Ansible checks whether that’s already the case, and changes things only where they’re not as desired.
Ansible can be used for considerably more complex setups than the one in this blog post — many, many servers of different types that all need different things doing to them — but I’ve made an effort at least to split up the playbook into plays relevant to basically any machine (hosts: all), or ones relevant just to the Raspberry Pi (hosts: raspberrypis), or ones specific to the rôle of being a system-test runner (hosts: testrunners).
I ended up with 40 or so tasks in the one playbook, which between them install all the needed system packages as root, then install Rust, Cargo and probe-rs as the laminar user, then set up the USB Ethernet adaptor as eth1 and run a DHCP server on it.
Declaring which actual machines make up “all”, “testrunners”, etc., is the job of a separate “inventory” file; the one included in the repository matches my setup at home. The inventory file is also the place to specify per-host information that shouldn’t be hard-coded into the tasks: in this case, the DHCP setup tasks need to know the MAC address of the USB Ethernet adaptor, but that’s host-specific, so it goes in the inventory file.
All the tasks in the main YAML file are commented, so I won’t re-explain them here, except to say that the “mark wifi optional”, “rename eth1”, and “set static IP for eth1” tasks do nothing to dispel my suspicion that Linux networking is nowadays just a huge seven-layer dip of xkcd-927 all the way down, with KDE and probably Gnome only adding their own frothy outpourings on top.
I added a simple Makefile to the systemtests/ansible directory, just to automate running the one playbook with the one inventory.
The name “Ansible” comes originally from science-fiction, where it’s used to mean a device that can communicate across deep space without experiencing speed-of-light latency. I only mention this because when communicating across about a metre of my desk, it’s bewilderingly slow — taking seconds to update each line of sshd_config. That’s about the same as speed-of-light latency would be if the test-runner was on the Moon.
But having said all that, there’s still value in just being able to re-run Ansible and know that everything is set consistently and repeatably.
I did wonder about running Ansible itself under CI — after all, it’s software, it needs to be correct when infrequently called upon, so it ought therefore to be automatically tested to avoid bugs creeping in. But running Ansible needs root (or sudo) access on the test-runner, which in turn means it needs SSH access to the test-runner as a user which is capable of sudo — and I don’t want to leave either of those capabilities lying around unencrypted-at-rest in CI. So for the time being it’s down to an unreliable human agent — that’s me — to periodically run make in the ansible directory.
3. Arrange that the CI server can SSH to the test-runner autonomously
Most of the hosts in this series of #homelab posts are set up with, effectively, zero-trust networking: they can only be accessed via SSH, and all SSH sessions start with me personally logged-in somewhere and running ssh-agent (and using agent-forwarding). But because the CI server needs to be able to start system-test jobs on the test-runner, it needs to be able to login via SSH completely autonomously.
This isn’t as alarming as it sounds, as the user it logs into (the laminar user on the test-runner) isn’t very privileged; in particular, it’s not in the sudo group and thus can’t use sudo at all. (The Ansible setup explicitly grants that user permissions to the hardware it needs to access.)
Setting this up is a bit like setting up your own SSH for the first time. First generate a key-pair:
ssh-keygen -t ed25519
— when prompted, give the empty passphrase, and choose “laminar-ssh” as the output file. The public key will be written to “laminar-ssh.pub”.
The public key needs to be added to ~laminar/.ssh/authorized_keys (American spelling!) on the test-runner; the Ansible setup already does this for my own CI server’s public key.
Once authorized_keys is in place, you can test the setup using:
ssh -i laminar-ssh laminar@scotch
Once you’re happy that it works, copy the file laminar-ssh as ~laminar/.ssh/id_ed25519 on the main CI server (not the test-runner!):
sudo cp ~peter/laminar-ssh ~laminar/.ssh/id_ed25519 sudo chown -R laminar.laminar ~laminar/.ssh sudo chmod 0700 ~laminar/.ssh
You can test that setup by using this command on the CI server (there should be no password or pass-phrase prompt):
sudo -u laminar ssh scotch
— indeed, you need to do this at least once, in order to reassure the CI server’s SSH client that you trust the test-runner’s host key.
4. Run a trivial CI job on the test-runner
Now that the laminar user on the CI server can SSH freely to scotch, what remains is mostly Laminar setup. This part is adapted very closely from Laminar’s own documentation: we set up a “context” for the test-runner, separate from the default context used by all the other jobs (because the test-runner can carry on when other CPU-intensive jobs are running on the CI server), then add a remote job that executes in that context.
| /var/lib/laminar/cfg/contexts/test-runner-scotch.conf |
|---|
EXECUTORS=1 |
| /var/lib/laminar/cfg/contexts/test-runner-scotch.env |
|---|
RUNNER=scotch |
The context is partly named after the test-runner host, but it also includes the name of the test-runner host as an environment variable. This means that the job-running scripts don’t need to hard-code that name.
As before, the actual job file in the jobs directory defers all the complexity to a generic do-system-tests script in the scripts directory:
| /var/lib/laminar/cfg/jobs/cotton-system-tests-dev.run |
|---|
#!/bin/bash -xe exec do-system-tests |
In keeping with the Laminar philosophy of not poorly reinventing things that already exist, Laminar itself has no built-in support for running jobs remotely — because that’s what SSH is for. This, too, is closely-inspired by the Laminar documentation:
| /var/lib/laminar/cfg/scripts/do-system-tests |
|---|
#!/bin/bash -xe ssh laminar@$RUNNER /bin/bash -xe << "EOF" echo 1 > /sys/class/gpio/gpio13/value uname -a run-parts /etc/update-motd.d sleep 20 echo 0 > /sys/class/gpio/gpio13/value EOF |
This, of course, doesn’t run any actual tests, but it provides validation that the remote-job mechanism is working. The LED attached to GPIO13 on the test-runner Raspberry Pi serves as the “testing in progress” indicator. (And the run-parts invocation is an Ubuntu-ism: it reproduces the “message of the day”, the welcome message that’s printed when you log in. Most of it is adverts these days.)
Tying the job to the context is the $JOB.conf file:
| /var/lib/laminar/cfg/jobs/cotton-system-tests-dev.conf |
|---|
CONTEXTS=test-runner-* |
Due to the judicious use of a wildcard, the job can run on any test-runner; my setup only has the one, but if you found yourself in a team with heavy contention for the system-test hardware, this setup would allow you to build a second, identical Raspberry Pi with all the same hardware attached — called shepherds, maybe — and add it as a separate Laminar context. Because Laminar runs a job as soon as any context it’s fit for becomes available, this would automatically split queued system-test jobs across the two test-runners.
With all of this in place, it’s time to trigger the job on the CI server:
laminarc queue cotton-system-tests-dev
After a few teething troubles, including the thing I mentioned above about making sure that the SSH client accepts scotch’s host key, I was pleased to see the “testing in progress” LED come on and the message-of-the-day spool out in the Laminar logs.
5. Package up the system-tests and run a real CI job on the test-runner
We didn’t come here just to read some Ubuntu adverts in the message-of-the-day. Now we need to do the real work of building Cotton for the STM32 target, packaging-up the results, and transferring them to the test-runner where they can be run on the target hardware. First we build:
| /var/lib/laminar/cfg/jobs/cotton-embedded-dev.run |
|---|
#!/bin/bash -xe
PROJECT=cotton
RUST=stable
BRANCH=${BRANCH-main}
SOURCE=/var/lib/laminar/run/$PROJECT/workspace
(
flock 200
cd $SOURCE/$PROJECT
git checkout $BRANCH
cd -
cp -al $SOURCE/$PROJECT $PROJECT
) 200>$SOURCE/lock
source $HOME/.cargo/env
rustup default $RUST
cd $PROJECT
cargo build -p systemtests -F arm,stm32f746-nucleo --all-targets
cargo test --no-run -p systemtests -F arm,stm32f746-nucleo 2> $ARCHIVE/binaries.txt
grep "Executable tests/" $ARCHIVE/binaries.txt | cut -d'(' -f 2 | cut -d')' -f 1 > binaries.2.txt
tar cf $ARCHIVE/binaries.tar `find cross/*/target -type f -a -executable \
| grep -v /deps/ | grep -v /build/` `cat binaries.2.txt`
laminarc queue cotton-system-tests-dev PARENT_RUN=$RUN
exec prune-archives cotton-embedded-dev 10 |
The actual build commands look much like many of the other Laminar jobs but with the extra Cargo features added which enable the cross-compiled targets; the interesting parts of this script come once the cargo build is done and the results must be tarred-up ready to be sent to the test-runner.
Finding all the target binaries is fairly easy using find cross/*/target, but we also need to find the host binary from the systemtests package. The easiest way to do that is to parse the output of cargo test --no-run, which includes lines such as:
Compiling systemtests v0.0.1 (/home/peter/src/cotton/systemtests)
Finished test [unoptimized + debuginfo] target(s) in 2.03s
Executable unittests src/lib.rs (target/debug/deps/systemtests-4a9b67de54149231)
Executable tests/device/main.rs (target/debug/deps/device-cfdcb3ff3e5eaaa5The line with “Executable tests” is the one we’re looking for. (The string of hex digits after the executable name changes every time the sources change.) It’s possible that we could “cheat” here and just pick the first file we find starting with target/debug/deps/device-, as this is CI so we’re always building from clean — but this is a more robust way of determining the most recent binary.
(You might feel that this section is a bit of a cop-out, a bit white-box: knowing that there’s only one host-side binary does make the packaging a lot easier. If there were a lot of host-side binaries to package, and this technique started creaking at the seams, I’d look into cargo-nextest which has features specifically designed for packaging and unpackaging suites of Cargo tests.)
Once everything the system-tests job will need is stored in $ARCHIVE/binaries.tar, we can trigger the system-tests job — making sure to tell it, in $PARENT_RUN, which build in the archive it should be testing. (Initially I had the system-tests job use “latest”, but that’s wrong: it doesn’t handle multiple queued jobs correctly, and has a race condition even without queued jobs. The “latest” archive is that of the most recent successfully-finished job — but the build job hasn’t yet finished at the time it triggers the test job.)
The final prune-archives command is something I added after the initial Laminar writeup when some of the archive directories (particularly doc and coverage) started getting big: it just deletes all but the most recent N non-empty archives:
| /var/lib/laminar/cfg/scripts/prune-archives |
|---|
#!/bin/bash -xe
PROJECT=$1
KEEP=${2-2}
cd /var/lib/laminar/archive/$PROJECT
for d in `find * -maxdepth 0 -type d -a \! -empty | sort -n | head -n -$KEEP`; do
rm -r /var/lib/laminar/archive/$PROJECT/$d
done |
No-one likes deleting data, but in this case older archives should all be recoverable at any time, if the need arises, just by building the source again at that revision.
Now the cotton-system-tests-dev.run job needs to pass the $PARENT_RUN variable on to the underlying script:
| /var/lib/laminar/cfg/jobs/cotton-system-tests-dev.run |
|---|
#!/bin/bash -xe exec do-system-tests cotton-embedded-dev $PARENT_RUN |
and the do-system-tests script can use it to recover the tarball and scp it off to the test-runner:
| /var/lib/laminar/cfg/scripts/do-system-tests |
|---|
#!/bin/bash -xe
PARENT_JOB=$1
PARENT_RUN=$2
scp /var/lib/laminar/archive/$PARENT_JOB/$PARENT_RUN/binaries.tar laminar@$RUNNER:
ssh laminar@$RUNNER /bin/bash -xeE << "EOF"
echo 1 > /sys/class/gpio/gpio13/value
cleanup_function() {
echo 0 > /sys/class/gpio/gpio13/value
exit 1
}
trap 'cleanup_function' ERR
export PS4='+ \t '
export PATH=/home/laminar/.cargo/bin:$PATH
rm -rf tests
mkdir -p tests/systemtests
( cd tests
tar xf ../binaries.tar
cd systemtests
export CARGO_MANIFEST_DIR=`pwd`
../target/debug/deps/device-* --list
../target/debug/deps/device-* --test
)
echo 0 > /sys/class/gpio/gpio13/value
EOF |
The rest of the script has also gained in features and complexity. It now includes a trap handler to make sure that the testing-in-progress LED is extinguished even if the tests fail with an error. (See here for why this requires the -E flag to /bin/bash.)
The script goes on to add timestamps to the shell output (and thus to the logs) by adding \t to PS4, and add the Cargo bin directory to the path (because that’s where probe-rs got installed by Ansible).
The tests themselves need to be executed as if from the systemtests directory of a full checkout of Cotton — which we don’t have here on the test-runner — so the directories must be created manually. With all that in place, we can finally run the host-side test binary, which will run all the device-side tests including flashing the STM32 with the binaries, found via relative paths from $CARGO_MANIFEST_DIR.
That’s a lot, but it is all we need to successfully list and run all the device-side tests on our test-runner. Here’s (the best part of) the Laminar logs from a successful run:
+ 16:46:54 ../target/debug/deps/device-453652d9c9dda7c1 --list stm32f746_nucleo::arm_stm32f746_nucleo_dhcp: test stm32f746_nucleo::arm_stm32f746_nucleo_hello: test stm32f746_nucleo::arm_stm32f746_nucleo_ssdp: test 3 tests, 0 benchmarks + 16:46:54 ../target/debug/deps/device-453652d9c9dda7c1 --test running 3 tests test stm32f746_nucleo::arm_stm32f746_nucleo_dhcp ... ok test stm32f746_nucleo::arm_stm32f746_nucleo_hello has been running for over 60 seconds test stm32f746_nucleo::arm_stm32f746_nucleo_ssdp has been running for over 60 seconds test stm32f746_nucleo::arm_stm32f746_nucleo_ssdp ... ok test stm32f746_nucleo::arm_stm32f746_nucleo_hello ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 66.37s + 16:48:00 echo 0
And now that everything’s working, we can add it to the chain of events that’s triggered whenever a branch is pushed to the CI server:
| /var/lib/laminar/cfg/job/cotton-dev.run |
|---|
#!/bin/bash -xe
BRANCH=${BRANCH-main}
do-checkout cotton $BRANCH
export LAMINAR_REASON="built $BRANCH at `cat git-revision`"
laminarc queue \
cotton-embedded-dev BRANCH=$BRANCH \
cotton-doc-dev BRANCH=$BRANCH \
cotton-grcov-dev BRANCH=$BRANCH \
cotton-msrv-dev BRANCH=$BRANCH \
cotton-beta-dev BRANCH=$BRANCH \
cotton-nightly-dev BRANCH=$BRANCH \
cotton-minver-dev BRANCH=$BRANCH
|
If you’re wondering about the -dev suffix on all those jobs: I set up Laminar with two parallel sets of identical build jobs. There’s the ones with -dev which are triggered when pushing feature branches, and the ones without the -dev which are triggered when pushing to main. This is arranged by the Git server post-receive hook:
| git/cotton.git/hooks/post-receive |
|---|
#!/bin/bash -ex
while read oldrev newrev ref
do
if [ "${ref:0:11}" == "refs/heads/" -a "$newrev" != "0000000000000000000000000000000000000000" ];
then
export BRANCH=${ref:11}
export LAMINAR_REASON="git push $BRANCH"
if [ "$BRANCH" == "main" ];
then
laminarc queue cotton BRANCH=$BRANCH
else
laminarc queue cotton-dev BRANCH=$BRANCH
fi
fi
done |
In a sense there’s no engineering need for this duplication: exactly the same actual work is undertaken in either case. But there’s a social need: feature branches (with -dev) aren’t expected to always pass all tests — indeed, such branches are often pushed for the express purpose of determining whether or not they pass all tests. But the main branch is expected to pass all tests, all the time, and a regression is to be taken seriously. That is: if the cotton-dev build or one of its downstreams fails, that’s got a very different implication from the cotton build or one of its downstreams failing. The cotton build itself should enjoy long, uninterrupted streaks of regression-free passes (and indeed it does; the last failure was in November 2023 due to this issue causing intermittent unit-test failures).
6. Go and sit down and think about what we’ve done
Well, what have we done? We have, over the course of three blog posts, taken a bit of knowledge of bash, cron, and SSH that we already had, then gone and learned a bit about Laminar, Ansible, and Cargo, and the “full stack” that we then engineered for ourselves using all that knowledge is this: any time we push a feature branch, we (eventually) get told automatically, as a straight yes/no answer, whether it’s OK for main or not. That’s an immensely powerful thing to have, an immensely useful property to have an oracle for.
Having that facility available is surely expected these days in other areas of software engineering — where test-harnesses are easier to create — but I hope I’ve demonstrated in these blog posts that even those working on embedded systems can also enjoy and benefit from the reliability (of main in particular) that’s due to this type of workflow.
(Other versions of the workflow could be constructed if preferred. Perhaps you don’t want every push to start a system-test sequence — in that case, you could either write a more complex Git post-receive hook, or set up an alternative Git remote, called perhaps “tests”, so that pushing to that remote kicked off the test sequence. Or you could tie the test sequence in to your pull-request process somehow.)
To repeat a line from the previous instalment, any software you need to test can be tested automatically in some way that is superior to not testing it at all. Compared to Rust’s unit tests, which are always just a cargo test away, it took us three quite detailed blog posts and a small pile of physical hardware to get us to the stage where this embedded code could be tested automatically. If I had to distill the message of these three posts from their ~11,000 words down to just six, they’d be: this sort of effort is worthwhile. If your product is, or runs on, a specific platform or piece of hardware, it’s worth spending a lot of effort arranging to test automatically on the actual target hardware, or as near to the actual target as is practical. (Sometimes embedded products are locked-down, fused-off, potted, or otherwise rendered inaccessible; testing, in that case, requires access to unlocked variants.)
That is to say: is the effort worth it for the cotton-ssdp crate — a few thousand lines of code, about 60% of which is already tests, and the rest of which has 100% test coverage? Arguably yes, but also arguably no, especially as almost all of cotton-ssdp can be tested in hosted builds. The cotton-ssdp crate has acted here more as a spike, a proof-of-concept. But the point is, the concept was proved, a baseline has now been set, and all the right testing infrastructure is in place if I want to write a power-aware RTOS, or implement HAL crates for some of these weird development boards in my desk drawer, or if I want to disrupt the way PAC crates are generated in order to improve the testing story of HAL crates. Now when I start doing those things, I can start defending the functionality with system-tests from the outset. If I want to do those more complex, more embedded-centric things — which I do — then all the effort expended so far will ultimately be very beneficial indeed. If you, too, aim to do complex or embedded-centric things, then similar levels of effort will benefit your projects.
6.1 War stories from the front lines of not-system-testing
I have some war stories for you. I heard tell of a company back in the day whose product, a hardware peripheral device, was (for sound commercial reasons) sold as working with numerous wonky proprietary Unixes. But for some of the more obscure platforms, there had been literally no automated testing: a release was declared by development, it was thrown over the wall into the QA department, and in due course a human tester would physically insert the CD-R into one of these wonky old machines and manually run through a test script ensuring that everything appeard to work. This was such a heavyweight process that it was run very rarely — meaning that, if an issue was found on, say, AIX, then the code change that caused it probably happened months ago and with significant newer work built on top of it. And of course such a discovery at “QA time” meant that the whole, lumbering manual release process had to be started again from scratch once the issue was fixed. This was exactly the pathology that CI was invented to fix! I mean, I’m pretty sure Laminar doesn’t support antediluvian AIX versions out of the box, but given the impact of any issues on release schedules, it was definitely worth their putting in quite significant development effort to bring the installation process under CI — automatically on main (at least nightly, if not on every push), and by request on any feature branch. (Developers need a way to have confidence they haven’t broken AIX before merging to main.) They should have asked themselves, “What can be done to automate testing of the install CD, in some way that is superior to not testing it at all?” — to place it under the control of a CI test-runner, as surely as the STM32F746-Nucleo is under the control of this Raspberry Pi? Well — what’s the simplest thing that can act as a fake CD-ROM drive well enough to fool an AIX box? Do those things have USB host? Can you bitbang SCSI-1 target on a Raspberry Pi? Would a BlueSCSI help? Or even if they were to decide that “CI-ing” the actual install CD image is too hard — how often are the failures specifically CD-related? Could they just SSH on the installer tarballs and test every other part of the process? How did this pathology last for more than one single release?
I also heard tell of a different company whose product was an embedded system, and was under automated test, including before merging to main — but following a recent “urgent” porting exercise (again for sound commercial reasons), many of the tests didn’t pass. The test harness they used supported marking tests as expected-failure — but no-one bothered doing so. So every test run “failed”, and developers had to manually pore over the entire list of test results to determine whether they’d regressed anything. In a sense the hard part of testing was automated, but the easy part not! This company had put in 99% of the effort towards automated testing, but were reaping just a tiny fraction of the benefits, because the very final step — looking at the test results and turning them into a yes/no for whether the code is OK for main — was not automated. How did this pathology last for more than one single afternoon?
6.2 People
The rhetorical-looking questions posed above about the AIX CI and the expected-fail tests (“How did these obviously wrong situations continue?”) did in fact have an answer in the real world. Indeed, it was the same answer in both cases: people.
In the first company, the head of QA presided over a large and (self-)important department — which it needed to be in order to have enough staff to manually perform quite so much automatable work. If QA was run with the attitude that human testers are exploratory testers, squirrelers-out of corner-cases, stern critics of developers’ assumptions — if the testers’ work-product, including during early-development-phase involvement, was more and better automated tests — then they probably wouldn’t need anything like so many testers, and the rôle of “Head of QA” would perhaps be viewed as a less big cheese than hitherto. Although the product quality would benefit, the company’s bottom-line would benefit, and even the remaining testers’ career-progression would benefit — the Head of QA’s incentives were misaligned with all of that, and they played the game they were given by the rules that they found to be in effect.
The second company seems harder to diagnose, but fundamentally
the questions are, “Who is in charge of setting the
quality bar for merges to main?” and “Who is in
charge of what happens when that bar is not met?”. Those
are likely to be two different people — they require very
different skills — but if you find that a gulf is opening up
between your team’s best practices and your
team’s typical practices, then both those people are needed
in order to bring the two closer together again. (In my own career
I’ve a few times acted as the first one, but I’ve never acted as
the second one: as Dr. Anthony Fauci famously didn’t say,
“I
don’t know how to explain to you that you should care
for other people software quality.”)
This post, and this blog, and this blogger, cannot help you to deal with people. But often the talking point of the people you need to convince (or whose boss you need to convince to overrule them) is that better automated testing isn’t technologically feasible, isn’t worth attempting. I hope I’ve done a little to dispel that myth at least.
No comments:
Post a Comment