System-testing embedded code in Rust, part two: Things I learned testing SSDP

Previously on #rust:

• Self-hosted CI for Rust and C++ using Laminar
• Things I learned writing an SSDP implementation in Rust
• Rust crate release checklist
• System-testing embedded code in Rust 1: Infrastructure

With the basic system-test infrastructure now in place thanks to the previous post in this series, it’s time to wire the STM32F746-Nucleo development board up to Ethernet and start testing actual code: the cotton-ssdp crate.

Having said that, in fact the first thing to do after plugging-in Ethernet, is to write a test that verifies basic connectivity. If packets can’t flow between the Nucleo and the rest of the network for any reason, then there’s no point disparaging the SSDP code for its failure to communicate. The basic test will establish that simple networking is operational: that the Ethernet interface sees link (i.e., that the Ethernet cable is actually connected to the Nucleo, and also to something at the other end), and also that DHCP can succeed and the Nucleo obtain an IP address.

And of course writing that code isn’t throwaway effort: every other network-related test will need to do all those things first before getting on with more specific tasks. So the setup code will form part of all subsequent SSDP test binaries.

As always, I encountered problems along the way, because Rust. But all of those problems eventually had solutions, also because Rust.

The aim of the tests

It’s worth just going over what the goals are here. Why spend development time on writing these tests, and on cabling up these test rigs? What is the payback?

My answer is, that I’d like to be able to work on cotton-ssdp, and eventually other similar crates, knowing that I have automated testing that verifies new functionality and defends against regressions in existing functionality. When I push a new feature branch to the CI server, I want it to tell me, as straightforwardly and clearly as possible, whether or not my changes are OK for main.

I don’t think the testing constitutes a promise that the code is completely bug-free (even less, that the functionality it provides is objectively useful). But it does, at the very least, constitute a promise that certain types or certain show-stopping severities of bug are absent. Tom DeMarco, writing in Peopleware, describes Gilb’s Law: “Anything you need to quantify can be measured in some way that is superior to not measuring it at all”. Something similar applies here: any software you need to test can be tested automatically in some way that is superior to not testing it at all.

As a target, “superior to doing nothing” is not a very high bar to clear. These system-tests aren’t very comprehensive in terms of, say, line coverage of the cotton-ssdp crate. But then the crate, after all, is thoroughly unit-tested. These system-tests are more about testing the platform integration code — here, with the RTIC real-time operating system, and the smoltcp networking stack — and about systematically verifying the crate’s original goal of being useful to implementers of embedded systems.

Concretely, the tests presented here really just check that the device can discover resources on the network, and advertise its own resources. Once that two-way communication is proven to work, everything else about exactly what is communicated, is already covered by the unit tests.

I’m not saying that the existence of a unit test automatically renders useless any system testing of the same function. Unit tests and system tests have different (human, organisational) readership — typically, unit tests are only interesting to developers, whereas system tests are often high-level enough and visible enough to serve as technology demonstrations to project managers and beyond — and both audiences are entitled to ask for and to see evidence of all claimed functionality. But in this case, the developers, project managers and beyond are all me, and anyway adding a huge variety of tests would clog up the narrative of this blog post, which focuses more on describing the framework.

Being a good citizen of Ethernet: MAC addresses

We’ll need to start by getting this Nucleo board onto the local Ethernet. In order to participate in SSDP, it’s going to need an IP address, as handed out by the DHCP server in my router. But in order to even participate in Ethernet enough to communicate with the DHCP server, it’s going to need its own Ethernet address. This is also called a hardware address or MAC address, it’s 48 bits (six bytes) long, every device on an Ethernet network has one, and it’s often printed on the back of routers and suchlike as twelve hex digits separated by colons. Some networking hardware comes with an officially-allocated MAC address built-in (as a company, you can get ranges of them allocated to you, like IP addresses) — but STM32s don’t, probably because ST Micro sell a lot of STM32s, many of them into designs (such as the Electric Imp) where they never even use their Ethernet circuitry, and it’d be a waste of a finite resource for ST Micro to allocate each one its own MAC address from the fixed pool.

For our purposes it’d be overkill to get an official address block allocated (though you’d need to if you intended to sell actual products), so it’s fortunate that an alternative way of obtaining an address is possible. One of those 48 bits is set to zero in every official (“Universally Administered”) address, but can be set to one to indicate a “Locally Administered” address, i.e. one chosen by the local network administrator. Which is also me! So in the tests, we set that bit, then pick a device-specific value for the other 47 bits (in fact 46 as there’s another reserved one), and use the result as our MAC address. So long as the 46 device-specific bits are chosen randomly enough, the chance of an accidental collision is suitably negligible.

So we need a calculation that always provides the same answer when performed on the same device, but always provides different answers when performed on different devices. Fortunately each STM32 does include a unique chip ID, burned into each individual die at chip-manufacture time, as described in the STM32F74x reference manual (“RM0385”) section 41.1.

But it’s not a good idea to just use the raw chip ID as the MAC address, for several reasons: it’s the wrong size, it’s quite predictable (it’s not 96 random bits per chip, it encodes the die position on the wafer, so two different STM32s might have IDs that differ only in one or two bits, meaning we can’t just pick any 46 bits from the 96 in case we accidentally pick seldom-changing ones) — and, worst of all, if anyone were to use the same ID for anything else later, they might be surprised if it were very closely correlated with the device’s MAC address.

So the thing to do, is to hash the unique ID along with a key, or salt, which indicates what we’re using it for. You can see this on Github or right here:

pub fn stm32_unique_id() -> &'static [u32; 3] { // SAFETY: this address only valid when running on STM32 unsafe { let ptr = 0x1ff0_f420 as *const [u32; 3]; &*ptr } } pub fn unique_id(salt: &[u8]) -> u64 { let id = stm32_unique_id(); let key1 = (u64::from(id[0]) << 32) + u64::from(id[1]); let key2 = u64::from(id[2]); let mut h = siphasher::sip::SipHasher::new_with_keys(key1, key2); h.write(salt); h.finish() } pub fn mac_address() -> [u8; 6] { let mut mac_address = [0u8; 6]; let r = unique_id(b"stm32-eth-mac").to_ne_bytes(); mac_address.copy_from_slice(&r[0..6]); mac_address[0] &= 0xFE; // clear multicast bit mac_address[0] |= 2; // set local bit mac_address }

This is quite a general mechanism for producing device-specific but deterministic unique IDs; it’s also used by the SSDP test for generating the UUID for the example resource. And it can be made more sophisticated by hashing-in more information. Want a unique, but deterministic, Wifi MAC address that’s per-network to defeat tracking? Hash in the Wifi network name, or the router's MAC address. (It’s harder to avoid tracking on Ethernet, because you need the MAC address before you even start using DHCP, i.e. before you know anything about the network you’re joining. But of course passive tracking isn’t really a threat model on Ethernet, where you must have chosen to plug in the cable.) Want different UUIDs for different UPnP services? Hash in the service name. One thing you mustn’t do, though, is to use these IDs as cryptographic identifiers, as the hash function in question hasn’t been analysed for the collision-resistance and irreversibility properties which you need for such identifiers.

Spike then refactor: Adding smoltcp support

Getting the cotton-ssdp crate tested on embedded devices, was always going to involve porting it to use smoltcp as an alternative to the standard-library socket implementation that it currently uses. I fondly imagined that the result would look like the idealised “triangle of initialisation”:

fn main() { let mut a = A::new(); let mut b = B::new(&a); let mut c = C::new(&a, &b); let mut d = D::new(&a, &b, &c); ... do_the_thing(&a, &b, &c, &d, ...); }

so, in this case, perhaps:

fn main() { let mut stm32 = Stm32::<STM32::F746, STM32::Power::Reliable3V3>::default(); let mut smoltcp = Smoltcp::new(stm32.ethernet()); run_dhcp_test(&mut stm32, &mut smoltcp); }

I didn’t quite get to that level of neatness — there’s a lot of boilerplate in most embedded software, and anyway that pseudocode appears to allocate everything on the stack, where overruns are a runtime failure, as opposed to being in the data segment, where overrun would be (as an improvement) a link-time failure.

But the first version of the DHCP test was over 550 lines of boilerplate, so most of the commits on the pdh-stm32-ssdp branch are just about tidying away the common code to make the intent of the test more obvious.

Once DHCP was working, implementing a simple test for SSDP could commence. (And adding a second network-related test spurred the factoring-out of yet more common code between the two.) Rather than immediately wade in to changing cotton-ssdp itself, though, I was able to use the exposed networking traits which cotton-ssdp already contained, to start implementing smoltcp support directly in the test. This was a fortuitous case of “pre-adaptation”: the abstractions that let cotton-ssdp’s core be agnostic on the matter of mio versus tokio, turned out to be exactly those needed for also abstracting away smoltcp. (Well, okay, that wasn’t completely fortuitous, as I did have embedded systems in mind even when developing hosted cotton-ssdp.)

Once the test was working, I could then move those trait implementations into cotton-ssdp, hiding them behind a new Cargo feature smoltcp in order not to introduce needless new dependencies for those using cotton-ssdp on hosted platforms.

I’m not sure I’m quite your (IPv4) type

If you look at the cotton-ssdp additions for smoltcp support, you’ll see that only a small part of it (lines 220-300) is the actual smoltcp API usage. A much larger part is taken up by conversions between types representing IP addresses.

The issue is, that networking APIs are typically system-specific, and not present on many embedded systems, so the Rust people very sensibly and usefully left those functions out of the embedded, no_std configuration. But, a bit less usefully for our purposes, they also left out the types representing IPv4 and IPv6 addresses. These types are not platform-specific — they’re straight from the RFCs — but, as they weren’t available to no_std builds of smoltcp, the smoltcp people were forced to invent their own versions.

Subsequently a crate called “no-std-net” was released, containing IP address types (structurally) identical to the standard-library ones (when built with no_std and just renaming the standard-library ones when built hosted. The cotton-ssdp crate uses the no-std-net names.

Now in fairness the Rust people did then realise that this situation wasn’t ideal, and standardised types are set to land in Rust 1.77 — but that’s much newer than most people’s minimum-supported-Rust-version (MSRV), and anyway tons of smoltcp users in the field are still using the smoltcp versions. So conversions were necessary.

Rust has very well-defined idioms for type conversion, via implementing the From trait — so on the face of it, all that’s needed is to implement From for the standard types on the smoltcp types, and vice versa. That doesn’t work, though, because of Rust’s “Orphan Rule”, which allows trait implementations only in the crate defining the trait or the one defining the type being extended — and cotton doesn’t define From, std, or smoltcp. The best we can do, it seems, is to invent yet another set of “Generic” IP address types so that we can define the conversions both ways. This leads to lots of double-conversions; stm32f746-nucleo-ssdp-rtic has the likes of

no_std_net::IpAddr::V4(GenericIpv4Address::from(ip).into())

and

GenericSocketAddr::from(sender.endpoint).into()

but it keeps the core code looking sane.

Parts of the generic-types code are more complex than I’d hoped because the smoltcp stack itself can be built either with or without IPv6 (and IPv4 for that matter), using Cargo features. As it stands, cotton-ssdp only uses IPv4, so it depends on smoltcp with the IPv4 feature. But users of cotton-ssdp will of course be using it in some wider system, and, because of the way Rust’s crate feature resolution works, if any part of that system enables the IPv6 feature of smoltcp, then everyone gets a smoltcp with IPv6 enabled. But the enumerated IP address types inside smoltcp — such as wire::Endpoint — change if IPv6 is enabled! That means that cotton-ssdp has no way of knowing whether its usage of smoltcp will result in it getting handed the one-variant version of those enumerated types, or two-variant versions. That’s why the conversions accepting those smoltcp types must look like this:

// smoltcp may or may not have been compiled with IPv6 support, and // we can't tell #[allow(unreachable_patterns)] match ip { wire::IpAddress::Ipv4(v4) => ... _ => ... }

in order to compile successfully however many variants ip has; if there’s only one variant, it needs the allow() in order to compile, but if there’s two or more, it needs the “_ =>” in order to compile.

“Don’t Panic” in large, friendly letters

The first version of the DHCP test worked much like the existing “Hello World” test, which was refactored to match: spawn probe-run as a child process, listen to (and run tests against) its trace output, and then shut it down (in a Drop handler) once all tests have passed.

This was a plausible first stab, and worked well every time that the tests passed, but turned out not to be sound if a test ever failed. (Edit: A previous version of this page blamed the way that a failing test panics, for not unwinding the stack, meaning Drop handlers don’t run. But that’s not actually the case — panics do unwind the stack and do run Drop handlers — so it’s now not clear why the following change was needed. But at least it’s still a valid way of doing it, even though it’s not required.)

I found the solution on the Eric Opines blog: run the test inside panic::catch_unwind(). That function takes a closure to run, and returns a Result indicating either successful exit or a (contained) panic. This required refactoring the DeviceTest struct to also use a closure, but actually the resulting tests look quite neat and well-defined — here’s “Hello World”:

fn arm_stm32f746_nucleo_hello() { nucleo_test( "../cross/stm32f746-nucleo/target/thumbv7em-none-eabi/debug/stm32f746-nucleo-hello", |t| { t.expect("Hello STM32F746 Nucleo", Duration::from_secs(5)); }, ); }

The two parameters to nucleo_test() are of course the compiled binary to run — remembering that the build system ensures that these are up-to-date before starting the test — and a closure containing the body of the test. The closure gets passed the DeviceTest object, on which the available methods are expect(), which waits up to the given timeout for a message to appear on the device’s (virtualised, RTT) standard output (and if the timeout elapses without seeing it, fails the test), and expect_stderr() which does just the same for the device’s standard-error stream.

Because panic::catch_unwind() requires its closure to be “unwind-safe”, so does nucleo_test(); so far, this hasn’t been an issue in practice, so I haven’t looked deeply into what to do about it otherwise. The DHCP test, at least on the host side, is just as straightforward:

fn arm_stm32f746_nucleo_dhcp() { nucleo_test( "../cross/stm32f746-nucleo/target/thumbv7em-none-eabi/debug/stm32f746-nucleo-dhcp-rtic", |t| { t.expect_stderr("(HOST) INFO success!", Duration::from_secs(30)); t.expect("DHCP config acquired!", Duration::from_secs(10)); }, ); }

The closure pattern was so appealing that I made the SSDP test use the same design — not least to ensure that it, too, was correctly shut down even following a failing test:

fn arm_stm32f746_nucleo_ssdp() { nucleo_test( "../cross/stm32f746-nucleo/target/thumbv7em-none-eabi/debug/stm32f746-nucleo-ssdp-rtic", |nt| { nt.expect_stderr("(HOST) INFO success!", Duration::from_secs(30)); nt.expect("DHCP config acquired!", Duration::from_secs(10)); ssdp_test( Some("cotton-test-server-stm32f746".to_string()), |st| { nt.expect("SSDP! cotton-test-server-stm32f746", Duration::from_secs(20)); st.expect_seen("stm32f746-nucleo-test", Duration::from_secs(10)); } ); } ); }

The implementation of ssdp_test() itself is a little more involved, because it must spawn a temporary background thread to start and run the host’s SSDP engine which communicates with the one on the device. The two parameters are an optional SSDP notification-type to advertise to the device, and a closure to contain the body of the test. Here the available method on the SsdpTest object passed to the closure, is expect_seen(), which waits with a timeout for someone on the network (hopefully, the device under test) to advertise a specific notification-type. Here the nt.expect() line checks that the device has seen the host’s advertisement, and the st.expect_seen() line checks that the host has seen the one from the device.

Those two events can occur in either order in practice, but both DeviceTest and SsdpTest buffer-up notifications, so that an expectation that has already come to pass before the expect call is made, completes immediately. In the future it might be interesting to investigate using async/await to express the asynchronous nature of this test more explicitly.

The SSDP test only works if the Nucleo board and the test running on the host, can exchange packets. Typically this means that they must be on the same Ethernet network — or, if the host is on Wifi, that the Wifi network must be bridged to the Ethernet (e.g., by cabling the Nucleo to one of the Ethernet LAN sockets on the Wifi router).

Putting it all together

As of the merge of the pdh-stm32-ssdp branch, the following command runs the system-tests on an attached STM32F746-Nucleo:

cargo test -F arm,stm32f746-nucleo

And for those without a Nucleo, the following commands still work, building all the device code but testing only the host code:

cargo build -F arm,stm32f746-nucleo cargo test -F arm

And for those without even the cross-compiler installed, which is probably most people, the following commands still work, building and testing only the host code:

cargo build cargo test cargo build-all-features --all-targets cargo test-all-features --all-targets

This all makes it easy for a developer to determine, before pushing to the central git server, whether their branch is likely to be OK for main — but the most definitive answer to that question is only available when going to the trouble of using a local Nucleo development board. If you’re working on something you think probably won’t affect embedded builds, what you really want is to not faff about with development boards (particularly multiple ones): what you want is for continuous integration to perform all of these system-tests as part of its mission to answer the question of whether your branch is OK for main. Adding a CI runner that can run these tests automatically on every push, is the topic of the third post in this series.

System-testing embedded code in Rust, part one: Infrastructure

Previously on #rust:

• Rust crate release checklist
• Things I learned writing an SSDP implementation in Rust
• Self-hosted CI for Rust and C++ using Laminar

One of the goals of the Rust crates I’ve been working on, cotton-netif and cotton-ssdp, is for them to be useful on embedded systems: microcontroller-based devices only capable of running simple real-time operating systems, as opposed to full-size Linux systems.

There is good support in Rust itself for targetting such platforms: it’s relatively easy to write such code (with the no_std attribute), and not even much harder to cross-compile it (using Cargo) and even run it on the target (the probe-rs folks do great work there). But if you’ve read some of the other posts here, you’ll be familiar with the idea that software isn’t done until it can be repeatably shown to be done. Someone — perhaps not Themis, the goddess of Justice, as pictured; more likely Laminar, the goddess of CI — must solemnly, dispassionately, objectively weigh the code’s activities against (some representation of) its specification, and hold it in judgement if it falls short.

Less fancifully, what’s needed is an automated way for CI (running on a rich, non-embedded host platform) to run the code on a genuine, embedded, target platform and check its functionality. Of course, it’s best to arrange that as as much code as possible is abstracted away from the hardware so that it can be unit-tested on the host, run through Miri and Valgrind and other dynamic-analysis tools on the host, and just plain debugged on the host, where everything is a little less awkward. But the proof of the pudding is still in the eating: only tests that run on the target can be the final arbiter.

(In particular, whether the target hardware appears well-documented or not, it’s all too easy to have misconceptions about how peripherals behave, leading to a situation where the code and the host-side unit-test agree about what’s going on, but they’re both wrong because the actual hardware does something completely different.)

At Electric Imp we had (what became) a large subsystem of Python scripts which our CI system ran, and which in turn installed the newly-built device firmware on some dozens of Imp devices, running through system-tests including thorough regression-testing of the wifi connectivity, the “curated” peripherals, and really the entire of the firmware functionality. This worked extremely well, and over the years caught simply oodles of bugs which had passed unit-testing but would have in some way scuppered our customers’ actual devices — but all of that Python always felt like an add-on to the main C++ build system, needing different skills to maintain. As Rust (or at least Cargo), by contrast, quite rightly represents testing as a first-class language feature, I wondered whether Cargo’s own facilities could be used to system-test embedded Rust without having to invent lots of extra infrastructure bolted on the side.

In this blog post I’ll focus on getting the testing infrastructure set up, literally just far enough to get a target-side test that prints “Hello, World” and a host-side test runner which checks that it has done so. Adding actual tests for the crates’ functionality (SSDP, to start with) will likely come in a subsequent post; CI considerations in yet another.

Goals of the Cotton automated system-tests

1. Joel Test #2 compatibility — Joel Spolsky, writing some years ago now, has some pithy questions to ask software development teams. “Can you make a build in one step?” is always valid to ask, for the reasons he lists — mostly about not forgetting intermediate steps — but also because in practice what’s easy is what most people will do. If the easy thing to do is run a command that builds absolutely everything, then most of the time developers will run that command. (Except perhaps if it starts to take too long, but that’s a different issue.) And the more that’s built by the one command that everybody uses each day, the harder it is to accidentally introduce a bug that affects some builds or facets of the system but not others.

2. Easy for me/anyone/CI to test everything on the host — Because most of the code compiles for the host (potentially even parts that are only useful on the target), host-side development should remain straightforward. In particular, it mustn’t require the presence of a cross-toolchain, or nightly Rust, or any target hardware.

   It certainly mustn’t require people using the crates from crates.io in normal host-side builds in the normal way, to install or attach anything special. (But Cargo makes sure of that anyway.)

   Someone wanting to work on a Cotton crate in combination with their own project, can check out the Cotton repository alongside their own, and use a path dependency when importing the Cotton crate (just as they would for a one-crate upstream repository).

3. Easy for me/anyone/CI to test everything on one target — Even if Cotton eventually targets many different embedded systems, almost all embedded developers will only have one type of hardware attached to their development host at any one time. It must be straightforward to run all the tests that can run on (say) an STM32F746-Nucleo, but none that requires different hardware.

4. Easy for me/anyone/CI to test one crate on one target — Eventually several crates (not just SSDP) will share the same system-test infrastructure; it must remain possible to test just one crate at a time, for the sake of development cycle time.

5. Possible for CI to test everything on many/all targets — Once Cotton targets several different embedded systems, each one should have its tests run by CI. But this mustn’t require one CI host per target — it must be possible to attach several development boards to the same CI host and have it run the right tests on the right boards. Notice that this goal is for it to be “possible”, not “easy”: having lots of different development boards attached is going to be uncommon, so it’s okay for it to need slightly more awkward setup.
6. Separation of concerns — The cotton-ssdp crate, say, can probably be tested on each of several different target devices. Adding a new target device mustn’t require changes to cotton-ssdp itself.

Outline of the solution

You can see the merge that creates this infrastructure at commit 181a8fdc and the whole tree at that commit here on Github.

There were definitely a few false starts along the way to achieving those goals. The first thing I tried was “per-package targets”; this is a Rust facility that in theory should make it possible to mark certain packages in a workspace as building for a different platform than the rest of the workspace. The inspiration for the feature was people building web apps where the server end compiles to x86_64 or ARM64 or whatever’s cheap in AWS these days, but the client end compiles to WASM to run in browsers. It’s more-or-less exactly what’s needed here too — but sadly it’s more complicated to implement than you’d first think, and is only available in nightly Rust where it doesn’t work very well. (When I tried it, cargo test kept trying to run my STM32 binaries on the host.)

Without per-package targets, I was going to need different invocations of Cargo for host and device builds (because each invocation of Cargo can only build for a single platform). So I tried having a build.rs build script that, when invoked for the host platform, re-runs Cargo for the device platform. Yee-hah, right? Sheer cowboyery. It doesn’t work, because Cargo deadlocks trying to rebuild the same crate it’s already building. Can’t really blame it, either.

I looked for a while into setting rustflags in Cargo.toml, but that can’t be set per-package in a single workspace, let alone per-test in a single package. Each different set of rustflags must currently be a separate invocation of Cargo.

The answer seems to be, to take parts of each of those ideas:

• Have the workspace as a whole continue to build native for the host,
• and have a build script that re-invokes Cargo for each target platform,
• but have each target platform’s root crate be separate, and never built for the host, using an exclusion in the workspace Cargo.toml:

  Cargo.toml
  [workspace] members = [ "cotton-netif", "cotton-ssdp", "systemtests", ] exclude = [ "cross", ]

The resulting workspace structure looks like this:

cotton ├── Cargo.toml ├── cotton-netif │   ├── Cargo.toml │   └── ... ├── cotton-ssdp │   ├── Cargo.toml │   └── ... ├── cross │   └── stm32f746-nucleo │       ├── .cargo │       │   └── config.toml │       ├── Cargo.toml │       ├── memory.x │       └── src │           └── bin │               └── hello.rs └── systemtests     ├── build.rs     ├── Cargo.toml     ├── src     │   └── lib.rs     └── tests         └── stm32f746-nucleo.rs

The cross subdirectory is excluded from the root workspace (in the root Cargo.toml), and the crates inside it (at present, only stm32f746-nucleo) are built by a recursive Cargo invocation in systemtests/build.rs; recursively invoking Cargo is a bit subtle, as you need to unset a bunch of environment variables in order that the sub-Cargo runs mostly as a new top-level Cargo (otherwise, the deadlocking issues reappear). Here’s the build-script section that achieves that, with the cross-compilation guarded by a (Cargo) feature called arm:

systemtests/build.rs (partial)

if env::var("CARGO_FEATURE_ARM").is_ok() { /* Run the inner Cargo without any Cargo-related environment variables * from this outer Cargo. */ let filtered_env: HashMap<String, String> = env::vars() .filter(|(k, _)| !k.starts_with("CARGO")) .collect(); let child = Command::new("cargo") .arg("build") .arg("-vv") .arg("--all-targets") .arg("--target") .arg("thumbv7em-none-eabi") .current_dir("../cross/stm32f746-nucleo") .env_clear() .envs(&filtered_env) .output() .expect("failed to cross-compile for ARM"); io::stdout().write_all(&child.stderr).unwrap(); io::stdout().write_all(&child.stdout).unwrap(); assert!(child.status.success()); }

The obvious downside here, is that build scripts aren’t run with a live terminal: unless the outer Cargo is invoked with -vv, the script’s standard output and standard error are written only to files, and shown only if the script fails. If the recursive Cargo invocation succeeds, all you see is a long pause in your build — though at least if the recursive invocation fails, you do see its error output.

For the time being, this acts as extra encouragement to keep complex code out of the only-compiled-for-target crates — though if need be you can always do a normal top-level Cargo build inside the STM32 crate, to see live standard output and standard error:

cargo -C cross/stm32f746-nucleo build --target thumbv7em-none-eabi

Host-side runner for a device-side test

To start with (and to validate the rest of the solution), there is only one actual system-test in the first merge: writing the “Hello World” binary onto an STM32F746-Nucleo development board, running it, and checking the output. Cargo will have run our build script before running the test, so we know that our device-side binaries have all been built and are up-to-date (another Joel Test benefit) — and it’s just a question of using (the excellent) probe-run, which is built on probe-rs and already knows about STM32 development boards, to write the binary to the STM32 chip and then run it. The STM32 binary uses defmt-rtt for its logging output, using the Cortex-M Real-Time Tracing system, support for which is again built-in to probe-run (no semihosting! no UARTs!), so the device side is as simple as this:

cross/stm32f746-nucleo/src/bin/hello.rs
#![no_std] #![no_main]   use defmt_rtt as _; // global logger use panic_probe as _; use cortex_m::asm;   #[cortex_m_rt::entry] fn main() -> ! { defmt::println!("Hello STM32F746 Nucleo!");   loop { asm::bkpt() } }

and the host side only a little less simple:

systemtests/tests/stm32f746-nucleo.rs

use assertables::*; use serial_test::*; use std::env; use std::path::Path; use std::process::Command;   use std::io::{self, Write};   #[test] #[serial] #[cfg_attr(miri, ignore)] fn arm_stm32f7_hello() { let elf = Path::new(env!("CARGO_MANIFEST_DIR")).join( "../cross/stm32f746-nucleo/target/thumbv7em-none-eabi/debug/hello", );   let mut cmd = Command::new("probe-run"); if let Ok(serial) = env::var("COTTON_PROBE_STM32F746_NUCLEO") { cmd.arg("--probe"); cmd.arg(serial); } let output = cmd .arg("--chip") .arg("STM32F746ZGTx") .arg(elf) .output() .expect("failed to execute probe-run");   println!("manifest: {}", env!("CARGO_MANIFEST_DIR")); println!("status: {}", output.status); io::stdout().write_all(&output.stderr).unwrap(); io::stdout().write_all(&output.stdout).unwrap(); assert!(output.status.success());   let stdout = String::from_utf8_lossy(&output.stdout); assert_contains!(stdout, "Hello STM32F746 Nucleo"); }

Notice the use of serial_test to defeat Rust’s default behaviour of running multiple integration tests in parallel — that wouldn’t end well if they were all competing for one single physical development board. (Although obviously there’s only one test at the moment.)

Clearly much more complex logging and checking could (and will) be implemented via the same mechanism, but as a proof-of-concept this suffices for this initial blog post.

Does this meet our goals?

1. ✅ Joel Test — the commands:

   cargo build
   cargo test
   cargo build-all-features --all-targets
   cargo test-all-features --all-targets
   

   all do as expected, and none need any cross-toolchains, special hardware, or even nightly Rust. (The all-features ones work because the arm feature is carefully excluded from all-features builds.) The same generic “CI for Rust” scripts that worked for simple old host-side Cotton, continue to work fine with the exciting new multi-platform Cotton.
2. ✅ Test everything on the host — as above, the easy, everyday commands build and test all the host-compatible crates. The device-only crates can be built with:

   cargo build -F arm
   cargo test -F arm
   

   This needs the thumbv7em-none-eabi target to be installed for the current Rust toolchain:

   rustup target add thumbv7em-none-eabi

   but does not need any hardware, nor nightly Rust.
3. ✅ Test everything on one target — The Cargo.toml in the systemtests declares a Cargo feature stm32f746-nucleo, which depends on feature arm and enables the integration test whose host side is shown above. So building and running this test (i.e., running all the tests for that particular development board) looks like this:

   cargo build -F arm,stm32f746-nucleo
   cargo test -F arm,stm32f746-nucleo
   

   This needs the cross-toolchain installed as above, and of course it needs an actual physical STM32F746-Nucleo development board attached via USB to the host computer running the tests.
4. ✅ Test one crate on one target — Because the system-tests infrastructure is shared, this would have to be accomplished by careful naming of tests, and the use of Cargo’s wildcard test name option:

   cargo test -F arm,stm32f746-nucleo --test '*ssdp*'
   

5. ✅ Test everything on many targets — the main issue here is that, if several different (probe-rs compatible) development boards are attached to the same host, the probe-run command needs to be told each time which one to use. This is the idea behind the optional COTTON_PROBE_STM32F746_NUCLEO environment variable seen in the host-side runner above: a CI or other setup that has several development boards attached, needs to specify using these environment variables the unique “probe identifier” of each one. This feature will get more of a workout in a future blog post when a second target platform is added.

6. ✅ Separation of concerns — So far, this is good but not perfect. Device-side code that can be compiled for the host goes in the top-level workspace (perhaps alongside host-side code that can’t be compiled for the device). Device-side code that can’t be compiled for the host, goes in one of the crates under the “cross” directory. (Maybe one day those too should become workspaces?) Entire device-side applications (or, at the very least, example ones) can live there too.

   At some level it would be appealing for (say) system-tests for SSDP to go somewhere under the cotton-ssdp create. But device-side tests are inherently device-specific, and given N crates and M devices and a potentially N×M-sized testing matrix, it seemed better to keep the tests by device rather than by crate-under-test, on the basis that all engineers working with embedded Rust (in the device directories) are surely also familiar with host-side Rust, but the reverse (engineers working on host-side Rust in, say, the SSDP crate being familiar with embedded Rust) seems less guaranteed.

   Also, all the code in the systemtests crate, including much of build.rs, is generic across any crate wanting system-testing, and isn’t Cotton-specific. As of this blog post, no better advice is being offered here than to copy-and-paste it into your own projects, but in the future it would be more Rust-like to offer this functionality in free-standing crates that other projects desiring system-testing could just import in the normal way.

That Github link once again

You can see the merge that creates this infrastructure at commit 181a8fdc and the whole tree at that commit here on Github.

Similar blogs elsewhere

Ferrous Systems have a series of blog posts covering very similar topics, though they use cargo-xtask to construct their single build commands.

Continue to...

• System-testing embedded code in Rust 2: Testing SSDP
• System-testing embedded code in Rust 3: A CI runner

Perceived angular bisection

Given collinear points A, B, C in the plane, the locus of points P such that $\angle \mathrm{APB}=\angle \mathrm{BPC}\ne 0$ is the unique generalised circle through B which inverts A to C.

Those angles are also trivially equal (both zero) for all P collinear with A, B, C and not between A and C. The locus of other points P is a true circle, except when $AB\; =\; BC$, in which case it is the straight line that is the perpendicular bisector of AC through B; "generalised" circle includes this case as a notional "circle of zero curvature".

Equivalently, the locus is the circle through B centred at a point O chosen such that $\mathrm{OA}.\mathrm{OC}={\mathrm{OB}}^2$ (again with the exception when $AB\; =\; BC$).

This result answers the question: given three points in a line and no other information — three lamp-posts in an otherwise-dark landscape, for instance — whereabouts are the viewpoints from which the middle one appears, through perspective, to be exactly halfway in-between the other two?

Proof

Consider such a point P. Without loss of generality, assume AB < BC (the AB=BC case is trivial, and the AB > BC case is symmetrical with this one).

Draw the unique circle through P and B whose centre is collinear with A, B, and C (e.g. by constructing the perpendicular bisector of PB and marking its centre where that line intersects with the line through A and C). Call this centre O.

Label the angles as shown: $$\begin{array}{cccc}\angle \mathrm{OPA}& =& c& \\ \angle \mathrm{APB}& =& d& \text{which is also}\angle BPC\\ \angle \mathrm{OCP}& =& e& \\ \angle \mathrm{OBP}& =& g& \end{array}$$

Then consider:

$$\begin{array}{cccc}\text{in △CPB,}& e+d+(\mathrm{180°}-g)& =& \mathrm{180°}\\ & e+d& =& g\\ \text{△POB is isosceles,}& c+d& =& g\\ \text{therefore,}& e& =& c\\ \text{i.e.,}& \angle \mathrm{OPA}& =& \angle \mathrm{OCP}\end{array}$$

But the triangles OAP and OPC also have the angle at O in common, so they have two angles in common, therefore they are similar, therefore the ratios of corresponding sides are equal: $\mathrm{OA}/\mathrm{OP}=\mathrm{OP}/\mathrm{OC}$.

And $\mathrm{OP}=\mathrm{OB}$, because both are radii of the circle, so $\mathrm{OA}/\mathrm{OB}=\mathrm{OB}/\mathrm{OC}$ or $\mathrm{OA}.\mathrm{OC}={\mathrm{OB}}^2$, which is one definition of the circle of inversion.

Note that the position of O is the same whichever P we pick; it doesn't depend on any of the angles. (This is easier to see if we rearrange $\mathrm{OA}.\mathrm{OC}={\mathrm{OB}}^2$ into $\mathrm{OA}=\frac{{\mathrm{AB}}^2}{\mathrm{BC}-\mathrm{AB}}$ — it's uniquely specified by the positions of A, B, and C.) So the circle does represent the locus of all P.

Bit off-topic for this blog though eh

Yes. But it is a problem I've been thinking about for a while — in fact, ever since I read that, given the original formulation (three lamp-posts on a darkened plain) you can't even tell from a photograph whether or not they're evenly-spaced, because perspective means that they could be anywhere. But if your photograph has four lamp-posts in, you can tell whether or not they're evenly-spaced, because the cross-ratio is projection-invariant.

So I wondered whether, given three lamp-posts, there was always a point from which they appear evenly-spaced. It felt sort-of plausible that there were such points, but I had no intuition what their locus looked like. The first time I attacked this problem I plotted the points by brute force, and was surprised to see an apparently perfect circle. With the cosine rule and lots of ghastly slog (and Wolfram Alpha) I found the equation of the circle and the position of O. But that was clearly not The Book's proof, so I shelved the draft blog post — for some years. Then I happened to be reading about circle inversion, and suddenly realised it was talking about the three-lamp-posts circle. Eventually I was able to use circle-inversion techniques to come up with a geometrical, not algebraic proof, which I'm much happier with.

And if you're dying to see how bad the ghastly brute-force proof looked, here's a part of it ($p$ is $\mathrm{AB}$ and $q$ is $\mathrm{BC}$):

$p^2{q}^2{x}^2-2p^{2}q{x}^3+p^2{x}^4+p^2{x}^2{y}^2+2p{q}^2{x}^3+2p{q}^{2}x{y}^2-4pq{x}^4-4pq{x}^2{y}^2+2p{x}^5+4p{x}^3{y}^2+2px{y}^4+q^2{x}^4+2q^2{x}^2{y}^2+q^2{y}^4-2q{x}^5-4q{x}^3{y}^2-2qx{y}^4+x^6+3x^4{y}^2+3x^2{y}^4+y^6=p^2{q}^2{x}^2-2p^{2}q{x}^3-2p^{2}qx{y}^2+p^2{x}^4+2p^2{x}^2{y}^2+p^2{y}^4+2p{q}^2{x}^3-4pq{x}^4-4pq{x}^2{y}^2+2p{x}^5+4p{x}^3{y}^2+2px{y}^4+q^2{x}^4+q^2{x}^2{y}^2-2q{x}^5-4q{x}^3{y}^2-2qx{y}^4+x^6+3x^4{y}^2+3x^2{y}^4+y^6$

(I didn't set that by hand; there used to be an online Latex-to-MathML converter, but it seems to have since been retired.)

Rust crate release checklist

Previously on #rust:

• Things I learned writing an SSDP implementation in Rust
• Self-hosted CI for Rust and C++ using Laminar

There’s been a few public releases now of the Rust crates I’ve been working on, cotton-netif and cotton-ssdp. The SSDP one even has a merged pull request from a contributor! But because it’s often a little while between releases, I struggle to remember all the steps required. (Not so many, as Cotton is no massive failer of the Joel Test, but there are a few moving parts just because of all the fine, free open-source tools in use.) This document collects them all in one place, as much for my own benefit as anyone else’s.

1. Check Github for third-party pull requests as I don’t want to annoy contributors by seeming to ignore their work. This check can be automated using a badge:
2. Update Cargo.toml and CHANGELOG.md for all packages being released, like in commit 4ba3675c. If the package being released is depended on by other parts of Cotton, update their Cargo.toml dependencies too, like in commit c989d9f1.
3. Check that everything is pushed upstream, both to self-hosted CI and to Github.

   git push main git push github main

4. Check that both CI pipelines are passing; again, there’s a badge for the Github one:

   If your CI includes statistical metrics (as opposed to pass/fail ones: coverage, for example), check that those are in acceptable ranges too.

5. Do a dry-run publish, remembering to cd to the crate directory, not the workspace root:

   cargo publish --dry-run

   Cargo will check that the package is buildable; if any errors occur, fix them and go back to Step 3.
6. Tag the release, using multiple tags if multiple crates are being released:

   git tag cotton-ssdp-0.0.3

7. Push the new tag to both upstreams:

   git push cotton-ssdp-0.0.3 git push github cotton-ssdp-0.0.3

8. Actually publish the crate on crates.io:

   cargo publish

   Hopefully there won’t be any errors, given that the dry-run succeeded.
9. Let any contributors know that their stuff is now in a release – if any pull requests have been merged, now is the time to let those contributors know that they can go back to using real upstream releases of your crate, and potentially stop maintaining their forks.

Three SSH settings which aren’t the default, but which you probably want

Previously on #homelab:

• Setting up a Raspberry Pi as home server
• Hub-and-spoke backups using Syncthing
• Self-hosted CI for Rust and C++ using Laminar
• Network firewall on a Raspberry Pi using OpenWRT

Everybody uses OpenSSH to securely log in to remote machines. And they have done for ages. But that’s actually a problem, because OpenSSH has been around for so long that some of the security decisions made earlier in the project’s history no longer match current best practices. Here are a few things which would probably be the default if OpenSSH was starting out today, but which – for sound backward-compatibility reasons – you’ll need to arrange for yourself.

“ssh-add -c”

One criticism of the ssh-agent system is that when using it, you lose visibility of exactly when you’re signing for things. One way to mitigate this is, when using ssh-add to add local private keys to the agent, adding the -c option. This makes ssh-agent ask for confirmation (on console or in a pop-up dialog) every time a private-key operation is requested; unexpected pop-ups could be a sign that nefarious software is trying to use your key.

This also somewhat mitigates the risk, when using the agent-forwarding (ssh -A) feature, of attacks by a malicious actor who has root on the remote computer.

“ssh-add -D” on screen lock

On most reasonable systems, you have to give your local Unix password to unlock the screen once the screensaver has kicked-in. But if you feel that your SSH private key is more valuable than your local password – which you probably do, otherwise you wouldn’t’ve bothered encrypting your SSH private key in the first place – then that’s effectively a privilege-escalation attack: if you’ve left a ssh-agent session running, then knowing only your local password gives an attacker login ability using your SSH private key.

The obvious way to mitigate that, is to empty all saved keys from the ssh-agent session, every time the screen locks. Honestly it’s a bit surprising that KDE and Gnome don’t already have this built-in – but (at least in KDE) there’s a hook that lets you do just that.

In KDE “System Settings”, go to “Notifications” then next to “Applications:” click “Configure...”. Scroll down to “Screen Saver” and click “Configure Events...”. In the resulting pop-up window, choose “Screen locked” then below tick “Run command” and enter:

/usr/bin/ssh-add -D

Alternatively, manually add the following to ~/.config/ksmserver.notify:

[Event/locked] Action=Execute Execute=/usr/bin/ssh-add -D Logfile= Sound= TTS=

Under Gnome it’s less straightforward; there’s a script available called lockheed, but as it stands it only listens for unlock events; perhaps it would be possible to modify it to listen for lock events too. (Or perhaps wiping the keys from ssh-agent only on unlock is actually okay.)

“ssh-keygen -a 1000”

Over the years, OpenSSH has gone through a few design iterations on how to encrypt SSH private keys – in other words, what the passphrase you give to unlock the key actually does. If your private key file (usually ~/.ssh/id_rsa) starts with

-----BEGIN RSA PRIVATE KEY----- Proc-Type: 4,ENCRYPTED ...

then it is encrypted in a very obsolete way using PKCS#1; such passphrases are vulnerable to brute-forcing, should an attacker get their hands on the file. If, alternatively, it starts with:

-----BEGIN ENCRYPTED PRIVATE KEY----- MII...

then it’s PKCS#8, which is better but still not great. What you hope to see is the OpenSSH key format, which allows specifying the number of rounds of key-derivation function to use (i.e., how much work an attacker would have to do per guess in order to brute-force the passphrase):

-----BEGIN OPENSSH PRIVATE KEY-----

This particular recommendation slightly undermines the title of the blog post, because the OpenSSH format, which you probably want, is the default when creating new keys nowadays – but only since the OpenSSH 7.8 release of 2018-Aug-24, and OpenSSH does not upgrade from one format to another automatically. (Which is the Right Answer of course, as doing so would break compatibility with older versions if an upgrade ever had to be rolled-back.) This means that if your key was created with an older version of OpenSSH, the encryption used was likely to be, and is likely to still remain, one of the weaker forms.

Fortunately, there’s an OpenSSH facility to update just the encryption of any private key to the newest, most secure format, without altering the actual key:

ssh-keygen -p -f id_rsa -a 1000

This command upgrades the passphrase protection. It asks for the old passphrase (to decrypt the key) and then twice for the new passphrase (to re-encrypt it). If you’re confident that your old private key hasn’t leaked anywhere, you can re-use the same passphrase. (If you aren’t confident of that, you probably need to generate all-new keys anyway.) The -a 1000 sets the number of rounds to 1,000 – up from the default of 16. On this oldish Core-i7 machine, the setting of 1,000 makes checking the passphrase take about ten seconds. (Whether successful or unsuccessful!) This is a slight annoyance for you, but each time you’re waiting those ten seconds you can be thinking about how any attacker trying to brute-force your passphrase will be using up all that CPU on every single wrong guess they make.

Unfortunately, although the number of rounds is stored unencrypted in the key file, there appears to be no straightforward way of reading it out again directly. Following the directions given in a stackoverflow answer, you can use this command to get a hex dump of the base64-decoded encrypted key structure:

cat id_rsa | head -n -1 | tail -n +2 | base64 -d | hexdump -C | head

On a key I had lying around, this produced the bytes shown below. What you’re looking for is the string “bcrypt”, followed by 24 bytes you don’t care about (the KDF-descriptor length 0x0000_0018, the salt length 0x0000_0010, and the salt itself as 16 random bytes) followed by a 32-bit big-endian value which is the number of rounds:

00000000 6f 70 65 6e 73 73 68 2d 6b 65 79 2d 76 31 00 00 |openssh-key-v1..| 00000010 00 00 0a 61 65 73 32 35 36 2d 63 74 72 00 00 00 |...aes256-ctr...| 00000020 06 62 63 72 79 70 74 00 00 00 18 00 00 00 10 1c |.bcrypt.........| 00000030 d1 ab a0 6b cd 50 a7 8e 01 8c 9a f7 98 32 a6 00 |...k.P.......2..| 00000040 00 00 10 00 00 00 01 00 00 00 33 00 00 00 0b 73 |..........3....s|

In this file it’s 16 (0x0000_0010), the default. Once I’d run the ssh-keygen command on it, it instead appears as 1,000 (0x0000_03e8):

00000000 6f 70 65 6e 73 73 68 2d 6b 65 79 2d 76 31 00 00 |openssh-key-v1..| 00000010 00 00 0a 61 65 73 32 35 36 2d 63 74 72 00 00 00 |...aes256-ctr...| 00000020 06 62 63 72 79 70 74 00 00 00 18 00 00 00 10 cb |.bcrypt.........| 00000030 d4 4a be 47 b2 26 c9 15 d4 8d 0d d0 36 4c 62 00 |.J.G.&......6Lb.| 00000040 00 03 e8 00 00 00 01 00 00 00 33 00 00 00 0b 73 |..........3....s|

Whenever you need to generate a new SSH key, the ssh-keygen command accepts the -a 1000 option in that case too.

Bonus fourth SSH thing while you’re here

The encfs-agent script lets you set up EncFS encrypted filesystems in such a way that you can use ssh-agent signing operations to unlock (mount) them – with no need to remember or enter a separate passphrase. Use it with ssh-add -c!

Note that the filesystem then remains mounted/decrypted until manually unmounted; ssh-agent is only consulted during the mount operation. The ssh-add -D command does not unmount the filesystem (analogously, it doesn’t close existing SSH sessions either). If you want screen-locking to umount these filesystems, consider the command:

umount -a -t fuse.encfs

That’s a slightly dangerous thing to do, though; it’s not guaranteed to work if any process is holding the filesystem open (by holding a file on it open, or having a current-directory inside it). Even if it does work, running processes might get very confused – perhaps, for instance, by writing important files into the unencrypted mount-point directory outside the EncFS, instead of inside the EncFS where you wanted them.

LATER EDIT: Extra bonus fifth SSH thing

TIL you can put

AddKeysToAgent confirm

in your .ssh/config and it’ll automatically do the equivalent of “ssh-add -c” for any key whose passphrase you supply to an ordinary ssh invocation. Semi-life-changing given the number of times I think my key is in ssh-agent, but it isn’t, and I end up having to enter the passphrase all over again...