Things I learned writing an SSDP implementation in Rust

Previously on #rust:

• Self-hosted CI for Rust and C++ using Laminar

A while ago I wrote Chorale, a collection of open-source media-library and networking software, mostly based on the UPnP (Universal Plug-’n’-Play) AV standards (but including other things as well, such as support for Rio Receiver and Empeg car-player devices, whose firmware I’d worked on at those companies, an even longer while ago).

Chorale never got much interest (or, as the tagline on this very blog puts it, “Waits for audience applause ... not a sossinge”); this is no doubt partly due to (closed) cloud streaming platforms for music hugely overtaking in popularity all dealings with (open) local media libraries, but no doubt also due to the completely zero amount of effort I actually spent promoting Chorale (or even, really, mentioning its existence).

So I’ve resolved to be a bit more talkative about open-source things I do from here on in. Inspired by a post by Simon Willison, I’m trying to adopt his mantra that anything you do, at least in the software world, isn’t finished until you’ve told people that you’ve done it – especially if part of the reason you’re doing it in the first place, is in the hope of spreading useful information and good practices.

In this post I’ll tell you what I did, when I went to implement one of the first, lowest-level parts of the UPnP protocol suite in Rust: SSDP. This is the backstory of the cotton-ssdp crate.

Background: a Simple Service Discovery Protocol

SSDP, the Simple Service Discovery Protocol, is aimed at making it easy for separately-purchased and separately-designed devices to discover each others’ resources over a local network, so that they can later interact using higher-level protocols such as UPnP itself. A resource might be a streaming-media server, or a router, or a network printer, or anything else that someone might want to search for or enumerate on a network. Like most protocols, there are two basic rôles involved: a server advertises the availability of a resource, and a client discovers available resources.

What precisely is advertised, or discovered, is, for each resource, a unique identifier for that particular resource (Unique Service Name, USN), an identifier for the type of resource (Notification Type, NT), and the location of the resource in the form of a URL.

SSDP is mainly used by UPnP systems, such as for media libraries and local streaming of music and video – it is how diskless media players such as the Roku Soundbridge discover the available music servers – but the mechanism is quite generic, and could as easily be used for any type of device or resource that must be discoverable over a network. Effectively it enables the automatic self-assembly of (locally-) distributed systems, including in ad hoc settings which don’t necessarily have expert network administrators close at hand to configure them manually.

For a learning project in Rust, SSDP is ideal as it’s about as simple as a network protocol could possibly be without being uninteresting. SSDP servers send out advertisements over UDP multicast to a “well-known” multicast group, which can be picked up passively by clients in that group; and clients can also actively send out search requests, also over UDP multicast, to which servers will respond.

There is no Internet RFC as such for SSDP – merely some expired drafts. The protocol is, instead, documented in the UPnP Device Architecture documents.

SSDP is very similar in motivation, and (it turns out) fairly similar in implementation, to the Multicast DNS (mDNS) protocol – but is not directly compatible.

Particular goals of the cotton-ssdp crate

In no specific order:

1. Produce a useful SSDP implementation, which other Rust users who need SSDP might reasonably hear about and decide to use;
2. Personally learn more about Rust and about the packaging, distributing, and publicising of Rust crates;
3. Implement both client and server;
4. Support use both in full-size/desktop/hosted environments, and in embedded/freestanding/no_std environments;
5. Do the Right Thing on multi-homed hosts (ones with multiple active network interfaces), which were the source of a lot of complexity in Chorale and in proprietary network protocols which I’ve worked on in my day job;
6. Follow, and indeed embody, if not evangelise, software engineering best practices: lots of documentation, lots of tests.

Things I learned about

1. Effective techniques for getting good test coverage
2. Automate your quality checks as far as possible
3. Not picking a lane when it comes to third-party ecosystems
4. Effective techniques for multi-homed hosts
5. The journey to no_std
6. Using the remarkable free tools available

1. Effective techniques for getting good test coverage

Coming from a C++ background, I was very impressed with the “batteries-included” nature of the Rust (Cargo) test setup. It’s as easy to start writing tests as it is to start writing the code in the first place – which is a Good Thing, as it means that every project can start out on the right side of history by including good testing from the very beginning.

But even here, there are better and worse ways to write tests, and it’s worth learning how to structure your tests in order to be most effective. As well as including the test harness itself, the Rust/Cargo system has baked into it the difference between unit tests and integration tests: outside the Rust world, this is a distinction about which philosophical debates can be had, but in Rust these two terms refer to two very specific and distinct things. When I started out in Rust, I was bundling all the tests into a mod tests in the same .rs file as the implementation: these are Rust’s unit tests. And I soon got in a pickle, because on the one hand I wanted to mock out (or dependency-inject) fake versions of some of the dependencies (specifically, system calls such as setsockopt whose failure is hard to arrange for in a test, but needs to be handled if it does happen in reality), but on the other hand I didn’t want the hassle of reimplementing the actual functionality of the real calls when testing the success case. The Rust mocking library mockall, for instance, replaces the real dependency completely when compiled in cfg(test) mode, as is done when compiling unit tests.

But Rust was way ahead of me here, and had already solved that problem. Tests for the success case – tests which can be written against the public API of the library – can be written as Rust integration tests, in the crate’s tests directory. Those are linked against the real version of the crate, compiled without cfg(test), so they get the actual, non-mock dependencies.

(As it stands, cotton-ssdp does not use mockall, but that’s because I had difficulty with “false negatives” in code-coverage, where it said not all regions were covered, but then didn’t show any actually non-covered regions when drilling-down into the source file. I suspect, but can’t prove, that this is because the coverage tool gets confused when a piece of code is monomorphised twice, once for real and once for mock, and can’t always correctly match up equivalent regions between the two.)

1.1 An example of dependency injection

Here’s the code I first wrote for creating a new UDP socket and setting the particular socket options that cotton-ssdp will need:

fn new_socket(port: u16) -> Result<mio::net::UdpSocket, std::io::Error> { let socket = socket2::Socket::new( socket2::Domain::IPV4, socket2::Type::DGRAM, None, )?; socket.set_nonblocking(true)?; socket.set_reuse_address(true)?; let addr = std::net::SocketAddrV4::new(std::net::Ipv4Addr::UNSPECIFIED, port); socket.bind(&socket2::SockAddr::from(addr))?; setsockopt(socket.as_raw_fd(), Ipv4PacketInfo, &true)?; let socket: std::net::UdpSocket = socket.into(); Ok(mio::net::UdpSocket::from_std(socket)) }

Now how does one test the error code paths from set_nonblocking and friends – all those question-marks? Well, one way is to use dependency injection – that is, to arrange that in unit-test builds, a “fake” version of the system-call is made (one which, for testing purposes, can be arranged to fail on demand), yet in the finished crate the actual, non-artificially-failing system-call is of course used. That change looks like this:

type NewSocketFn = fn() -> std::io::Result<socket2::Socket>; type SockoptFn = fn(&socket2::Socket, bool) -> std::io::Result<()>; type RawSockoptFn = fn(&socket2::Socket, bool) -> Result<(), nix::errno::Errno>; type BindFn = fn(&socket2::Socket, std::net::SocketAddrV4) -> std::io::Result<()>;   fn new_socket_inner( port: u16, new_socket: NewSocketFn, nonblocking: SockoptFn, reuse_address: SockoptFn, bind: BindFn, ipv4_packetinfo: RawSockoptFn, ) -> std::io::Result<mio::net::UdpSocket> { let socket = new_socket()?; nonblocking(&socket, true)?; reuse_address(&socket, true)?; bind( &socket, std::net::SocketAddrV4::new(std::net::Ipv4Addr::UNSPECIFIED, port), )?; ipv4_packetinfo(&socket, true)?; Ok(mio::net::UdpSocket::from_std(socket.into())) }   fn new_socket(port: u16) -> Result<mio::net::UdpSocket, std::io::Error> { new_socket_inner( port, || { socket2::Socket::new( socket2::Domain::IPV4, socket2::Type::DGRAM, None, ) }, socket2::Socket::set_nonblocking, socket2::Socket::set_reuse_address, |s, a| s.bind(&socket2::SockAddr::from(a)), |s, b| setsockopt(s.as_raw_fd(), Ipv4PacketInfo, &b), ) }

Now all the tests can call new_socket_inner, passing versions of new_socket, nonblocking, and so on which are as fallible as can be – but users of the real library are non the wiser; indeed, the API they see has not changed. (The actual tests are visible in commit d6cf9af2; they subsequently moved around a bit as later refactorings were done.)

This is also a standard dependency-injection technique in C++, but one of the hidden wonders of Rust is that the language design allows for a compiler to make lots of optimisations that, through technicalities, might be harder to prove correct in C++ (it’s not like I’ve quizzed the designers on whether this was an explicit goal of the design – but things have worked out that way so often, that really it must have been). The standard compiler is very good at optimising away abstractions like new_socket_inner: it spots that (in non-cfg(test) builds) the only call site of new_socket_inner is in new_socket and it inlines it completely; it also inlines all the lambdas (closures) and turns them back into direct function calls. In this case I checked, and the two versions of the code shown above produce identical assembler: a literally zero-cost abstraction.

Good code coverage is not, of course, any indicator of good code or even of good testing. But poor code coverage is very much an indicator of poor testing – so, much like a compiler warning, it’s always something worth looking at just in case it indicates a real underlying problem of poor code.

2. Automate your quality checks as far as possible

Engineering is the art of the repeatable; art is the engineering of the unrepeatable. Or, put differently: if it costs you actual human effort to make sure an engineered product, such as a particular version of a piece of software, is proper and correct, then that effort itself is, in a sense, written on the wind – as it tells no-one anything about whether any other versions of that software are proper or correct. So effort is usually much better applied on automating quality-assurance work than it is on performing (rote) quality-assurance work; it’s both dehumanising and non-cost-effective to send a person to do the job of a machine. (Though of course some quality-assurance work isn’t automatable: such work is often known as exploratory testing and works best when viewed as an exercise in creatively finding corner-cases or oversights in the automated test suite, which can then be automated in their turn.)

But there’s more to quality-assurance than testing. It’s also a discipline of quality-assurance to run code through analysis tools, to pick up on any potential errors. In the C++ world it’s long been received wisdom to enable most or all available compiler warnings, across multiple alternative C++ compilers if possible: the small iatrogenic downside of avoiding false positives from compiler warnings, is outweighed by the benefits accrued to the project each time a warning pinpoints an actual mistake. In Rust, compiler warnings are usually very good and actionable (and peculiar idioms for avoiding them seem rare), and Clippy lints even more so. The “cargo clippy” command could equally aptly have been called “cargo clear-up-my-misconceptions-about-how-this-language-works”, although “clippy” is definitely easier to type.

The principle of “the more automation the better” applies of course to continuous integration (automatic building and testing) too. An earlier post on this blog described how to set up a local CI system, though for packages that aren’t aimed at embedded systems, CI services are also available in the cloud (and some SaaS engineers speak as if the term “CI” implies a cloud-based service, having seen no other design). Projects hosted in public Github repositories can take advantage of Github’s CI infrastructure (and should do, even if only as a quality filter on incoming pull requests); more on that later on.

Once a crate’s functionality starts to be split up into Cargo “features”, configuring CI to test all feature combinations can be a chore. The cargo-all-features crate can help with this, by running “cargo build” (or check, or test) on all combinations of features in one command. This also helps to surface any issues where features are interdependent, but where that interdependency isn’t spelled out in Cargo.toml. (For instance, in cotton-ssdp, the “async” feature doesn’t compile without the “std” feature enabled, so Cargo.toml specifies that dependency.) Running “cargo test-all-features” takes, in theory, exponential time in the number of features – but, at least in cotton-ssdp’s case, the great bulk of the work is done just once, and the 2ⁿ-1 additional builds are all very fast.

At the top of cotton-ssdp’s lib.rs are the lines

#![warn(missing_docs)] #![warn(rustdoc::missing_crate_level_docs)]

which automate some quality checks on the documentation. Also assisting the documentation are these lines

#![cfg_attr(nightly, feature(doc_auto_cfg))] #![cfg_attr(nightly, feature(doc_cfg_hide))] #![cfg_attr(nightly, doc(cfg_hide(doc)))]

which, between them, arrange for those “only available with feature blah” labels on optional structs and methods in the docs.rs documentation – see later on for how to get docs.rs documentation builds to set the “nightly” attribute, as it isn’t built-in.

One last quality check, if you can call it that, is provided by Rust’s borrow-checker and ownership rules. A common refrain from Rust developers is, “Oh, I’m currently fighting the borrow-checker”. But what this reminds me of most, is an anecdote about a famous bull-fighter being asked what exercises he does to keep in shape for his toreadoring: “Exercises? There is something you don’t understand, my friend. I don’t wrestle the bull.” The anecdote appears in DeMarco and Lister’s “Peopleware” (and, slightly worryingly, apparently nowhere else on the whole of Google) – where the authors use it to suggest that if some activity feels like you’re wrestling a bull, you’re probably not going about it in the most efficient or masterful way. Hand-to-hand combat with the borrow-checker can sometimes be reframed as not heeding its suggestion that the design you’re pursuing could be done differently with more clarity around memory-safety, and thus better reliability or maintainability. The borrow-checker is like a compiler warning, but for your design instead of your implementation, and it’s actionable in the same way: it’s telling you that you might be able to barrel on regardless, but you should really stop and think about whether you’re doing things the best way.

Though admittedly, by that theory, it’s a shame that the borrow-checker only offers that suggestion at quite a late stage of the design process (viz., during actual coding); it’s a bit like being told, “You can’t lay that next brick there.” – “Why not?” – “Because the house that you’re building is going to be ugly.” It would have been better if the ugliness of the house had been spotted at the blueprint stage! There’s a gap in the market, and indeed in the discourse as a whole, for a book on Rust design patterns which would help stave off these issues at an earlier conceptual stage.

3. Not picking a lane when it comes to third-party ecosystems

There are two, not very compatible, ways of writing Rust networking code: synchronous and asynchronous. When hoping to write a library that’s useful to people (one of the stated goals), it makes most sense to me to try and be useful on both sides of this divide. After all, SSDP itself can be boiled down to, packets arrive, other packets go out in response (or in response to a timer expiring). That’s a usefully narrow-sounding interface. So what cotton-ssdp does, is hide all the actual SSDP work away in a struct called Engine, and offer two as-thin-as-possible wrappers over it: one for the synchronous (mio) case (Service) and a second for the asynchronous (tokio) case (AsyncService).

This also involved abstracting over the difference between tokio::net::UdpSocket and mio::net::UdpSocket, using traits which are implemented for both and over which the relevant methods of Engine are generic:

/// Notify the `Engine` that data is ready on one of its sockets pub fn on_data<SCK: udp::TargetedSend + udp::Multicast>( &mut self, buf: &[u8], socket: &SCK, wasto: IpAddr, wasfrom: SocketAddr, )

Both the Engine and UDP abstractions will hopefully also turn out to be exactly what’s needed when porting SSDP to embedded platforms using smoltcp or a similar embedded networking stack. (So in fact, it’s not quite right to say that the SSDP implementation is “hidden” away in Engine – Engine is part of the public library API.)

4. Effective techniques for multi-homed hosts

Networking applications that run on multi-homed hosts – machines with more than one TCP/IP-capable network interface – often need to take particular care over which networks services are offered to: it’s commonplace for a host such as a firewall to have interfaces onto networks with different security policies. (This is so rarely dealt with well, that it’s also commonplace for such setups to resort to containerisation to ensure that certain services are only offered on certain interfaces – which does have the advantage that all such containers or services can then have their network access configured in the same way, rather than with service-specific ways.)

But supporting multi-homed hosts well in a networking server isn’t a huge challenge, so long as it’s thought-about early in the design process. The fundamental idea is always to keep track of which network interface is being used to talk to any given client; Linux, for instance, assigns long-lived “interface indexes” (smallish integers) to each available hardware interface. Knowing which interface packets arrived on (especially in the UDP case) gives the server enough information to know how to send its reply – and in the particular case of a discovery protocol such as SSDP, can inform the content of the reply.

For example, if the SSDP resource being advertised is, from the server’s point-of-view, http://127.0.0.1:3333/test, and its Ethernet IP address is 192.168.1.3, and its wifi IP address is 10.0.4.7, then anyone listening to SSDP on Ethernet should be told that the resource is at http://192.168.1.3:3333/test and anyone listening on wifi should instead be told http://10.0.4.7:3333/test.

In a UDPv4 service such as SSDP, the Linux IP_PKTINFO facility can be used to obtain, at recvmsg time, the interface index on which a packet arrived; similarly, sendmsg can be told which interface to send on. (Fortunately, unlike SO_BINDTODEVICE, this does not require CAP_SYS_RAW, although that restriction on SO_BINDTODEVICE was lifted in kernel 5.7.) The SSDP server can then determine which of the host’s IP addresses that particular client should use, and rewrite the resource URL accordingly when replying.

With care, this can even be used to do the Right Thing when more than one network interface has the same IP network and netmask, just so long as they still have different IP addresses – a situation which is particularly likely to arise if RFC3927 “Zeroconf” networking is in use on multiple physical networks.

Although cotton-ssdp is currently IPv4-only, it appears that IPv6 has noticed and ameliorated the same problem, with APIs taking IPv6 link-local addresses also taking a “Scope ID” which is intended to be used very similarly to this use of the interface-index.

(Every time I see “multi-homed” written down, even if I wrote it myself, I read it the first time as “multi-horned”, which is a metaphor so much more apposite that it makes me wonder whether “homed” was just a typo that stuck.)

5. The journey to no_std

Rust draws a formal distinction between (what in the C world are called) hosted and freestanding implementations: hosted implementations are servers, desktops, and mobile platforms with the usual wide range of standard library facilities, and freestanding implementations are embedded systems where even standard library facilities are not necessarily available. (In fact C draws this distinction too, but as the standard C library is itself so low-level, only the most deeply-embedded systems are freestanding by C’s definition.)

Rust target platforms indicate their freestanding nature with “-none-” as the OS part of their target triple, such as in “thumbv7em-none-eabi”; Rust crates targeting such platforms indicate that by a crate-level #![no_std] attribute. In a no_std crate, much of the standard library is still available (very core parts such as Result and Option are in a crate called core, which works even if no heap is available; facilities that require an allocator live in a second crate called alloc), but anything which requires system-calls is not available.

As cotton-ssdp is aimed at both hosted and embedded users, it needs to be possible (but optional) to compile it in no_std mode. The standard idiom for this, is to have a (Cargo) feature called std, which is usually enabled by default, but can be disabled to produce a no_std-compatible crate. This is taken care of in the code by adding, at the top of lib.rs, the line...

#![cfg_attr(not(feature = "std"), no_std)]

...and then in Cargo.toml, declaring std as a feature and making any non-no_std-compatible dependencies optional and dependent on std; the Cargo.toml changes are in commit 47a0290d, which also marks the examples and integration-tests as depending on std.

The issues that might arise when enabling no_std mode in a crate revolve around: system types whose names (libraries) are different; system types which aren’t available at all; dependencies which themselves need to have their no_std build enabled; and dependencies which have no no_std mode, which must be either encapsualated away or worked around altogether. All of those issues cropped up in cotton-ssdp.

The entire journey to no_std support can be seen in the pull request, though it might look a bit odd because in fact I only created the PR retrospectively, after the branch was already merged, for the purposes of this blog post; that seemed the easiest way to get all the commits listed in one place. The eleven commits on that branch are (in chronological order):

1. 364e7c5 Split UDP implementations into core, std, mio, tokio – once again, something seemed a bit tiresome (annotating everything in udp.rs with cfg(feature = "std")) but that tiresomeness actually represented a “compiler warning” about the design; the UDP traits belong in a separate source file to the std-using implementations, and they in turn belong in separate source files to each other. Now only the mod declarations need to have cfg annotations attached!
2. e590562 Make message.rs no_std-compatible – some of the std facilities previously used here are available in core or alloc, but HashMap is not (because it relies on a random number generator), and nor is io::Cursor (as it uses io::Error, which is in std because it needs to be OS-specific in order to represent system-call errors). Fortunately, reimplementing Cursor from scratch is not hard.
3. 0380edf Make UDP errors more ergonomic (and also more no_std-compatible) – rather than forcing SSDP errors into the io::Error template, use a custom error class (preparatory to cfg-ing out io::Error altogether). This actually makes the crate’s errors easier to use, by following the suggestions in in a recent blog post I read about modularising errors.
4. 47a0290 Featureise std, sync, and async – this is the above-mentioned commit which actually introduces the cargo-level features. As well as enabling no_std, making “async” a feature means that clients who need only the synchronous version, do not need to pull in the sizeable tokio dependencies. This commit also adds the no_std_net crate, which usefully provides the common IP address types in no_std builds, while being a pass-through for the standard-library ones in std builds. (A future release of Rust will move those types to core, but I didn’t want to wait.)
5. fc344f0 Update dependencies – fixes test breakage from the previous commit, as cotton-ssdp’s tests require cotton-netif’s sync feature, not just its std one. Fortunately, using cargo test-all-features made this issue show up straightaway.
6. 0005fe0 Factor RefreshTimer out of Engine – it’s desirable for Engine to compile in no_std mode, because that’s intended to be the basis even of embedded implementations of SSDP. But Engine used std::time::Instant, which (especially in the now method) relies on system-specific clock facilities (std::time::Duration does not rely on those facilities, and is available in no_std builds as core::time::Duration). In practice, most embedded platforms support some variant of rtic_monotonic::Monotonic for clock facilities (a list of implementations of that trait for common microcontrollers can be found on crates.io), but abstracting over both that and the standard library felt like an abstraction too far, so instead this commit moves the whole refresh-timer mechanism out into a separate struct, so that an embedded implementation depending on Monotonic can be provided later and just slotted-in.
7. 6899933 Actually set no_std for non-feature-"std" – this is where the crate is first built with no_std set, which necessitated use of the alloc crate to find String and Format, as well as cfg’ing off some uses of std that couldn’t be ported. In particular, the variant of the udp::Error enum which wraps an underlying system-call error is not, of course, no_std. The [#non_exhaustive] tag tells the Rust compiler that more variants (perhaps one encapsulating a smoltcp error) may be added in future.
8. 5ac330e Remove "url" crate from public API – this crate, a hard dependency used within Engine (for rewriting resource URLs on a per-IP-address basis, see above) which does not compile under no_std. Fortunately only a tiny fraction of the crate’s functionality is actually used in cotton-ssdp – functionality which can, as the least-worst option, just be reimplemented from scratch. But the first thing to do is eliminate the url::Url type from the public interface; this is a breaking change which would cause a semver major change, if it weren’t for the fact that the semver major already needed changing due to other API changes since 0.0.1.
9. 38694d1 Remove usage of "url" crate – this is where the url.set_ip_host call is replaced with a simpler equivalent. Notably this new code has a bug, only fixed later in b92ded1, which causes it to do the Wrong Thing with any URL containing an explicit port (such as, er, the example ones discussed above where the need for the rewriting is explained).
10. 261a6bb Replace HashMap usage with BTreeMap – as mentioned, HashMap is not available in no_std, but BTreeMap is, and has essentially the same interface.
11. 0faacd5 Actually compile ssdp including Engine with no_std – finally cargo build --no-default-features works (including Engine and Message). Moreover, so does cargo build --no-default-features --target thumbv7em-none-eabi – that’s a freestanding target with no std support whatsoever.

Oh, plus there’s another commit that wasn’t on that branch, but should have been:

12. b92ded1 Fix rewrite_host bogusly stripping ":port" from URLs – fixes a bug introduced when removing the “url” crate. It seems pretty likely that the only forms of URL that make any sense as SSDP locations start with http://127.0.0.1..., not even HTTPS – but the Principle of Least Surprise suggests that the code should at least not flake out if given one in a different form.

Between them, those commits hopefully tell the whole story of taking a crate initially written in hosted Rust, and making it work in both hosted and freestanding applications.

6. Using the remarkable free tools available

Most of the anecdotes in this post are hopefully applicable to both proprietary and open-source software development. But in the open-source case in particular, it’s worth looking into the freely-available online tools that support such development efforts. Even getting started in Rust development introduces everyone to crates.io and to docs.rs, but it’s also worth looking into:

Github Actions cloud CI

Continuous-integration servers run by Github, and automatically triggered on pushes to Github repositories (if your project contains YAML files for workflows, perhaps a bit like cotton’s one).

docs.rs

Documentation pages on docs.rs are built automatically for all crates published to crates.io. The build is always done with a nightly toolchain, so you can use nightly-only features such as doc_auto_cfg – just make sure you leave your crate buildable by stable and other non-nightly toolchains, perhaps by guarding the doc_auto_cfg attribute like this:

#![cfg_attr(nightly, feature(doc_auto_cfg))]

and then adding this to your Cargo.toml such that docs.rs builds always enable the “nightly” cfg:

[package.metadata.docs.rs] all-features = true rustdoc-args = ["--cfg", "nightly"]

deps.rs

Checks the dependencies in your Cargo.toml, and warns you about out-of-date versions (and warns you strongly about ones with known security problems). It’s wired up to Github, so can be used on any project published there. (For Cotton, that means there’s one page for the whole monorepo, not one page per crate, but it’s subdivided into the dependencies of each crate.)

codecov.io

Code-coverage as a service. You have to sign up for an account, but it’s free for public repositories. Support can be incorporated into Github CI workflows, which is how it works for cotton-ssdp.

The neat thing is that many of these services provide “badges”: little status icons that you can add to README.md on Github, or to any web page. The statuses below aren’t just a pretty picture: each one is the live status, on most of them you can click-through to the underlying data, and as you’re reading them right now, you might be noticing an issue before I do:

Cotton project:
cotton-netif:
cotton-ssdp:

Network firewall on a Raspberry Pi using OpenWRT

Previously on #homelab:

• Setting up a Raspberry Pi as home server
• Hub-and-spoke backups using Syncthing
• Self-hosted CI for Rust and C++ using Laminar

For a while I’ve been using a TP-Link TL-R860 router as the main firewall and switch for my home ethernet network. This has worked well and reliably (unlike the Linksys WRT54-G which I eventually put on a Christmas lights timer to reboot it every night), but it’s old now and is no longer getting security updates (if it ever did). Its HTTPS configuration UI understands nothing newer than SSL3.0, and I wouldn’t even know where to find a browser prepared to lower itself to that these days. As it’s the final line of defence for my home network, I wanted to upgrade to something a bit newer.

I’ve still got a handful of Raspberry Pi 3 devices knocking around, so I thought I’d press one of them into service as the new ethernet-to-ethernet firewall – plus, OpenWRT is open-source and widely used, and so probably more trustworthy than a proprietary router that’s (searches “Amazon” folder in Evolution) over ten years old now. And gigabit switches are cheap these days, which might provide a handy speedup, at least for internal traffic, from the 10/100 switch in the TP-Link. That means that the goal is for the network to look like this:

So here’s a complete list of parts:

• Raspberry Pi 3B (or a Raspberry Pi 4 if you can get one)
• Branded Raspberry Pi power supply
• A USB flash drive to be the root filesystem (it doesn’t need to be high-performance; I used a Sandisk one which is literally “Amazon’s Choice” for the search “flash drive”)
• A USB ethernet adaptor supported by Linux (I used an Amazon Basics one)
• A gigabit ethernet switch (any one will do; I picked a Zyxel one mostly because I wanted one with the LEDs on the front and the sockets on the back)

Just as when setting up a home server, if you’re using a Raspberry Pi 3 you’ll need to enable boot-from-USB; the process is described in the linked post and you’ll need to boot once from a microSD card to be able to make the configuration change. The Raspberry Pi 4, though, can boot from USB out-of-the-box.

Alternatively, you could just stick with a microSD card for the whole process, but it’s persistently rumoured that running a Raspberry Pi from a microSD card is bad for long-term reliability, and that running from USB is better.

This blog post will cover:

1. Installing OpenWRT
2. Adding a second ethernet interface via USB
3. The security setup
4. Conclusion

The security setup will, as in previous posts, follow the principle of minimising potential attack surface. This post is as much about writing down for my own benefit what I did here, as it is for any wider purpose!

1. Installing OpenWRT

OpenWRT is at heart just a very custom Linux distribution, and it’s installed onto the Raspberry Pi like any other Linux. The OpenWRT folks have a specific page on using the Raspberry Pi. Download the latest “Firmware OpenWRT Install” img.gz file from the Installation section there; when I did it, that was version 22.03.3.

Install it on your flash drive (or microSD card) using rpi-imager (on Ubuntu, sudo apt install rpi-imager). Click “Choose OS” then “Use custom” then select the img.gz file you just downloaded. The click “Choose storage” and select your USB flash drive (or microSD card); be careful because other disks that you don’t want wiped may also be listed. Then click “Write”.

The default image contains a boot partition that assumes that the root filesystem will be on the microSD card (/dev/mmcblk0p2); if you’re using a USB flash drive then that’s not going to work. So in that case you need to mount the boot partition from the flash drive, and edit cmdline.txt to change the part of the command-line that specifies the root partition. Find the part that says root=/dev/mmcblk0p2 and change it to root=/dev/sda2. Don’t alter the rest of the command-line. Unmount the partition and unplug the flash drive.

With that flash drive, your Raspberry Pi is now able to run OpenWRT. Initially the new OpenWRT install will set itself up with IP address 192.168.1.1 on the Raspberry Pi ethernet interface; you can connect to this directly with an ethernet cable from a laptop (or desktop), setting the laptop’s IP address to 192.168.1.2. You can then use SSH from the laptop to log into the Raspberry Pi:

ssh root@192.168.1.1

Now would be a good time, as OpenWRT itself now suggests, to change the root password from the default blank one.

The Raspberry Pi isn’t yet a very useful firewall, though, because it only has the one ethernet port. It does additionally have wifi, but that’s not enabled by default and anyway I’m specifically after an ethernet-to-ethernet firewall here. So it’s time to add the second ethernet interface.

2. Adding a second ethernet interface via USB

The goal is for the Raspberry Pi’s built-in ethernet to be the inside of the firewall, connected to the switch, and for the USB ethernet to be the outside, connected to my ISP’s router.

Parts of this section are based on Vladimír Záhradník’s article on Medium (also here) – but, like me, you’ll need to go off-piste from that script a little if it’s awkward to connect the firewall to the internet straightaway.

And it’s likely that the firewall won’t be on the internet straightaway, because although the laptop can use SSH or HTTP to talk to the Raspberry Pi, the laptop probably won’t be set up to forward packets, so the Raspberry Pi itself will not be able to access the internet.

The first thing to do is work out what Linux driver is needed for your chosen USB etthernet adaptor; the easiest way to do that, is just plug it in to an existing Linux box and see what module gets auto-loaded; in my case it was ax88179_178a.

You need to find the equivalent driver package starting from downloads.openwrt.org: I clicked on “OpenWRT 22.03.3” then “bcm27xx” then “bcm2710” (that’s for Raspberry Pi 3, for a 4 it would be “bcm2711”), then “packages”, then I looked along the packages starting with “kmod-usb-net” until I found one with “ax88179” in the name: this turned out to be kmod-usb-net-asix-ax88179_5.10.161-1_aarch64_cortex-a53.ipk.

Now from a fully-operational OpenWRT install, I could just install that using “opkg install kmod-usb-net-asix-ax88179” and it would fetch and install it directly; for this initial setup, though, I had to download it manually onto the laptop, then scp it to the firewall, then install locally:

on the laptop	scp kmod-usb-net-asix-ax88179_5.10.161-1_aarch64_cortex-a53.ipk root@192.168.1.1:
	ssh root@192.168.1.1
now on the Pi	opkg install ./kmod-usb-net-asix-ax88179_5.10.161-1_aarch64_cortex-a53.ipk

And the opkg command failed: it needed some further modules, some dependencies. An online opkg would fetch and install dependencies automatically, but I had to do that manually, by repeating the download-and-scp process for the following:

• kmod-libphy_5.10.161-1_aarch64_cortex-a53.ipk
• kmod-mii_5.10.161-1_aarch64_cortex-a53.ipk
• kmod-usb-net_5.10.161-1_aarch64_cortex-a53.ipk

Once those were installed, the main ax88179 package could also be successfully installed.

When I did this, the ethernet adaptor was still not being detected, and it took me far too long to realise that it was still plugged into the laptop from when I was working out what driver it needed. As soon as I plugged it back into the Raspberry Pi, it worked fine and appeared in OpenWRT’s web UI as eth1 (with eth0 being the built-in ethernet).

3. The security setup

Almost everything else can now be configured from the web UI. Under the “Network” menu, choose “Interfaces”, then “Add new interface”, call it “wan”, choose “DHCP client”, and assign it to eth1.

Under “Network” and “Firewall” the default will probably already be correct: LAN-to-WAN traffic gets forwarded, WAN-to-LAN traffic gets rejected except that masquerading (i.e., NAT) is enabled.

Under “System” and “Administration” you can set up SSH access: have it listen on the LAN interface only and give it your SSH public key. Once you’re sure that SSH public-key authentication is working, you can disable SSH password authentication. (In the worst-case scenario, you can still log in to the firewall on console, using a keyboard and monitor.)

Under “HTTP(S) Access”, make sure “Redirect to HTTPS” is turned off; this is less secure but we’ll be re-securing it a bit later.

Back under “Network” and “Interfaces”, choose “Edit” for the LAN interface and you can set the IP address and configure the DHCP server. You get a warning message when changing the settings, because that’s the interface from which you’re using the web UI, but if you’re confident in your settings then make the change; worst case you might need to unplug and re-plug the network cable in order for the laptop to realise it needs to renegotiate its own IP address on ethernet.

You can now connect the firewall’s WAN interface to the upstream (ISP) router and use SSH to log in to the firewall (via its LAN interface) under its final LAN IP address. The firewall now has internet connectivity via the upstream router, so now is a good time to log in and install any additional packages you might need:

opkg install nano

Now is also a good time to set up anything required on the upstream router – for instance a static DHCP address, or any port forwarding.

I made one final security-related configuration change. By default the web UI listens on all interfaces, on both IPv4 and IPv6. That’s not terrible, because a further login is still required to actually do anything, and because firewall rules on the WAN interface will drop connection requests anyway. But just to reduce attack surface, I changed that to only listen for connections from localhost. To do that, SSH into the firewall and edit /etc/config/httpd. As first installed, it will have lines a bit like these:

/etc/config/uhttpd
... list listen_http 0.0.0.0:80 list listen_http [::]:80 list listen_https 0.0.0.0:443 list listen_https [::]:443 ...

And to change it to listen on localhost only, replace those four lines with this one (leaving the rest of the file unchanged):

/etc/config/uhttpd
... list listen_http '127.0.0.1:80' ...

Now the only way to access the web UI is via SSH tunnelling – in other words, only an attacker with SSH credentials can change anything. So I use a script to run the SSH command, in order to avoid remembering the command parameters (“donk” is the hostname of the firewall):

ssh-donk
#!/bin/bash # 8001: openwrt exec ssh -t \ -L 8001:localhost:80 \ root@donk

And with that running on the laptop, the OpenWRT web UI is accessible in the laptop’s web browser as http://localhost:8001.

4. Conclusion

Setting up a Raspberry Pi 3 as a firewall was relatively straightforward. OpenWRT is well-supported and gets ongoing security maintenance, and is much less likely to contain unexpected backdoors (or even bugs) than an ancient proprietary firewall. And its performance is good – or at least, better than the dusty old TP-Link thing. From inside the firewall, my laptop now gets 94Mbits/s download, 20Mbits/s upload – up from 28 down and 8 up when using the previous firewall. Quite possibly a Raspberry Pi 4 could do even better; 94Mbits/s is very close to saturating the Raspberry Pi 3’s onboard 10/100 ethernet interface, whereas the Raspberry Pi 4 has gigabit ethernet onboard.

Self-hosted CI for Rust and C++ using Laminar

Previously on #homelab:

• Setting up a Raspberry Pi as home server
• Hub-and-spoke backups using Syncthing

I’ve been a keen user of the Jenkins continuous-integration server (build daemon) since the days when it was called Hudson. I set it up on a shadow IT basis under my desk at Displaylink, and it was part of Electric Imp’s infrastructure from the word go. But on the general principle that you don’t appreciate a thing unless you know what the alternatives are like, I’ve recently been looking at Laminar for my homelab CI setup.

Laminar is a much more opinionated application, mostly in the positive sense of that term, than Jenkins. If Jenkins (as its icon suggests) is an obsequious English butler or valet, then Laminar is the stereotype of the brusque New Yorker: forgot to mark your $JOB.init script as executable? “Hey pal. I’m walking here.”

But after struggling occasionally with the complexity and sometimes opacity of Jenkins (which SSH key is it using?) the simplicity and humility of Laminar comes as a relief. Run a sequence of commands? That’s what a shell is for; it’s not Laminar’s job; Laminar just runs a single script. Run a command on a timer? That’s what cron (or anacron) is for; it’s not Laminar’s job; Laminar provides a trigger command that you can add to your crontab.

So what does it provide? Mostly sequencing, monitoring, statistics-gathering, artifact-gathering, and a web UI. (Unlike Jenkins, the web UI is read-only – but as it exposes the contents of all packages built using it, it’s still best to keep it secure.) I have mine set up to, once a week, do a rustup update and then check that all my projects still build and pass their tests with the newest nightly build (and beta, and stable, and the oldest supported version). It’s very satisfying to glance at the Laminar page and be reassured that everything still builds and works, even if I’ve been occupied with other things that week. (And conversely, on the rare occasions when a new nightly breaks something, at least I find out about it early, as opposed to it suddenly being in my way at a time when I’m filled with the urge to be writing some new thing.)

This blog post will cover:

1. Installing Laminar
2. CI for Chorale, a C++ package
3. CI for Cotton, a Rust package
4. Setting up Git to build on push
5. CI for rustup

You should probably skim at least the first part of the C++ section even if you’re mostly interested in Rust, as it introduces some basic Laminar concepts and techniques.

By the way, it’s reasonable to wonder whether, or why, self-hosted CI is even a thing, considering that Github Actions offer free CI for open-source projects (up to a certain, but very generous, usage limit). One perfectly adequate answer is that the hobby of #homelab is all about learning how things work – learning which doesn’t happen if someone else’s cloud service is already doing all the work. But there are other good answers too: eventually (but not in this blog post) I’m going to want CI to run tests on embedded systems, STM32s and RP2040s and the like – real physical hardware, which is attached to servers here but very much not attached to Github’s CI servers. (Emulation can cover some of this, but not for instance driver work, where the main source of bugs is probably misconceptions about how the actual hardware works.) Yet a third reason is trust: for a released open source project there’s, by definition, no point in keeping the source secret. But part of the idea of these blog posts is to document best practices which commercial organisations, too, might wish to adopt – and they might have very different opinions on uploading their secret sauce to third-party services, even ones sworn to secrecy by NDA. And even a project determined to make the end result open-source, won’t necessarily be making all their tentative early steps out in the open. Blame Apple, if you like, for that attitude; blame their habit of saying, “By the way, also, this unexpected thing. And you can buy it today!”

1. Installing Laminar

This is the part where it becomes clear that Laminar is quite a niche choice of CI engine. It is packaged for both Debian and Ubuntu, but there is a bug in both the Debian and Ubuntu packages – it’s not upstream, it’s in the Debian patchset – which basically results in nothing working. So you could either follow the suggestions in the upstream bug report of using a third-party Ubuntu PPA or the project’s own binary .deb releases, or you could do what I did and install the broken Ubuntu package anyway (to get the laminar user, the systemd scripts, etc. set up), then build Laminar 1.2 from upstream sources and install it over the top.

Either way, if you navigate to the Laminar web UI (on port 8080 of the server) and see even the word “Laminar” on the page, your installation is working and you’ve avoided the bug.

The default install sets the home directory for the laminar user to /var/lib/laminar; this is the Debian standard for system users, but to make things less weird for some of the tools I’m expecting Laminar to run (e.g., Cargo), I changed it (in /etc/passwd) to be /home/laminar.

2. CI for Chorale, a C++ package

I use Laminar to do homelab builds of Chorale, a C++ project comprising a UPnP media-server, plus some related bits and pieces. For the purposes of this blog post, it’s a fairly typical C++ program with a typical (and hopefully pretty sensible) Autotools-based build system.

Laminar is configured, in solid old-school Unix fashion, by a collection of text files and shell scripts. These all live (in the default configuration) under /var/lib/laminar/cfg, which you should probably chown -R to you and also check into a Git repository, to track changes and keep backups. (The installation process sets up a user laminar, which should remain a no-login user.)

All build jobs execute in a context, which allows sophisticated build setups involving multiple build machines and so on; for my purposes everything executes in a simple context called local:

/var/lib/laminar/cfg/contexts/local.conf
EXECUTORS=1 JOBS=*

This specifies that only one build job can be run at a time (but it can be any job), overriding Laminar’s default context which allows for up to six executors: it’s just a Raspberry Pi, after all, we don’t want to overstress it.

2.1 C++: building a package

When it comes to the build directories for its build jobs, Laminar is much more disciplined (or, again, opinionated) than Jenkins: it keeps a rigid distinction between (a) the build directory itself, which is temporary, (b) a persistent directory shared between runs of the same job, the workspace, and (c) a persistent directory dedicated to each run, the archive. So the usual pattern is to keep the git checkout in the workspace (to save re-downloading the whole repo each time), then each run can do a git pull, make a copy of the sources into the build directory, do the build (leaving the workspace with a clean checkout), and finally store its built artifacts into the archive. All of which is fine except for the very first build, which needs to do the git clone. In Laminar this is dealt with by giving the job an init script (remember to mark it executable!):

/var/lib/laminar/cfg/jobs/chorale.init
#!/bin/bash -xe git clone /home/peter/git/chorale.git .

as well as its normal run script (which also needs to be marked executable):

/var/lib/laminar/cfg/jobs/chorale.run
#!/bin/bash -xe ( flock 200 cd $WORKSPACE/chorale git pull --rebase cd - cp -al $WORKSPACE/chorale chorale ) 200>$WORKSPACE/lock cd chorale libtoolize -c aclocal -I autotools autoconf autoheader echo timestamp > stamp-h.in ./configure make -j4 release cp chorale*.tar.bz2 $ARCHIVE/ laminarc queue chorale-package

There’s a few things going on here, so let’s break it down. The business with flock is a standard way, suggested in Laminar’s own documentation, of arranging that only one job at a time gets to execute the commands inside the parentheses – this isn’t necessarily likely to be an issue, as we’ve set EXECUTORS=1, but git would get in such a pickle if it happened that it’s a sensible precaution anyway. These protected commands update the repository from upstream (here, a git server on the same machine), then copy the sources into the build directory (via hard-linking, cp’s -l, to save time and space).

Once that’s done, we can proceed to do the actual build; the commands from libtoolize as far as make are the standard sequence for bootstrapping an Autotools-based C++ project from bare sources. (It’s not exactly Joel Test #2 compliant, mostly for Autotools reasons, although at least any subsequent builds from the same tree would be single-command.)

Chorale, as is standard for C++ packages, is released and distributed as a source tarball, which in this case is produced by the release target in the Makefile. The final cp command copies this tarball to the Laminar archive directory corresponding to this run of this job. (The archive directory will have a name like /var/lib/laminar/archive/chorale/33, where the “33” is a sequential build number.)

The final command, laminarc for “laminar client”, queues-up the next job in the chain, testing the contents of the Chorale package. (The bash -xe at the top, ensures that if the build process produces any errors, the script will terminate with an error and not get as far as kicking off the test job.)

That’s all that’s needed to set up a simple C++ build job – Laminar doesn’t have any concept of registering or enrolling a job; just the existence of the $JOB.run file is enough for the job to exist. To run it (remembering that the web UI is read-only), execute laminarc queue chorale and you should see the web UI spring into life as the job gets run. Of course, it will fail if any of the prerequisites (gcc, make, autoconf, etc.) are missing from the build machine; add them either manually (sudo apt-get install ...) or perhaps using Chef, Ansible or similar. Once the build succeeds (or fails) you can click around in the web UI to find the logs or perhaps download the finished tarball.

2.2 C++: running tests

The next job in the chain, chorale-package, tests that the packaging process was successful (and didn’t leave out any important files, for instance); it replicates what the user of Chorale would do after downloading a release. This time the run script gets the sources not from git, but from the package created by (the last successful run of) the chorale job, so no init script is needed:

/var/lib/laminar/cfg/jobs/chorale-package.run
#!/bin/bash -xe PACKAGE=/var/lib/laminar/archive/chorale/latest/chorale-*.tar.bz2 tar xf $PACKAGE cd chorale* ./configure make -j4 EXTRA_CCFLAGS=-Werror make -j4 EXTRA_CCFLAGS=-Werror check laminarc queue chorale-gcc12 chorale-clang

Like a user of Chorale, the script just untars the package and expects configure and make to work. The build fails if that doesn’t happen. This job also runs Chorale’s unit-tests using make check. This time, we build with the C++ compiler’s -Werror option, to turn all compiler warnings into hard errors which will fail the build.

If everything passes, it’s clear that everything is fine when using the standard Ubuntu C++ compiler. The final two jobs, kicked-off whenever the chorale-package job succeeds, build with alternative compilers just to get a second opinion on the validity of the code (and to avoid unpleasant surprises when the standard compiler is upgraded in subsequent Ubuntu releases):

/var/lib/laminar/cfg/jobs/chorale-gcc12.run
#!/bin/bash -xe PACKAGE=/var/lib/laminar/archive/chorale/latest/chorale-*.tar.bz2 tar xf $PACKAGE GCC_FLAGS="-Werror" cd chorale* ./configure CC=gcc-12 CXX=g++-12 make -j4 CC=gcc-12 EXTRA_CCFLAGS="$GCC_FLAGS" make -j4 CC=gcc-12 EXTRA_CCFLAGS="$GCC_FLAGS" GCOV="gcov-12" check

New compiler releases sometimes introduce new, useful warnings; this script is a good place to evaluate them before adding them to configure.ac. Similarly, the chorale-clang job checks that the sources compile with Clang, a compiler which has often found issues that G++ misses (and vice versa). Clang also has some useful extra features, the undefined-behaviour sanitiser and address sanitiser, which help to detect code which compiles but then can misbehave at runtime:

/var/lib/laminar/cfg/jobs/chorale-clang.run

#!/bin/bash -xe PACKAGE=/var/lib/laminar/archive/chorale/latest/chorale-*.tar.bz2 tar xf $PACKAGE cd chorale* ./configure CC=clang CXX=clang++ # -fsanitize=thread incompatible with gcov # -fsanitize=memory needs special libc++ # for SANE in undefined address ; do CLANG_FLAGS="-Werror -fsanitize=$SANE -fno-sanitize-recover=all" make -j4 CC=clang EXTRA_CCFLAGS="$CLANG_FLAGS" make -j4 CC=clang EXTRA_CCFLAGS="$CLANG_FLAGS" GCOV="llvm-cov gcov" tests make clean done

If the Chorale code passes all of these hurdles, then it’s probably about as ready-to-release as it’s possible to programmatically assess.

3. CI for Cotton, a Rust package

All the tools and dependencies required to build a typical C++ package are provided by Ubuntu packages and are system-wide. But Rust’s build system encourages the use of per-user toolchains and utilities (as well as per-project dependencies). So before we do anything else, we need to install Rust for the laminar user – i.e., as the laminar user – which requires a moment’s thought, as we carefully set up laminar to be a no-login user. So we can’t just su to laminar and run rustup-init normally; we have to use su to execute one command at a time from a normal user account.

So start by downloading the right rustup-init binary for your system – here, on a Raspberry Pi, that’s the aarch64-unknown-linux-gnu one. But then execute it (and then use it to download extra toolchains) as the laminar user (bearing in mind that rustup-init’s careful setup of the laminar user’s $PATH will not be in effect):

$ $ $ $ $ $

sudo -u laminar /home/peter/rustup-init sudo -u laminar /home/laminar/.cargo/bin/rustup toolchain install beta sudo -u laminar /home/laminar/.cargo/bin/rustup toolchain install nightly sudo -u laminar /home/laminar/.cargo/bin/rustup toolchain install 1.56 sudo -u laminar /home/laminar/.cargo/bin/rustup +nightly component add llvm-tools-preview sudo -u laminar /home/laminar/.cargo/bin/cargo install rustfilt

The standard rustup-init installs the stable toolchain, so we just need to add beta, nightly, and 1.56 – that last chosen because it’s Cotton’s “minimum supported Rust version” (MSRV), which in turn was selected because it was the first version to support the 2021 Edition of Rust, and that seemed to be as far back as it was reasonable to go. We also install llvm-tools-preview and rustfilt, which we’ll be using for code-coverage later.

So to the $JOB.run scripts for Cotton. What I did here was notice that I’ve actually got a few different Rust packages to build, and they all need basically the same things doing to them. So I took advantage of the Laminar /var/lib/laminar/cfg/scripts directory, and made all the infrastructure common among all the Rust packages. When running a job, Laminar arranges that the scripts directory is on the shell’s $PATH (and note that it’s in the cfg directory, so will be captured and versioned if you set that up as a Git checkout). This means that, as far as Cotton is concerned – after an init script that’s really just like the C++ one:

/var/lib/laminar/cfg/jobs/cotton.init
#!/bin/bash -xe git clone /home/peter/git/cotton.git .

– the other build scripts come in pairs: one that’s Cotton-specific but really just runs a shared script which is generic across projects, and then the generic one which does the actual work. We’ll look at the specific one first:

/var/lib/laminar/cfg/jobs/cotton.run
#!/bin/bash -xe BRANCH=${BRANCH-main} do-checkout cotton $BRANCH export LAMINAR_REASON="built $BRANCH" laminarc queue cotton-doc BRANCH=$BRANCH \ cotton-grcov BRANCH=$BRANCH \ cotton-1.56 BRANCH=$BRANCH \ cotton-beta BRANCH=$BRANCH \ cotton-nightly BRANCH=$BRANCH

The assignment to BRANCH is a Bash-ism which means, “use the variable $BRANCH if it exists, but if it doesn’t exist, default to main”. This is usually what we want (in particular, a plain laminarc queue cotton will build main), but making it flexible will come in handy later when we build the Git push hook. All the actual building is done by the do-checkout script, and then on success (remembering that bash -xe means the script gets aborted on any failures) we go on to queue all the downstream jobs. Note that when parameterising jobs using laminarc’s VAR=VALUE facility, each VAR applies only to one job, not to all the jobs named.

The do-checkout script is very like the one for Chorale, including the flock arrangement to serialise the git operations, and differing only in that it takes the project and branch to build as command-line parameters – and of course includes the usual Rust build commands instead of the C++/Autotools ones. (This time we can take advantage of rustup-init’s (Cargo’s) $PATH setup, but only if we source the environment file directly.)

/var/lib/laminar/cfg/scripts/do-checkout
#!/bin/bash -xe PROJECT=$1 BRANCH=$2 # WORKSPACE is predefined by Laminar itself ( flock 200 cd $WORKSPACE/$PROJECT git fetch git checkout $BRANCH git pull --rebase cd - cp -al $WORKSPACE/$PROJECT $PROJECT ) 200>$WORKSPACE/lock source $HOME/.cargo/env rustup default stable cd $PROJECT cargo build --all-targets cargo test

Notice that this job explicitly uses the stable toolchain, minimising the chance of version-to-version breakage. We also want to test on beta, nightly, and MSRV though, which is what three of those downstream jobs are for. Here I’ll just show the setup for nightly, because the other two are exactly analogous. Again there’s a pair of scripts; firstly, there’s the specific one:

/var/lib/laminar/cfg/jobs/cotton-nightly.run
#!/bin/bash -xe exec do-buildtest cotton nightly ${BRANCH-main}

Really not much to see there. All the work, as before, is done in the generic script, which is parameterised by project and toolchain:

/var/lib/laminar/cfg/scripts/do-buildtest
#!/bin/bash -xe PROJECT=$1 RUST=$2 BRANCH=$3 SOURCE=/var/lib/laminar/run/$PROJECT/workspace ( flock 200 cd $SOURCE/$PROJECT git checkout $BRANCH cd - cp -al $SOURCE/$PROJECT $PROJECT ) 200>$SOURCE/lock source $HOME/.cargo/env rustup default $RUST cd $PROJECT cargo build --all-targets --offline cargo test --offline

Here we lock the workspace again, just to avoid any potential clashes with a half-finished git update, but we don’t of course do another git update – we want to build the same version of the code that we just built with stable. For similar reasons, we run Cargo in offline mode, just in case anyone published a newer version of a dependency since we last built.

That’s the cotton-beta, cotton-nightly, and cotton-1.56 downstream jobs dealt with. There are two more: cotton-doc and cotton-grcov, which deal with cargo doc and code coverage respectively. The documentation one is the more straightforward:

/var/lib/laminar/cfg/jobs/cotton-doc.run
#!/bin/bash -xe exec do-doc cotton ${BRANCH-main}

And even the generic script (parameterised by project) is quite simple:

/var/lib/laminar/cfg/scripts/do-doc
#!/bin/bash -xe PROJECT=$1 BRANCH=$2 SOURCE=/var/lib/laminar/run/$PROJECT/workspace ( flock 200 cd $SOURCE/$PROJECT git checkout $BRANCH cd - cp -al $SOURCE/$PROJECT $PROJECT ) 200>$SOURCE/lock source $HOME/.cargo/env rustup default stable cd $PROJECT cargo doc --no-deps --offline cp -a target/doc $ARCHIVE

It much resembles the normal build, except for running cargo doc instead of a normal build. On completion, though, it copies the finished documentation into Laminar’s $ARCHIVE directory, which makes it accessible from Laminar’s web UI afterwards.

The code-coverage scripts are more involved, largely because I couldn’t initially get grcov to work, and ended up switching to using LLVM’s own coverage tools instead. (But the scripts still have “grcov” in the names.) Once more the per-project script is simple:

/var/lib/laminar/cfg/jobs/cotton-grcov.run
#!/bin/bash -xe exec do-grcov cotton ${BRANCH-main}

And the generic script does the bulk of it (I cribbed this recipe from the rustc book, q.v.; I didn’t come up with it all myself):

/var/lib/laminar/cfg/scripts/do-grcov

#!/bin/bash -xe PROJECT=$1 BRANCH=$2 SOURCE=/var/lib/laminar/run/$PROJECT/workspace ( flock 200 cd $SOURCE/$PROJECT git checkout $BRANCH cd - cp -al $SOURCE/$PROJECT $PROJECT ) 200>$SOURCE/lock source $HOME/.cargo/env rustup default nightly cd $PROJECT export RUSTFLAGS="-Cinstrument-coverage" export LLVM_PROFILE_FILE="$PROJECT-%p-%m.profraw" cargo test --offline --lib rustup run nightly llvm-profdata merge -sparse `find . -name '*.profraw'` -o cotton.profdata rustup run nightly llvm-cov show \ $( \ for file in \ $( \ cargo test --offline --lib --no-run --message-format=json \ | jq -r "select(.profile.test == true) | .filenames[]" \ | grep -v dSYM - \ ); \ do \ printf "%s %s " -object $file; \ done \ ) \ --instr-profile=cotton.profdata --format=html --output-dir=$ARCHIVE \ --show-line-counts-or-regions --ignore-filename-regex='/.cargo/' \ --ignore-filename-regex='rustc/'

Honestly? Bit of a mouthful. But it does the job. Notice that the output directory is set to Laminar’s $ARCHIVE directory so that, again, the results are viewable through Laminar’s web UI. (Rust profiling doesn’t produce branch coverage as such, but “Region coverage” – which counts what a compiler would call basic blocks – amounts to much the same thing in practice.) The results will look a bit like this:

Why yes, that is very good coverage, thank you for noticing!

4. Setting up Git to build on push

So far in our CI journey, we have plenty of integration, but it’s not very continuous. What’s needed is for all this mechanism to swing into action every time new code is pushed to the (on-prem) Git repositories for Chorale or Cotton.

Fortunately, this is quite straightforward – or, at least, good inspiration is available online. Pushes to the Git repository for Cotton can be hooked by adding a script as hooks/post-receive under the Git server’s cotton.git directory (the hooks directory is probably already there). In one of those Git features that at first makes you think, “this is a bit over-engineered”, but then makes you realise, “wait, this couldn’t actually be made any simpler while still working in full generality”, the Git server passes to this script, on its standard input, a line for every Git “ref” being pushed – for these purposes, refs are mostly branches – along with the Git revisions at the old tip and new tip of the branch.

Laminar comes with an example hook which builds every commit on every branch pushed. I admire this but don’t follow it; it’s great for preserving bisectability, but seems like it would lead to a lot of interactive rebasing every time a feature branch is rebased on a later main – not to mention a lot of building by the CI server. So the hook I actually use just builds the tip of each branch:

git/cotton.git/hooks/post-receive
#!/bin/bash -ex while read oldrev newrev ref do if [ "${ref:0:11}" == "refs/heads/" ]; then export BRANCH=${ref:11} export LAMINAR_REASON="git push $BRANCH" laminarc queue cotton BRANCH=$BRANCH fi done

The LAMINAR_REASON appears in the web UI and indicates which branch each run is building:

5. CI for rustup

The final piece of the puzzle, at least for today, is the continuous integration of Rust itself. As new nightlies, and betas, and even stable toolchains come out, I’d like it to be a computer’s job, not that of a person, to rebuild everything with the new version. (Especially if that person would be me.)

This too, however, is straightforward with all the infrastructure put in place by the rest of this blog post. All that’s needed is a new job file which runs rustup-update:

/var/lib/laminar/cfg/jobs/rustup-update.run
#!/bin/bash -ex export LAMINAR_REASON="rustup update" source $HOME/.cargo/env rustup update laminarc queue cotton assay sparkle

The rustup update command updates all the toolchains; once that is done, the script queues-up builds of all the Rust packages I build. I schedule a weekly build in the small hours of Thursday morning, using cron:

edit crontab using “crontab -e”
0 3 * * 4 LAMINAR_REASON="Weekly rustup" laminarc queue rustup-update

With a bit of luck, this means that by the time I sit down at my desk on Thursday morning, all the jobs have run and Laminar is showing a clean bill of health. As I’ve been playing with Rust for quite a long elapsed time, but really only taking it seriously in quite occasional bursts of energy, having everything kept up-to-date automatically during periods when I’m distracted by a different shiny thing is a real pleasure.

Hub-and-spoke backups using Syncthing

Syncthing is a synchronisation tool which can keep files and directories synchronised across several computers. This post is about how I set it up for secure backups of various Linux boxes, including off-site backup. It is based on Syncthing’s own excellent documentation and mostly documents the particular configuration I use to maintain various security constraints:

No computer holds, at rest, credentials to log into any other

SSH public keys are stored encrypted, with a passphrase required for usage.

No port except (somewhat unavoidably) SSH is exposed to the internet

Syncthing seems well-maintained and safe. But there’s no point offering an attack surface of both Syncthing and SSH, when I can just hide Syncthing inside SSH.

No computer holds, at rest, unencrypted files

The computers involved all either have full-disk encryption or home-directory encryption enabled. Information needing extra security (financial data etc.) is kept in an additional layer of encryption using EncFS and/or encfs-agent, and only opened when required; Syncthing sychronises the encrypted form.

I’ve got Syncthing installed on a total of four computers, in a hub-and-spoke manner:

• A Linux desktop, where I do most of my work;
• A Linux laptop (it’s a Lenovo Thinkpad);
• A Raspberry Pi that’s my home server;
• A second Raspberry Pi at a friend’s house as an off-site backup.

For the purposes of this post, we’ll call these boxes desktop, laptop, central, and offsite respectively. The hub of the hub-and-spoke setup is central; the other three only ever synchronise with central and not with each other. Both laptop and desktop connect inbound to central; central connects outbound to offsite. Each interaction is a full two-way sync, but in my case almost all new data flows from desktop and laptop to central and then offsite; almost no new data flows in the other direction.

Syncthing has its own package repository for Debian/Ubuntu, so installing it (on both clients and servers) involves following the instructions at https://apt.syncthing.net; there are packages for ARM64 Ubuntu (the Raspberry Pi) as well as AMD64 Ubuntu (for desktop and laptop). It typically runs in the background and is operated using a web GUI on port 8384; by default this (quite rightly) listens only for connections from localhost. On central, Syncthing starts as a service at boot time, and on the other boxes it is manually run when needed – this is a backup-on-demand setup, not automated backup.

Syncthing apparently also runs on Windows and Macintosh, but all the boxes I need to use it on run Linux.

Hub setup

The server, central, runs headless. To interact with the Syncthing web GUI from desktop, I need to SSH to central and use port forwarding. A script, on desktop, called ssh-central provides this:

ssh-central (1/2)
#!/bin/bash # 8000: central syncthing UI exec ssh -A -X -t \ -L localhost:8000:localhost:8384 \ peter@central

Alternatively, options could be added in ~/.ssh/config for host central, but doing it this way means that I can still SSH to central normally, with no port forwards, when needed.

Once that script is running and has connected (offering a shell prompt on central in your terminal window), going to https://localhost:8000 in your web browser will bring up Syncthing’s user interface; again consult Syncthing’s own documentation for what you’re looking at there.

By default Syncthing exposes the synchronisation protocol to all comers on a given port; this isn’t horrific as there is mutual authentication and a key exchange required before any actual synchronisation happens – but, on the general principle of minimising attack surface, I set it up to work like the GUI: available only to localhost, and requiring anyone else to set up SSH port forwarding in order to connect. From the Actions menu choose Settings, then Connections, and set the “Sync Protocol Listen Addresses” to tcp4://127.0.0.1:22000 and turn off NAT traversal, local discovery, and global discovery:

Adding “dial-in” spokes (workstations)

Although I’m using Syncthing in, effectively, a client/server mode, the protocol itself treats all connections as symmetrical peers. This means that two ports need to be forwarded over SSH by desktop: one for desktop’s Syncthing to connect to central’s Syncthing, and another in the opposite direction. (Honestly, I don’t see why it can’t do everything it needs over just one bidirectional TCP stream, but all the documentation suggests that it wants two.)

So this means that each workstation needs its own port allocated on the server, to be the listening address for server-to-workstation connections. I keep track of these in a file called lordwarden.txt on central...

lordwarden.txt
sync-port desktop 22001 laptop 22002 offsite 22003

...but that’s because I have an inveterate fondness for appalling puns, and you should pick a more sensible filename.

Once you’ve picked a port, you can write the do-syncthing script for each workstation. Here’s the one on desktop:

do-syncthing (desktop)
#!/bin/bash -x ssh -N \ -L localhost:22001:localhost:22000 \ -R localhost:22001:localhost:22000 \ peter@central & SSHPID=$! syncthing kill $SSHPID echo Done!

This sets up the SSH tunnel and starts the local (to desktop) instance of Syncthing. While this is running in a terminal, going to https://localhost:8384/ in your web browser will show the Syncthing user interface.

Here you need to set up the “Sync Protocol Listen Addresses” just like above, then (from the main screen) choose “Add Remote Device”. You’ll need to enter central’s Syncthing “Device ID”, which you can find in central’s user interface by opening the Actions menu and choosing “Show ID”. (I always just copy and paste the textual form and don’t bother with the QR code.) Then in the “Advanced” tab of the “Add Device” dialog, enter tcp://127.0.0.1:22001 – the port you chose for it in lordwarden.txt – under “Addresses” and then click “Save” to confirm.

It may take Syncthing a little while to successfully connect, but eventually central’s Syncthing will ask you to confirm the new peer’s identity, and the two boxes will be connected.

By default Syncthing just synchronises a default directory under $HOME, but you can add others and choose which peers to share them with. One useful directory to share is $HOME/.local/share/evolution/mail/local, which is the Evolution mail client’s “On This Computer” top-level maildir directory. The maildir format (unlike mbox) is very amenable to being shared in this way without risking corruption – though Evolution doesn’t always notice when the currently-selected folder is changed behind its back, and you have to select a different folder and then click back again.

Once you’ve finished setting-up and synchronising directories, go to the “Actions” menu on desktop and choose “Shutdown”; the do-syncthing shell-script will then exit cleanly.

On subsequent invocations of do-syncthing, there is of course nothing further to configure, and Syncthing will immediately perform the synchronisation, offering progress reports through its user interface.

Adding the “dial-out” spoke (offsite)

This is just a little harder, because you aren’t sitting in front of the box, because it’s off-site.

For the purposes of this post I’m going to gloss over the process of setting up a Raspberry Pi, installing it at a friend’s house, arranging that it has a port tunnelled through the friend’s firewall, and signing up with a free dynamic DNS service such as DuckDNS (no commercial relationship, just a satisfied customer) to give it a resolvable domain-name that will stay updated whenever your friend’s internet provider changes their IP address. So I’ll just assume that you’ve arranged that offsite.duckdns.org resolves to a box running an SSH server accessible on port 20202. Bear in mind that if you’ve set up your off-site Raspberry Pi to need a disk-encryption passphrase on every boot, you’ll need to visit your friend’s house to re-enter the passphrase every time they have a power cut!

(You could, of course, equally well use a hosting service or cloud service for your off-site backup – though, depending on how much you trust your hosting provider, you might want to look into Syncthing’s untrusted devices functionality. At time of writing, that is marked as being for beta/testing only.)

The security constraints at the top of this post, exclude the possibility that central statically has any credentials to log in to offsite. So to synchronise the two, you’ll need to sit at desktop or laptop with ssh-agent running, SSH to central with ssh-agent forwarding (the -A option visible in the ssh-central script), and then SSH again from central to offsite.

The script that connects from central to offsite needs to forward two Syncthing ports (as noted in lordwarden.txt) plus the port for the Syncthing user interface on offsite:

sync-offsite (central)
#!/bin/bash exec ssh -L 127.0.0.1:22003:127.0.0.1:22000 \ -R 127.0.0.1:22003:127.0.0.1:22000 \ -L 127.0.0.1:8040:127.0.0.1:8384 \ -p 20202 peter@offsite.duckdns.org syncthing

But of course that only forwards the user interface as far as central; to get it all the way to desktop where you can actually see it, you need to add another port forward to desktop’s ssh-central script:

ssh-central (2/2)
#!/bin/bash # 8000: central syncthing UI # 8040: offsite syncthing UI exec ssh -A -X -t \ -L localhost:8000:localhost:8384 \ -L localhost:8040:localhost:8040 \ peter@central

Armed with all of this, you can sit at desktop, run ssh-central to log in to central, and once there run sync-offsite to reach offsite. At this point offsite’s Syncthing can be reached in your web browser on desktop as https://localhost:8040, and you can set about configuring it. (But if offsite is a Raspberry Pi 3, it will take a little while to respond to the initial requests.) Set its “Sync Protocol Listen Addresses” to tcp4://127.0.0.1:22000 as above, and then add central as a new device, entering tcp://127.0.0.1:22000 as its “Addresses”. You might also need to tell central that offsite’s address is tcp://127.0.0.1:22003. Again it might take a little while to connect, but once it has you can tell it which folders to share and let the synchronisation proceed to completion. Once everything is synchronised, go to the “Actions” menu on offsite and choose “Shutdown”; the sync-offsite shell-script will then exit cleanly.

Again, any further invocations of ssh-central and sync-offsite will immediately start the synchronisation process without any more configuration needed.

Raspberry Pi as home server, with Ubuntu 22.04, USB boot, and headless FDE

Recently, and it might be said precipitately, Arch Linux ARM ended all support for ARMv5 architectures, including the Tonidoplug (Marvell Kirkwood) device which was acting as my home server. So after looking around a little for cheap, low-power ARM servers, I landed on the Raspberry Pi 3, largely because I already had a couple in a drawer. (As indeed does everybody else, which bodes well for avoiding surprise deprecations by distros.)

I thought I’d take the opportunity to improve the security story while I was at it, including adding Full Disk Encryption (FDE) using LUKS. But how is that to work on a headless (no screen or keyboard attached) server? How is the decryption key or passphrase to be entered?

Perhaps I could SSH into the box and type in the passphrase. But if it needs the passphrase to decrypt the root filesystem, how is the SSH server running? The answer is that it’s possible to set up Dropbear, a lightweight SSH server, so that it runs in the kernel’s initrd before the root filesystem is mounted. Ubuntu provides handy packages for dropbear-initramfs and cryptsetup-initramfs to make this easier.

The usual (and easiest) way to set up FDE on an x86 or x86_64 server is by using the normal Ubuntu installer. But installing Ubuntu on a Raspberry Pi involves its own custom installer, and there isn’t an easily-ticked option that just does it for you. So you’ll need to do it manually, as outlined in the following description. (Which might also be of use to anyone upgrading any other Ubuntu installation to use FDE – though be warned that it’s not possible to upgrade in-place, and we manage it on the Raspberry Pi because we’re also moving to a new root filesystem.)

To follow these steps you’ll need:

1. A Raspberry Pi 3B. (A Raspberry Pi 4, with its USB3 controller, would have higher performance – but they’re hard to get hold of at the moment.)
2. A proper power supply for the Raspberry Pi. Raspberry Pis are notoriously picky about power supplies, especially when driving USB bus-powered devices, so I used the branded Raspberry Pi one. Note that the Raspberry Pi 4 would need a different power supply.
3. An existing PC able to write images to a microSD card; I used Ubuntu but other Linuxes, Windows and Macintosh can also be made to work. (Many laptops, and some monitors, have SD slots, but you might still need a microSD-to-SD or microSD-to-USB adaptor.)
4. A microSD card – I used an 8GB one. It’s probably only needed during setup, so you could borrow one from a phone or MP3 player – but it will be erased during the process, so don’t use a vital one. (The “probably” is because, if your USB SSD is not quite Raspberry Pi compatible, you might need to keep it as a first-stage bootloader.)
5. A USB SSD, to be the main storage for your new server – I used a Samsung T7. (For some reason I assumed it would be 2.5in-sized, but of course it’s not, it’s smaller – about the size of the Raspberry Pi’s own PCB – which will make the overall thing much neater and tidier.)

Out of the box, the Raspberry Pi will only boot from a microSD card, so the first course of action is to enable booting from USB devices. This might not appear essential – microSD cards are very cheap, and it could easily be left in even with most of the data on the USB – but whether it’s the Raspberry Pi’s fault or that of the microSD card, running a Raspberry Pi from microSD has a bad reputation for long-term reliability. Similarly, using a device marketed as a USB SSD is likely to mean that it’s intended for heavier use than a USB thumb drive, as well as potentially being higher-performance.

Installing Ubuntu on a microSD card

Ubuntu have a good tutorial which you should just follow. The steps are basically these:

1. Put the microSD card into the adaptor and insert into your PC.
2. Install rpi-imager (on Ubuntu, sudo snap install rpi-imager) and run it.
3. Click “Choose OS”, then “Other general purpose OS”, then “Ubuntu”, then pick Ubuntu Server 20.04 or 22.04 64-bit from the list. (If you want 22.04 and it’s not on the list yet, download ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz from the Ubuntu release site and choose “Use custom” from the menu; you don’t need to un-XZ the img.xz file as rpi-imager will use the compressed version just fine.)
4. Click “Choose Storage” then choose your microSD card from the list (carefully! important other disks that you don’t want wiped will also be listed).
5. Write the image to the microSD card, then insert it back into the Pi and power-on the Pi to boot it.

The only wrinkle I found, was discovering the Raspberry Pi’s IP address after it came up on the network for the first time. The suggested

arp -na | grep -i "b8:27:eb"

wasn’t finding it, even following

ping -b [my local broadcast address]

and the only answer seemed to be attempting SSH connections to addresses in my DHCP range in turn until one responded.

The next thing is to enable USB booting. There is a tutorial at thepi.io, but it is Raspbian-centric; for Ubuntu use, firstly note that on Ubuntu 20.04 only the config file to modify is not /boot/config.txt but /boot/firmware/usercfg.txt. Also, the mentioned vcgencmd command is not installed by default in Ubuntu, but it’s available via sudo apt install libraspberrypi-bin.

Settings in usercfg.txt are only applied to the actual non-volatile configuration at boot time, so the reboot in the middle of that tutorial isn’t optional. In summary, the steps are, for 20.04:

sudo apt update
sudo apt upgrade
echo program_usb_boot_mode=1 | sudo tee -a /boot/firmware/usercfg.txt
sudo reboot
... log in again and ...
sudo apt install libraspberrypi-bin
vcgencmd otp_dump | grep 17

or for 22.04:

sudo apt update
sudo apt upgrade
echo program_usb_boot_mode=1 | sudo tee -a /boot/firmware/config.txt
sudo reboot
... log in again and ...
sudo apt install libraspberrypi-bin
vcgencmd otp_dump | grep 17

Either way the final output should be the value 3020000a.

If you have several Raspberry Pi devices to deal with, boot this image on all of them at this stage to update the non-volatile configuration of each of them; that means you can just swap USB devices later.

You might also want to change the hostname and default username at this point from the current ubuntu@ubuntu, here’s an example for bob@porpentine:

sudo hostnamectl set-hostname porpentine
sudo adduser bob
sudo adduser bob sudo
sudo adduser bob admin

Don’t delete the ubuntu user until you’re sure that you can SSH in as bob and use sudo!

Moving the Ubuntu installation to an encrypted USB SSD

Much of this is based on Hamy’s (non-Raspberry-Pi-based) headless FDE tutorial, but with adaptations for the Raspberry Pi scenario.

1. Plug the USB SSD into the Pi and check (using cat /proc/partitions) that it appears as /dev/sda. (If your SSD was supplied pre-formatted, you might have /dev/sda1 too; that’s fine as we’re going to delete it.)
2. Now install the necessary initramfs packages:

   sudo apt install dropbear-initramfs cryptsetup-initramfs busybox-initramfs

3. You’ll need to do a little setup for dropbear: first, find the SSH authorized_keys file corresponding to the keys/users that you’d like to be able to unlock the server, and copy it into the Dropbox initramfs directory: on 20.04 this is /etc/dropbear-initramfs:

   sudo cp .ssh/authorized_keys /etc/dropbear-initramfs/

   although on 22.04 it’s /etc/dropbear/initramfs:

   sudo cp .ssh/authorized_keys /etc/dropbear/initramfs/

   (Watch out, as the default Ubuntu 20.04 installation puts an empty file in ~/.ssh/authorized_keys – make sure you’re using a real one, probably scp’d on from an existing Linux installation).

   Now you’ll need to edit the configuration file – the available editor is nano. On Ubuntu 20.04 it’s:

   sudo nano /etc/dropbear-initramfs/config

   and on 22.04 it’s:

   sudo nano /etc/dropbear/initramfs/dropbear.conf

   In either case, uncomment the #DROPBEAR_OPTIONS= line and set it to:

   DROPBEAR_OPTIONS="-p 999 -s -j -k"

   This sets the listening port to 999 (from the default 22); doing this means that your SSH client won’t get confused that it’s connecting to a “different” SSH server on the same IP and port as the normal Ubuntu SSH server. The remaining options increase security by disabling password logins and port forwarding.

4. Now you need to partition the USB SSD. On the Pi, run sudo fdisk /dev/sda. Using fdisk is beyond the scope of this note, but you want to delete any existing partitions and add two more: /dev/sda1 the same size as /dev/mmcblk0p1 (which will be small, 256Mbytes), FAT formatted, for the bootloader and ramdisk, plus /dev/sda2, as yet unformatted, for the rest of the disk. The resulting partition table (“p” command in fdisk) should look a bit like this:

   Command (m for help): p
   Disk /dev/sda: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors
   Disk model: PSSD T7
   Units: sectors of 1 * 512 = 512 bytes
   Sector size (logical/physical): 512 bytes / 512 bytes
   I/O size (minimum/optimal): 512 bytes / 512 bytes
   Disklabel type: dos
   Disk identifier: 0x404755d2
   
   Device     Boot   Start        End    Sectors  Size Id Type
   /dev/sda1  *      2048     526335     524288   256M  c W95 FAT32 (LBA)
   /dev/sda2       526336 1953525167 1952998832 931.3G 83 Linux
   
   Filesystem/RAID signature on partition 1 will be wiped.

   This should look broadly similar to the partition table of the existing microSD card installation (fdisk -l /dev/mmcblk0):

   Disk /dev/mmcblk0: 7.41 GiB, 7948206080 bytes, 15523840 sectors
   Units: sectors of 1 * 512 = 512 bytes
   Sector size (logical/physical): 512 bytes / 512 bytes
   I/O size (minimum/optimal): 512 bytes / 512 bytes
   Disklabel type: dos
   Disk identifier: 0xf66f0719
   
   Device         Boot  Start      End  Sectors  Size Id Type
   /dev/mmcblk0p1 *      2048   526335   524288  256M  c W95 FAT32 (LBA)
   /dev/mmcblk0p2      526336 15523806 14997471  7.2G 83 Linux

5. Now let’s actually make the encrypted block device that will contain the root filesystem. (This section is adapted from an Alpine Linux tutorial.) Lay out the device using that tutorial’s “optimised for security” settings:

   sudo cryptsetup -v -c aes-xts-plain64 -s 512 --hash sha512 --pbkdf pbkdf2 --iter-time 5000 --use-random luksFormat /dev/sda2

   You will need to enter (for the first time!) the decryption passphrase, and repeat it to guard against mistypings.

   If we just wanted one encrypted partition, we might be nearly done here. But to lay out multiple partitions inside the encrypted block device, we need to also use LVM (Logical Volume Manager). We create a “PV” (Physical Volume) inside the encrypted block device, then a “VG” (Volume Group) inside the PV, then two “LVs” (Logical Volumes) inside the VG. LVM can do much more sophisticated things than we’re using it for here!

   

   First, open the LUKS device:

   sudo cryptsetup luksOpen /dev/sda2 lvmcrypt

   – the passphrase is of course required. Now the encrypted version of the block device is available as /dev/mapper/lvmcrypt.

   Then, create the LVM PV and the VG inside it:

   sudo pvcreate /dev/mapper/lvmcrypt
   sudo vgcreate vg0 /dev/mapper/lvmcrypt

   Now we can create the partitions we actually want, as LVs inside that VG. We’ll create two: a swap partition and a root partition:

   sudo lvcreate -L 16G vg0 -n swap
   sudo lvcreate -l 100%FREE vg0 -n root

   We size the swap partition at 16 Gbytes, and the root partition as the whole rest of the device – note the tricksy difference between “-l” and “-L”. Our two new partitions appear as /dev/vg0/swap and /dev/vg0/root, as can be verified by sudo lvscan. (They also appear as /dev/mapper/vg0-swap and /dev/mapper/vg0-root.)

   Now we can actually do something with those partitions:

   sudo mkfs.ext4 /dev/vg0/root
   sudo mkswap /dev/vg0/swap

   To use the new root filesystem, we need to manually add entries to /etc/fstab and /etc/crypttab. So in /etc/fstab change the “/” entry to read:

   /dev/mapper/vg0-root / ext4 defaults 0 1

   and in /etc/crypttab add:

   lvmcrypt /dev/sda2 none luks,discard,initramfs

   The “initramfs” is important – the initramfs setup scripts realise that it’s important to unlock the root filesystem at startup, but /dev/sda2 isn’t (at the time you’re executing these commands) the current root filesystem – so if you don’t go out of your way to tell it that unlocking /dev/sda2 is important, the initramfs setup scripts will omit the necessary components to unlock it!

   We need to copy the root filesystem into the new root filesystem; there are various ways to do this, but I just did:

   sudo mount /dev/vg0/root /mnt
   sudo rsync -xrpvltoD / /mnt/

   You’ll also need to edit the kernel command line to tell it where the root is: edit /boot/firmware/cmdline.txt to add a “root=” parameter – and, just to be sure, an “ip=dhcp” parameter:

   net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/vg0-root rootfstype=ext4 elevator=deadline rootwait fixrtc ip=dhcp

   During experimentation, it might be convenient to setup two alternative cmdline.txt files – cmdssd.txt and cmdmmc.txt, say – so that switching back and forth can be done using a simple “cp” command such as can be found in an initramfs rescue shell!

6. Remake the initramfs incorporating these changes (on a Raspberry Pi, running update-initramfs also copies the image to the firmware partition):

   sudo update-initramfs -u

7. Check that your initramfs actually contains a crypttab – this caused me a lot of frustration on the way to figuring this out:

   unmkinitramfs /boot/firmware/initrd.img tmp-init/
   cat tmp-init/cryptroot/crypttab

   The output should be one line much like /etc/crypttab (for me, it had changed the device name to a UUID, but the effect is the same; use the blkid command to check whether it picked the right UUID).

8. Now it’s the moment of truth – time to reboot the Pi and see whether it comes up with Dropbear waiting for your passphrase.

   sudo shutdown -r now

   Your SSH session will of course be disconnected. So, once the Raspberry Pi has rebooted (ethernet LEDs go off, then on again), connect to the initramfs Dropbear server we installed:

   ssh -p 999 root@192.168.168.103

   If all is well, you will see this, or something very like it:

   BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.4) built-in shell (ash)
   Enter 'help' for a list of built-in commands.
   
   #

   (If all is not well, you might reluctantly need to attach a monitor and keyboard to the Raspberry Pi to see what went wrong.)

   But in the happy case, the Raspberry Pi has loaded its initramfs, but not mounted root yet. So you have very limited tools to play with, but the one thing you should have is the cryptroot-unlock command. This will ask you for your passphrase, open the root filesystem, and allow booting to proceed (your SSH connection will be closed). You’ll need to repeat this login and unlock process every time the Raspberry Pi reboots – and every time there’s a power-cut!

9. As it stands, the Raspberry Pi is booting from microSD, but then mounting its root filesystem from the USB SSD. The final stage is to clean this up by letting the Raspberry Pi boot straight from USB. So we need the /boot/firmware partition to be on the USB SSD, as /dev/sda1. Because we arranged that /dev/sda1 and /dev/mmcblk0p1 are exactly the same size, you can just use dd to copy the filesystem across:

   sudo dd if=/dev/mmcblk0p1 of=/dev/sda1
   sudo umount /boot/firmware
   sudo mount /dev/sda1 /boot/firmware
   ls /boot/firmware

   If your list of files includes lots of “dtb” files and so on, everything went to plan. So edit /etc/fstab one last time, and change the /boot/firmware line to say:

   /dev/sda1 /boot/firmware vfat defaults 0 1

   Ubuntu 20.04 (only) adds a further wrinkle here that is not needed in Raspbian or Ubuntu 22.04 builds. So far we’ve made sure that the Raspberry Pi onboard firmware is happy to boot from USB, and we’ve made sure the kernel is happy that its root is on USB. But on a default Ubuntu 20.04 installation, U-Boot will still torpedo us. Unlike on Raspbian, the kernel= lines in /boot/firmware/usercfg.txt do not name the actual kernel, they name the U-Boot secondary bootloader. And U-Boot, at least in the Ubuntu 20.04 Raspberry Pi images, only knows how to boot from microSD. But fortunately we can bypass U-Boot altogether (it’s not clear what benefit it’s giving us here). This is done by editing /boot/firmware/usercfg.txt and

   • every time there’s a kernel= line, changing it to:

     kernel=vmlinuz
     initramfs initrd.img followkernel

   • commenting-out the device_tree_address line:

     # device_tree_address=0x03000000

   The top of the resulting file should look like this:

   [pi4]
   kernel=vmlinuz
   initramfs initrd.img followkernel
   # kernel=uboot_rpi_4.bin
   
   [pi2]
   kernel=vmlinuz
   initramfs initrd.img followkernel
   # kernel=uboot_rpi_2.bin
   
   [pi3]
   kernel=vmlinuz
   initramfs initrd.img followkernel
   # kernel=uboot_rpi_3.bin
   
   [pi0]
   kernel=vmlinuz
   initramfs initrd.img followkernel
   
   [all]
   # device_tree_address=0x03000000

   This change has already been made in the Ubuntu 22.04 images, so those edits are not needed.

10. Finally, it is the moment of, uh, further truth. Shut down the Pi:

    sudo shutdown -h now

    And, once it’s off (the green LED goes off and stays off), unplug or switch off power and remove the microSD card. Now power it back on again. Once the Ethernet LEDs come on, you’ll need to repeat the login-and-unlock process. But now you should have a fully armed and operational, full-disk-encrypted, empty Raspberry Pi server at your command!

    If at this stage nothing happens – the Ethernet LEDs do not come on, bearing in mind it can take nearly a minute – (and if you have a Raspberry Pi 3), it may be that your USB SSD is not compatible with the onboard bootloader. In that case you can either keep the microSD card inserted indefinitely (and revert the /etc/fstab change in Step 9), or look into the Raspberry Pi’s “Special bootcode.bin-only boot mode”.

    The latter is preferable, as it means the microSD card is never written to, which hopefully means it never risks corruption. You can either use the same microSD card, or a smaller one – though the smallest microSD cards readily available are 2Gbyte, of which bootcode.bin will take up just 52Kbytes, or 0.003%...

11. And there’s one final caveat: I did this and it works. But it seems that the Samsung USB SSD uses a lot of power – so much so, that even with the branded power supply, running it makes USB+5V droop so far that no other USB peripheral will work at the same time including the branded Raspberry Pi keyboard. For true headless use this is of course fine, but it does limit any additional uses of the same Raspberry Pi. It’s likely that using a powered USB hub would fix this issue.