 |
I thought I’d take the opportunity to improve the security story while I was at it, including adding Full Disk Encryption (FDE) using LUKS. But how is that to work on a headless (no screen or keyboard attached) server? How is the decryption key or passphrase to be entered?
Perhaps I could SSH into the box and type in the passphrase. But if it needs the passphrase to decrypt the root filesystem, how is the SSH server running? The answer is that it’s possible to set up Dropbear, a lightweight SSH server, so that it runs in the kernel’s initrd before the root filesystem is mounted. Ubuntu provides handy packages for dropbear-initramfs and cryptsetup-initramfs to make this easier.
The usual (and easiest) way to set up FDE on an x86 or x86_64 server is by using the normal Ubuntu installer. But installing Ubuntu on a Raspberry Pi involves its own custom installer, and there isn’t an easily-ticked option that just does it for you. So you’ll need to do it manually, as outlined in the following description. (Which might also be of use to anyone upgrading any other Ubuntu installation to use FDE – though be warned that it’s not possible to upgrade in-place, and we manage it on the Raspberry Pi because we’re also moving to a new root filesystem.)
To follow these steps you’ll need:
- A Raspberry Pi 3B. (A Raspberry Pi 4, with its USB3 controller, would have higher performance – but they’re hard to get hold of at the moment.)
- A proper power supply for the Raspberry Pi. Raspberry Pis are notoriously picky about power supplies, especially when driving USB bus-powered devices, so I used the branded Raspberry Pi one. Note that the Raspberry Pi 4 would need a different power supply.
- An existing PC able to write images to a microSD card; I used Ubuntu but other Linuxes, Windows and Macintosh can also be made to work. (Many laptops, and some monitors, have SD slots, but you might still need a microSD-to-SD or microSD-to-USB adaptor.)
- A microSD card – I used an 8GB one. It’s probably only needed during setup, so you could borrow one from a phone or MP3 player – but it will be erased during the process, so don’t use a vital one. (The “probably” is because, if your USB SSD is not quite Raspberry Pi compatible, you might need to keep it as a first-stage bootloader.)
- A USB SSD, to be the main storage for your new server – I used a Samsung T7. (For some reason I assumed it would be 2.5in-sized, but of course it’s not, it’s smaller – about the size of the Raspberry Pi’s own PCB – which will make the overall thing much neater and tidier.)
Out of the box, the Raspberry Pi will only boot from a microSD card, so the first course of action is to enable booting from USB devices. This might not appear essential – microSD cards are very cheap, and it could easily be left in even with most of the data on the USB – but whether it’s the Raspberry Pi’s fault or that of the microSD card, running a Raspberry Pi from microSD has a bad reputation for long-term reliability. Similarly, using a device marketed as a USB SSD is likely to mean that it’s intended for heavier use than a USB thumb drive, as well as potentially being higher-performance.
Installing Ubuntu on a microSD card
Ubuntu have a good tutorial which you should just follow. The steps are basically these:
- Put the microSD card into the adaptor and insert into your PC.
- Install rpi-imager (on Ubuntu, sudo snap install rpi-imager) and run it.
- Click “Choose OS”, then “Other general purpose OS”, then “Ubuntu”, then pick Ubuntu Server 20.04 or 22.04 64-bit from the list. (If you want 22.04 and it’s not on the list yet, download ubuntu-22.04-preinstalled-server-arm64+raspi.img.xz from the Ubuntu release site and choose “Use custom” from the menu; you don’t need to un-XZ the img.xz file as rpi-imager will use the compressed version just fine.)
- Click “Choose Storage” then choose your microSD card from the list (carefully! important other disks that you don’t want wiped will also be listed).
- Write the image to the microSD card, then insert it back into the Pi and power-on the Pi to boot it.
The only wrinkle I found, was discovering the Raspberry Pi’s IP address after it came up on the network for the first time. The suggested
arp -na | grep -i "b8:27:eb"wasn’t finding it, even following
ping -b [my local broadcast address]
and the only answer seemed to be attempting SSH connections to addresses in my DHCP range in turn until one responded.
The next thing is to enable USB booting. There
is a
tutorial at thepi.io, but it is Raspbian-centric; for Ubuntu
use, firstly note that on Ubuntu 20.04 only the config file to
modify is not /boot/config.txt
but /boot/firmware/usercfg.txt. Also, the mentioned
vcgencmd command is not installed by default in Ubuntu,
but it’s available via sudo apt install
libraspberrypi-bin.
Settings in usercfg.txt are only applied to the actual non-volatile configuration at boot time, so the reboot in the middle of that tutorial isn’t optional. In summary, the steps are, for 20.04:
sudo apt updateor for 22.04:
sudo apt upgrade
echo program_usb_boot_mode=1 | sudo tee -a /boot/firmware/usercfg.txt
sudo reboot
... log in again and ...
sudo apt install libraspberrypi-bin
vcgencmd otp_dump | grep 17
sudo apt updateEither way the final output should be the value 3020000a.
sudo apt upgrade
echo program_usb_boot_mode=1 | sudo tee -a /boot/firmware/config.txt
sudo reboot
... log in again and ...
sudo apt install libraspberrypi-bin
vcgencmd otp_dump | grep 17
If you have several Raspberry Pi devices to deal with, boot this image on all of them at this stage to update the non-volatile configuration of each of them; that means you can just swap USB devices later.
You might also want to change the hostname and default username at this point from the current ubuntu@ubuntu, here’s an example for bob@porpentine:
sudo hostnamectl set-hostname porpentineDon’t delete the ubuntu user until you’re sure that you can SSH in as bob and use sudo!
sudo adduser bob
sudo adduser bob sudo
sudo adduser bob admin
Moving the Ubuntu installation to an encrypted USB SSD
Much of this is based on Hamy’s (non-Raspberry-Pi-based) headless FDE tutorial, but with adaptations for the Raspberry Pi scenario.
- Plug the USB SSD into the Pi and check (using cat /proc/partitions) that it appears as /dev/sda. (If your SSD was supplied pre-formatted, you might have /dev/sda1 too; that’s fine as we’re going to delete it.)
- Now install the necessary initramfs packages:
sudo apt install dropbear-initramfs cryptsetup-initramfs busybox-initramfs
You’ll need to do a little setup for dropbear: first, find the SSH authorized_keys file corresponding to the keys/users that you’d like to be able to unlock the server, and copy it into the Dropbox initramfs directory: on 20.04 this is /etc/dropbear-initramfs:
sudo cp .ssh/authorized_keys /etc/dropbear-initramfs/
although on 22.04 it’s /etc/dropbear/initramfs:sudo cp .ssh/authorized_keys /etc/dropbear/initramfs/
(Watch out, as the default Ubuntu 20.04 installation puts an empty file in ~/.ssh/authorized_keys – make sure you’re using a real one, probably scp’d on from an existing Linux installation).
Now you’ll need to edit the configuration file – the available editor is nano. On Ubuntu 20.04 it’s:
sudo nano /etc/dropbear-initramfs/config
and on 22.04 it’s:sudo nano /etc/dropbear/initramfs/dropbear.conf
In either case, uncomment the #DROPBEAR_OPTIONS= line and set it to:
DROPBEAR_OPTIONS="-p 999 -s -j -k"
This sets the listening port to 999 (from the default 22); doing this means that your SSH client won’t get confused that it’s connecting to a “different” SSH server on the same IP and port as the normal Ubuntu SSH server. The remaining options increase security by disabling password logins and port forwarding.
- Now you need to partition the USB SSD. On the Pi,
run sudo fdisk /dev/sda. Using fdisk is beyond the scope
of this note, but you want to delete any existing partitions and
add two more: /dev/sda1 the same size as /dev/mmcblk0p1 (which
will be small, 256Mbytes), FAT formatted, for the bootloader and
ramdisk, plus /dev/sda2, as yet unformatted, for the rest of the
disk. The resulting partition table (“p” command in fdisk) should
look a bit like this:
Command (m for help): p Disk /dev/sda: 931.53 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: PSSD T7 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x404755d2 Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 526335 524288 256M c W95 FAT32 (LBA) /dev/sda2 526336 1953525167 1952998832 931.3G 83 Linux Filesystem/RAID signature on partition 1 will be wiped.
This should look broadly similar to the partition table of the existing microSD card installation (fdisk -l /dev/mmcblk0):
Disk /dev/mmcblk0: 7.41 GiB, 7948206080 bytes, 15523840 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0xf66f0719 Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 * 2048 526335 524288 256M c W95 FAT32 (LBA) /dev/mmcblk0p2 526336 15523806 14997471 7.2G 83 Linux
- Now let’s actually make the encrypted block device that will
contain the root filesystem. (This section is adapted
from an
Alpine Linux tutorial.) Lay out the device using that
tutorial’s “optimised for security” settings:
sudo cryptsetup -v -c aes-xts-plain64 -s 512 --hash sha512 --pbkdf pbkdf2 --iter-time 5000 --use-random luksFormat /dev/sda2
You will need to enter (for the first time!) the decryption passphrase, and repeat it to guard against mistypings.
If we just wanted one encrypted partition, we might be nearly done here. But to lay out multiple partitions inside the encrypted block device, we need to also use LVM (Logical Volume Manager). We create a “PV” (Physical Volume) inside the encrypted block device, then a “VG” (Volume Group) inside the PV, then two “LVs” (Logical Volumes) inside the VG. LVM can do much more sophisticated things than we’re using it for here!
First, open the LUKS device:
sudo cryptsetup luksOpen /dev/sda2 lvmcrypt
– the passphrase is of course required. Now the encrypted version of the block device is available as /dev/mapper/lvmcrypt.
Then, create the LVM PV and the VG inside it:
sudo pvcreate /dev/mapper/lvmcrypt
sudo vgcreate vg0 /dev/mapper/lvmcryptNow we can create the partitions we actually want, as LVs inside that VG. We’ll create two: a swap partition and a root partition:
sudo lvcreate -L 16G vg0 -n swap
sudo lvcreate -l 100%FREE vg0 -n rootWe size the swap partition at 16 Gbytes, and the root partition as the whole rest of the device – note the tricksy difference between “-l” and “-L”. Our two new partitions appear as /dev/vg0/swap and /dev/vg0/root, as can be verified by sudo lvscan. (They also appear as /dev/mapper/vg0-swap and /dev/mapper/vg0-root.)
Now we can actually do something with those partitions:
sudo mkfs.ext4 /dev/vg0/root
sudo mkswap /dev/vg0/swapTo use the new root filesystem, we need to manually add entries to /etc/fstab and /etc/crypttab. So in /etc/fstab change the “/” entry to read:
/dev/mapper/vg0-root / ext4 defaults 0 1
and in /etc/crypttab add:
lvmcrypt /dev/sda2 none luks,discard,initramfs
The “initramfs” is important – the initramfs setup scripts realise that it’s important to unlock the root filesystem at startup, but /dev/sda2 isn’t (at the time you’re executing these commands) the current root filesystem – so if you don’t go out of your way to tell it that unlocking /dev/sda2 is important, the initramfs setup scripts will omit the necessary components to unlock it!
We need to copy the root filesystem into the new root filesystem; there are various ways to do this, but I just did:
sudo mount /dev/vg0/root /mnt
sudo rsync -xrpvltoD / /mnt/You’ll also need to edit the kernel command line to tell it where the root is: edit /boot/firmware/cmdline.txt to add a “root=” parameter – and, just to be sure, an “ip=dhcp” parameter:
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/vg0-root rootfstype=ext4 elevator=deadline rootwait fixrtc ip=dhcp
During experimentation, it might be convenient to setup two alternative cmdline.txt files – cmdssd.txt and cmdmmc.txt, say – so that switching back and forth can be done using a simple “cp” command such as can be found in an initramfs rescue shell!
- Remake the initramfs incorporating these changes (on a
Raspberry Pi, running update-initramfs also copies the image to
the firmware partition):
sudo update-initramfs -u
- Check that your initramfs actually contains a crypttab – this
caused me a lot of frustration on the way to figuring this out:
unmkinitramfs /boot/firmware/initrd.img tmp-init/
cat tmp-init/cryptroot/crypttabThe output should be one line much like /etc/crypttab (for me, it had changed the device name to a
UUID , but the effect is the same; use the blkid command to check whether it picked the rightUUID ). - Now it’s the moment of truth – time to reboot the Pi and
see whether it comes up with Dropbear waiting for your
passphrase.
sudo shutdown -r now
Your SSH session will of course be disconnected. So, once the Raspberry Pi has rebooted (ethernet LEDs go off, then on again), connect to the initramfs Dropbear server we installed:
ssh -p 999 root@192.168.168.103
If all is well, you will see this, or something very like it:
BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.4) built-in shell (ash) Enter 'help' for a list of built-in commands. #
(If all is not well, you might reluctantly need to attach a monitor and keyboard to the Raspberry Pi to see what went wrong.)
But in the happy case, the Raspberry Pi has loaded its initramfs, but not mounted root yet. So you have very limited tools to play with, but the one thing you should have is the cryptroot-unlock command. This will ask you for your passphrase, open the root filesystem, and allow booting to proceed (your SSH connection will be closed). You’ll need to repeat this login and unlock process every time the Raspberry Pi reboots – and every time there’s a power-cut!
- As it stands, the Raspberry Pi is booting from microSD, but then
mounting its root filesystem from the USB SSD. The final stage
is to clean this up by letting the Raspberry Pi boot straight
from USB. So we need the /boot/firmware partition to be on the
USB SSD, as /dev/sda1. Because we arranged that
/dev/sda1 and /dev/mmcblk0p1 are exactly the same size, you can
just use dd to copy the filesystem across:
sudo dd if=/dev/mmcblk0p1 of=/dev/sda1
sudo umount /boot/firmware
sudo mount /dev/sda1 /boot/firmware
ls /boot/firmwareIf your list of files includes lots of “dtb” files and so on, everything went to plan. So edit /etc/fstab one last time, and change the /boot/firmware line to say:
/dev/sda1 /boot/firmware vfat defaults 0 1
Ubuntu 20.04 (only) adds a further wrinkle here that is not needed in Raspbian or Ubuntu 22.04 builds. So far we’ve made sure that the Raspberry Pi onboard firmware is happy to boot from USB, and we’ve made sure the kernel is happy that its root is on USB. But on a default Ubuntu 20.04 installation, U-Boot will still torpedo us. Unlike on Raspbian, the kernel= lines in /boot/firmware/usercfg.txt do not name the actual kernel, they name the U-Boot secondary bootloader. And U-Boot, at least in the Ubuntu 20.04 Raspberry Pi images, only knows how to boot from microSD. But fortunately we can bypass U-Boot altogether (it’s not clear what benefit it’s giving us here). This is done by editing /boot/firmware/usercfg.txt and
- every time there’s a kernel= line, changing it to:
kernel=vmlinuz
initramfs initrd.img followkernel - commenting-out the device_tree_address line:
# device_tree_address=0x03000000
The top of the resulting file should look like this:
[pi4] kernel=vmlinuz initramfs initrd.img followkernel # kernel=uboot_rpi_4.bin [pi2] kernel=vmlinuz initramfs initrd.img followkernel # kernel=uboot_rpi_2.bin [pi3] kernel=vmlinuz initramfs initrd.img followkernel # kernel=uboot_rpi_3.bin [pi0] kernel=vmlinuz initramfs initrd.img followkernel [all] # device_tree_address=0x03000000
This change has already been made in the Ubuntu 22.04 images, so those edits are not needed.
- every time there’s a kernel= line, changing it to:
- Finally, it is the moment of, uh, further truth. Shut
down the Pi:
sudo shutdown -h now
And, once it’s off (the green LED goes off and stays off), unplug or switch off power and remove the microSD card. Now power it back on again. Once the Ethernet LEDs come on, you’ll need to repeat the login-and-unlock process. But now you should have a fully armed and operational, full-disk-encrypted, empty Raspberry Pi server at your command!
If at this stage nothing happens – the Ethernet LEDs do not come on, bearing in mind it can take nearly a minute – (and if you have a Raspberry Pi 3), it may be that your USB SSD is not compatible with the onboard bootloader. In that case you can either keep the microSD card inserted indefinitely (and revert the /etc/fstab change in Step 9), or look into the Raspberry Pi’s “Special bootcode.bin-only boot mode”.
The latter is preferable, as it means the microSD card is never written to, which hopefully means it never risks corruption. You can either use the same microSD card, or a smaller one – though the smallest microSD cards readily available are 2Gbyte, of which bootcode.bin will take up just 52Kbytes, or 0.003%...
- And there’s one final caveat: I did this and it works. But it seems that the Samsung USB SSD uses a lot of power – so much so, that even with the branded power supply, running it makes USB+5V droop so far that no other USB peripheral will work at the same time including the branded Raspberry Pi keyboard. For true headless use this is of course fine, but it does limit any additional uses of the same Raspberry Pi. It’s likely that using a powered USB hub would fix this issue.
