Hub-and-spoke backups using Syncthing

Syncthing is a synchronisation tool which can keep files and directories synchronised across several computers. This post is about how I set it up for secure backups of various Linux boxes, including off-site backup. It is based on Syncthing’s own excellent documentation and mostly documents the particular configuration I use to maintain various security constraints:

No computer holds, at rest, credentials to log into any other

SSH public keys are stored encrypted, with a passphrase required for usage.

No port except (somewhat unavoidably) SSH is exposed to the internet

Syncthing seems well-maintained and safe. But there’s no point offering an attack surface of both Syncthing and SSH, when I can just hide Syncthing inside SSH.

No computer holds, at rest, unencrypted files

The computers involved all either have full-disk encryption or home-directory encryption enabled. Information needing extra security (financial data etc.) is kept in an additional layer of encryption using EncFS and/or encfs-agent, and only opened when required; Syncthing sychronises the encrypted form.

I’ve got Syncthing installed on a total of four computers, in a hub-and-spoke manner:

• A Linux desktop, where I do most of my work;
• A Linux laptop (it’s a Lenovo Thinkpad);
• A Raspberry Pi that’s my home server;
• A second Raspberry Pi at a friend’s house as an off-site backup.

For the purposes of this post, we’ll call these boxes desktop, laptop, central, and offsite respectively. The hub of the hub-and-spoke setup is central; the other three only ever synchronise with central and not with each other. Both laptop and desktop connect inbound to central; central connects outbound to offsite. Each interaction is a full two-way sync, but in my case almost all new data flows from desktop and laptop to central and then offsite; almost no new data flows in the other direction.

Syncthing has its own package repository for Debian/Ubuntu, so installing it (on both clients and servers) involves following the instructions at https://apt.syncthing.net; there are packages for ARM64 Ubuntu (the Raspberry Pi) as well as AMD64 Ubuntu (for desktop and laptop). It typically runs in the background and is operated using a web GUI on port 8384; by default this (quite rightly) listens only for connections from localhost. On central, Syncthing starts as a service at boot time, and on the other boxes it is manually run when needed – this is a backup-on-demand setup, not automated backup.

Syncthing apparently also runs on Windows and Macintosh, but all the boxes I need to use it on run Linux.

Hub setup

The server, central, runs headless. To interact with the Syncthing web GUI from desktop, I need to SSH to central and use port forwarding. A script, on desktop, called ssh-central provides this:

ssh-central (1/2)
#!/bin/bash # 8000: central syncthing UI exec ssh -A -X -t \ -L localhost:8000:localhost:8384 \ peter@central

Alternatively, options could be added in ~/.ssh/config for host central, but doing it this way means that I can still SSH to central normally, with no port forwards, when needed.

Once that script is running and has connected (offering a shell prompt on central in your terminal window), going to https://localhost:8000 in your web browser will bring up Syncthing’s user interface; again consult Syncthing’s own documentation for what you’re looking at there.

By default Syncthing exposes the synchronisation protocol to all comers on a given port; this isn’t horrific as there is mutual authentication and a key exchange required before any actual synchronisation happens – but, on the general principle of minimising attack surface, I set it up to work like the GUI: available only to localhost, and requiring anyone else to set up SSH port forwarding in order to connect. From the Actions menu choose Settings, then Connections, and set the “Sync Protocol Listen Addresses” to tcp4://127.0.0.1:22000 and turn off NAT traversal, local discovery, and global discovery:

Adding “dial-in” spokes (workstations)

Although I’m using Syncthing in, effectively, a client/server mode, the protocol itself treats all connections as symmetrical peers. This means that two ports need to be forwarded over SSH by desktop: one for desktop’s Syncthing to connect to central’s Syncthing, and another in the opposite direction. (Honestly, I don’t see why it can’t do everything it needs over just one bidirectional TCP stream, but all the documentation suggests that it wants two.)

So this means that each workstation needs its own port allocated on the server, to be the listening address for server-to-workstation connections. I keep track of these in a file called lordwarden.txt on central...

lordwarden.txt
sync-port desktop 22001 laptop 22002 offsite 22003

...but that’s because I have an inveterate fondness for appalling puns, and you should pick a more sensible filename.

Once you’ve picked a port, you can write the do-syncthing script for each workstation. Here’s the one on desktop:

do-syncthing (desktop)
#!/bin/bash -x ssh -N \ -L localhost:22001:localhost:22000 \ -R localhost:22001:localhost:22000 \ peter@central & SSHPID=$! syncthing kill $SSHPID echo Done!

This sets up the SSH tunnel and starts the local (to desktop) instance of Syncthing. While this is running in a terminal, going to https://localhost:8384/ in your web browser will show the Syncthing user interface.

Here you need to set up the “Sync Protocol Listen Addresses” just like above, then (from the main screen) choose “Add Remote Device”. You’ll need to enter central’s Syncthing “Device ID”, which you can find in central’s user interface by opening the Actions menu and choosing “Show ID”. (I always just copy and paste the textual form and don’t bother with the QR code.) Then in the “Advanced” tab of the “Add Device” dialog, enter tcp://127.0.0.1:22001 – the port you chose for it in lordwarden.txt – under “Addresses” and then click “Save” to confirm.

It may take Syncthing a little while to successfully connect, but eventually central’s Syncthing will ask you to confirm the new peer’s identity, and the two boxes will be connected.

By default Syncthing just synchronises a default directory under $HOME, but you can add others and choose which peers to share them with. One useful directory to share is $HOME/.local/share/evolution/mail/local, which is the Evolution mail client’s “On This Computer” top-level maildir directory. The maildir format (unlike mbox) is very amenable to being shared in this way without risking corruption – though Evolution doesn’t always notice when the currently-selected folder is changed behind its back, and you have to select a different folder and then click back again.

Once you’ve finished setting-up and synchronising directories, go to the “Actions” menu on desktop and choose “Shutdown”; the do-syncthing shell-script will then exit cleanly.

On subsequent invocations of do-syncthing, there is of course nothing further to configure, and Syncthing will immediately perform the synchronisation, offering progress reports through its user interface.

Adding the “dial-out” spoke (offsite)

This is just a little harder, because you aren’t sitting in front of the box, because it’s off-site.

For the purposes of this post I’m going to gloss over the process of setting up a Raspberry Pi, installing it at a friend’s house, arranging that it has a port tunnelled through the friend’s firewall, and signing up with a free dynamic DNS service such as DuckDNS (no commercial relationship, just a satisfied customer) to give it a resolvable domain-name that will stay updated whenever your friend’s internet provider changes their IP address. So I’ll just assume that you’ve arranged that offsite.duckdns.org resolves to a box running an SSH server accessible on port 20202. Bear in mind that if you’ve set up your off-site Raspberry Pi to need a disk-encryption passphrase on every boot, you’ll need to visit your friend’s house to re-enter the passphrase every time they have a power cut!

(You could, of course, equally well use a hosting service or cloud service for your off-site backup – though, depending on how much you trust your hosting provider, you might want to look into Syncthing’s untrusted devices functionality. At time of writing, that is marked as being for beta/testing only.)

The security constraints at the top of this post, exclude the possibility that central statically has any credentials to log in to offsite. So to synchronise the two, you’ll need to sit at desktop or laptop with ssh-agent running, SSH to central with ssh-agent forwarding (the -A option visible in the ssh-central script), and then SSH again from central to offsite.

The script that connects from central to offsite needs to forward two Syncthing ports (as noted in lordwarden.txt) plus the port for the Syncthing user interface on offsite:

sync-offsite (central)
#!/bin/bash exec ssh -L 127.0.0.1:22003:127.0.0.1:22000 \ -R 127.0.0.1:22003:127.0.0.1:22000 \ -L 127.0.0.1:8040:127.0.0.1:8384 \ -p 20202 peter@offsite.duckdns.org syncthing

But of course that only forwards the user interface as far as central; to get it all the way to desktop where you can actually see it, you need to add another port forward to desktop’s ssh-central script:

ssh-central (2/2)
#!/bin/bash # 8000: central syncthing UI # 8040: offsite syncthing UI exec ssh -A -X -t \ -L localhost:8000:localhost:8384 \ -L localhost:8040:localhost:8040 \ peter@central

Armed with all of this, you can sit at desktop, run ssh-central to log in to central, and once there run sync-offsite to reach offsite. At this point offsite’s Syncthing can be reached in your web browser on desktop as https://localhost:8040, and you can set about configuring it. (But if offsite is a Raspberry Pi 3, it will take a little while to respond to the initial requests.) Set its “Sync Protocol Listen Addresses” to tcp4://127.0.0.1:22000 as above, and then add central as a new device, entering tcp://127.0.0.1:22000 as its “Addresses”. You might also need to tell central that offsite’s address is tcp://127.0.0.1:22003. Again it might take a little while to connect, but once it has you can tell it which folders to share and let the synchronisation proceed to completion. Once everything is synchronised, go to the “Actions” menu on offsite and choose “Shutdown”; the sync-offsite shell-script will then exit cleanly.

Again, any further invocations of ssh-central and sync-offsite will immediately start the synchronisation process without any more configuration needed.