Moving my package tests from Docker to systemd-nspawn

Hello internet! I recently moved my pkgbuilds testing environment from using Docker to plain systemd-nspawn containers. To me, this switch has definitely been an upgrade. I now have a simple, concrete tool to start local containers whose file systems live in the pkgbuilds directory. My decision to ditch Docker is a very personal one and I understand if you believe that what follows is a lot of work for very little gain.

During this transition, however, I found one difficult obstacle to overcome: a lack of concrete user documentation for creating and spinning up containers. The systemd-nspawn (nspawn for short) manual is wonderfully detailed, but I missed being able to read articles focusing on setting your system up with ease.

Today I'm trying to share this knowledge with you while the process is still fresh in my brain. Keep in mind that the examples are going to be based on Arch Linux, so a few details may be different in your particular system. Let's begin!

Bur first, a note...

Docker is good enough

I really mean this. My previous solution was based on a simple Docker environment and it worked like a charm.

The workflow consisted on updating paur (my Python AUR build script) in the container and calling docker run passing the package I wanted to test as an argument. In hindsight, I can see that I could have skipped building new images for every paur update by binding the script as a volume. I think this is a good time to say that I'm not an expert on this topic, just a volunteer maintainer!

What made me consider ditching Docker was this ugly table:

❯ df -h / /home
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2   47G   29G   16G  65% /
/dev/nvme0n1p3  413G  371G   22G  95% /home

I don't have much disk space in my productive laptop by design, as I work better with light set-ups (mostly because I don't have disk space for games, oops). As Docker stores its data en /var, my system disk was getting low on space. I'm sure I could have changed the data path, but I was already interested in nspawn.

The fact that the systemd-nspawn already comes with the systemd package is a huge plus too. Having a built-in solution I can run on virtually any Linux machine is a big advantage to consider. With these things in mind, I took on the task to move my environment to one based on nspawn.

The journey of moving to nspawn

If you wish to build a package from scratch...

...you must first create the container. This is well documented in the Arch Wiki for both Arch Linux and other distros. The idea is that we set up a directory tree that resembles a Linux file system so we can later tell the kernel to use it as a container. For this, we use pacstrap and tell it to install the minimum set of packages required by our environment:

sudo mkdir "Container" # Required as pacstrap will refuse to create it!
sudo pacstrap -K -c "Container" base base-devel git pacman-contrib python sudo

You will immediately notice one advantage over Docker: the initial package installation makes use of your system's package cache. This is much faster and considerate on the Arch servers that performing a full download every time you need to create a new container!

When the file system is ready, the Container/ directory will contain a file system populated with the packages you have installed. If you want to play with your new system, you can spin the container up right away:

sudo systemd-nspawn -D "Container"

You will be logged in as root in /root and will be able to use it as if it was a new Arch Linux system. If you ever wish to remove the container, you can just remove its root directory and create a new one. Alas, however far you can get by manually configuring the system, it won't be reproducible, so we need a way to automate the system's set-up. Let's look at a way of achieving this.

Installing and running a set-up script

My set-up script for the container looks like this:

#!/usr/bin/env sh

sed -i \
    's/#\?ParallelDownloads.*/ParallelDownloads=50\nILoveCandy/' \
    /etc/pacman.conf

pacman -Rcnsu --noconfirm $(pacman -Qtdq) || true
pacman -Sc --noconfirm

useradd -m "mrpaur" -u 1000
echo "mrpaur ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

First, it enables parallel downloads and cute progress bars for pacman, just like in my host machine. Then, I clean all leftover packages from the installation and the pacman cache to save some space. The last two lines create the mrpaur user and give it access to sudo. Simple enough, right?

Note, tough, that useradd will most likely assign uid 1000 to the user it creates if no -u option is passed. Since you may need to share file permissions with the users in the container, you can edit the uid the user will have to match yours. You can programmatically find your uid with id -u.

In order to run this script, we must first copy it into the container. As I manage the creation of the container with a script, what I actually do is redirect a function's output to the file:

# init_script is a function that echoes the script text shown above
init_script | sudo tee "Container/root/init.sh"
sudo chmod +x "Container/root/init.sh"

Of course, you can have actual script files in a directory ready to be copied. Bonus points if they're prefixed by a number to be run in order! Copying all scripts from a directory would look like this:

# Execute permissions are assumed, no chmod required
sudo cp -r "init-scripts" "Container/root/init-scripts"

Now we need a way to run the script file. Previously, we ran nspawn without arguments, which let us enter a login shell with root. This time, we're going to pass nspawn the command we wish to run inside the container.

However, we actually want to run more than one command! In order to keep the container clean we are going to run the init script and then remove it. To do so, we can use the sh command, which runs a command string if -c is passed:

sudo systemd-nspawn -D "Container" sh -c "/root/init.sh; rm /root/init.sh"

Using absolute paths will get burdensome really soon, so we can use the --chdir option to specify the working directory we wish to start the container in:

sudo systemd-nspawn -D "Container" --chdir="/root" sh -c " \
    ./init.sh; \
    rm init.sh"

Or, if you copied multiple files:

sudo systemd-nspawn -D "Container" --chdir="/root" sh -c " \
    init-scripts/*; \
    rm -rf init-scripts"

You could also bind the init-scripts/ directory and avoid having to remove it after execution. We will look at a way of doing this later, but I wanted to show this use case too as it's also a valid way of achieving the same result. Now, let's take a look at how we can do actual work with the container.

Running tests inside the container

We now have a container we can use to run programs. The next step is to call the paur testing script and let it modify our host file system.

To achieve this, we want to use the --bind option, which uses the same syntax as Docker volumes: <host-dir>:<targe-dir>, where host-dir is the directory in our file system we want to bind to the container and target-dir is the directory in the container it will be bound to. For example, here's how we can use --bind to add paur to the container's $PATH:

sudo systemd-nspawn --bind="paur:/usr/bin/paur" paur

Binding the script file instead of installing it in a previous step has the advantage of letting us modify it without needing to reinstall it to see the changes in action.

For our use case, we also want to bind the packages/ directory that paur uses to test the installers. We can do this by adding --bind="packages:/home/mrpaur/packages, as we know that packages exists in the host's current working directory. We also add the option -x, a.k.a. --ephemeral, to make sure that the changes in the container will be discarded once we finish the task at hand. This is necessary to keep a clean environment between package tests.

The last touch we need is to run the tests as the mrpaur user, as makepkg will refuse to run them as root. We do this with the option -u mrpaur. This is when the uid we set before comes in handy: as long as our host user has permission to read an write the files, the container's will have them as well. The command that runs paur is the most complex, but after reading this article you should understand it well:

sudo systemd-nspawn -xD "Container" \
    --bind="paur:/usr/bin/paur" \
    --bind="packages:/home/mrpaur/packages" \
    -u "mrpaur" \
    --chdir="/home/mrpaur/packages" \
    paur manim # To test manim, for example

A last note about colours

It's possible that the container's output is formatted with a particular blue background. If this bothers you visually or makes your saved logs harder to parse, you can set the environment variable SYSTEMD_COLORS=0. Remember to use sudo -E to replicate your user's environment in the one used by sudo:

SYSTEMD_COLORS=0 sudo -E systemd-nspawn your-args-here

Closing remarks

As you may have noticed, this article explores a very simple set-up to get started with nspawn. The requirements I have for my package testing environment are minimal due to the very nature of the task, so we haven't discussed more complex configuration needs like networking or graphical environments. What I wanted to make clear instead is that nspawn is an excellent out-of-the-box solution for simple use cases.

I haven't had the need to test the system's extensibility and scalability. In fact, writing this article has made me simplify the container script! If you want to a layered, distributable container environment, Docker may be a better tool for your use case. On the other hand, if you want to spin up small containers for simple tasks, nspawn is very easy to set up and run without additional dependencies.

There's one thing that is an advantage for me but can be a show stopper for many users: nspawn's set-up is 100% artisanal-shell-based. On the one hand, you don't need to learn an abstraction language to define your containers. On the other, container definitions are undoubtedly less skimmable and lack Docker's layering system. With more verbosity comes more flexibility, but also more responsibility to get the set-up right.

I will surely keep using nspawn to test my packages in the future, as it doesn't take space in my system disk, which is the biggest advantage for me personally. The fact that it can be configured through shell scripts also makes it easier to understand to me, as I'm already fluent with the shell and understand how all the pieces work together. In conclusion, I find nspawn a fine container tool and I think that you can make good use of it too if you ever need a quick container and can't or don't want to use Docker.

Thank you for reading and have a great day!