At the end of Dockerfiles - redux, I said I'd technically made myself a very small Linux distribution, just one with "no installer, no kernel" and "no intention whatsoever of becoming a Linux distribution". I also said that was probably how these things start. Well, I now have a kernel and an installer, so I think I have to stop pretending, although I still think it's more of a remix.

What, why?

I've said before that my servers are cattle, not pets. That's mostly true of what runs on them: everything is a container, the containers are built from my own packages, the compose files describe how they fit together, and Adze redeploys them when a new image lands. The bit that isn't true is the host itself. Each one was installed and updated by hand, and none was quite the same as any of the others. They don't even run the same distribution: some run Arch, some run Alpine. To me, this still felt more like a small herd of pets that I didn't love enough to give names.

Having just rebuilt everything around "software is a package, an image is a list of packages", the hosts were suddenly the odd ones out. The host OS was the only thing I ran that I couldn't describe in a single file and rebuild from scratch. I had been watching bootc for a while. The idea is that an operating system is just a container image. You build it the same way you build any other image, push it to a registry, and the server pulls it, stages it, and boots into it next time. If the new one is broken, you boot the old one. There's no package manager running on the host and nothing gets changed in place; the whole OS is replaced as a unit. bootc is almost entirely a Fedora and CentOS project.

After my Dockerfiles redux, everything I was running server-side was Alpine-based. Every container uses the same Alpine base, the same conventions, and the same package repository, which is why I'd started deploying my newer servers with Alpine. It did work surprisingly well, but OpenRC always caught me off guard and there were still weird musl or BusyBox bugs to look out for. The other week, however, I noticed that systemd had started work on musl support, and Alpine had started some work on systemd packages.

I already had a registry, an Alpine repository, and a tool that builds images from a list of packages. It suddenly felt like taking that next step wasn't going to be that much work. So, in typical Greboid style, why pick the easy road? Why not Alpine?!

How

I knew it would be a pile of yaks, so I started with the first yak and let it show me where I needed to go.

First up, systemd. I started with the upstream packages and a lot of coaxing in the form of -D flags pointing systemd at where Alpine actually keeps things. It failed because it wanted D-Bus, and Alpine's D-Bus package uses eudev rather than udev, so I stripped D-Bus support completely.

Next, I needed an initramfs that understood composefs so I could use bootc. Rather than trying to figure out how to port a whole lot of code, I thought I'd just install Dracut. Alpine even had a package for this. Amazing! Except installing Dracut dragged in eudev rather than udev, which wasn't going to work with systemd. On to the next yak. Dracut: a couple of options, build against musl, nice and simple. Then bootc: no Alpine build and no upstream support for musl. I disabled SELinux, which I hate (and Alpine doesn't support), got some helpful hints from an LLM, and had a patched version up and running in no time. It takes ages to build, but otherwise it was very straightforward. I booted an Alpine live CD in a VM, added my Forgejo package repository, installed Dracut, built an initramfs, and bodged it and a kernel into a disk image. I wasn't aiming at a bootc install yet, just proof that systemd and the kernel worked. Boot: nothing. Change some Dracut options, reboot: nothing.

At this point, I decided I was going to be cobbling a lot of disk images together, so I scripted the process of building them and booting the VM.

Many reboots later, I finally worked out how to get it booting, but the Alpine kernel package puts the kernel in the wrong place for bootc to use. So, a kernel package. I was loath to build a kernel; that seemed like a waste. Instead, a repackage: osimage-kernel takes Alpine's linux-stable package and repacks it into the layout bootc expects: the kernel in /usr/lib/modules/<version>/vmlinuz rather than /boot, the modules next to it, and a symlink so Alpine's kmod can still find them. The same melange build produces the initramfs as a subpackage, so the kernel and the initramfs that boots it always come out of one build at one version.

It took more reboots to figure out what you actually need to boot a system. First, it complained that UTF-8 was enabled on the ESP partition, and then that it couldn't activate things via D-Bus. Sigh. I knew I'd regret disabling it. Back to systemd to re-enable the D-Bus support I'd removed earlier. Reboot... Alpine's D-Bus package doesn't support systemd, so it can't be socket-activated. dbus: simple file, point at Alpine locations, add systemd support. The UTF-8 fix ended up being a bootc patch. I could have changed the kernel, but that would mean rebuilding it rather than repackaging it, and I'm building bootc regardless.

Not a Distribution

In the last post, I wrote "I am not attempting to rebuild Alpine. This is mostly a message to myself." I'd like to point out that I still haven't packaged BusyBox, and I still don't compile a kernel!

The image itself is an apko config, the same as every container. It lists the repositories, the packages and the accounts, and that's it:

packages:
  - systemd=262.1-r6
  - bootc=1.17.3-r2
  - osimage-kernel=7.1.5-r11
  - osimage-kernel-initramfs=7.1.5-r11
  - osimage-config=1.0.0-r21
  - tailscale=1.104.5-r1
  - docker=29.9.3-r0
  ...

The rule from the containers applies here too. If you want something in the image, it needs to be a package. That includes configuration. All the systemd drop-ins, kernel arguments, network config and lists of enabled services live in osimage-config, which is a package like anything else.

It effectively defines the defaults for how a machine should behave. It tells systemd which services to start and how they should run, sets up networking, and supplies the kernel arguments needed to boot the image. It also means that enabling a service or changing a systemd setting is a package change, not something I SSH into each machine to do. Rebuild the package, rebuild the image, and every machine gets the same configuration on its next upgrade.

The important distinction is that this is configuration for the OS, not for an individual machine. Hostnames and machine IDs are handled separately, but everything that's supposed to be identical lives here. Nothing gets compiled, copied or scripted at image build time; apko installs packages into a filesystem and the filesystem is the OS.

This is the same thing I liked about the containers. The image definition is boring. All the interesting parts are in the packages, where they get built once, versioned, and reused by anything that needs them.

There are now two images. minimal is the OS that gets installed, and installer is the live environment that installs it, built from mostly the same packages. Everything they share lives in a common repository, so the image repositories themselves are just an apko.yaml, a workflow, and a README.

Immutability

The root filesystem is composefs. /usr is read-only, which I expected. /etc wasn't, to start with. The first time I booted it, I realised I could still change things in there by mistake, so I made /etc read-only too. The consequences were the bit I hadn't thought through: long-term gain, short-term pain.

Every machine gets exactly the same /etc, and upgrades replace it wholesale. Nothing can be tweaked by hand on one machine and then forgotten about, so config drift has nowhere to happen. This is lovely right up until you want a machine to have its own hostname, or its own machine ID, or a root password.

BusyBox's passwd doesn't work because it writes a temporary file next to /etc/shadow. hostnamectl doesn't work. echo name > /etc/hostname doesn't work. Even systemd can't create its own service users at boot, so they're baked into the image definition instead.

The fix is that the two files that actually need to differ between machines—the hostname and the machine ID—live in /var/lib/osimage/etc/, and the initramfs bind-mounts them read-only over their /etc counterparts before systemd starts. To change the hostname, you edit the file in /var and reboot.

Passwords started out needing more than that. My first attempt was osimage-passwd, which rewrote a copy of shadow in /var in place, took the same options as BusyBox's passwd, and installed itself as /usr/bin/passwd in place of the BusyBox applet. It worked, but it was a lot of effort to keep something I didn't really want: a password on a server I only ever reach over SSH.

So I've since thrown it away. doas is now built against PAM (doas-pam), and its PAM config uses pam_rssh, which authenticates against the SSH agent I've forwarded in my session. The keys it accepts live in the read-only /etc, so nobody can add their own, and the initramfs locks every password in shadow at boot. There's no password to set, change or forget; if I can SSH in with my key, I can doas.

How to get it on the server

You can't install a bootable OS image without something to boot first, so there's an installer ISO. It boots a live copy of the installer image into RAM and runs osimage-install, which started as a shell script and is now a Go program, because of course it is.

It asks which disk to use, a hostname, and whether to bring Tailscale along. There used to be a root password prompt too, but with doas-pam and every password locked at boot, it was asking for something that would never be used, so it's gone. The Tailscale option is my favourite bit. The installer logs itself into my tailnet (via a QR code on the console if needed), so I can do the rest of the install over Tailscale SSH. When it's finished, it copies its own Tailscale state onto the new disk. The installer is running from RAM and is gone the moment the machine reboots, so the installed system boots up as the same node, already on the tailnet, and I never have to type anything on the server's console.

Building the ISO was by far the most annoying part. A hybrid ISO that boots as both a CD and a USB stick on UEFI systems sounds like a solved problem, and it is, as long as you do exactly what everyone else does. Local QEMU booted every variation I tried. A virtual server on a remote host booted none of them, for a series of reasons that I now know far more about than I ever wanted to, involving El Torito boot catalogue layouts and a 16-bit sector count field. I tried lots and lots of options, uploaded them to the provider, mounted the ISO, rebooted the server... nothing. So, so many times.

Then I discovered the remote host was caching the attached ISO by filename. Several of the builds that didn't work probably worked absolutely fine; it was just booting the old ISO every time. Every ISO now gets a fresh dated filename, which has nothing to do with good versioning practice.

Automated

CI builds the image and pushes it to Forgejo's container registry with a dated tag and latest. Once a day, every machine runs osimage-upgrade --apply, which pulls the new image, stages it, and reboots into it. Nobody asks me first. Each machine ends up running whatever I pushed last.

Letting every server reboot itself into an image I built that afternoon sounds like a terrible idea. What makes it "safe" is what happens when it goes wrong. New boot entries are given two attempts. If the kernel panics, it reboots after 10 seconds; if it hangs, the hardware watchdog resets it. Two failed boots and the bootloader falls back to the previous version on its own. If it boots but a service fails, a health check notices, rolls back, and reboots.

There are tests for all of this that boot VMs, install the image, push a deliberately broken version, and check that the machine ends up back where it started. Those run locally rather than in CI for now, mostly because I haven't wanted to work out nested virtualisation in the runners.

osimage-upgrade itself is only a small wrapper. After bootc upgrade stages the new image, it has to be finalised with bootc composefs-finalize-staged before a reboot will actually boot into it. That's a separate command, and I kept forgetting to run it, so now a script remembers for me. It also only reboots if there's actually something new to boot into; bootc upgrade --apply reboots regardless, which isn't what you want from something that runs every day.

So, the list from the last post. Almost no packages: no longer true. No installer: there's an installer and an ISO. No kernel: there's a kernel, repackaged, with its own initramfs. A slightly patched toolchain: now it's also a slightly patched init system and a slightly patched bootloader stack. No intention whatsoever of becoming a Linux distribution: I've got an organisation in Forgejo called osimage and a base image with a version number.

But a server is now a disk I can wipe and reinstall from an ISO in a few minutes. It comes up on the tailnet with the same name, and then the compose files get deployed onto it and started. Upgrading the OS works the same way as upgrading a container: push an image, and by the next day the machine is running it. If it breaks, it goes back.

Every machine is identical, and none of them is special. Other approaches might be considerably more sensible, but this one is much more me.

What next?

I think the next step is to automate getting the compose files onto the right servers. Then I'm not sure I'll need to SSH into a server for management at all.

I haven't tackled Secure Boot yet. I'll probably look at it, though; something about having a certified boot path alongside a reproducible image feels nice.

It also needs a real name at some point, rather than osimage, but I'm bad at naming things, so that may take longer than either.