vmd(8) + autoinstall(8) + dhcpd(8) = <3

,

TL;DR: As part of redoing my network at home and for hosting stuff I migrated almost everything to OpenBSD, and as it turns out, vmd(8) is a potent hypervisor. I built my setup so I can automatically deploy new virtual machines.

During my recent experiments with OpenBSD I learned about how far the hypervisor created by the OpenBSD project, vmd, has come since I first read about it in 2018. I was intrigued, because I wanted to separate some stuff I had running at home, but I really did not feel like going down the QEMU-route, and I personally don’t like to use Docker.

What eventually draw me in fully was a talk given by Mischa Peters, the guy who runs openbsd.amsterdam, about how he utilizes different components available in the base system of OpenBSD to simplify creating virtual machines for customers. One comment stuck with me - I can’t remember the exact wording, but it was along the lines of “It’s nearly automated, I have to press ‘A’, but I am fine with that.”

Because my requirements were simpler than his I was sure that I would be able to fully automate the process, allowing me to create a new virtual machine by simply passing a couple of variables to an Ansible-playbook. So I started reading blog posts and manpages.

In my head, I came up with the following plan:

  • Create a bridge-interface on the host machine
  • Define a new virtual machine in /etc/vm.conf to use said interface, with a statically configured MAC-address
  • Use dhcpd(8) to assign a static IP-address to the configured MAC-address
  • Use httpd(8) to serve a configuration file for autoinstall(8) to the virtual machine
  • Wait a minute, then happily log into the new machine

Coming from the Linux-world, having built something similar for other systems of mine, I was hesitant. This seemed to easy and simple. But as it turns out, this setup really does work - well, with a small twist, but I’ll get to that at the end of this post, in which I will roughly describe how everything is configured to achieve what I want.

If you want to follow along, there are a couple of assumptions I’m making:

  • You run a recent version of OpenBSD; I tested this with OpenBSD 7.1-RELEASE
  • You use a bridge-interface and control the DHCP-daemon for your local network
    • I am relatively sure that this can be implemented with other network configurations and NAT-wizardry, but I haven’t tried that out

A last word of warning: As cool as I find this setup, there are a few limitations. Consider the following before deciding on trying to build a similar setup to the one I describe:

  • This only works for OpenBSD. You absolutely can run Linux with vmd(8), but this requires manual installation.
  • vmd(8) currently only allows for a single vCPU per virtual machine; as far as I understand it this limitation should eventually be lifted, and there might already be patches for it, but my current knowledge of OpenBSD is too limited to try that out.

As I mentioned, this configuration needs a bridge-interface. So I added the following to /etc/hostname.bridge0:

add em0
up

And brought it up by running sh /etc/netstart bridge0. Then I went ahead and defined the new virtual machine in /etc/vm.conf. As with a lot of tools written by the OpenBSD project, the syntax is simple, human-readable and well-explained by the manpage. I am typing these words on the machine that looks like this in the configuration:

switch "uplink" {
        interface bridge0
}

vm "shell.int.gmmi.me" {
        owner gmmi
        boot "/bsd.rd"
        memory 2G
        disk "/home/gmmi/vm/shell.int.gmmi.me.img"

        interface {
                up
                switch "uplink"
                lladdr "02:00:00:29:AE:4F"
        }
}

The disk-file can be created with vmctl create -s 20G /path/to/file.img. There is support for .qcow2, however I could not get it to run in an automated fashion due to the kernel throwing a panic about the file system being empty.

After a doas rcctl reload vmd this machine will be available and when started ..

  • .. use the recovery-kernel to boot into the installer
  • .. use the configured disk
  • .. have network access through the bridge

But before I could start it, I configured dhcpd(8) to hand out a static IP-address, adding the following configuration block to my already existing subnet declaration in /etc/dhcpd.conf (and reloaded the daemon):

host shell.int.gmmi.me {
        hardware ethernet 02:00:00:29:AE:4F;
        fixed-address 192.168.1.34;
        option host-name "shell.int.gmmi.me";
	filename "auto_install";
}

Technically this step is not needed. In contrary to PXE-boot, there are no special options necessary. As long as the machine is able to get a valid IP-address that can reach the default gateway - which is where autoinstall(8) seems to look for its configuration file - you are fine. But I like predictability, which also makes configuring A-records easier

Next I wrote a configuration file for autoinstall(8). If you have never heard of that software before, the manpage explains it quite well:

autoinstall allows unattended installation or upgrade of OpenBSD by automatically responding to installer questions with answers from a response file. autoinstall uses DHCP to discover the location of the response file and HTTP to fetch the file. If that fails, the installer asks for the location which can either be a URL or a local path.

The manpage (obviously) contains a further bit of relevant information:

If filename is auto_install, then the URLs tried are, in order:

  • http://server/MAC_address-install.conf
  • http://server/hostname-install.conf
  • http://server/install.conf

So I created the following file, named 02:00:00:29:AE:4F-install.conf in the document root of httpd(8):

System hostname = shell.int.gmmi.me
Password for root = $mytotallyrealpasswordhashprovidedbyencrypt(6)
Change the default console to com0 = yes
Which speed should com0 use = 115200
Setup a user = gmmi
Password for user = $mytotallyrealpasswordhashprovidedbyencrypt(6)
Public ssh key for user = ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0AvQ66C663MV5+ZEoSRbQH0+wD/r2qr2n7uQ1V2GsaURQbgfyGMkykOJAES9BKwiC71WF7fgkxR9yyQryOCd7SgvLNxEF8hoC3I0M3XwxmIti0eoMIGkoIhseSqyzi/F5LINObqBeymJQGQ0ApfT2khFpfckRbZIPDiM+fBtZj/JMgjqRcC7lH6lci27Q0IYNdOUDBbHKQ7cBJbgglsSAfKd+B3Ex4O0PJb9MDlPZIxy3h4wf3BGvUVFnNTqsKK5ElxOU9bl+CBskZXlN4RGSK5uvzwxJUp2VDtdU5772fImto/ihKNpizDjiTnw3/gJjXYNPuzm12poRBcIxI28KkFJMgX+bXZf2QWELBihT1wHfr7kJsFK/5E1VcaXIQXhtzjC+AHLIvHfZpM0C2tGYpglPRY+wlaDPjJqkm8hczgQwgynJ80L4uvs0kOGQS96LVLkeBH0vupVi3mUb/t8ALPVH0ZO8BOxF+xxxxhn5Hu5XsOT6BaXRsnsG9MEufOCZYBZuw+bNGy3ujZTUDH0XcZ9iQgumVUyMgBje6XfO9QjeYlbxx0/w1uYMNeyasETqaqEIe5IOEVUlh5nQnTkqQP58FMddwBcj8e5eeE4B/uMUeVZ0/11uLTBPU0iyvDVGNapAkxDCvkm6Hg7C0RB89Ybo6JcAtQadvarAxcP6F7Q==
What timezone are you in = Europe/Berlin
Location of sets = http
HTTP Server = cdn.openbsd.org

Basically this is nothing more than human-readable anwers to the questions usually asked by the OpenBSD-installer. And in order to serve this file I added the following lines to /etc/httpd.conf, before reloading the daemon:

server "default" {
    listen on * port 80
    root "/htdocs/"
}

Now I can start the virtual machine via vmctl start -c 2, and instruct the installer to run an automated install by pressing ‘A’. After that is done I replace the kernel for the virtual machine in /etc/vm.conf with /bsd.sp, because otherwise every boot would end up in recovery mode, rendering the virtual machine relatively useless.

I already hear you question how that is fully automated - and yes, you are right, this is very much not it. There’s pressing keys and reconfiguring daemons involved.

The above state was relatively easy to achieve, but removing the last manual steps from the process cost me a couple of days and a lot of thinking, tinkering and desperation.

In the end I was on the right path, trying to utilize tmux(1) to programmatically send keystrokes to a terminal, but I was only able to fully solve the problem when I stumbled upon a blog post which not only contained the correct syntax I was to use for tmux(1), but also solved my problem of having to manually re-configure the kernel after the automated installation was done.

I wasn’t entirely truthful when showing the contents of /etc/vm.conf at the beginning. The relevant snippet of my actual file looks like this:

vm "shell.int.gmmi.me" {
        disable
        owner gmmi
        **boot "/bsd.shell.int.gmmi.me"**
        memory 2G
        disk "/home/gmmi/vm/shell.int.gmmi.me.img"

        interface {
                up
                switch "uplink"
                lladdr "02:00:00:29:AE:4F"
        }
}

Note the name of the kernel. That path is in actuality a symlink that is created when the script doing the installation of the virtual machine is run. The script, which I run through Ansible, looks like this:

doas ln -f /bsd.rd /bsd.{{ item.shortname }}

tmux new -s autoinstall_{{ item.shortname }} -d
tmux send-keys -t autoinstall_{{ item.shortname }}:0 "vmctl start {{ item.shortname }} -c" C-m
sleep 10
tmux send-keys -t autoinstall_{{ item.shortname }}:0 "A" C-m
sleep 5
tmux kill-session -t autoinstall_{{ item.shortname }}
doas ln -f /bsd.sp /bsd.{{ item.shortname }}

What this does is create a symlink to the recovery-kernel for the virtual machine to boot from, then start the installation in an instance of tmux(1), and after that is done it re-creates the symlink, this time pointing to the ‘regular’ kernel. And with this final step the installation is now truly hands-off. \o/

This setup is obviously far from perfect. Improvements that I can think of from the top of my head could be:

  • There is no access-control to the configuration files for autoconf(8), because I consider my local network to be trusted, to a reasonable extent
  • Don’t use an official mirror, use your own mirror of the sets
  • Allow for custom partitioning schemes using disklabel(8)

I hope this post is helpful for some people. I learned a lot trying to build this, and it felt good to accomplish this. If there are any questions, recommendations or even corrections - as I mentioned earlier, I am not that knowledgeable about OpenBSD yet, so I might have gotten things wrong - please feel free to reach out to me through the channels described here.

As per usual, credit where it’s due. Besides the excellent manpages provided by the OpenBSD project, a lot of ideas and configuration snippets have been gathered from the following sources: