This week I was working on a QEMU-based integration test suite and needed to customize the base QCoW2 images shipped by the Ubuntu project for use in my test harness.

Previously I would have treated this in a similar manner to EC2 and AMIs: launching a VM, SSH’ing into it and applying tweaks, and cloning it to create our desired VM image.

Unsurprisingly there is a better way in the form of virt-customize which can modify disk images by installing packages, editing files, and other settings in place.

Here’s an example that enables SSH, configures the public key for the root user, and applies a static IP network configuration.

virt-customize -a noble-server-cloudimg-amd64.img \
  --run-command "ssh-keygen -A" \
  --run-command 'systemctl enable ssh' \
  --ssh-inject 'root:file:ci.pub' \
  --run-command 'systemctl enable systemd-networkd' \
  --run-command 'mkdir -p /etc/systemd/network' \
  --run-command 'cat > /etc/systemd/network/10-enp0s2.network << "EOF"
[Match]
Name=enp0s2

[Network]
Address=192.0.2.3/28
Gateway=192.0.2.1
DNS=8.8.8.8
EOF'

How does it work?

I was curious to understand how all this works under the hood and decided to read the manpages for virt-customize and its libguestfs and supermin dependencies. The guestfs internals manpage was particularly helpful in understanding how all the parts fit together.

Here’s what I learned: libguestfs is a library that provides a way of reading and modifying virtual machine images of different formats. It does this by launching an “appliance” (a type of small virtual machine) with QEMU, mounting the disk within that appliance, and making its contents available to the main process via an RPC API. The use of QEMU to run the appliances means that libguestfs can read and write any format that QEMU is compatible with including QCoW2, VirtualBox, VMWare, and others.

Here’s an ASCII art diagram from the documentation

 ┌───────────────────┐
 │ main program      │
 │                   │
 │                   │           child process / appliance
 │                   │          ┌──────────────────────────┐
 │                   │          │ qemu                     │
 ├───────────────────┤   RPC    │      ┌─────────────────┐ │
 │ libguestfs  ◀╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍╍▶ guestfsd        │ │
 │                   │          │      ├─────────────────┤ │
 └───────────────────┘          │      │ Linux kernel    │ │
                                │      └────────┬────────┘ │
                                └───────────────│──────────┘
                                                │
                                                │ virtio-scsi
                                         ┌──────┴──────┐
                                         │  Device or  │
                                         │  disk image │
                                         └─────────────┘

An example program using libguestfs to create an /etc/flag file in a given disk image.

#include <stdio.h>
#include <stdlib.h>
#include <guestfs.h>

int main(int argc, char *argv[])
{
    guestfs_h *g;
    if (argc != 2)
    {
        fprintf(stderr, "usage: %s <disk-image>\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    const char *disk = argv[1];

    /* create the guestfs handle */
    g = guestfs_create();
    if (g == NULL)
    {
        fprintf(stderr, "failed to create guestfs handle\n");
        exit(EXIT_FAILURE);
    }

    /* add the disk image */
    if (guestfs_add_drive(g, disk) == -1)
    {
        fprintf(stderr, "failed to add drive\n");
        exit(EXIT_FAILURE);
    }

    /* launch the appliance */
    if (guestfs_launch(g) == -1)
    {
        fprintf(stderr, "failed to launch\n");
        exit(EXIT_FAILURE);
    }

    /* mount the root filesystem */
    if (guestfs_mount(g, "/dev/sda1", "/") == -1)
    {
        fprintf(stderr, "failed to mount\n");
        exit(EXIT_FAILURE);
    }

    /* write our flag */
    if (guestfs_write(g, "/etc/flag", "hello world", 11) == -1)
    {
        fprintf(stderr, "failed to write file\n");
        exit(EXIT_FAILURE);
    }

    /* teardown */
    guestfs_umount(g, "/");
    guestfs_shutdown(g);
    guestfs_close(g);

    printf("created /etc/flag in %s\n", disk);
    return 0;
}

$ gcc -o create_flag create_flag.c $(pkg-config --cflags --libs libguestfs)
$ time ./create_flag disk.img
created /etc/flag in disk.img
./create_flag disk.img  0.04s user 0.03s system 4% cpu 1.741 total

The appliances libguestfs creates are separate to your virtual machines. libguestfs launches a new ephemeral QEMU child process for each main program caller and tears it down when finished. The documentation warns that you must halt any VM guests that use the disk before you run virt-customize as otherwise the simultaneous writes can lead to disk corruption.

supermin

libguestfs uses supermin to create its child appliances. supermin works by storing a minimal manifest of required packages and other dependencies and then assembling these into a bootable filesystem at runtime by extracting only the components required from the host system’s package cache and using the host system’s kernel.

In practice this means you can distribute appliance templates that weigh under 200KB and that boot in a second or two.

$ supermin --prepare bash coreutils -o supermin.test.d
$ du -hs supermin.test.d
168K    supermin.test.d
$ supermin --build supermin.test.d -f ext2 -o supermin.test.d
$ du -hs supermin.test.d/root
749M    supermin.test.d/root
$ time guestfish -a supermin.test.d/root run
guestfish -a supermin.test.d/root run  0.02s user 0.02s system 2% cpu 1.660 total