Running Virtual Machines on Linux

Virtual machines are useful for running other operating systems within your computer and for testing and sandboxing system-level software. This post is about running VMs locally on Linux: how to get a usable disk image and how to connect to it over SSH. It's not as trivial as it sounds.

The VM ecosystem has evolved over the last decades. QEMU/KVM is still the easiest way to get started on Linux, but many of the ecosystem's projects are designed for running cloud services rather than for desktop or casual server use. This post aims to provide simple instructions for running VMs without installing large software stacks. It covers running VMs with and without cloud-init and libvirt.

This post deals with a few challenges:

These instructions are Linux, Debian, and amd64-centric because that's what I'm most familiar with. There are many alternatives and options at every level of the stack. This post aims to provide a reasonable starting point, not the most optimal configuration.

nocloud images

This section describes how to start a basic interactive VM. To achieve this, we'll use QEMU/KVM directly and use Debian's nocloud image. This image allows root to log in with no password and does not have cloud-init installed. (The next section deals with cloud-init images.)

First, ensure the host CPU has virtualization extensions enabled. Almost all modern amd64/x86-64 CPUs support virtualization, but some systems have this disabled in the UEFI settings. For AMD, the extensions are called SVM or AMD-V, and should cause an svm flag to show up in /proc/cpuinfo when enabled. For Intel, the extensions are called VT-x and should cause a vmx flag to show up in /proc/cpuinfo when enabled.

grep -m 1 -P '^flags\b.*\b(svm|vmx)\b' /proc/cpuinfo

If you get no output, reboot into your UEFI settings, enable the setting, and check again.

Then, install the packages on the host for KVM/QEMU:

sudo apt install ovmf qemu-system-gui qemu-system-x86 qemu-utils

Download the Debian 12 nocloud VM image, resize the virtual disk, and run the VM:

curl -LO https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-nocloud-amd64.qcow2
cp -i debian-12-nocloud-amd64.qcow2 test.qcow2
qemu-img resize test.qcow2 32G
kvm -m 4G -nic user,hostfwd=tcp::2200-:22 test.qcow2

The options for running KVM/QEMU are vast. Refer to the man page and docs as needed.

In the graphical window that pops up, log into the VM as root with no password.

Note: If the VM grabs (steals) your mouse and keyboard, there's a magical key combination to escape, which is Ctrl-Alt-G for me. Check the window titlebar for a hint if that doesn't work.

Install an SSH server in the VM:

apt update
apt instal --yes openssh-server

Now, you have some authentication options:

  • Ideally, authorize an SSH key, for example by downloading it from GitHub:

    mkdir -m 700 -p .ssh
curl https://github.com/⟪USER⟫.keys | tee -a .ssh/authorized_keys

  • Or if that's inconvenient, set a root password:

    passwd
echo 'PermitRootLogin yes' > /etc/ssh/sshd_config.d/10rootpassword.conf
systemctl restart ssh

  • Or if you don't care for this VM at all and are confident in the security of your host's network, allow SSH as root with no password:

    cat > /etc/ssh/sshd_config.d/10insecure.conf <<END
PermitRootLogin yes
PermitEmptyPasswords yes
END
systemctl restart ssh

Now, you can SSH from the host:

ssh root@localhost -p 2200

From inside the VM, resize the root partition and filesystem:

apt install --yes cloud-guest-utils
growpart /dev/sda 1
resize2fs /dev/sda1

And upgrade stale packages:

apt upgrade --yes

Now your VM is ready to use.

You can shut it down gracefully or kill the kvm process to stop the VM. Then you can remove its disk image.

cloud-init images

This section describes how to run cloud-init images, which are provided by many distributions and operating systems. cloud-init is software inside the VM that runs at boot to discover configuration provided to it from the outside world. This requires a little more setup before starting the VM, but then the VM will be better configured on startup.

There are various ways to inject the cloud-init configuration. This post uses the simplest NoCloud data source with an extra attached drive. The extra drive (like a virtual CD-ROM or thumb drive) is given a volume label of CIDATA so cloud-init inside the VM can discover it during boot and look for configuration files inside.

One added benefit of using cloud-init is that these images typically have cloud-initramfs-growroot, which means they will automatically grow their root partition and filesystem if the virtual disk has extra space (at least on the first boot). This saves us some steps.

In this section, we'll use Debian's generic image, which contains cloud-init. There is no way to log into this image without using cloud-init.

The nocloud terminology is confusing: Debian nocloud images do not contain cloud-init. Debian generic images contain cloud-init and can use its NoCloud data source. No clouds were harmed in the writing of this blog post.

Debian also offers a genericcloud image, but I don't recommend it. It's smaller than the generic image by omitting a bunch of drivers (about 330 MB vs 420 MB). However, these drivers can be useful, even with KVM/QEMU:

  • QEMU emulates an e1000 driver for the NIC by default, and the genericcloud image doesn't have that driver. (You can use a virtio NIC to work around this by passing model=virtio-net-pci as an option to -nic.)
  • The way that cloud-install presents its cloud-init configuration drive (as used in the next section) also requires drivers, so the genericcloud images will fail to find the drive.
  • I don't know what other QEMU devices might have missing drivers and cause headaches in the future.

Download the Debian 12 generic VM image and resize the virtual disk:

curl -LO https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2
cp -i debian-12-generic-amd64.qcow2 test.qcow2
qemu-img resize test.qcow2 32G

At this point, if you run kvm like before, you'll observe that you can't log in. This can be frustrating if cloud-init somehow isn't working.

Next, set up a cloud-init configuration to set a password and authorize an SSH key. This happens in a file called user-data. The user-data file format is YAML with an extra #cloud-config header. This post uses a more JSON-like format to prevent whitespace errors; as YAML is a superset of JSON, this is allowed. These options apply to the default user, which varies per distro. The default username is debian for Debian images.

mkdir -p cidata
tee cidata/user-data <<END
#cloud-config
{
    "password": "p4ssw0rd",
    "chpasswd": {
        "expire": false,
    },
    "ssh_authorized_keys": [
        "$(cat ~/.ssh/id_ed25519.pub)",
    ],
}
END

Also, create a blank meta-data file, since that's required:

echo > cidata/meta-data

The cloud-init documentation describes the possible fields.

To get these files to the VM, they'll need to be packaged into an ISO or VFAT filesystem with the volume label of CIDATA. Rather than do that manually, QEMU can create a VFAT drive from a directory. The QEMU invocation is a mouthful, but it's still fairly convenient.

kvm -m 4G \
    -nic user,hostfwd=tcp::2200-:22 \
    test.qcow2 \
    -drive file=fat:./cidata,format=vvfat,if=virtio,label=CIDATA

If all goes well, you can log in interactively as user debian with password p4ssw0rd (as set in user-data), and that user can sudo without a password. You can also use SSH with your SSH key:

ssh debian@localhost -p 2200

Thanks to cloud-init, the root partition and filesystem have already been expanded to the available disk image size.

Upgrade stale packages:

sudo apt update
sudo apt upgrade --yes

Now your VM is ready to use.

Next time you run the VM, you don't need to provide the CIDATA image, since its job is done.

libvirt

This section runs VMs using libvirt, which is a way of managing VMs without lengthy KVM invocations. libvirt isn't necessary, but it's probably a better approach if you want to manage long-lived VMs with a variety of operating systems. It also configures KVM with more modern defaults.

Libvirt is a collection of software, named after the underlying library:

  • virsh is its main command-line interface.
  • virt-install is a command-line tool to create the VMs (as this is difficult to do with virsh directly).
  • virt-viewer provides a virtual keyboard, video, and mouse.
  • virt-manager is a GUI for managing VMs. While you can use the GUI to do most of this, this post focuses on the command-line tools.

Libvirt can be used in a system-wide mode (qemu:///system) or for your local user (qemu:///session). This post uses the user mode. (The system mode may be useful to bridge your VMs to the host's network. To use the system mode, you'd need to install libvirt-daemon-system and add your user to the libvirt group.)

Unfortunately, different libvirt tools have different default modes:

  • virsh defaults to qemu:///session.
  • virt-install defaults to qemu:///system.
  • virt-manager defaults to showing both and connecting to neither.
  • virt-viewer defaults to qemu:///session.

You can pass --connect qemu:///session to any of these tools. You may want to set up shell aliases for convenience.

Install the host packages:

sudo apt install \
    gir1.2-spiceclientgtk-3.0 \
    libvirt-clients \
    libvirt-daemon \
    virtinst \
    virt-manager \
    virt-viewer

Create a VM using virt-install based on Debian's generic image. This uses the image downloaded and the cloud-init configuration files created in the previous section.

cp -i debian-12-generic-amd64.qcow2 test.qcow2
qemu-img resize test.qcow2 32G
virt-install --connect qemu:///session \
    --cloud-init 'disable=on,meta-data=./cidata/meta-data,user-data=./cidata/user-data' \
    --disk test.qcow2 \
    --import \
    --memory 4096 \
    --name test \
    --os-variant debian11

This uses the debian11 OS variant since the osinfo-db package in Debian 12 does not currently know about Debian 12. The OS variant doesn't appear to matter much when importing a disk image.

Use the key combination Ctrl-] to exit the virsh console. You can pass --autoconsole none to virt-install if you don't want to be dropped into the console.

Since virt-install supports cloud-init, we didn't need to have QEMU present a CIDATA drive. Actually, we don't even need the YAML files for simple settings. The following is usually sufficient:

--cloud-init "disable=on,clouduser-ssh-key=$HOME/.ssh/id_ed25519.pub"

Forward a host port to SSH to the VM:

virsh qemu-monitor-command --hmp test 'hostfwd_add tcp::2200-:22'

Since libvirt doesn't currently support forwarding host ports, you'll need to run that hostfwd_add command every time you run the VM. Note that test in the command is the name of the VM. This workaround is thanks to Adam Spiers' blog post from 2012.

Then run:

ssh debian@localhost -p 2200

Upgrade stale packages:

sudo apt update
sudo apt upgrade --yes

Now your VM is ready to use.

These are some useful libvirt commands:

  • List VM statuses:

    virsh list --all

  • Boot an existing VM:

    virsh start ⟪NAME⟫

  • View a VM's text console:

    virsh console ⟪NAME⟫

  • View a VM's graphical console:

    virt-viewer ⟪NAME⟫

  • Shut down a VM gracefully:

    virsh shutdown ⟪NAME⟫

  • Shut down a VM forcefully:

    virsh destroy ⟪NAME⟫

  • Delete a VM and its disks:

    virsh undefine ⟪NAME⟫ --remove-all-storage

And recall that virt-manager is a useful GUI that also provides all this virsh and virt-viewer functionality and more.

Someday/maybe

  • virt-manager issue #143 would allow using cloud-init when creating new VMs from the GUI.
  • libvirt issue #285 would support forwarding host ports.
  • virt-customize appears to directly modify VM disk image files and may be an alternative to cloud-init. I haven't tried it.
  • The libvirt SSH proxy should allow SSH to VMs over VSOCK instead of TCP. This would remove the need for port forwarding for SSH. However, it's not packaged for Debian or widely supported in VM images yet.

Other distributions and operating systems

Many operating systems offer cloud images and cloud-init support. This section documents a few that I've tried.

Debian

Debian images for Debian 12 (Bookworm) were covered above. For other releases, see https://cloud.debian.org/images/cloud/.

The default user is debian.

Use --os-variant debian11 as the closest version known to Debian 12.

Fedora

These images work with and require cloud-init. For other releases, see https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/. For more info, see https://fedoraproject.org/cloud/download.

Fedora 41:

curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/41/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-41-1.4.x86_64.qcow2

The default user is fedora.

Use --os-variant fedora37 as the closest version known to Debian 12.

FreeBSD

These images allow logging in interactively as root with no password, and they grow their root filesystem automatically. Despite its name, even the BASIC-CLOUDINIT images do not appear to run cloud-init yet. For other releases, see https://download.freebsd.org/releases/VM-IMAGES/.

FreeBSD 14.1 BASIC-CLOUDINIT with ZFS variant:

curl -L https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-BASIC-CLOUDINIT-zfs.qcow2.xz | \
    xz -d > FreeBSD-14.1-RELEASE-amd64-BASIC-CLOUDINIT-zfs.qcow2

FreeBSD 14.1 standard with ZFS variant:

curl -L https://download.freebsd.org/releases/VM-IMAGES/14.1-RELEASE/amd64/Latest/FreeBSD-14.1-RELEASE-amd64-zfs.qcow2.xz | \
    xz -d > FreeBSD-14.1-RELEASE-amd64-zfs.qcow2

Use --os-variant freebsd13.1 as the closest version known to Debian 12.

Ubuntu

These images work with and require cloud-init (which is a Canonical project). For other releases, see https://cloud-images.ubuntu.com/.

Ubuntu 24.04 LTS (Noble):

curl -L -o noble-server-cloudimg-amd64.qcow2 \
    https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

Ubuntu 24.10 (Oracular):

curl -L -o oracular-server-cloudimg-amd64.qcow2 \
    https://cloud-images.ubuntu.com/oracular/current/oracular-server-cloudimg-amd64.img

The default user is ubuntu.

Use --os-variant ubuntu22.10 as the closest version known to Debian 12.