Cloud on Bastien Traverse

Optimized cloud-init templates on Proxmox

Fri, 04 Oct 2024 00:00:00 +0200

There are already quite a few resources out there demonstrating how to create a cloud-init enabled VM template in Proxmox. Here are the ones I mainly used to discover the topic, and which I suggest you go through because what follows depends on them:

Proxmox FAQ, wiki and mostly identical official documentation on Cloud-Init support
Perfect Proxmox Template with Cloud Image and Cloud Init (YouTube, Techno Tim 2022-03)

What those and many similar resources give are step-by-step instructions divided in as many commands to facilitate understanding. What I haven’t seen so far though, is an all-in-one, optimized command to do the same thing, so here’s my contribution to the field:

qm create 1000 \
    --name debian12-cloud \
    --description "Debian 12 cloud-init template" \
    --template 1 \
    --ostype l26 \
    --machine q35 \
    --cpu host \
    --cores 2 \
    --memory 4096 \
    --balloon 512 \
    --scsihw virtio-scsi-single \
    --scsi0 local-lvm:0,import-from=/path/to/debian-12-generic-amd64.qcow2,discard=on,iothread=1,ssd=1 \
    --net0 virtio,bridge=vmbr0 \
    --tablet 0 \
    --rng0 source=/dev/urandom \
    --boot order=scsi0 \
    --vga serial0 --serial0 socket \
    --ide2 local-lvm:cloudinit \
    --ciuser myuser \
    --cipassword changeme \
    --sshkey /path/to/your-public.key \
    --ciupgrade 0 \
    --ipconfig0 ip=dhcp

The same thing as a one-liner for the latest Ubuntu:

qm create 2000 --name ubuntu-server-2404-cloud --description "Ubuntu Server 24.04 cloud-init template"  --template 1 --ostype l26 --machine q35 --cpu host --cores 2 --memory 4096 --balloon 512 --scsihw virtio-scsi-single --scsi0 local-lvm:0,import-from=/path/to/ubuntu-24.04-server-cloudimg-amd64.img,discard=on,iothread=1,ssd=1 --net0 virtio,bridge=vmbr0 --rng0 source=/dev/urandom --tablet 0 --boot order=scsi0 --vga serial0 --serial0 socket --ide2 local-lvm:cloudinit --ciuser myuser --cipassword changeme --sshkey /path/to/your-public.key --ciupgrade 0 --ipconfig0 ip=dhcp

Note that you cannot copy-paste those blindly, you have to adjust a few parameters to your local environment (especially the VMID, disk image and SSH key paths).

Follows a description of relevant options at the exclusion of self-evident ones (name, description, cores, memory…), as well as some possible variations you might want.

Generic options

qm create 1000: the Proxmox CLI command to create a VM. Replace 1000 by the VMID of your choice (must be ≥ 100)
--template 1: directly convert the created VM to a template

--ostype l26: hint to optimize for a Linux 2.x-6.x-based system
--machine q35: use a modern machine type
--cpu host: pass-through the host CPU type to make all its features available in the VM
--balloon 512: when set to a lower value than memory, enables dynamic memory allocation
--scsihw virtio-scsi-single: the most performant SCSI controller, especially when combined with iothread=1 (see next point)
--scsi0 local-lvm:0,import-from=/path/to/debian-12-generic-amd64.qcow2,iothread=1,discard=on,ssd=1:
- import (i.e. copy) the referenced cloud image as the VM disk
  - replace /path/to/ with the full path to where you downloaded the cloud image (which you should have already done by now if you have followed the resources linked above 😉)
- configure it with performance (IO Thread), thin-provisioning and SSD-optimized settings
  - remove discard=on and/or ssd=1 if not applicable to your storage
--tablet 0: one of the lesser-known performance tips but one of the most important! Disables the USB tablet device only needed when connecting via the integrated console to guests with a GUI (e.g. Ubuntu Desktop). Reported to have a big performance impact.
--rng0 source=/dev/urandom (optional): provides a virtual hardware random number generator to get entropy from the host system (can speed things up during the first boot)

Up to here were performance-related options applicable to all VM templates, not only cloud-init ones. Here comes the cloud-init specific bits:

--boot order=scsi0: apparently speeds up booting for cloud-init enabled images
--vga serial0 --serial0 socket: creates the serial connection expected by most cloud images in their “native” cloud environments; also useful to monitor and troubleshoot the boot process via the Proxmox console
- verified to work with Debian 12 and Ubuntu 24.04 server cloud images; remove if causing issues with the image you’re using
--ide2 local-lvm:cloudinit: creates the required cloud-init “CD-ROM” drive
--ciuser myuser (optional): provides a custom username for the user account provisioned by cloud-init; without it the account name will depend on the distribution’s default (debian for Debian, ubuntu for Ubuntu… Check your cloud image docs about this)
--cipassword changeme (optional): generally not needed nor recommended, but useful for quickly making sure everything is all right the first few times over; afterwards use a SSH key instead
--sshkey /path/to/your-public.key (required if not setting a password): the authorized SSH public key that will be placed in the user account created by cloud-init
--ciupgrade 0 (optional): disable automatically upgrading packages during first boot; useful to speed things up during testing, afterwards remove it/set it to 1 (the default) if you want “always fresh” clones (which is probably a smart choice)
--ipconfig0 ip=dhcp: cloud-init in Proxmox doesn’t have a network configuration by default, so use this to let DHCP handle it or use something like --ipconfig0 ip=10.0.10.123/24,gw=10.0.10.1 for static config. Can be done later for each VM individually, just don’t leave it empty otherwise they won’t have any network by default.

If you already have custom cloud-init snippets, specify them via --cicustom "user=<volume>,network=<volume>,meta=<volume>", e.g. --cicustom "user=local:snippets/user-config.yaml".

If you do, make sure you have the equivalents of the Proxmox cloud-init options above set in your custom config, because using a custom user snippet overrides the complete user config set in the GUI or config! Yeah I know, it sucks and it’s not documented, boo Proxmox.

Fortunately, as mentioned in the docs the GUI config can be dumped to serve as a base for custom configs:

qm cloudinit dump 1000 user
qm cloudinit dump 1000 network

Note

Unlike Proxmox’ implementation, when using --cicustom and in the absence of network configuration, the image’s cloud-init process will generate a network configuration that will issue a DHCP request on a “first” network interface. So if DHCP is what you want, you don’t have to supply a "network:..." snippet besides the "user:..." one.

Post-creation steps

The only thing that cannot be done in the same step (due to using import-from) is resizing the disk image. I personally prefer doing it on the cloned VMs rather than on the template itself to reduce cloning time and adjust the size depending on the VM’s needs, but there is also a case to be made to do it on the template directly.

So in my case I first clone the template to a new VM:

qm clone 1000 150 --full --name "debian12-cloud"

Note

Besides being generally recommended for VMs you will keep around, it seems we can only use a full clone when using --scsihw virtio-scsi-single as without the --full option I get:

Linked clone feature is not supported for drive 'scsi0'

YMMV.

Then expand its disk size:

qm resize 150 scsi0 15G

And then we’re ready to fire up the VM!

Double-click on the VM name (or on the “Console” button at the top right) instead of going through the integrated VM-specific menu “Console” element, because the pop-up window it opens can be resized fullscreen and allows to scroll back the buffer and read the boot log if necessary.

After checking everything works, you may want to stop and destroy this test VM:

qm stop 150
qm destroy 150 --purge --destroy-unreferenced-disks 1

Now you can do the final adjustments to your template (e.g. remove --cipassword, --ciupgrade 0 etc.) and you are ready to rock the cloud-init lifestyle in Proxmox! ☁️🤘🕺

UEFI variant

Generally I try to use as modern a stack as is reasonable, because software written in the last few years is more likely to be tested with it than a more legacy stack.

But I have realized that UEFI is much less commonplace in virtualized environments than on bare metal, making it less tested and I’d say, slightly less supported overall (case in point: it’s still not the default in QEMU/Proxmox).

However it is easy enough to use it in our templates by adding the following options:

--bios ovmf --efidisk0 local-lvm:0,efitype=4m,size=4M,pre-enrolled-keys=0

The only thing to note is that pre-enrolled-keys=0 disables Secure Boot, which trips up all the distros that don’t want to play the Microsoft game (Arch Linux being a notable one for me). Leave the parameter out or switch its value to 1 for a Secure Boot-enabled template (confirmed working with Ubuntu for example)!

The QEMU Guest Agent conundrum

By default, no cloud images I know of come with qemu-guest-agent preinstalled, but it’s pretty useful on Proxmox.

To install it in your cloud images, you basically have two options:

Install and use libguestfs’ virt-customize to the cloud images themselves, as illustrated in this random blog post I found
Let cloud-init do it during the first boot of each cloned VM using a custom cloud-init snippet: see this SuperUser answer for an example. The required lines to add to your user-config.yaml are:
```
#cloud-config
...
package_update: true
packages:
  - qemu-guest-agent

runcmd:
  - systemctl enable --now qemu-guest-agent
```

Remember that using a custom user snippet overrides the complete user config set in the GUI or config, so those lines must be added to your complete user snippet!

In this case add --cicustom "user=local:snippets/user-config.yaml" and--agent 1,fstrim_cloned_disks=1 when creating the template (see the docs for details).

Tips & tricks

don’t use Debian genericcloud image: its kernel is optimized for Azure & AWS environments and in my tests, didn’t work with Proxmox. I had started with this one (being fooled by the wording on the download page ("genericcloud: Similar to generic. Should run in any virtualised environment. Is smaller than generic by excluding drivers for physical hardware"), spent quite a bit of time troubleshooting the VM booting but cloud-init not kicking in, until I eventually tried the generic image where everything worked perfectly. The Debian wiki actually sets the record straight:

The generic image uses Debian’s standard Linux kernel packages, while the genericcloud image uses the cloud kernel build. The cloud kernel disables a large number of device drivers and primarily targets the Amazon EC2 and Microsoft Azure VM device models. It may be usable in other environments, but for maximum compatibility we recommend using the generic images.

While troubleshooting I’ve seen plenty of other reports of people having issues making the genericcloud image work with Proxmox, while it worked for some others… generic is the reliable, consistant option. ’nough said!
you can get Proxmox to display .qcow2 images alongside regular .iso in its GUI, by simply suffixing/replacing their extension with .img (like Ubuntu does). It’s a regex issue ¯\_(ツ)_/¯

Using a container to sidestep a forgotten password in CasaOS

Sat, 03 Feb 2024 00:00:00 +0100

Problem statement

As part of dabbling with self-hosting again, I installed CasaOS on an Oracle Cloud free Ampere instance to try it out.

After setting it aside for a few weeks, when I logged in via SSH and tried to use sudo I realized I absolutely couldn’t remember my user’s password 😅

The standard operating procedure in this case is to either reboot the machine on a live system and use that to chroot into the local install, or fiddle with GRUB rescue/kernel command line.¹

But since I could still install containers through CasaOS web interface, I thought I’d use this opportunity to explore ways to recover my sudo access/change my password without rebooting (which could be nice in case it is essential to avoid downtime).

Edit: As I was finishing this post, I realized that the third constraint listed below was incorrect, and as a result a less cumbersome resolution was possible. So read on if you are interested in the learning journey, or jump straight to the simplified solution.

The constraints

the CLI/SSH access is “useless”: my user is not setup to interact with the Docker socket without elevated privileges (which is usually a good thing)
besides its App Store content, CasaOS allows to manually install containers using a Compose file or a docker run command (which is then composerized): see official video or screenshots of the process
~~its interface however doesn’t allow to docker exec or interact with containers in any other way; the installed containers must expose a web interface for us to be able to interact with them.~~ As it turns out, CasaOS interface does offer access to containers’ console (equivalent to a docker exec -it <containe> /bin/sh), at least for apps installed from its official Store, via the app Settings > “Terminal and Logs” icon.

Access to a container shell and logs in CasaOS interface

Hacking around

Because I’m pretty new to this, my first idea was to start a container which would give me a root shell with access to the docker socket, and then use that environment to start a second privileged container (Docker-in-Docker style) from which I would mount the host filesystem and use {ch,}passwd or such… Yes, I felt very smart thinking this up 😅

Unaware of my mistaken approach, I looked online for a “web terminal container” and the first result on StartPage was the web terminal GitHub repo, which did what I needed (expose a root shell in the browser) but hadn’t been updated since August 2021 😕

Aside

Docker Hub search is appalling: searching for web terminal gave me a bunch of completely unrelated results. I had to put a dash between the two words to get anywhere, but then the two first results hadn’t been updated in 2 years either…

Searching for web shell and "web shell" didn’t give better results, while web-shell and webshell led to very outdated stuff without any description. I can’t believe there isn’t a reference implementation of this stuff!!

I also looked up ttyd, the software used by web-terminal, and here the first result was fresh from a few days ago; however there was no mention of “How to use this with Docker” , so I wasn’t sure a simple docker run would achieve the desired result… Had I checked its Dockerfile, I would have seen that its ENTRYPOINT/CMD was, in fact, starting up the service 🙃

Lo and behold, I took the “risk” (after checking the image’s Dockerfile) and entered the following command in CasaOS Docker CLI interface:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 7681:7681 raonigabriel/web-terminal:latest

After setting up the access port in CasaOS interface and confirming everything was okay, I clicked “Install” and connected to my server’s port 7681 in a browser… Success, I was in!

For the next stage of my plan I ran the following command inspired by this StackExchange thread:

docker run -ti --privileged --net=host --pid=host --ipc=host --volume /:/host busybox chroot /host
/bin/sh: docker: not found

Crap, I didn’t even check that the container had docker installed 🤦 You can tell I really don’t know what I am doing 😂

Of course when I subsequently tried to install docker…

root@1d0fa4b64c55:/$ apk add docker
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
  docker (missing):
    required by: world[docker]
root@1d0fa4b64c55:/$ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 36 distinct packages available

A quick search seems to indicate that the image is simply too old. Building an updated image was out of the scope I had set for this experiment, so I paused and took time to think.

That’s when it occurred to me that this “nested container” approach was completely useless, and would most likely not work since at that point you have the first container’s virtualized filesystem namespace acting as a buffer between the host and the DinD container… Complete misdirection, backing up!

The solution

After thinking it through some more, I realized one could achieve the desired outcome by simply mounting the host / read-write as an attached volume 😁

docker run -d -v /:/host -p 7681:7681 raonigabriel/web-terminal:latest

Et voilà, I finally had a root shell from which I could chroot into the host and update the user password and/or configure password-less sudo (which amounts to the same). Or really, do (nearly) everything to the host system 😨

Root shell provided by the web-terminal container

Initially I thought it would be necessary to use a privileged container, but trying it showed that wasn’t the case. I guess it’s because I was only editing files/using regular utilites, and not trying to create new devices/nodes etc.

The solution: simplified

This is what happens when you don’t know your tools enough… You miss very obvious pathways that lead to simpler solutions 😁

Indeed, there is no need to install a third-party container. It is enough to add the /:/host volume to an existing app installed from CasaOS official App Store (e.g. NextCloud), and after making sure in its settings it runs with the root UID/GID, we can use the interface to connect the container’s console!

Root shell inside nextcloud container

Trying to refine the solution

Now that I was there, I felt like I should have been able to use a simple busybox image to execute a command non-interactively directly from docker run/Compose file, eliminating the need to find an image that exposes a web service or to connect to the container’s console.

After a bit of fiddling, I came up with the following invocation:

docker run -v /etc/sudoers:/host/etc/sudoers busybox /bin/sh -c echo '%sudo  ALL=(ALL) NOPASSWD: ALL' >> /host/etc/sudoers

But CasaOS interface constantly threw an error. Probably the redirection in the command, but no amount of quoting led me to a successful run.

At that point I threw the towel in: I had recovered access to my sudo access and learnt quite a few things along the way. Time to wrap up.

Takeways

Docker default security story is scary: because the daemon runs as root by default, the mere ability to run containers (even without --privileged) and mount any desired host path as a volume gives root-equivalent status
Due to this, access to CasaOS interface is equivalent to root access on the server

That’s all folks, thanks for reading this account I hope you enjoyed!

See e.g. https://gcore.com/learning/how-to-reset-password-in-linux/ ↩︎