Tech on Bastien Traverse

Optimized cloud-init templates on Proxmox

Fri, 04 Oct 2024 00:00:00 +0200

There are already quite a few resources out there demonstrating how to create a cloud-init enabled VM template in Proxmox. Here are the ones I mainly used to discover the topic, and which I suggest you go through because what follows depends on them:

Proxmox FAQ, wiki and mostly identical official documentation on Cloud-Init support
Perfect Proxmox Template with Cloud Image and Cloud Init (YouTube, Techno Tim 2022-03)

What those and many similar resources give are step-by-step instructions divided in as many commands to facilitate understanding. What I haven’t seen so far though, is an all-in-one, optimized command to do the same thing, so here’s my contribution to the field:

qm create 1000 \
    --name debian12-cloud \
    --description "Debian 12 cloud-init template" \
    --template 1 \
    --ostype l26 \
    --machine q35 \
    --cpu host \
    --cores 2 \
    --memory 4096 \
    --balloon 512 \
    --scsihw virtio-scsi-single \
    --scsi0 local-lvm:0,import-from=/path/to/debian-12-generic-amd64.qcow2,discard=on,iothread=1,ssd=1 \
    --net0 virtio,bridge=vmbr0 \
    --tablet 0 \
    --rng0 source=/dev/urandom \
    --boot order=scsi0 \
    --vga serial0 --serial0 socket \
    --ide2 local-lvm:cloudinit \
    --ciuser myuser \
    --cipassword changeme \
    --sshkey /path/to/your-public.key \
    --ciupgrade 0 \
    --ipconfig0 ip=dhcp

The same thing as a one-liner for the latest Ubuntu:

qm create 2000 --name ubuntu-server-2404-cloud --description "Ubuntu Server 24.04 cloud-init template"  --template 1 --ostype l26 --machine q35 --cpu host --cores 2 --memory 4096 --balloon 512 --scsihw virtio-scsi-single --scsi0 local-lvm:0,import-from=/path/to/ubuntu-24.04-server-cloudimg-amd64.img,discard=on,iothread=1,ssd=1 --net0 virtio,bridge=vmbr0 --rng0 source=/dev/urandom --tablet 0 --boot order=scsi0 --vga serial0 --serial0 socket --ide2 local-lvm:cloudinit --ciuser myuser --cipassword changeme --sshkey /path/to/your-public.key --ciupgrade 0 --ipconfig0 ip=dhcp

Note that you cannot copy-paste those blindly, you have to adjust a few parameters to your local environment (especially the VMID, disk image and SSH key paths).

Follows a description of relevant options at the exclusion of self-evident ones (name, description, cores, memory…), as well as some possible variations you might want.

Generic options

qm create 1000: the Proxmox CLI command to create a VM. Replace 1000 by the VMID of your choice (must be ≥ 100)
--template 1: directly convert the created VM to a template

--ostype l26: hint to optimize for a Linux 2.x-6.x-based system
--machine q35: use a modern machine type
--cpu host: pass-through the host CPU type to make all its features available in the VM
--balloon 512: when set to a lower value than memory, enables dynamic memory allocation
--scsihw virtio-scsi-single: the most performant SCSI controller, especially when combined with iothread=1 (see next point)
--scsi0 local-lvm:0,import-from=/path/to/debian-12-generic-amd64.qcow2,iothread=1,discard=on,ssd=1:
- import (i.e. copy) the referenced cloud image as the VM disk
  - replace /path/to/ with the full path to where you downloaded the cloud image (which you should have already done by now if you have followed the resources linked above 😉)
- configure it with performance (IO Thread), thin-provisioning and SSD-optimized settings
  - remove discard=on and/or ssd=1 if not applicable to your storage
--tablet 0: one of the lesser-known performance tips but one of the most important! Disables the USB tablet device only needed when connecting via the integrated console to guests with a GUI (e.g. Ubuntu Desktop). Reported to have a big performance impact.
--rng0 source=/dev/urandom (optional): provides a virtual hardware random number generator to get entropy from the host system (can speed things up during the first boot)

Up to here were performance-related options applicable to all VM templates, not only cloud-init ones. Here comes the cloud-init specific bits:

--boot order=scsi0: apparently speeds up booting for cloud-init enabled images
--vga serial0 --serial0 socket: creates the serial connection expected by most cloud images in their “native” cloud environments; also useful to monitor and troubleshoot the boot process via the Proxmox console
- verified to work with Debian 12 and Ubuntu 24.04 server cloud images; remove if causing issues with the image you’re using
--ide2 local-lvm:cloudinit: creates the required cloud-init “CD-ROM” drive
--ciuser myuser (optional): provides a custom username for the user account provisioned by cloud-init; without it the account name will depend on the distribution’s default (debian for Debian, ubuntu for Ubuntu… Check your cloud image docs about this)
--cipassword changeme (optional): generally not needed nor recommended, but useful for quickly making sure everything is all right the first few times over; afterwards use a SSH key instead
--sshkey /path/to/your-public.key (required if not setting a password): the authorized SSH public key that will be placed in the user account created by cloud-init
--ciupgrade 0 (optional): disable automatically upgrading packages during first boot; useful to speed things up during testing, afterwards remove it/set it to 1 (the default) if you want “always fresh” clones (which is probably a smart choice)
--ipconfig0 ip=dhcp: cloud-init in Proxmox doesn’t have a network configuration by default, so use this to let DHCP handle it or use something like --ipconfig0 ip=10.0.10.123/24,gw=10.0.10.1 for static config. Can be done later for each VM individually, just don’t leave it empty otherwise they won’t have any network by default.

If you already have custom cloud-init snippets, specify them via --cicustom "user=<volume>,network=<volume>,meta=<volume>", e.g. --cicustom "user=local:snippets/user-config.yaml".

If you do, make sure you have the equivalents of the Proxmox cloud-init options above set in your custom config, because using a custom user snippet overrides the complete user config set in the GUI or config! Yeah I know, it sucks and it’s not documented, boo Proxmox.

Fortunately, as mentioned in the docs the GUI config can be dumped to serve as a base for custom configs:

qm cloudinit dump 1000 user
qm cloudinit dump 1000 network

Note

Unlike Proxmox’ implementation, when using --cicustom and in the absence of network configuration, the image’s cloud-init process will generate a network configuration that will issue a DHCP request on a “first” network interface. So if DHCP is what you want, you don’t have to supply a "network:..." snippet besides the "user:..." one.

Post-creation steps

The only thing that cannot be done in the same step (due to using import-from) is resizing the disk image. I personally prefer doing it on the cloned VMs rather than on the template itself to reduce cloning time and adjust the size depending on the VM’s needs, but there is also a case to be made to do it on the template directly.

So in my case I first clone the template to a new VM:

qm clone 1000 150 --full --name "debian12-cloud"

Note

Besides being generally recommended for VMs you will keep around, it seems we can only use a full clone when using --scsihw virtio-scsi-single as without the --full option I get:

Linked clone feature is not supported for drive 'scsi0'

YMMV.

Then expand its disk size:

qm resize 150 scsi0 15G

And then we’re ready to fire up the VM!

Double-click on the VM name (or on the “Console” button at the top right) instead of going through the integrated VM-specific menu “Console” element, because the pop-up window it opens can be resized fullscreen and allows to scroll back the buffer and read the boot log if necessary.

After checking everything works, you may want to stop and destroy this test VM:

qm stop 150
qm destroy 150 --purge --destroy-unreferenced-disks 1

Now you can do the final adjustments to your template (e.g. remove --cipassword, --ciupgrade 0 etc.) and you are ready to rock the cloud-init lifestyle in Proxmox! ☁️🤘🕺

UEFI variant

Generally I try to use as modern a stack as is reasonable, because software written in the last few years is more likely to be tested with it than a more legacy stack.

But I have realized that UEFI is much less commonplace in virtualized environments than on bare metal, making it less tested and I’d say, slightly less supported overall (case in point: it’s still not the default in QEMU/Proxmox).

However it is easy enough to use it in our templates by adding the following options:

--bios ovmf --efidisk0 local-lvm:0,efitype=4m,size=4M,pre-enrolled-keys=0

The only thing to note is that pre-enrolled-keys=0 disables Secure Boot, which trips up all the distros that don’t want to play the Microsoft game (Arch Linux being a notable one for me). Leave the parameter out or switch its value to 1 for a Secure Boot-enabled template (confirmed working with Ubuntu for example)!

The QEMU Guest Agent conundrum

By default, no cloud images I know of come with qemu-guest-agent preinstalled, but it’s pretty useful on Proxmox.

To install it in your cloud images, you basically have two options:

Install and use libguestfs’ virt-customize to the cloud images themselves, as illustrated in this random blog post I found
Let cloud-init do it during the first boot of each cloned VM using a custom cloud-init snippet: see this SuperUser answer for an example. The required lines to add to your user-config.yaml are:
```
#cloud-config
...
package_update: true
packages:
  - qemu-guest-agent

runcmd:
  - systemctl enable --now qemu-guest-agent
```

Remember that using a custom user snippet overrides the complete user config set in the GUI or config, so those lines must be added to your complete user snippet!

In this case add --cicustom "user=local:snippets/user-config.yaml" and--agent 1,fstrim_cloned_disks=1 when creating the template (see the docs for details).

Tips & tricks

don’t use Debian genericcloud image: its kernel is optimized for Azure & AWS environments and in my tests, didn’t work with Proxmox. I had started with this one (being fooled by the wording on the download page ("genericcloud: Similar to generic. Should run in any virtualised environment. Is smaller than generic by excluding drivers for physical hardware"), spent quite a bit of time troubleshooting the VM booting but cloud-init not kicking in, until I eventually tried the generic image where everything worked perfectly. The Debian wiki actually sets the record straight:

The generic image uses Debian’s standard Linux kernel packages, while the genericcloud image uses the cloud kernel build. The cloud kernel disables a large number of device drivers and primarily targets the Amazon EC2 and Microsoft Azure VM device models. It may be usable in other environments, but for maximum compatibility we recommend using the generic images.

While troubleshooting I’ve seen plenty of other reports of people having issues making the genericcloud image work with Proxmox, while it worked for some others… generic is the reliable, consistant option. ’nough said!
you can get Proxmox to display .qcow2 images alongside regular .iso in its GUI, by simply suffixing/replacing their extension with .img (like Ubuntu does). It’s a regex issue ¯\_(ツ)_/¯

Using a container to sidestep a forgotten password in CasaOS

Sat, 03 Feb 2024 00:00:00 +0100

Problem statement

As part of dabbling with self-hosting again, I installed CasaOS on an Oracle Cloud free Ampere instance to try it out.

After setting it aside for a few weeks, when I logged in via SSH and tried to use sudo I realized I absolutely couldn’t remember my user’s password 😅

The standard operating procedure in this case is to either reboot the machine on a live system and use that to chroot into the local install, or fiddle with GRUB rescue/kernel command line.¹

But since I could still install containers through CasaOS web interface, I thought I’d use this opportunity to explore ways to recover my sudo access/change my password without rebooting (which could be nice in case it is essential to avoid downtime).

Edit: As I was finishing this post, I realized that the third constraint listed below was incorrect, and as a result a less cumbersome resolution was possible. So read on if you are interested in the learning journey, or jump straight to the simplified solution.

The constraints

the CLI/SSH access is “useless”: my user is not setup to interact with the Docker socket without elevated privileges (which is usually a good thing)
besides its App Store content, CasaOS allows to manually install containers using a Compose file or a docker run command (which is then composerized): see official video or screenshots of the process
~~its interface however doesn’t allow to docker exec or interact with containers in any other way; the installed containers must expose a web interface for us to be able to interact with them.~~ As it turns out, CasaOS interface does offer access to containers’ console (equivalent to a docker exec -it <containe> /bin/sh), at least for apps installed from its official Store, via the app Settings > “Terminal and Logs” icon.

Access to a container shell and logs in CasaOS interface

Hacking around

Because I’m pretty new to this, my first idea was to start a container which would give me a root shell with access to the docker socket, and then use that environment to start a second privileged container (Docker-in-Docker style) from which I would mount the host filesystem and use {ch,}passwd or such… Yes, I felt very smart thinking this up 😅

Unaware of my mistaken approach, I looked online for a “web terminal container” and the first result on StartPage was the web terminal GitHub repo, which did what I needed (expose a root shell in the browser) but hadn’t been updated since August 2021 😕

Aside

Docker Hub search is appalling: searching for web terminal gave me a bunch of completely unrelated results. I had to put a dash between the two words to get anywhere, but then the two first results hadn’t been updated in 2 years either…

Searching for web shell and "web shell" didn’t give better results, while web-shell and webshell led to very outdated stuff without any description. I can’t believe there isn’t a reference implementation of this stuff!!

I also looked up ttyd, the software used by web-terminal, and here the first result was fresh from a few days ago; however there was no mention of “How to use this with Docker” , so I wasn’t sure a simple docker run would achieve the desired result… Had I checked its Dockerfile, I would have seen that its ENTRYPOINT/CMD was, in fact, starting up the service 🙃

Lo and behold, I took the “risk” (after checking the image’s Dockerfile) and entered the following command in CasaOS Docker CLI interface:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 7681:7681 raonigabriel/web-terminal:latest

After setting up the access port in CasaOS interface and confirming everything was okay, I clicked “Install” and connected to my server’s port 7681 in a browser… Success, I was in!

For the next stage of my plan I ran the following command inspired by this StackExchange thread:

docker run -ti --privileged --net=host --pid=host --ipc=host --volume /:/host busybox chroot /host
/bin/sh: docker: not found

Crap, I didn’t even check that the container had docker installed 🤦 You can tell I really don’t know what I am doing 😂

Of course when I subsequently tried to install docker…

root@1d0fa4b64c55:/$ apk add docker
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
  docker (missing):
    required by: world[docker]
root@1d0fa4b64c55:/$ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 36 distinct packages available

A quick search seems to indicate that the image is simply too old. Building an updated image was out of the scope I had set for this experiment, so I paused and took time to think.

That’s when it occurred to me that this “nested container” approach was completely useless, and would most likely not work since at that point you have the first container’s virtualized filesystem namespace acting as a buffer between the host and the DinD container… Complete misdirection, backing up!

The solution

After thinking it through some more, I realized one could achieve the desired outcome by simply mounting the host / read-write as an attached volume 😁

docker run -d -v /:/host -p 7681:7681 raonigabriel/web-terminal:latest

Et voilà, I finally had a root shell from which I could chroot into the host and update the user password and/or configure password-less sudo (which amounts to the same). Or really, do (nearly) everything to the host system 😨

Root shell provided by the web-terminal container

Initially I thought it would be necessary to use a privileged container, but trying it showed that wasn’t the case. I guess it’s because I was only editing files/using regular utilites, and not trying to create new devices/nodes etc.

The solution: simplified

This is what happens when you don’t know your tools enough… You miss very obvious pathways that lead to simpler solutions 😁

Indeed, there is no need to install a third-party container. It is enough to add the /:/host volume to an existing app installed from CasaOS official App Store (e.g. NextCloud), and after making sure in its settings it runs with the root UID/GID, we can use the interface to connect the container’s console!

Root shell inside nextcloud container

And that is the Easy Way© to get a root shell on your server using containers!

Trying to refine the solution

Now that I was there, I felt like I should have been able to use a simple busybox image to execute a command non-interactively directly from docker run/Compose file, eliminating the need to find an image that exposes a web service or to connect to the container’s console.

After a bit of fiddling, I came up with the following invocation:

docker run -v /etc/sudoers:/host/etc/sudoers busybox /bin/sh -c echo '%sudo  ALL=(ALL) NOPASSWD: ALL' >> /host/etc/sudoers

But CasaOS interface constantly threw an error. Probably the redirection in the command, but no amount of quoting led me to a successful run.

At that point I threw the towel in: I had recovered access to my sudo access and learnt quite a few things along the way. Time to wrap up.

Takeways

Docker default security story is scary: because the daemon runs as root by default, the mere ability to run containers (even without --privileged) and mount any desired host path as a volume gives root-equivalent status
Due to this, access to CasaOS interface is equivalent to root access on the server

That’s all folks, thanks for reading this account I hope you enjoyed!

See e.g. https://gcore.com/learning/how-to-reset-password-in-linux/ ↩︎

Graphical management interfaces for Linux servers

Mon, 11 Dec 2023 00:00:00 +0100

Terminology considerations

“Server management interface”, “control panel”, “dashboard”… While investigating Free and Open Source graphical server management software, I had trouble establishing the categorical difference between the different solutions I was reading about.

My use case being a “single user homelab/server” and not a multi-tenancy reseller/web hosting solution, I initially left aside the well established category of control panels (with the exception of HestiaCP).

The projects that caught my interest were Cockpit, HestiaCP, Portainer, CapRover, CasaOS, Cloudron, and to a lesser extent, Yacht.

But try as I might, I just couldn’t wrap my head around what precisely distinguished them. Of course I could see that they had different focus: some were more sysadmin/general server management-oriented while others were more app- and Docker-oriented; but I lacked a general category theory for them.

My search-fu not yielding anything relevant, I thought I’d probe ChatGPT about it. This resulted in a most enlightening conversation, as it answered with a distinction that felt like a Eureka moment to me:

The solutions you’ve mentioned, including Portainer, HestiaCP, CapRover, CasaOS, Cloudron, and Yacht, represent a different category of software compared to the server management tools like Cockpit, Webmin, Ajenti, and others.

It then went on to provide the following breakup:

Server Management Tools (e.g. Cockpit, Webmin, Ajenti):

Purpose:

Primarily focused on server administration and configuration.

Provide a broad range of controls over system settings, users, and services.

Complexity:

Varies from beginner-friendly to advanced, catering to users with different skill levels.

Interface Type:

Often web-based but can include desktop GUI options.

Aimed at managing servers directly, not specifically for application or container orchestration.

Scalability:

Typically suitable for small to large enterprises.

Designed for overall server management, including security, networking, and system configuration.

Community Support:

Active open-source communities provide support and contribute to feature development.

Integration:

Extensive third-party module or plugin support for customization.

May include features for managing server-wide tasks and configurations.

Application and Container Orchestration Tools (e.g. Portainer, HestiaCP, CapRover, CasaOS, Cloudron, Yacht):

Purpose:

Focused on managing and orchestrating applications, containers, and services.

Streamlined for deploying and managing web applications and services in containerized environments.

Complexity:

Typically designed to be user-friendly, catering to developers and administrators with varying skill levels.

Interface Type:

Primarily web-based interfaces, emphasizing container and application management.

Often include features for easy deployment, scaling, and monitoring of applications.

Scalability:

Suited for small to medium-scale applications and services.

Designed for ease of use in deploying and managing containerized applications.

Community Support:

Active communities, often developer-focused, providing support and contributing to the development of container-related features.

Integration:

Strong integration with containerization technologies (e.g., Docker, Kubernetes).

Emphasis on managing application stacks, updating applications, and maintaining service availability.

Note that:

the content of some categories don’t really fit their header (e.g. “Interface Type”)
HestiaCP is lumped in the “Application and Container Orchestration Tools” category, despite its features being arguably more control panel-y/towards server management.

This is to just highlight the very reasonable limits of ChatGPT (3.5), in what was otherwise a super useful answer.

Coming up with a unified categorization

I kept poking at the prompt until I managed to formulate a General Category Theory of Server Management Graphical User Interfaces™, inclusive of control panels and mere dashboards. Here’s the result:

The terms “server management tools”, “control panels”, “application and container orchestration tools” and “server dashboards” are sometimes used interchangeably, but they often refer to different aspects of managing and monitoring servers. Here’s how you can distinguish between them:

Server Management Tools:

Purpose: Server management tools are software applications designed to facilitate the configuration, administration, and monitoring of servers. They offer a range of features for tasks such as user management, service configuration, and system monitoring.

Scope: Server management tools encompass a broad set of functionalities related to overall server administration, configuration, and maintenance.

Examples: Cockpit, Webmin, Ajenti and ISPConfig are examples of server management tools.

Control Panels:

Purpose: Control panels are specialized interfaces that provide centralized control over specific server functionalities, often in a web-based environment. They are designed to simplify the management of specific server aspects, such as web hosting, email services, or database management.

Scope: Control panels are more focused and may not cover all aspects of server management. They often excel in specific use cases and are user-friendly for individuals who may not have extensive server administration experience.

Examples: HestiaCP, Froxlor and Virtualmin are examples of control panels.

Application and Container Orchestration Tools:

Purpose: These tools aim to streamline application deployment, scale applications automatically to handle varying workloads, provide centralized management for containers, and automate various aspects of deployment and updates, ultimately enhancing overall efficiency.

Scope: Their scope includes container orchestration, service discovery, load balancing, health monitoring, and automated rollbacks and updates. They focus on managing and coordinating the deployment, scaling, and operation of application containers in a containerized environment, ensuring seamless and efficient operations.

Examples: Portainer, CapRover and Yacht are examples of application and container orchestration tools.

Server Dashboards:

Purpose: Server dashboards are visual representations of real-time or historical server data, allowing users to monitor key metrics and performance indicators. They provide a quick overview of the server’s health and status.

Scope: Dashboards are primarily focused on providing visibility into server performance metrics and may not offer the same depth of configuration and administration features as management tools or control panels.

Examples: Grafana, Kibana, and custom dashboards created using tools like Prometheus or Nagios are examples of server dashboards.

In summary, while there is some overlap in functionality, server management tools generally provide a comprehensive set of features for overall server administration. Control panels are more specialized, focusing on specific server functionalities. Application and container orchestration tools are geared towards managing and orchestrating applications and containers, ensuring efficient deployment, scaling, and maintenance. Server dashboards, on the other hand, offer visual representations of server performance metrics for monitoring purposes. The choice between them depends on the specific needs and preferences of the user or organization.

So, even though ChatGPT kind of threw my FOSS requirement to the wind by mentioning famous proprietary panels (that I replaced here), all in all I feel like we did a great job coming up with a good typology of graphical, general purpose GNU/Linux server management solutions (what a mouthful 😅).

Category members

Now that our categories are well defined, it is time to flesh them out by filling them with our candidates!

Without further ado, here’s a curated list with stuff somewhat beyond my initial use case:

Server Management Tools:

Cockpit Project
- License: LGPL 2.1
- Purpose: Web-based server administration and monitoring.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for both SMEs and large enterprises.
- Community Support: Growing community.
- Integration: Focus on integration with systemd and containers.
- GitHub Repository
Webmin
- License: BSD
- Purpose: Comprehensive web-based server configuration and management.
- Complexity: Varies from beginner-friendly to advanced.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs and large enterprises.
- Community Support: Active community.
- Integration: Extensive third-party module support.
- GitHub Repository
Ajenti
- License: AGPLv3
- Purpose: Web-based server administration with support for plugins.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Supports various plugins for extended functionality.
- GitHub Repository
ISPConfig
- License: BSD
- Purpose: All-in-one server management panel.
- Complexity: Intermediate to Advanced.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs and large enterprises.
- Community Support: Active community.
- Integration: Strong support for multi-server setups.
- GitHub Repository
Zentyal
- License: GPLv2
- Purpose: Linux Small Business Server.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Offers a wide range of server modules.
- GitHub Repository

Web hosting/Control Panels

HestiaCP
- License: GPLv3
- Purpose: Web hosting control panel.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages website hosting, email services, and server configurations.
- GitHub Repository
Froxlor
- License: GPL
- Purpose: Server management panel with a focus on web hosting and server administration.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages web hosting, domain configurations, and server settings.
- GitHub Repository
Virtualmin
- License: GPL
- Purpose: Web hosting control panel with a focus on managing virtual hosts.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages website hosting, email services, and server configurations.
- GitHub Repository (Note: Virtualmin is primarily available through its official website, and its source code is available on GitHub for specific components.)
CyberPanel
- License: GPL
- Purpose: Web hosting control panel with a focus on performance and security.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages website hosting, email services, and server configurations.
- GitHub Repository (Note: CyberPanel is primarily available through its official website, and its source code is available on GitHub.)
AlternC
- License: GPLv3
- Purpose: Web hosting control panel with a focus on environmental responsibility.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- GitHub Repository

Application/Container Orchestration Tools:

Portainer
- License: AGPLv3
- Purpose: Docker container management.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages Docker containers with a user-friendly interface.
- GitHub Repository
CapRover
- License: Apache License 2.0
- Purpose: Automated deployment for web apps.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Focus on automated deployment of web applications.
- GitHub Repository
CasaOS
- License: AGPLv3
- Purpose: Server management and automation.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Provides server management and automation features.
- GitHub Repository
Cloudron
- License: Proprietary
- Purpose: Self-hosted web app platform.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages self-hosted web applications.
- GitHub Repository
Yacht
- License: MIT
- Purpose: Docker management with a focus on simplicity.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Simplifies Docker container management.
- GitHub Repository

Now it’s experimentation time! 🥳

PS: If you need even more, here is a random article listing 30 similar solutions including many dashboards.

Happy hacking!

Moving an existing install into a virtual machine

Tue, 05 Apr 2022 00:00:00 +0200

This a follow-up to my tale of a system image.

For fun and convenience, I now wanted to have a bootable image of my old laptop.

So at first I thought of making a Live ISO out of it, but didn’t know the name of the process. I tried a few searches like “convert a configured system into ISO” or “create live ISO from running system”, but what I found looked strangely difficult and didn’t seem to really address my need.

Then in the course of writing the aforementioned post, I was reminded of an Archwiki article I had seen some time ago: Moving an existing install into (or out of) a virtual machine. This was exactly what I wanted, as I didn’t need a Live ISO so much as a VM that I could boot on my main machine whenever I wanted to check something, or bask nostalgically into the glow of my old wallpaper.

Looking into it, I stumbled upon presentation slides by Clonezilla and learned that this process had a name: “Physical-to-Virtual”, or P2V for short (here is another article by IBM from 2009 about this topic).

The ultimate OS conservation project: virtualizing to immortality

Indeed, with a VM image of your old install you can easily keep it around forever, as with the specifications of the machine written down into an .xml file or in a QEMU command line invocation, you can spin one up at will.

I’ll use this opportunity to simplify my system stack, namely to rid it of both LUKS and LVM and consolidate all the existing logical volumes into one single filesystem.

So how do we go about turning our physical system into a functional VM? Here is a rough outline of the process:

Preparing the VM image
Transferring the data
Making the necessary adjustments
Profit

Note

There are different ways to approach steps 1 & 2 depending on what you are working with, e.g. your target hypervisor, what kind of backup/file transfer program is available to you and whether or not you need to modify the image.

If you need to do any modifications to the partitions/filesystems, forget about Clonezilla/partclone/FSArchiver savepart mode and use a file-level transfer method.

Since this is my case I’ll describe this approach, but if e.g. for you the physical and virtual machines are connected via the network and no modifications are needed, then Clonezilla remote device cloning could work great.

(Heck, if no modifications are needed you could even just use a loopback file mounted on /home/partimag in the live environment on the physical machine and have Clonezilla make a local clone!)

Here I will target QEMU for the hypervisor as that’s what I use, but only little adjustments should be required for other hypervisors.

All right, let’s delve into it!

Step 1: Preparing the VM image

Here we will create and prepare the image file which will be used as the VM drive.

In this case I will directly create and work with QEMU qcow format, but will also illustrate how to do it all with a raw image file which can be later converted to the desired format if needed.

Creating the file

First we create an image of an appropriate size¹.

qcow image:

qemu-img create -f qcow2 -o cluster_size=2M --prealocation=falloc p2v.qcow2 15G

The -o cluster_size=2M option is used to increase performance.

Then we make the image available to the system:

sudo modprobe nbd max_part=1
sudo qemu-nbd -c /dev/nbd0 p2v.qcow2

Raw image:

On file systems supporting it (ext4, XFS, Btrfs, FAT – but not exFAT), fallocate is the best choice:

fallocate -l 20G p2v.img

For systems without fallocate but sparse file support, use truncate:

truncate -s 20G p2v.img

If even sparse files are too much to ask, then it’s time to get the disk destroyer out (warning: might take a while):

dd if=/dev/zero of=p2v.img bs=1M count=20000

If you really don’t want to know/choose and have QEMU installed, let it handle the details for you:

qemu-img create p2v.img 20G

Partitioning

Now to partition our image. It is easier to use the same partition table type as your physical machine (MBR or GPT), except if you want to convert it of course.

Warning

All of the following commands must be run with elevated privileges. Be careful with what you execute. Replace /dev/nbd0 with p2v.img (or whatever you named it) if using a raw disk image.

MBR

Here is the simplest way I found to create a single partition with MBR scheme aligned on the 1 MB boundary (with the default partition type for GNU/Linux).

printf ',' | sfdisk /dev/nbd0

Crazy right? If that (rightly) scares you, I’ve got an equivalent, more spelled-out parted one-liner:

parted --script --align optimal /dev/nbd0 mklabel msdos mkpart primary 1 100%

GPT

And here are their GPT equivalents (with sgdisk thrown into the mix):

printf ',' | sfdisk --label gpt /dev/nbd0
sgdisk --largest-new 0 /dev/nbd0
parted --script --align optimal /dev/nbd0 mklabel gpt mkpart $name 1 100%

See the $name in that parted command here? That’s because parted requires (yes, requires) that a partition be given a name in GPT mode, even though that’s absolutely not part of any specification… So do as you please to christen the first realm of this new digital kingdom!

Note however that if you are planning to use UEFI, a second EFI System Partition (ESP) is required as well. Here are three more one-liners to create both partitions in one go with a 512 MiB ESP, and the rest for the second partition (source for sfdisk example) :

printf ',512M,U\n,,' | sfdisk --label gpt /dev/nbd0
sgdisk -n 1:0:+512MiB -t 1:ef00 -n 2:0:0 /dev/nbd0
parted --script --align optimal /dev/nbd0 mklabel gpt mkpart EFI 0% 513MiB mkpart System 513MiB 100% set 1 esp

Formatting

Before mounting, we need to format our newly created partition(s) with a filesystem. It’s certainly a safe choice to go with the same one as the source system, albeit not strictly required. Ext4 example:

sudo mkfs.ext4 /dev/nbd0p1

If using a raw image file, we first need to use losetup to make the partition(s) available to the system:

sudo losetup --partscan --find --show --nooverlap p2v.img
--> /dev/loop0

From here on replace /dev/nbd0 with /dev/loop0 (or whatever the command returned).

In case of UEFI, the ESP needs to be formatted as FAT (FAT32 recommended):

sudo mkfs.fat -F32 /dev/nbd0p1
sudo mkfs.ext4 /dev/nbd0p2

Mounting

That’s it, now we can finally mount our image. For BIOS (single partition):

sudo mount /dev/nbd0p1 /mnt

For UEFI (two partitions), replace $ESP with the desired mount point:

sudo mount /dev/nbd0p2 /mnt
sudo mkdir -p /mnt/$ESP
sudo mount /dev/nbd0p1 !$

That’s it for the preparations, we are now ready to transfer the source system content!

Step 2: Transferring the data

This part depends on how you realized your full system backup on the source machine. For completeness’ sake and to create the counterpart to the previous article, I’ll show the restoration process of the various tools presented in the linked section.

partclone

Word of warning concerning partclone: it’s not good at restoring to a device smaller than the source, even if the actual content would fit in. Despite its offering of a “Don’t check device size and free space” (-C, --nocheck) option, my experience matched the web reports saying that restoration fails with a target seek ERROR:Invalid argument. Clonezilla FAQ even warns against this use case.

However, for completeness’ sake I’ll leave a quick restoration example to illustrate how to restore from a zstd-compressed image:

umount -R /mnt
zstdcat /path/to/partclone.img.zst | partclone.restore -o /dev/nbd0pY

Replace the Y at the end with the appropriate partition number for your use case.

Tip

partclone offers a --restore_raw_file option that aims at “creating special raw file for loop device”. This image wouldn’t have a partition table though so it’s not useful for our use case, but it’s still good to know it’s there 😉️

FSArchiver

I have a sweet spot for FSArchiver due to its versatility and very complete feature set. The only real downside I found to it are its exclusion rule patterns being less than clear.

Being a file-aware tool instead of block-aware one like partclone, it has absolutely no issues restoring to a destination smaller than the source (it is even one of the suggested workaround in the aforementioned Clonezilla FAQ entry!).

If you made a backup using its savepart mode, see the official docs for nice examples illustrating the different restoration cases.

On the other hand, if like me you used its savedir mode so as to have more flexibility, restoring is as simple as:

fsarchiver --jobs=$(nproc) restdir /path/to/backup.fsa /mnt/

May St IGNUcius bless FSarchiver and its author/team! 😇️

SquashFS

Another very straightforward option:

unsquashfs -force -dest /mnt/  backup.squashfs

Bonus points go to SquashFS compared with FSArchiver and tar because we can very simply access the archived content without having to restore/extract it: a simple mount backup.squashfs /path/to/mount/point suffices to make it browsable (with very good performance) 👍️

Rsync & tar

rsync is just about reversing the source and destination in its invocation, and tar about switching -c, --create with -x, --extract:

rsync -aHAXUUSh --info=progress2 --partial /path/to/backup/dir/* /mnt/
tar -x --zstd -p --acls --xattrs --atime-preserve=system -f /path/to/backup.tar.zst /mnt/

Alright, now that we have transferred the content of our physical system into the VM image, let’s move to the final adjustments.

Step 3: Adjusting the system config

Now it is time to chroot into our restored environment and make the needed adjustments. This step is very much dependent on your source system configuration and which modifications you wish to implement (if any).

arch-chroot /mnt

systemd-nspawn can replace arch-chroot if not using an Arch-based distro.

In my case, it equated to removing/commenting out references to the former LVM/LUKS devices in /etc/{fstab,crypttab} and configuring a single mount:

sed -i 's/^\([^#].*\)/# \1/g' /etc/crypttab
sed -i 's/^\([^#].*\)/# \1/g' /etc/fstab
echo "/dev/vda1  /  ext4  noatime  0  1" >> /etc/fstab

The two sed invocations comment out “active” lines while leaving blank lines alone.

For GPT replace the last line with:

printf "/dev/vda2  /  ext4  noatime  0  1\n/dev/vda1  /  vfat  defaults  0  2\n" >> /etc/fstab

Network-wise, I had nothing to do since NetworkManager was already handling automatic network configuration with DHCP.

I also removed the now unneeded lvm2, encrypt, keyboard and keymap mkinitcpio hooks as well as the i915 module from /etc/mkinitcpio.conf and regenerated the initramfs.

Finally, I removed the cryptdevice= kernel command line parameter from /etc/default/grub, installed GRUB in the image file boot sector and regenerated its config.

grub-install --target=i386-pc /dev/nbd0
sed -i '/cryptdev/s/.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/' /etc/default/grub
grub-mkconfig -o /boot/grub/grub.cfg

Don’t forget to replace references to nbd0 (or loop0) in /boot/grub/grub.cfg with whatever disk naming you will use (in my case /dev/vda):

sed -i 's/nbd0/vda/g' /boot/grub/grub.cfg

Then exit the chroot, unmount the image file and remove the nbd device and module:

exit
sudo umount -R /mnt/
sudo qemu-nbd -d /dev/nbd0
sudo modprobe -r nbd

Or loop mount:

losetup -d /dev/loop0

(Optionally, convert the image to a format suitable to your target hypervisor)

And voilà, you are now ready to try out your brand new virtualized system! Hopefully it will boot on the first try, otherwise just make the image available again as before, mount it and chroot in it to troubleshoot the issue.

Enjoy your nascent hall of past digital abodes! 😉️

Edit (2024-07-09): I’ve just stumbled upon virt-p2v from libguestfs, which is a “GUI interface to convert a physical machine to run as virtual machine on KVM”. It is a companion front-end to virt-v2v, and “comes as an ISO, CD or PXE image that can be booted on physical machines to virtualize those machines”. Looks like a great alternative to try as well!

At a minimum it should be able to accommodate all the data you’re going to transfer from the physical system, plus a little bit of spare space for system maintenance. ↩︎

Offline Arch install with local pacman cache

Sat, 12 Mar 2022 00:00:00 +0100

While developing my Arch VM script, I had to go through the installation over and over to test every change.

I am tethered to my phone for Internet connectivity, and the quality and speeds are… How should I put it? Underwhelming to say the least (I live in a mountainous rural area).

So while doing so, I grew increasingly wary of the long wait for packages to download and the 287 MB worth (gah!) of data it would chip off my 40 GB plan each and every time. What I needed was a way to provide a local package cache to pacman.

Amazingly, but certainly not surprisingly, there is a great article on the Arch wiki for that.

However, due to my target file system being exFAT (cough ventoy shenanigans, cough) and some file names containing the forbidden : character, this solution didn’t work for me.

Also because I am using GNOME Boxes, sharing a folder with the host was out of the equation as that entails installing some SPICE-related packages which pull in an entire graphical stack as dependencies, hence defeating the purpose.

Setting up SSH access to the guest requires quite some fiddling, and anyway it wasn’t immediately clear to me how I would instruct pacstrap to tap into the network cache, or it would have likely required some modifications to the live pacman.conf and I wanted to avoid all that complexity¹.

Remastering the ISO was another option that I left out because I wanted the script to work with a vanilla install. No, what I was really looking for was something of the utmost simplicity².

What I finally came up with, and not to toot my own horn but I think it is pretty darn brillant (considering I haven’t seen it mentioned anywhere online when I researched the topic³), is a bind mounted squashfs⁴ package cache placed on an external medium like a USB flash drive. This in turn can easily be shared in GNOME Boxes even without the SPICE packages installed in the guest!

Here is a rundown of the process:

create a SquashFS archive of /var/cache/pacman/pkg on a system containing the desired package set (I reused a just-installed VM to match its exact minimal set but you can go wild and copy everything from your main rig, that’ll give you a larger cache):
```
mksquashfs /var/cache/pacman/pkg/ pkg.sqsh \
    -comp zstd \
    -no-exports \
    -progress \
    -mem 1G \
    -not-reproducible
```

copy the SquashFS file on the external medium you will mount during the install
boot into the Arch Live environment and create two directories, one for your external medium and one for the archive:
```
mkdir /tmp/{usb,squash}
```

Note

Do not use /mnt if you plan to use it as the installation mount point.

mount the external medium containing the squash file and then the squash file itself:
```
mount /dev/XXXY /tmp/usb
mount /tmp/usb/pkg.sqsh /tmp/squash
```
finally, and that’s what’s doing the trick for pacstrap, bind-mount /tmp/squash on archiso’s /var/cache/pacman/pkg:
```
mount --bind /tmp/squash /var/cache/pacman/pkg
```

That’s it, now you just have to add -c to your pacstrap invocation:

pacstrap -c /mnt base linux [...]

Downloads will be skipped (as long as package versions in the squashfs archive are not older than the synced databases’ ones), and now you can carry on with your installation, contented by the blissful release of not having to suffer through the hell of this perpetual cycle of reincar… Err I mean wasteful downloading of already acquired packages 😊️

Happy hacking!

Just as I was bringing the finishing touches to this article, I realized my blunder: what I needed was an access from the guest to the host, not the contrary. This would allow me to do something like an SSHFS mount or even serving the cache via a simple web server. Follow-up article to come on how easy it actually is to connect from a guest to the host in GNOME Boxes 😁️ ↩︎
Yeah, no seriously, in my use case that was the web server/SSHFS mount. But let’s just say this article was actually meant to be for a fully offline install… 😂️ ↩︎
I’ll admit this may only be due to my constraints being ridiculously specific, but maybe for once I can dream of having come up with a truly original solution, wouldn’t you agree ? 😀️ ↩︎
The idea of using SquashFS instead of a more common loop-mounted filesystem-in-a-file is what I felt was the real feat of ingeniosity here. And yes, I need to pat myself on the back from time to time, is it something wrong? 😉️ ↩︎

Ejecting and reattaching a USB drive from the CLI

Thu, 24 Feb 2022 00:00:00 +0100

Setup: I was booted on archlinux-2022.02.01-x86_64.iso using a multiboot USB flash drive made with Ventoy 1.0.65, and had selected the “Copy to RAM” option in the ISO menu with the idea of reusing the drive as a backup destination for a system backup.

The issue: I was unable to normally eject the drive, both eject and udisksctl power-off (after installing udisks2) gave the error Device or resource busy, and lsof | grep -e ventoy -e sdb didn’t return anything. Here is what it looked like:

root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
│ └─ventoy    254:0    0 812.3M  1 dm    <-- problematic entry
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

root@archiso ~ # eject -v /dev/sdb
eject: device name is `/dev/sdb'
eject: /dev/sdb: not mounted
eject: /dev/sdb: is whole-disk device
eject: cannot open /dev/sdb: Device or resource busy

root@archiso ~ # eject -v /dev/mapper/ventoy
eject: device name is `/dev/mapper/ventoy'
eject: /dev/mapper/ventoy: not mounted
eject: /dev/mapper/ventoy: is whole-disk device
eject: /dev/mapper/ventoy: is not hot-pluggable device

root@archiso ~ # ls -l /dev/mapper/ventoy  
lrwxrwxrwx 1 root root 7 Feb 24 20:24 /dev/mapper/ventoy -> ../dm-0

root@archiso ~ # ls -l /dev/dm-0
brw-rw---- 1 root disk 254, 0 Feb 24 20:24 /dev/dm-0

root@archiso ~ # cat /sys/class/block/dm-0/dm/name
ventoy

So, I wasn’t sure what was causing the issue, and I didn’t want to simply pull the drive out as it equates to playing Russian roulette with your devices (do you like getting the rug pulled under you or being knocked inconscious by a stealthy sucker punch? Well, neither do most filesystems).

At first I had worked around the issue by using this trick :

echo offline > /sys/block/sdb/device/state
echo 1 > /sys/block/sdb/device/delete

See this post on the Arch forums for a more thorough procedure along the same vein.

Still, I wasn’t entirely convinced: it didn’t address the root cause of the issue.

Digging a little deeper led me to dmsetup:

root@archiso ~ # dmsetup table
boot-arch: 0 516096 linear 8:1 2048
ventoy: 0 1663640 linear 8:17 59447552

root@archiso ~ # dmsetup info /dev/mapper/ventoy  
Name:              ventoy
State:             ACTIVE (READ-ONLY)
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      254, 0
Number of targets: 1

Well, that READ-ONLY state was already good news, but checking dmsetup manpage I saw it offered a remove command:

root@archiso ~ # dmsetup remove ventoy
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

Victory, the mapping is gone! And now…

root@archiso ~ # eject /dev/sdb -v
eject: device name is `/dev/sdb'
eject: /dev/sdb: not mounted
eject: /dev/sdb: is whole-disk device
eject: /dev/sdb: trying to eject using CD-ROM eject command
eject: CD-ROM eject command failed
eject: /dev/sdb: trying to eject using SCSI commands
eject: SCSI eject succeeded

root@archiso ~ # lsblk  
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
sr0            11:0    1  1024M  0 rom

Success! I saw the LED on my thumb drive flash quickly, and then it fell silent. But wait, what is all this red in my logs?

archiso kernel: sd 6:0:0:0: [sdb] Media removed, stopped polling
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: ldm_validate_partition_table(): Disk read failed.
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel:  sdb: unable to read partition table
archiso kernel: sd 6:0:0:0: [sdb] Media removed, stopped polling
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: ldm_validate_partition_table(): Disk read failed.
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel:  sdb: unable to read partition table
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed

Hum, looks like eject is doing something kind of dirty… Let’s move ahead for now, we’ll come back to it later.

Now, since this is a laptop laying on my desk, I certainly can manually unplug the drive before plugging it back again to make it available. But what if I am lazy, or the machine is faraway, or I just want to take extra care of that super frail USB plug and prevent a superfluous actuation cycle?

Reattaching an ejected USB device from the CLI

Here we are going to make use of the ability to bind and unbind drivers from devices manually from user space. First, get the drive Bus number:

root@archiso ~ # lsusb  
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive  <-- here is my device
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Then Port for “Bus 4 Device 3”:

root@archiso ~ # lsusb -t  
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M  <-- spot it?
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

Now that I know that my USB device is identified as 4-1 (bus-port) (this can be confirmed by checking dmesg | grep usb-storage), we can make it go through an unbinding/binding cycle:

root@archiso ~ # echo "4-1" > /sys/bus/usb/drivers/usb/unbind
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sr0            11:0    1  1024M  0 rom  
root@archiso ~ # echo "4-1" > /sys/bus/usb/drivers/usb/bind  
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

Et voilà, the flash drive is once again available for mount without having to physically remove it!

Note

Trying to bind directly after eject without unbind leads to a write error: device or resource busy . So now I am really left wondering what eject is doing to my drive…

An alternative to manually writing to /sys/bus/usb/drivers/usb/ is to use usb_modeswitch (included on the Arch installation media). In that case we only need the ID pair from lsusb:

root@archiso ~ # lsusb  
...
Bus 004 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive
                       ==== ==== ------ ---------
                                      |         |
root@archiso ~ # usb_modeswitch -v 0x0781 -p 0x5580 --reset-usb
Look for default devices ...
 Found devices in default mode (1)
Access device 003 on bus 004
Get the current device configuration ...
Current configuration number is 1
Use interface number 0
 with class 8
Warning: no switching method given. See documentation
Reset USB device .
 Device was reset
-> Run lsusb to note any changes. Bye!

Props to this AskUbuntu answer, confirmed and extended via this Stackoverflow one.

Wrap up

After a few trial and errors, I realized ejecting the drive was entirely unnecessary: I could get away with just dmsetup remove ventoy and an unbind/bind cycle.

Now if you ever have to cleanly power-off a device on the command line, here is the ultimate method:

% udisksctl power-off -h
Utilisation :
  udisksctl power-off [OPTION…]

Safely power off a drive.

Options :
  -p, --object-path         Object path for ATA device
  -b, --block-device        Device file for ATA device
  --no-user-interaction     Do not authenticate the user if needed

E.g. udisksctl power-off -b /dev/sdb.

For the life of me I don’t know why this utility isn’t included by default on the arch installation media, but in any case you are just one pacman -S udisks2 away from safely-removed device nibbana.

That’s all folks for today’s learning diary. Thanks for reading and take care of your devices!

A system imaging tale: LUKS, Clonezilla and friends

Mon, 14 Feb 2022 00:00:00 +0100

Today’s project: make a system image of my previous laptop for archival purposes before refurbishing it.

Conditions:

make a space efficient image, i.e. not a block-level copy: no need to retain encryption or logical volume setup, only their content matters. However:
all standard and extended file attributes must be correctly preserved, so that if I ever want to boot this image the system will be in a correct state.

The plan: boot up an Arch ISO, open the encrypted drives, set everything up for clonezilla – profit.

Here is how it actually went.

An unexpected obstacle to boot: Ventoy shenanigans

In order to boot into the Arch live environment, I unsheathed my very dear and trusted Sandisk Extreme multiboot USB thumb drive (not the currently sold, lower performing one but the ever chart-topping one from about 10 years ago), which had been irreproachably and impeccably fulfilling its duty in the two years since I had discovered Ventoy, the multiboot solution which surpassed them all and ended my long standing quest to find an easy to use, low-maintenance and low-friction multiboot USB solution. And yes, this was the longest sentence of the entire article 😁

Just recently I had updated it to the latest version using the provided Linux script, which completed without issue. But as soon as I booted on it, I was faced with an error: Warning! This is not a standard device and is not officially supported. Um, beg your pardon? It was working fine those past two years, so certainly it was the update which broke something…

Searching for this error, I eventually found this “kill switch” the dev introduced slightly after I used the software, to try and contain the flood of users with non-standard installs complaining to them when something got wrong.

I can’t really fault them for that, but here is the thing: mine was a pure product of the official tool from back in 2020, with no deviation from the prescribed layout/FS whatsoever. The first partition was starting at the 2048/1 MB boundary and formatted with the default exFAT, the second partition was 65536 sector large, the drive had an MBR partition table and all and all…

# fdisk -l /dev/sda
Disk /dev/sda: 59.63 GiB, 64023257088 bytes, 125045424 sectors
Disk model: Extreme  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd1a119ab

Device     Boot     Start       End   Sectors  Size Id Type
/dev/sda1  *         2048 124979887 124977840 59.6G  7 HPFS/NTFS/exFAT
/dev/sda2       124979888 125045423     65536   32M ef EFI (FAT-12/16/32)>

The official command line tool didn’t complain either while upgrading or when run manually:

# ventoy -l /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Ventoy Version in Disk: 1.0.65
Disk Partition Style  : MBR
Secure Boot Support   : NO

Indeed but nope, I got axed by the kill switch all the same. Fuck artificial restrictions and treacherous computing. And I am burned by a free software project at that… What to expect from this world.

Anyhoo, after transfering out my 24 GB of GNU/Linux ISOs (God bless USB 3.0 & quality flash memory speeds), I forced a reinstall with the -I switch:

# ventoy -I /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Disk : /dev/sda
Size : 59 GB
Style: MBR


Attention:
You will install Ventoy to /dev/sda.
All the data on the disk /dev/sda will be lost!!!

Continue? (y/n) y

All the data on the disk /dev/sda will be lost!!!
Double-check. Continue? (y/n) y

Create partitions on /dev/sda by parted in MBR style ...
Done
Wait for partitions ...
partition exist OK
create efi fat fs /dev/sda2 ...
mkfs.fat 4.2 (2021-01-31)
success
Wait for partitions ...
/dev/sda1 exist OK
/dev/sda2 exist OK
partition exist OK
Format partition 1 /dev/sda1 ...
mkexfatfs 1.3.0
Creating... done.
Flushing... done.
File system created successfully.
mkexfatfs success
writing data to disk ...
sync data ...
esp partition processing ...
Open ventoy efi file 0x610ac0
ventoy x64 efi file size 1757184 ...
Open bootx64 efi file 0x610ac0
Open ventoy ia32 efi file 0x610f10
ventoy efi file size 1183744 ...
Open bootia32 efi file 0x610ac0

Install Ventoy to /dev/sda successfully finished.

sudo ventoy -I /dev/sda  5,11s user 2,58s system 19% cpu 39,813 total

After transferring my files back and rebooting on the flash drive, everything was back in order and I was ready to rock 🤟

Optional step: setting up a comfortable environment

Being console-based, the archiso environment is somewhat austere. So the first thing I usually do is to set up SSH access so as to connect from my main machine:

# set french keymap
# the "-latin9" part is only needed if using e.g. accents
loadkeys fr-latin9
# directly connect to a known Wifi SSID
iwctl station wlan0 connect <SSID>
# Enable NTP to have accurate time
timedatectl set-timezone <Area/City>
timedatectl set-ntp true
# set root passhrase to enable SSH connection
passwd
# start SSH service
systemctl start sshd
# get IP address
iwctl station wlan0 show | grep IP | awk '{print $3}'
# or without iwd:
ip a | grep wlan | awk 'FNR == 2 {print substr($2, 1, length($2)-3)}'

Finally, connect to the live environment via SSH:

ssh -o PreferredAuthentications=password \
    -o PubkeyAuthentication=no \
    -o UserKnownHostsFile=/dev/null \
    root@<other-machine-IP>

👉️ The options used allow not to be affected by the local SSH configuration (SSH keys…), as well as well not polluting the local KnownHosts file with this one-time connection.

That’s it, now we can resume with the common procedure!

Clonezilla and LUKS/LVM

In order to efficiently image the system, we first need to open any LUKS container and make sure all the LVs are displayed correctly:

cryptsetup open --allow-discards /dev/sdb2 os
lvs

Then we create the /home/partimag directory (hardcoded backup destination in Clonezilla) and mount our drive on it:

mkdir /home/partimag
mount -o noatime /dev/sdc1 !$

Note

When booting the archiso, I selected the “Copy to RAM” option so as to free up the drive and reuse it as backup destination.

However, in order to do that I first needed to get rid of a lingering /dev/mapper/ventoy mapping on its first partition, which made eject/udisksctl throw errors despite the fact that it wasn’t mounted and lsof | grep -e ventoy -e sdb didn’t return anything. I didn’t want to pull it out on the off chance it’d corrupt something…

I wrote a separate post which goes into the details, but the gist is a simple dmsetup remove ventoy took care of the mapping, and then I was able to normally mount the drive.

And now it is on! Launch clonezilla from the command line and follow the on-screen instructions depending on your needs. Some personal recommendations:

use expert mode
avoid the 4G backup splitting by entering a large number when prompted (like 100000)
select “parallel zstd” (-z9p) as the compression algorithm

Or if you feel like keeping it CLI-only…

/usr/bin/ocs-sr \
    --use-partclone \
    --confirm \
    --clone-hidden-data \
    --zstdmt-compress
    --image-size 1000000 \
    --fsck-src-part \
    --skip-enc-ocs-img \
    --postaction choose \
    saveparts <backup-name> sdb1 sdb2

This is the long form equivalent of the

/usr/bin/ocs-sr -q2 -c -j2 -z9p -i 1000000 -fsck -senc -p choose saveparts <backup-name> sdb1 sdb

that got copied in a file under /tmp after validating my options in the TUI.

Then we just sit back and relax… Wait, what? dd mode? Why aren’t my OS logical volumes backed up by the regular free space-aware partclone mode? I explicitly decrypted the containers for that purpose, and saw the LVs being listed during the preparation screens! Although it’s true I could only select whole partitions to be backed up and not specific logical volumes, and saw a line flash by saying something like Shutting down the Volume manager… Oh come on.

One Ctrl+c later the backup of /dev/sdb2 was cancelled, but the unencrypted /dev/sdb1 with the arch LV (the only one in the boot VG) proceeded without issue. What the heck clonezilla?

Hum alright, according to this 2014 thread with the developer’s input, it seems this isn’t as straightforward as it looks. And it happened even much more recently to another user (although according to the dev’s answer it shouldn’t be the case), so anyway… Time for some manual partclone fun I guess!

System imaging, the manual way

Full disclosure: I had already stumbled upon the “manual partclone required” caveat when trying to efficiently backup LUKS-encrypted systems with Clonezilla, so I wasn’t really at a loss. All in all it’s pretty simple, you just have to use partclone like so:

partclone.$fstype --clone --source ... --output ...

Wich gave in my case:

partclone.ext4 --clone --source /dev/mapper/arch-root --output <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl.img

Tip

If you want to have the same nice ncurses-based UI as Clonezilla does, add the -N/--ncurses option (example).

Though if we really want to mimick Clonezilla and have space-efficient backups, we gotta add compression to the mix!

Compressing partclone images with zstd

Being a devout Zstandard Zealot© (this post summarizes well my feelings – no affiliation with the author), here is how to interface it with partclone for a nicely multi-threaded compression:

partclone.ext4 --clone --source /dev/mapper/arch-root | zstdmt -o <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl-img.zst

Info

From zstd man page: zstdmt is equivalent to zstd -T0, enabling auto-adjusted multi-threading.

If you are like me and have several LVs to clone and their names follow a pattern, you can use a for loop to optimize things a bit:

for i in arch-{root,home,opt,var} boot-arch; do partclone.ext4 --clone --source /dev/mapper/"${i}" | zstdmt -o <backup path>/$(date -I)-"${i}".ext4-ptcl-img.zst; done

Et voilà, space-efficient compressed images of your decrypted LVs!

Tip

Someone made a gist using somewhat older commands and touching upon restoration etc… Give it a look if you want to review the process!

Extra: comparing the process with different tools

A very precious thing in this world is diversity. Of people, ideas, food… And backup tools. Here is a rundown of the same task undertaken with four of them:

FSArchiver

The first time I realized I couldn’t use Clonezilla to properly back up my decrypted LVs, I stumbled upon a great alternative to partclone: FSArchiver.

The main reason for using it over the former? FSArchiver:

directly integrates multi-threaded compression with zstd (and recommends it over any others)
allows to specify multiple filesystems to be archived at once
also works with directories (creating a compressed and checksummed tarball of sorts)
allows to restore to a different filesystem and creating it in one go
allows to restore to a smaller partition by default, whereas partclone needs the -C option (and often just ends on a seek error)
accepts exclusion patterns to filter what’s archived/restored (the syntax is a bit tricky though)
offers to optionally encrypt data in the archive
and is generally really careful with data integrity… If that wasn’t already reason enough 😉

Its multi-threaded compression-ready invocation goes something like this:

fsarchiver --zstd=3 --jobs=$(nproc) savefs <backup path>/<archive-name>.fsa <list of devices to backup> --exclude=<exclude pattern>

--zstd=3 stands for zstd-compression at level 3
--jobs=$(nproc) works exactly like the eponymous make switch, enabling as many threads as there are logical cores on your system.

So for example:

fsarchiver -Z3 -j$(nproc) -v savefs <backup-path>/old-laptop.fsa /dev/mapper/arch-{root,home,opt,var} /dev/mapper/boot-arch --exclude="lost+found" --exclude="/var/cache/pacman/pkg/*"

The -v/--verbose option is useful to get a progress report, however it will list every file being archived and as such likely blow away your terminal history.

BTW, here is why clonezilla defaults to partclone instead of fsarchiver in the words of its author. Tldr: partclone supports pipes/stdin/stdout which are needed for other Clonezilla features, while fsarchiver doesn’t. Tough life.

Note however that FSArchiver (just like partclone) doesn’t deal with recreating a partition table on the destination, or any other form of partitioning/preparing the drive such as LUKS/LVM: you gotta do all that manually up to the point where you are ready to restore filsystems (that is to say, fsarchiver isn’t a disk cloning solution).

Also, you cannot merge the content of separate partitions created this way into one destination (e.g. if you want to consolidate separate LVs into one FS): they will overwrite each other. For that you’d rather use the savedir/restdir mode, which is akin to a tarball on steroids.

Squashfs

I discovered this method by stumbling upon this gem on the Arch wiki, “Full system backup with SquashFS”, and although the article is in dire need of a proper rewrite (which I’ve added to my todo list) I really wanted to give it a try.

In the words of TLDP, SquashFS is

“a read-only file system that lets you compress whole file systems or single directories, write them to other devices/partitions or to ordinary files, and then mount them directly (if a device) or using a loopback device (if it is a file). […] For archiving purposes, SquashFS gives you a lot more flexibility and performance speed than a tarball archive.”

Its selling points to me are:

random access & transparent decompression: no need to decompress and extract the entire image to browse its content (this is huge, especially when talking about gigabyte-sized images…)
file-level deduplication
exclusion patterns
zstd support 😊

However be aware that it doesn’t support ACLs, so if you rely on them you should look somewhere else.

After quickly perusing its manpage I gave it a go:

mount -o ro /dev/mapper/arch-root /mnt
mount -o ro /dev/mapper/boot-arch /mnt/boot
for i in home opt var; do mount -o ro /dev/mapper/arch-"${i}" /mnt/$i
cd /mnt

mksquashfs ./ <backup-path>/old-laptop.sqsh \
    -comp zstd \
    -no-exports \
    -progress \
    -mem 6G \
    -not-reproducible \
    -e var/cache/pacman/pkg

Results:

Squashfs 4.0 filesystem, zstd compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 3912057.66 Kbytes (3820.37 Mbytes)
	35.64% of uncompressed filesystem size (10975717.90 Kbytes)
Inode table size 2886293 bytes (2818.65 Kbytes)
	24.42% of uncompressed inode table size (11819496 bytes)
Directory table size 3606490 bytes (3521.96 Kbytes)
	36.39% of uncompressed directory table size (9910813 bytes)
Xattr table size 621 bytes (0.61 Kbytes)
	28.98% of uncompressed xattr table size (2143 bytes)

To put things in perspective:

root@archiso ~ # du -sh /mnt
16G	/mnt
root@archiso ~ # du -sh /mnt/var/cache/pacman/pkg
4.5G	/mnt/var/cache/pacman/pkg
root@archiso ~ # du -h <backup-path>/old-laptop.sqsh
3.8G	<backup-path>/old-laptop.sqsh

So it went from 16-4.5=11.5 GiB to 3.8… Nice!

It produced a slightly smaller archive than both partclone and fsarchiver (even when accounting for the non-excluded pacman cache), but it took 11.5 minutes. After testing, fsarchiver savedir took 3min08s and produced a 4.2 GiB image (with --exclude='*.pkg.tar.zst'), but then fsarchiver doesn’t produce browsable archives and doesn’t do any deduplication 😊

Rsync

One step more remote from system imaging, the ever powerful Swiss-army knife of file transfer can evidently also be used to create full system backups. Here is my personal take on it:

rsync \
    --archive \
    --hard-links \
    --acls \
    --xattrs \
    --atimes \
    --open-noatime \
    --sparse \
    --human-readable \
    --info=progress2 \
    --partial \
    --exclude={"var/cache/pacman/pkg/*","*/lost+found"} \
    /mnt/* /path/to/backup/dir

Or as a one-liner:

rsync -aHAXUUSh --info=progress2 --partial --exclude={"var/cache/pacman/pkg/*","*/lost+found"} /mnt/* /path/to/backup/dir

See its man page for details on the options used. Note that the repeated -U is not a mistake but a shortcut to enable both --atimes and --open-noatime 😉

However, while doing so I realized rsync wasn’t exactly the best solution when the destination filesystem is {ex,}FAT… Timestamps and permissions/ACLs/xattr won’t carry over correctly¹ 😒 This is what got me for the very first time interested in using…

Tar

O venerable Lord of computerized backups! Thy name is reminiscent of the glorious days of yore, when backups were mere tape archives…

Okay, but how does it fare in 2022 for complete system backup with high fidelity? After delving into its manpage this is what I came up with:

cd /mnt
tar --create \
    --preserve-permissions \
    --acls \
    --xattrs \
    --atime-preserve=system \
    --exclude={"var/cache/pacman/pkg/*","*lost+found"} \
    -I zstdmt \
    -f /path/to/backup/dir/<backup-name>.tar.zst \
    *
tar: /var/run/nscd/socket: socket ignored
...
# repeat dozens of times

Hmmm okay, apart from socket files which apparently clearly don’t matter, what about the completeness of the backup? Sections verify and compare from tar manual cover this question. A few pitfalls to be aware of however:

--verify (only for archive creation) doesn’t work with compressed archives; if you don’t mind you can first create and verify an uncompressed archive, then compress it in a second step…
--compare “ignores files in the file system that do not have corresponding members in the archive”, so if something was left over during the backup, no errors will be reported.

If you have the space and will for it, you could also just extract the archive and diff --recursive it against the source, but all in all I feel this is an aspect where the original (and still current) purpose of tar as a tape specific archiving tool, makes itself felt.

From my superficial testing everything seemed mostly alright, but I can’t shake the feelings that tar isn’t making as complete a clone as fsarchiver or rsync (see its limitations too). But that’s probably because my beard isn’t grey enough I guess… 😁

Wrap up

In conclusion, if you are in a situation where Clonezilla doesn’t work for you (most notably when using LUKS), my recommendation would be to turn to either FSArchiver, partclone or SquashFS. They seem to provide the most complete and reliable feature-set of the lot for imaging systems in a correct and efficient way. Rsync and tar being more general purpose tools, they require more options not to leave something out or otherwise shoot yourself in the foot.

Also, while laboring over this article I realized that while it was all fine and dandy to have a static archive of my old system, the next step was to make an interactive image of it that I could boot on whenever I want to see which GTK theme I used, or otherwise take a nostalgic whiff and relapse in my former computing abode…

This reminded me of the Moving an existing install into a virtual machine article, and I have an entire article coming up with the fruits of this endeavour: creating a VM image from the backups made here, testing in the process the oh-so-important restoration side of all the different tools used here (which taught me that an un-restored backup is no backup at all), making the required adjustments to be able to boot from it… Stay stuned for the second part! [edit: there it goes]

Thanks for reading and happy imaging 🙏

One workaround is to use a loop device file, i.e. create a file sufficiently large to hold your backup, partition it, activate it via losetup, then finally format it with a more amenable filesystem and mount it as the backup destination (example). ↩︎

Parallel ffmpeg stream manipulation

Fri, 29 Oct 2021 00:00:00 +0200

This post will illustrate how we can apply different operations (audio transcoding, subtitles selection, video, chapters and fonts passthrough) on multistream MKVs with ffmpeg, while optimizing for multicore systems.

More precisely, here is what we are trying to achieve:

transcode the Japanese audio track from a entire anime season, from a superfluously lossless Flac to the more efficient Opus format¹
shed the useless (to me) English audio track (also in Flac, and 5.1…), as well as the unneeded “Signs-only” subtitle tracks
pass-through the video track
retain all attachments (fonts, chapters…)
preserve file modification times
replace the original files with the new ones
do all this as fast and efficiently as possible 😎

When starting, I knew how to accomplish this in two steps using two different tools:

Identify and remove the unwanted streams with mkvmerge^(source):

mkvmerge -i *.mkv
File '01.mkv': container: Matroska
Track ID 0: video (HEVC/H.265/MPEG-H)
Track ID 1: audio (Flac)
Track ID 2: audio (Flac)
Track ID 3: subtitles (SubStationAlpha)
Track ID 4: subtitles (SubStationAlpha)
Track ID 5: subtitles (HDMV PGS)
Track ID 6: subtitles (HDMV PGS)
Attachment ID 1: type 'application/vnd.ms-opentype', size 30712 bytes, file name 'font.ttf'
(...)

for f in *.mkv; do mkvmerge -o "remuxed/$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; done

Note

Here I was able to use !eng because the audio track was labeled as such, but we could achieve the same with numerical index (just as we are doing for subs).

Use ffmpeg to transcode the remaining audio track to Opus, using a generous 192 kbps (96k per audio channel) bitrate and then do the cleanup:
```
for f in remuxed/*.mkv; do ffmpeg -i "$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done
```
Where -map 0 ports over all input streams, -c copy keeps them intact, and -c:a libopus -b:a 192k modulates this by transcoding all audio streams (in our case, the one we kept) to 192k Opus. Finally, touch -r updates our file modtime to reflect the original one, and mv overwrite the source with the modified file (warning: make sure to always test on a copy first, otherwise data loss will incur!).

From there it was rather easy to merge it all in one fell for loop:

for f in *.mkv; do \
    mkvmerge -o "tmp_$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; \
    ffmpeg -i "tmp_$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp2_$f"; \
    touch -r "$f" "tmp2_$f"; \
    mv "tmp2_$f" "$f"; \
    rm "tmp_$f"; \
done

However, now the next step was to get rid of the superfluous mkvmerge and do it all with the Ever Almighty Multimedia Swiss-Army Knife®, ffmpeg. For that I had to dig quite a bit, but here is the result:

for f in *.mkv; do ffmpeg -i "$f" -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done

As before we first select all streams via -map 0, then make use of negative mapping (notice the minus sign at the front of the subsequent -map values) to get rid of the unwanted tracks (second audio track, second and fourth sub tracks… Beware, this time we are using ffmpeg’s indexes/IDs which are relative to the category and 0-indexed). Then it is as before, we copy all remaining streams without changing them (-c copy) except audio ones which are transcoded to Opus.

Now this is all well and dandy, but this command runs sequentially, iterating over each file while queuing up the others, disregarding our multicore CPU architecture and thus slowing down the process. How can we run all this concurrently, dispatching a job to each CPU thread?

GNU Parallel² to the rescue!

parallel ffmpeg -i {} -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k tmp_{}';' touch -r {} tmp_{}';' mv tmp_{} {} ::: *.mkv

Note

Of course, overall time will also depend on your storage read/write speeds…

Et voilà, that’s how you efficiently and stylishly reduce a bloated 10.9GB series to a lean and mean 4.8GB one. Great success! 👍👏🤗

See Why you should never use FLAC (for that use case that is) ↩︎
See this Unix SE answer for why Parallel is superior to Xargs in this matter. ↩︎

Editor-induced bugs

Sun, 02 May 2021 00:00:00 +0200

While working on this site, I suddenly noticed that an extra space was added after any link, but it was only visible when that link was followed by another character like a parenthesis.

I dug hard and on a hunch, found that it was caused by my recent addition of a render hook for links. I had simply followed the instructions and added the following snippet in layouts/_default/_markup/render-link.html:

<a href="{{ .Destination | safeURL }}"{{ with .Title}} title="{{ . }}"{{ end }}{{ if strings.HasPrefix .Destination "http" }} target="_blank" rel="noopener"{{ end }}>{{ .Text | safeHTML }}</a>

Eventually, I found this Hugo bug that enlightened me: turns out some text editors automatically add a newline at the end of the file on save to comply with the POSIX standard definition of a line (source).

This other StackOverflow thread lists some Gedit bugs where devs don’t want to add at least a GUI option to enable/disable the automatic addition of newline (the option now exists in gsettings), nor to simply display it… Some more discussion of this issue on Hugo Forums.

The Hugo-specific solution was to add a Go template snippet at the end of the render hooks which removes trailing newlines (credit):

diff --git a/layouts/_default/_markup/render-link.html b/layouts/_default/_markup/render-link.html
index f04b2e3bda072e15cf8ac7787e37e68744a76048..06727fbe334a084ad29a29a85caf8f3a4fd55bda 100644
--- a/layouts/_default/_markup/render-link.html
+++ b/layouts/_default/_markup/render-link.html
@@ -1 +1,2 @@
 <a href="{{ .Destination | safeURL }}"{{ with .Title}} title="{{ . }}"{{ end }}{{ if strings.HasPrefix .Destination "http" }} target="_blank" rel="noopener"{{ end }}>{{ .Text | safeHTML }}</a>
+{{- /* This comment removes trailing newlines. */ -}}

Information technology is such a complex interaction of countless moving parts, that I often wonder how it can work and produce any results at all!

Cryptocurrenshit

Tue, 27 Apr 2021 00:00:00 +0200

From Cryptocurrency is an abject disaster by Drew DeVault:

Cryptocurrency is one of the worst inventions of the 21st century. I am ashamed to share an industry with this exploitative grift. It has failed to be a useful currency, invented a new class of internet abuse, further enriched the rich, wasted staggering amounts of electricity, hastened climate change, ruined hundreds of otherwise promising projects, provided a climate for hundreds of scams to flourish, created shortages and price hikes for consumer hardware, and injected perverse incentives into technology everywhere. Fuck cryptocurrency.

Ditto.

Further reading on this topic on Wikipedia, and for an illustration of the abuse it is causing see this Hacker News thread.

2021-07-15 The creator of Dogecoin shared his view on cryptocurrencies:

After years of studying it, I believe that cryptocurrency is an inherently right-wing, hyper-capitalistic technology built primarily to amplify the wealth of its proponents through a combination of tax avoidance, diminished regulatory oversight and artificially enforced scarcity.

Full Twitter thread / HN submission.

2021-10-07 Bitcoin is a Ponzi scheme.

2021-10-13 Cryptocurrencies expose you to the risk of developing a form of gambling addiction (on a personal note, I had these exact same symptoms the one and only time I got into the cryptocurren-shit show, when trying to convert in fiat Lumens received from a Keybase initiative. Damn did I get a bad couple of hours stressing over all the parameters and trying to maximize profit…).

2022-01-15 … And it’s not like it was still crypto early days!

2022-02-12 A very excellent scholarly lecture: Can We Mitigate Cryptocurrencies’ Externalities?

2022-02-15 Here is a Twitter thread where a prominent cryptocurrenshit holder explains how he was victim of a refined scam and very nearly lost all of his millions.

takeaway n°1: even being super knowledgeable about this space doesn’t shield you from those tricky schemes. So what about the average Joes and Janes?
takeaway n°2: Ethereum-adjacent discourse is straight up alien speak to me. Even though I generally feel rather proficient on the intellectual level, I’m still dumbfounded when I read the kind of mumble-jumble this thread is riped with. And in a very different way than when I’m reading something technical in a field I am not acquainted with, like say particle physics or philology. The closest thing I can compare it with is global finance, which to me says a lot about the habitus that governs the field of cryptocurrency…

Hugo on Vercel

Mon, 26 Apr 2021 00:00:00 +0200

As an attempt to diversify, I have been using Vercel to build and host this website since its inception, as opposed to going with the more common choice of Netlify.

After being pleasantly surprised with the initial setup-and-first-build flow (the automatic HTTPS on custom domains is awesome), I stumbled upon something that baffled me: the default Hugo version Vercel uses to build your site dates from September 2019, and you have to jump through hoops to find out about it.

Getting the version of Hugo used to build your site

This came out due to the many discrepancies I was seeing between my local dev version (using the latest and shiniest v0.82.1) and the deployed site. It took me a couple of weeks before I realized that Vercel was not using the latest stable version as I was naïvely thinking, but was in fact using version… 0.58.2 from 2019-09-13!! 😨

In order to get this basic information, you need to change the build command to something like:

hugo version && hugo -D --gc

I’d hope Vercel would make this the default build command, as this undocumented^[1][2] peculiarity made my first blog-and-deploy experience far more annoying than it ought to be.

I really don’t understand why they’d use such an outdated version, without even so much as a warning. Quite some things happened in Hugo-land since the somewhat “dramatic” changes in v0.60, which may have prompted this version pinning.

Tip

If someone at Vercel reads this, please either use the latest version by default or include the one you are using in your Docs and offer a way to specify a “latest” as the HUGO_VERSION environment variable.

Blogging from a smartphone for tech-savvy people

Tue, 21 Jan 2020 00:00:00 +0100

In this post we will see how to create and publish a blog using a smartphone, all the while flexing some serious tech chops.

Requirements:

a smartphone running Android
the Termux app with git and hugo installed (pkg install git hugo) and properly configured to access the local storage
a git client (Play store link) like the older but quite functional Forker app for easy graphical interactions with git (commit, push…): especially useful on the long run for a smoother publishing experience, but you may also exclusively use the command line for increased hacker-cred
a text editor: I strongly recommend MiXplorer (free on XDA Labs or support version in the Play Store), which provides editors suitable for both code and text and is an amazing file explorer all around; Markor is a good second choice
accounts on compatible git hosting and CDN deployment services (I’ll be using GitLab and Vercel): although the CDN part is optional, it provides some nice benefits and a smoother experience overall than e.g. GitHub/GitLab Pages
a mobile browser (I recommend Firefox for Android) and of course… Internet access 😁

Instructions:

create a new empty project on GitLab and clone it in Termux/Forker (tip: use the HTTPS URL in Termux to avoid having to install and setup openssh)
in Termux, cd to the repo folder (~/storage/shared/Android/data/ch.phcoder.jigit/files/repo/*name* if using Forker) and run hugo create site . -f in it
follow Hugo Quick Start Guide to add a theme and some content to your site
check that everything is working as intended by running hugo server -D and opening http://localhost:1313 in your browser
then commit and push your site to GitLab through Termux/Forker
now follow the official Vercel guide for deploying a Hugo site
finally, enjoy your new blog created on your smartphone and available worldwide! 👍

Going further:

customize your site to suit your needs
set up a custom domain and enjoy the automatic HTTPS (really smoothly done by Vercel) to be a good netizen

Now whenever you want to add some content to your blog, just create a new post in the proper folder either using hugo new posts/post_title in Termux or directly via your text editor, commit and push your changes and your updated site will be automagically deployed. You are officially equipped to blog on the go while roaming the vast world! Great success✌

Data transmission

Wed, 29 Jul 2015 00:00:00 +0200

Everything “digital”, everything related to computers is based on signal and quantization theories.

Today I was considering the following question: “How do electricity and light convey information?”. It seemed incomprehensible to me that we could vehicle structured information via those raw streams of particles. How is it possible to give electrons a specific meaning, like to register this text I am writing and send it over various types of wire (copper and optic fiber)?

I picture the phone line getting out of my house and my home modem-router plugged in it. Everything Internet related goes through this one copper wire. How is it possible that TV shows, distro images and other emails and web pages are transported by mere unformed electrons? This all seems really mysterious to me, and piqued my interest tonight.

So after searching a bit I ended up on Wikipedia’s data transmission article via the marvelous process of serendipity, and all my answers started getting answered.

(To be fair, I got a quick and useful answer from this site which gathers a few key concepts about electricity and drops in its intro the core of our question).

So as a matter of fact and to cut the suspense out, information can be coded into electricity and light not by modifying the inner particles but by shaping variations of their flow and parsing those on the receiving end.

In other words, standardized operations are applied to transported streams that make up the encoding.

And for the full blown explanation: physical layer (OSI model), Wikipedia.

Addendum (2022-03-11): There is a wonderful book about this exact topic that often gets recommended on Hacker News, Code: The Hidden Language of Computer Hardware and Software by Charles Petzold. It starts off from the predecessors of digital signal, Morse code, Braille and other ways of encoding information through normalized signals, and then retraces the evolution from there all the way to modern computing. Truly a didactic masterpiece interspersed with many other helpful experiments/practical bits.

ffmpeg set sub as default

Fri, 01 May 2015 00:00:00 +0200

Copy of my answer to the question “Is there an option in ffmpeg to specify a subtitle track that should be shown by default?” on superuser:

No, there isn’t an option in ffmpeg to specify a subtitle track that should be shown by default.
(ffmpeg version 2.6.1 built with gcc 4.9.2 (GCC) 20150304 (prerelease) as default on Arch, see end of answer for compile flags)

Resources highlighting the lack of command line options for setting default/forced subtitles stream in ffmpeg include the following:

this Oct. 2012 mailing list thread (which delves in the matroska format and explains that ffmpeg doesn’t write in the required fields)

this Jan. 2013 thread (shows the lack of interest of ffmpeg devs for this)

this Jan. 2014 thread (asks about flipping the default flag in Matroska container, to no avail)

this very recent forum post (MP4, zeranoe ffmeg forums)

Finally, this question on StackOverflow asks the same as you but for mkv, and highlights the fact that ffmpeg is the culprit since mkvpropedit (part of mkvtoolnix) can do it.

However, there is a potential workaround
This March 2013 ffmpeg bug report (still open) deals with mkv and default flag for subtitles and explains something that we can use to circumvent this limitation in some scenarios:

The Matroska specification - http://matroska.org/technical/specs/index.html (search for FlagDefault) - specifies that the value for the default flag is “default” if nothing is specified, the demuxer therefore marks all subtitle tracks as “default”.

So ffmpeg may write the default flag on our subtitles stream in some cases; I tried it with different input video files and got mixed results:
with the Big Buck Bunny AVI Mpeg4 and a random .srt file, the remuxed mkv subtitles stream did have the default flag:
  $ ffmpeg -i big_buck_bunny_480p_surround-fix.avi -i subtitle.srt -c copy test.mkv
  $ ffmpeg -i test.mkv
  Input #0, matroska,webm, from 'test.mkv':
    Metadata:
      ENCODER         : Lavf56.25.101
    Duration: 00:28:31.40, start: 0.000000, bitrate: 1028 kb/s
      Stream #0:0: Video: mpeg4 (Simple Profile), yuv420p, 854x480 [SAR 1:1 DAR 427:240], 24 fps, 24 tbr, 1k tbn, 24 tbc (default)
      Stream #0:1: Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s (default)
      Stream #0:2: Subtitle: subrip (default)
however I couldn’t reproduce this with a sample mp4-contained H264 sample neither with a personal mp4, so I guess mp4 --> mkv doesn’t work. But what if…?
  $ ffmpeg -i sample.mp4 -c copy sample.avi
  $ ffmpeg -i sample.avi -i subtitle.srt -c copy sample.mkv
  $ ffmpeg -i sample.mkv
  ...
Input #0, matroska,webm, from 'sample.mkv':
  Metadata:
	ENCODER         : Lavf56.25.101
  Duration: 00:28:31.40, start: 0.000000, bitrate: 2 kb/s
	Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(tv, bt709), 560x320, SAR 1:1 DAR 7:4, 60 fps, 60 tbr, 1k tbn, 60 tbc (default)
	Stream #0:1: Audio: aac (LC), 48000 Hz, mono, fltp (default)
	Stream #0:2: Subtitle: subrip (default)
VICTORY! Doing mp4 --> avi and then avi + srt --> mkv gets the subtitles stream on by default and VLC displays them correctly.

Sure that’s very roundabout and I’d rather not think of the information loss in the process, but at least it works.

Summary

ffmpeg is definitely not adapted right now for setting streams as default or forced,

mkv is probably better than mp4 as an origin container, since it has a known tool doing the job fine (mkvpropedit from mkvtoolnix).

ffmpeg version details:
ffmpeg version 2.6.1 built with gcc 4.9.2 (GCC) 20150304 (prerelease) configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-runtime-cpudetect --enable-shared --enable-swresample --enable-vdpau --enable-version3 --enable-x11grab

Tech on Bastien Traverse

Optimized cloud-init templates on Proxmox

Generic options

Performance related options

Cloud-init related options

Post-creation steps

UEFI variant

The QEMU Guest Agent conundrum

Tips & tricks

Using a container to sidestep a forgotten password in CasaOS

Problem statement

The constraints

Access to a container shell and logs in CasaOS interface

Hacking around

The solution

Root shell provided by the web-terminal container

The solution: simplified

Root shell inside nextcloud container

Trying to refine the solution

Takeways

Graphical management interfaces for Linux servers

Terminology considerations

Coming up with a unified categorization

Category members

Server Management Tools:

Web hosting/Control Panels

Application/Container Orchestration Tools:

Moving an existing install into a virtual machine

The ultimate OS conservation project: virtualizing to immortality

Step 1: Preparing the VM image

Creating the file

qcow image:

Raw image:

Partitioning

MBR

GPT

Formatting

Mounting

Step 2: Transferring the data

partclone

FSArchiver

SquashFS

Rsync & tar

Step 3: Adjusting the system config

Offline Arch install with local pacman cache

Ejecting and reattaching a USB drive from the CLI

Reattaching an ejected USB device from the CLI

Wrap up

A system imaging tale: LUKS, Clonezilla and friends

An unexpected obstacle to boot: Ventoy shenanigans

Optional step: setting up a comfortable environment

Clonezilla and LUKS/LVM

System imaging, the manual way

Compressing partclone images with zstd

Extra: comparing the process with different tools

FSArchiver

Squashfs

Rsync

Tar

Wrap up

Parallel ffmpeg stream manipulation

Editor-induced bugs

Cryptocurrenshit

Hugo on Vercel

Getting the version of Hugo used to build your site

Blogging from a smartphone for tech-savvy people

Requirements:

Instructions:

Going further:

Data transmission

ffmpeg set sub as default

No, there isn’t an option in ffmpeg to specify a subtitle track that should be shown by default.

However, there is a potential workaround

Summary

ffmpeg version details: