Learning-Diary on Bastien Traverse

Using a container to sidestep a forgotten password in CasaOS

Sat, 03 Feb 2024 00:00:00 +0100

Problem statement

As part of dabbling with self-hosting again, I installed CasaOS on an Oracle Cloud free Ampere instance to try it out.

After setting it aside for a few weeks, when I logged in via SSH and tried to use sudo I realized I absolutely couldn’t remember my user’s password 😅

The standard operating procedure in this case is to either reboot the machine on a live system and use that to chroot into the local install, or fiddle with GRUB rescue/kernel command line.¹

But since I could still install containers through CasaOS web interface, I thought I’d use this opportunity to explore ways to recover my sudo access/change my password without rebooting (which could be nice in case it is essential to avoid downtime).

Edit: As I was finishing this post, I realized that the third constraint listed below was incorrect, and as a result a less cumbersome resolution was possible. So read on if you are interested in the learning journey, or jump straight to the simplified solution.

The constraints

the CLI/SSH access is “useless”: my user is not setup to interact with the Docker socket without elevated privileges (which is usually a good thing)
besides its App Store content, CasaOS allows to manually install containers using a Compose file or a docker run command (which is then composerized): see official video or screenshots of the process
~~its interface however doesn’t allow to docker exec or interact with containers in any other way; the installed containers must expose a web interface for us to be able to interact with them.~~ As it turns out, CasaOS interface does offer access to containers’ console (equivalent to a docker exec -it <containe> /bin/sh), at least for apps installed from its official Store, via the app Settings > “Terminal and Logs” icon.

Access to a container shell and logs in CasaOS interface

Hacking around

Because I’m pretty new to this, my first idea was to start a container which would give me a root shell with access to the docker socket, and then use that environment to start a second privileged container (Docker-in-Docker style) from which I would mount the host filesystem and use {ch,}passwd or such… Yes, I felt very smart thinking this up 😅

Unaware of my mistaken approach, I looked online for a “web terminal container” and the first result on StartPage was the web terminal GitHub repo, which did what I needed (expose a root shell in the browser) but hadn’t been updated since August 2021 😕

Aside

Docker Hub search is appalling: searching for web terminal gave me a bunch of completely unrelated results. I had to put a dash between the two words to get anywhere, but then the two first results hadn’t been updated in 2 years either…

Searching for web shell and "web shell" didn’t give better results, while web-shell and webshell led to very outdated stuff without any description. I can’t believe there isn’t a reference implementation of this stuff!!

I also looked up ttyd, the software used by web-terminal, and here the first result was fresh from a few days ago; however there was no mention of “How to use this with Docker” , so I wasn’t sure a simple docker run would achieve the desired result… Had I checked its Dockerfile, I would have seen that its ENTRYPOINT/CMD was, in fact, starting up the service 🙃

Lo and behold, I took the “risk” (after checking the image’s Dockerfile) and entered the following command in CasaOS Docker CLI interface:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 7681:7681 raonigabriel/web-terminal:latest

After setting up the access port in CasaOS interface and confirming everything was okay, I clicked “Install” and connected to my server’s port 7681 in a browser… Success, I was in!

For the next stage of my plan I ran the following command inspired by this StackExchange thread:

docker run -ti --privileged --net=host --pid=host --ipc=host --volume /:/host busybox chroot /host
/bin/sh: docker: not found

Crap, I didn’t even check that the container had docker installed 🤦 You can tell I really don’t know what I am doing 😂

Of course when I subsequently tried to install docker…

root@1d0fa4b64c55:/$ apk add docker
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
  docker (missing):
    required by: world[docker]
root@1d0fa4b64c55:/$ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 36 distinct packages available

A quick search seems to indicate that the image is simply too old. Building an updated image was out of the scope I had set for this experiment, so I paused and took time to think.

That’s when it occurred to me that this “nested container” approach was completely useless, and would most likely not work since at that point you have the first container’s virtualized filesystem namespace acting as a buffer between the host and the DinD container… Complete misdirection, backing up!

The solution

After thinking it through some more, I realized one could achieve the desired outcome by simply mounting the host / read-write as an attached volume 😁

docker run -d -v /:/host -p 7681:7681 raonigabriel/web-terminal:latest

Et voilà, I finally had a root shell from which I could chroot into the host and update the user password and/or configure password-less sudo (which amounts to the same). Or really, do (nearly) everything to the host system 😨

Root shell provided by the web-terminal container

Initially I thought it would be necessary to use a privileged container, but trying it showed that wasn’t the case. I guess it’s because I was only editing files/using regular utilites, and not trying to create new devices/nodes etc.

The solution: simplified

This is what happens when you don’t know your tools enough… You miss very obvious pathways that lead to simpler solutions 😁

Indeed, there is no need to install a third-party container. It is enough to add the /:/host volume to an existing app installed from CasaOS official App Store (e.g. NextCloud), and after making sure in its settings it runs with the root UID/GID, we can use the interface to connect the container’s console!

Root shell inside nextcloud container

And that is the Easy Way© to get a root shell on your server using containers!

Trying to refine the solution

Now that I was there, I felt like I should have been able to use a simple busybox image to execute a command non-interactively directly from docker run/Compose file, eliminating the need to find an image that exposes a web service or to connect to the container’s console.

After a bit of fiddling, I came up with the following invocation:

docker run -v /etc/sudoers:/host/etc/sudoers busybox /bin/sh -c echo '%sudo  ALL=(ALL) NOPASSWD: ALL' >> /host/etc/sudoers

But CasaOS interface constantly threw an error. Probably the redirection in the command, but no amount of quoting led me to a successful run.

At that point I threw the towel in: I had recovered access to my sudo access and learnt quite a few things along the way. Time to wrap up.

Takeways

Docker default security story is scary: because the daemon runs as root by default, the mere ability to run containers (even without --privileged) and mount any desired host path as a volume gives root-equivalent status
Due to this, access to CasaOS interface is equivalent to root access on the server

That’s all folks, thanks for reading this account I hope you enjoyed!

See e.g. https://gcore.com/learning/how-to-reset-password-in-linux/ ↩︎

Offline Arch install with local pacman cache

Sat, 12 Mar 2022 00:00:00 +0100

While developing my Arch VM script, I had to go through the installation over and over to test every change.

I am tethered to my phone for Internet connectivity, and the quality and speeds are… How should I put it? Underwhelming to say the least (I live in a mountainous rural area).

So while doing so, I grew increasingly wary of the long wait for packages to download and the 287 MB worth (gah!) of data it would chip off my 40 GB plan each and every time. What I needed was a way to provide a local package cache to pacman.

Amazingly, but certainly not surprisingly, there is a great article on the Arch wiki for that.

However, due to my target file system being exFAT (cough ventoy shenanigans, cough) and some file names containing the forbidden : character, this solution didn’t work for me.

Also because I am using GNOME Boxes, sharing a folder with the host was out of the equation as that entails installing some SPICE-related packages which pull in an entire graphical stack as dependencies, hence defeating the purpose.

Setting up SSH access to the guest requires quite some fiddling, and anyway it wasn’t immediately clear to me how I would instruct pacstrap to tap into the network cache, or it would have likely required some modifications to the live pacman.conf and I wanted to avoid all that complexity¹.

Remastering the ISO was another option that I left out because I wanted the script to work with a vanilla install. No, what I was really looking for was something of the utmost simplicity².

What I finally came up with, and not to toot my own horn but I think it is pretty darn brillant (considering I haven’t seen it mentioned anywhere online when I researched the topic³), is a bind mounted squashfs⁴ package cache placed on an external medium like a USB flash drive. This in turn can easily be shared in GNOME Boxes even without the SPICE packages installed in the guest!

Here is a rundown of the process:

create a SquashFS archive of /var/cache/pacman/pkg on a system containing the desired package set (I reused a just-installed VM to match its exact minimal set but you can go wild and copy everything from your main rig, that’ll give you a larger cache):
```
mksquashfs /var/cache/pacman/pkg/ pkg.sqsh \
    -comp zstd \
    -no-exports \
    -progress \
    -mem 1G \
    -not-reproducible
```

copy the SquashFS file on the external medium you will mount during the install
boot into the Arch Live environment and create two directories, one for your external medium and one for the archive:
```
mkdir /tmp/{usb,squash}
```

Note

Do not use /mnt if you plan to use it as the installation mount point.

mount the external medium containing the squash file and then the squash file itself:
```
mount /dev/XXXY /tmp/usb
mount /tmp/usb/pkg.sqsh /tmp/squash
```
finally, and that’s what’s doing the trick for pacstrap, bind-mount /tmp/squash on archiso’s /var/cache/pacman/pkg:
```
mount --bind /tmp/squash /var/cache/pacman/pkg
```

That’s it, now you just have to add -c to your pacstrap invocation:

pacstrap -c /mnt base linux [...]

Downloads will be skipped (as long as package versions in the squashfs archive are not older than the synced databases’ ones), and now you can carry on with your installation, contented by the blissful release of not having to suffer through the hell of this perpetual cycle of reincar… Err I mean wasteful downloading of already acquired packages 😊️

Happy hacking!

Just as I was bringing the finishing touches to this article, I realized my blunder: what I needed was an access from the guest to the host, not the contrary. This would allow me to do something like an SSHFS mount or even serving the cache via a simple web server. Follow-up article to come on how easy it actually is to connect from a guest to the host in GNOME Boxes 😁️ ↩︎
Yeah, no seriously, in my use case that was the web server/SSHFS mount. But let’s just say this article was actually meant to be for a fully offline install… 😂️ ↩︎
I’ll admit this may only be due to my constraints being ridiculously specific, but maybe for once I can dream of having come up with a truly original solution, wouldn’t you agree ? 😀️ ↩︎
The idea of using SquashFS instead of a more common loop-mounted filesystem-in-a-file is what I felt was the real feat of ingeniosity here. And yes, I need to pat myself on the back from time to time, is it something wrong? 😉️ ↩︎

Ejecting and reattaching a USB drive from the CLI

Thu, 24 Feb 2022 00:00:00 +0100

Setup: I was booted on archlinux-2022.02.01-x86_64.iso using a multiboot USB flash drive made with Ventoy 1.0.65, and had selected the “Copy to RAM” option in the ISO menu with the idea of reusing the drive as a backup destination for a system backup.

The issue: I was unable to normally eject the drive, both eject and udisksctl power-off (after installing udisks2) gave the error Device or resource busy, and lsof | grep -e ventoy -e sdb didn’t return anything. Here is what it looked like:

root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
│ └─ventoy    254:0    0 812.3M  1 dm    <-- problematic entry
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

root@archiso ~ # eject -v /dev/sdb
eject: device name is `/dev/sdb'
eject: /dev/sdb: not mounted
eject: /dev/sdb: is whole-disk device
eject: cannot open /dev/sdb: Device or resource busy

root@archiso ~ # eject -v /dev/mapper/ventoy
eject: device name is `/dev/mapper/ventoy'
eject: /dev/mapper/ventoy: not mounted
eject: /dev/mapper/ventoy: is whole-disk device
eject: /dev/mapper/ventoy: is not hot-pluggable device

root@archiso ~ # ls -l /dev/mapper/ventoy  
lrwxrwxrwx 1 root root 7 Feb 24 20:24 /dev/mapper/ventoy -> ../dm-0

root@archiso ~ # ls -l /dev/dm-0
brw-rw---- 1 root disk 254, 0 Feb 24 20:24 /dev/dm-0

root@archiso ~ # cat /sys/class/block/dm-0/dm/name
ventoy

So, I wasn’t sure what was causing the issue, and I didn’t want to simply pull the drive out as it equates to playing Russian roulette with your devices (do you like getting the rug pulled under you or being knocked inconscious by a stealthy sucker punch? Well, neither do most filesystems).

At first I had worked around the issue by using this trick :

echo offline > /sys/block/sdb/device/state
echo 1 > /sys/block/sdb/device/delete

See this post on the Arch forums for a more thorough procedure along the same vein.

Still, I wasn’t entirely convinced: it didn’t address the root cause of the issue.

Digging a little deeper led me to dmsetup:

root@archiso ~ # dmsetup table
boot-arch: 0 516096 linear 8:1 2048
ventoy: 0 1663640 linear 8:17 59447552

root@archiso ~ # dmsetup info /dev/mapper/ventoy  
Name:              ventoy
State:             ACTIVE (READ-ONLY)
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      254, 0
Number of targets: 1

Well, that READ-ONLY state was already good news, but checking dmsetup manpage I saw it offered a remove command:

root@archiso ~ # dmsetup remove ventoy
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

Victory, the mapping is gone! And now…

root@archiso ~ # eject /dev/sdb -v
eject: device name is `/dev/sdb'
eject: /dev/sdb: not mounted
eject: /dev/sdb: is whole-disk device
eject: /dev/sdb: trying to eject using CD-ROM eject command
eject: CD-ROM eject command failed
eject: /dev/sdb: trying to eject using SCSI commands
eject: SCSI eject succeeded

root@archiso ~ # lsblk  
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
sr0            11:0    1  1024M  0 rom

Success! I saw the LED on my thumb drive flash quickly, and then it fell silent. But wait, what is all this red in my logs?

archiso kernel: sd 6:0:0:0: [sdb] Media removed, stopped polling
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: ldm_validate_partition_table(): Disk read failed.
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel:  sdb: unable to read partition table
archiso kernel: sd 6:0:0:0: [sdb] Media removed, stopped polling
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: ldm_validate_partition_table(): Disk read failed.
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel:  sdb: unable to read partition table
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed

Hum, looks like eject is doing something kind of dirty… Let’s move ahead for now, we’ll come back to it later.

Now, since this is a laptop laying on my desk, I certainly can manually unplug the drive before plugging it back again to make it available. But what if I am lazy, or the machine is faraway, or I just want to take extra care of that super frail USB plug and prevent a superfluous actuation cycle?

Reattaching an ejected USB device from the CLI

Here we are going to make use of the ability to bind and unbind drivers from devices manually from user space. First, get the drive Bus number:

root@archiso ~ # lsusb  
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive  <-- here is my device
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Then Port for “Bus 4 Device 3”:

root@archiso ~ # lsusb -t  
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M  <-- spot it?
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

Now that I know that my USB device is identified as 4-1 (bus-port) (this can be confirmed by checking dmesg | grep usb-storage), we can make it go through an unbinding/binding cycle:

root@archiso ~ # echo "4-1" > /sys/bus/usb/drivers/usb/unbind
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sr0            11:0    1  1024M  0 rom  
root@archiso ~ # echo "4-1" > /sys/bus/usb/drivers/usb/bind  
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

Et voilà, the flash drive is once again available for mount without having to physically remove it!

Note

Trying to bind directly after eject without unbind leads to a write error: device or resource busy . So now I am really left wondering what eject is doing to my drive…

An alternative to manually writing to /sys/bus/usb/drivers/usb/ is to use usb_modeswitch (included on the Arch installation media). In that case we only need the ID pair from lsusb:

root@archiso ~ # lsusb  
...
Bus 004 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive
                       ==== ==== ------ ---------
                                      |         |
root@archiso ~ # usb_modeswitch -v 0x0781 -p 0x5580 --reset-usb
Look for default devices ...
 Found devices in default mode (1)
Access device 003 on bus 004
Get the current device configuration ...
Current configuration number is 1
Use interface number 0
 with class 8
Warning: no switching method given. See documentation
Reset USB device .
 Device was reset
-> Run lsusb to note any changes. Bye!

Props to this AskUbuntu answer, confirmed and extended via this Stackoverflow one.

Wrap up

After a few trial and errors, I realized ejecting the drive was entirely unnecessary: I could get away with just dmsetup remove ventoy and an unbind/bind cycle.

Now if you ever have to cleanly power-off a device on the command line, here is the ultimate method:

% udisksctl power-off -h
Utilisation :
  udisksctl power-off [OPTION…]

Safely power off a drive.

Options :
  -p, --object-path         Object path for ATA device
  -b, --block-device        Device file for ATA device
  --no-user-interaction     Do not authenticate the user if needed

E.g. udisksctl power-off -b /dev/sdb.

For the life of me I don’t know why this utility isn’t included by default on the arch installation media, but in any case you are just one pacman -S udisks2 away from safely-removed device nibbana.

That’s all folks for today’s learning diary. Thanks for reading and take care of your devices!

A system imaging tale: LUKS, Clonezilla and friends

Mon, 14 Feb 2022 00:00:00 +0100

Today’s project: make a system image of my previous laptop for archival purposes before refurbishing it.

Conditions:

make a space efficient image, i.e. not a block-level copy: no need to retain encryption or logical volume setup, only their content matters. However:
all standard and extended file attributes must be correctly preserved, so that if I ever want to boot this image the system will be in a correct state.

The plan: boot up an Arch ISO, open the encrypted drives, set everything up for clonezilla – profit.

Here is how it actually went.

An unexpected obstacle to boot: Ventoy shenanigans

In order to boot into the Arch live environment, I unsheathed my very dear and trusted Sandisk Extreme multiboot USB thumb drive (not the currently sold, lower performing one but the ever chart-topping one from about 10 years ago), which had been irreproachably and impeccably fulfilling its duty in the two years since I had discovered Ventoy, the multiboot solution which surpassed them all and ended my long standing quest to find an easy to use, low-maintenance and low-friction multiboot USB solution. And yes, this was the longest sentence of the entire article 😁

Just recently I had updated it to the latest version using the provided Linux script, which completed without issue. But as soon as I booted on it, I was faced with an error: Warning! This is not a standard device and is not officially supported. Um, beg your pardon? It was working fine those past two years, so certainly it was the update which broke something…

Searching for this error, I eventually found this “kill switch” the dev introduced slightly after I used the software, to try and contain the flood of users with non-standard installs complaining to them when something got wrong.

I can’t really fault them for that, but here is the thing: mine was a pure product of the official tool from back in 2020, with no deviation from the prescribed layout/FS whatsoever. The first partition was starting at the 2048/1 MB boundary and formatted with the default exFAT, the second partition was 65536 sector large, the drive had an MBR partition table and all and all…

# fdisk -l /dev/sda
Disk /dev/sda: 59.63 GiB, 64023257088 bytes, 125045424 sectors
Disk model: Extreme  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd1a119ab

Device     Boot     Start       End   Sectors  Size Id Type
/dev/sda1  *         2048 124979887 124977840 59.6G  7 HPFS/NTFS/exFAT
/dev/sda2       124979888 125045423     65536   32M ef EFI (FAT-12/16/32)>

The official command line tool didn’t complain either while upgrading or when run manually:

# ventoy -l /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Ventoy Version in Disk: 1.0.65
Disk Partition Style  : MBR
Secure Boot Support   : NO

Indeed but nope, I got axed by the kill switch all the same. Fuck artificial restrictions and treacherous computing. And I am burned by a free software project at that… What to expect from this world.

Anyhoo, after transfering out my 24 GB of GNU/Linux ISOs (God bless USB 3.0 & quality flash memory speeds), I forced a reinstall with the -I switch:

# ventoy -I /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Disk : /dev/sda
Size : 59 GB
Style: MBR


Attention:
You will install Ventoy to /dev/sda.
All the data on the disk /dev/sda will be lost!!!

Continue? (y/n) y

All the data on the disk /dev/sda will be lost!!!
Double-check. Continue? (y/n) y

Create partitions on /dev/sda by parted in MBR style ...
Done
Wait for partitions ...
partition exist OK
create efi fat fs /dev/sda2 ...
mkfs.fat 4.2 (2021-01-31)
success
Wait for partitions ...
/dev/sda1 exist OK
/dev/sda2 exist OK
partition exist OK
Format partition 1 /dev/sda1 ...
mkexfatfs 1.3.0
Creating... done.
Flushing... done.
File system created successfully.
mkexfatfs success
writing data to disk ...
sync data ...
esp partition processing ...
Open ventoy efi file 0x610ac0
ventoy x64 efi file size 1757184 ...
Open bootx64 efi file 0x610ac0
Open ventoy ia32 efi file 0x610f10
ventoy efi file size 1183744 ...
Open bootia32 efi file 0x610ac0

Install Ventoy to /dev/sda successfully finished.

sudo ventoy -I /dev/sda  5,11s user 2,58s system 19% cpu 39,813 total

After transferring my files back and rebooting on the flash drive, everything was back in order and I was ready to rock 🤟

Optional step: setting up a comfortable environment

Being console-based, the archiso environment is somewhat austere. So the first thing I usually do is to set up SSH access so as to connect from my main machine:

# set french keymap
# the "-latin9" part is only needed if using e.g. accents
loadkeys fr-latin9
# directly connect to a known Wifi SSID
iwctl station wlan0 connect <SSID>
# Enable NTP to have accurate time
timedatectl set-timezone <Area/City>
timedatectl set-ntp true
# set root passhrase to enable SSH connection
passwd
# start SSH service
systemctl start sshd
# get IP address
iwctl station wlan0 show | grep IP | awk '{print $3}'
# or without iwd:
ip a | grep wlan | awk 'FNR == 2 {print substr($2, 1, length($2)-3)}'

Finally, connect to the live environment via SSH:

ssh -o PreferredAuthentications=password \
    -o PubkeyAuthentication=no \
    -o UserKnownHostsFile=/dev/null \
    root@<other-machine-IP>

👉️ The options used allow not to be affected by the local SSH configuration (SSH keys…), as well as well not polluting the local KnownHosts file with this one-time connection.

That’s it, now we can resume with the common procedure!

Clonezilla and LUKS/LVM

In order to efficiently image the system, we first need to open any LUKS container and make sure all the LVs are displayed correctly:

cryptsetup open --allow-discards /dev/sdb2 os
lvs

Then we create the /home/partimag directory (hardcoded backup destination in Clonezilla) and mount our drive on it:

mkdir /home/partimag
mount -o noatime /dev/sdc1 !$

Note

When booting the archiso, I selected the “Copy to RAM” option so as to free up the drive and reuse it as backup destination.

However, in order to do that I first needed to get rid of a lingering /dev/mapper/ventoy mapping on its first partition, which made eject/udisksctl throw errors despite the fact that it wasn’t mounted and lsof | grep -e ventoy -e sdb didn’t return anything. I didn’t want to pull it out on the off chance it’d corrupt something…

I wrote a separate post which goes into the details, but the gist is a simple dmsetup remove ventoy took care of the mapping, and then I was able to normally mount the drive.

And now it is on! Launch clonezilla from the command line and follow the on-screen instructions depending on your needs. Some personal recommendations:

use expert mode
avoid the 4G backup splitting by entering a large number when prompted (like 100000)
select “parallel zstd” (-z9p) as the compression algorithm

Or if you feel like keeping it CLI-only…

/usr/bin/ocs-sr \
    --use-partclone \
    --confirm \
    --clone-hidden-data \
    --zstdmt-compress
    --image-size 1000000 \
    --fsck-src-part \
    --skip-enc-ocs-img \
    --postaction choose \
    saveparts <backup-name> sdb1 sdb2

This is the long form equivalent of the

/usr/bin/ocs-sr -q2 -c -j2 -z9p -i 1000000 -fsck -senc -p choose saveparts <backup-name> sdb1 sdb

that got copied in a file under /tmp after validating my options in the TUI.

Then we just sit back and relax… Wait, what? dd mode? Why aren’t my OS logical volumes backed up by the regular free space-aware partclone mode? I explicitly decrypted the containers for that purpose, and saw the LVs being listed during the preparation screens! Although it’s true I could only select whole partitions to be backed up and not specific logical volumes, and saw a line flash by saying something like Shutting down the Volume manager… Oh come on.

One Ctrl+c later the backup of /dev/sdb2 was cancelled, but the unencrypted /dev/sdb1 with the arch LV (the only one in the boot VG) proceeded without issue. What the heck clonezilla?

Hum alright, according to this 2014 thread with the developer’s input, it seems this isn’t as straightforward as it looks. And it happened even much more recently to another user (although according to the dev’s answer it shouldn’t be the case), so anyway… Time for some manual partclone fun I guess!

System imaging, the manual way

Full disclosure: I had already stumbled upon the “manual partclone required” caveat when trying to efficiently backup LUKS-encrypted systems with Clonezilla, so I wasn’t really at a loss. All in all it’s pretty simple, you just have to use partclone like so:

partclone.$fstype --clone --source ... --output ...

Wich gave in my case:

partclone.ext4 --clone --source /dev/mapper/arch-root --output <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl.img

Tip

If you want to have the same nice ncurses-based UI as Clonezilla does, add the -N/--ncurses option (example).

Though if we really want to mimick Clonezilla and have space-efficient backups, we gotta add compression to the mix!

Compressing partclone images with zstd

Being a devout Zstandard Zealot© (this post summarizes well my feelings – no affiliation with the author), here is how to interface it with partclone for a nicely multi-threaded compression:

partclone.ext4 --clone --source /dev/mapper/arch-root | zstdmt -o <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl-img.zst

Info

From zstd man page: zstdmt is equivalent to zstd -T0, enabling auto-adjusted multi-threading.

If you are like me and have several LVs to clone and their names follow a pattern, you can use a for loop to optimize things a bit:

for i in arch-{root,home,opt,var} boot-arch; do partclone.ext4 --clone --source /dev/mapper/"${i}" | zstdmt -o <backup path>/$(date -I)-"${i}".ext4-ptcl-img.zst; done

Et voilà, space-efficient compressed images of your decrypted LVs!

Tip

Someone made a gist using somewhat older commands and touching upon restoration etc… Give it a look if you want to review the process!

Extra: comparing the process with different tools

A very precious thing in this world is diversity. Of people, ideas, food… And backup tools. Here is a rundown of the same task undertaken with four of them:

FSArchiver

The first time I realized I couldn’t use Clonezilla to properly back up my decrypted LVs, I stumbled upon a great alternative to partclone: FSArchiver.

The main reason for using it over the former? FSArchiver:

directly integrates multi-threaded compression with zstd (and recommends it over any others)
allows to specify multiple filesystems to be archived at once
also works with directories (creating a compressed and checksummed tarball of sorts)
allows to restore to a different filesystem and creating it in one go
allows to restore to a smaller partition by default, whereas partclone needs the -C option (and often just ends on a seek error)
accepts exclusion patterns to filter what’s archived/restored (the syntax is a bit tricky though)
offers to optionally encrypt data in the archive
and is generally really careful with data integrity… If that wasn’t already reason enough 😉

Its multi-threaded compression-ready invocation goes something like this:

fsarchiver --zstd=3 --jobs=$(nproc) savefs <backup path>/<archive-name>.fsa <list of devices to backup> --exclude=<exclude pattern>

--zstd=3 stands for zstd-compression at level 3
--jobs=$(nproc) works exactly like the eponymous make switch, enabling as many threads as there are logical cores on your system.

So for example:

fsarchiver -Z3 -j$(nproc) -v savefs <backup-path>/old-laptop.fsa /dev/mapper/arch-{root,home,opt,var} /dev/mapper/boot-arch --exclude="lost+found" --exclude="/var/cache/pacman/pkg/*"

The -v/--verbose option is useful to get a progress report, however it will list every file being archived and as such likely blow away your terminal history.

BTW, here is why clonezilla defaults to partclone instead of fsarchiver in the words of its author. Tldr: partclone supports pipes/stdin/stdout which are needed for other Clonezilla features, while fsarchiver doesn’t. Tough life.

Note however that FSArchiver (just like partclone) doesn’t deal with recreating a partition table on the destination, or any other form of partitioning/preparing the drive such as LUKS/LVM: you gotta do all that manually up to the point where you are ready to restore filsystems (that is to say, fsarchiver isn’t a disk cloning solution).

Also, you cannot merge the content of separate partitions created this way into one destination (e.g. if you want to consolidate separate LVs into one FS): they will overwrite each other. For that you’d rather use the savedir/restdir mode, which is akin to a tarball on steroids.

Squashfs

I discovered this method by stumbling upon this gem on the Arch wiki, “Full system backup with SquashFS”, and although the article is in dire need of a proper rewrite (which I’ve added to my todo list) I really wanted to give it a try.

In the words of TLDP, SquashFS is

“a read-only file system that lets you compress whole file systems or single directories, write them to other devices/partitions or to ordinary files, and then mount them directly (if a device) or using a loopback device (if it is a file). […] For archiving purposes, SquashFS gives you a lot more flexibility and performance speed than a tarball archive.”

Its selling points to me are:

random access & transparent decompression: no need to decompress and extract the entire image to browse its content (this is huge, especially when talking about gigabyte-sized images…)
file-level deduplication
exclusion patterns
zstd support 😊

However be aware that it doesn’t support ACLs, so if you rely on them you should look somewhere else.

After quickly perusing its manpage I gave it a go:

mount -o ro /dev/mapper/arch-root /mnt
mount -o ro /dev/mapper/boot-arch /mnt/boot
for i in home opt var; do mount -o ro /dev/mapper/arch-"${i}" /mnt/$i
cd /mnt

mksquashfs ./ <backup-path>/old-laptop.sqsh \
    -comp zstd \
    -no-exports \
    -progress \
    -mem 6G \
    -not-reproducible \
    -e var/cache/pacman/pkg

Results:

Squashfs 4.0 filesystem, zstd compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 3912057.66 Kbytes (3820.37 Mbytes)
	35.64% of uncompressed filesystem size (10975717.90 Kbytes)
Inode table size 2886293 bytes (2818.65 Kbytes)
	24.42% of uncompressed inode table size (11819496 bytes)
Directory table size 3606490 bytes (3521.96 Kbytes)
	36.39% of uncompressed directory table size (9910813 bytes)
Xattr table size 621 bytes (0.61 Kbytes)
	28.98% of uncompressed xattr table size (2143 bytes)

To put things in perspective:

root@archiso ~ # du -sh /mnt
16G	/mnt
root@archiso ~ # du -sh /mnt/var/cache/pacman/pkg
4.5G	/mnt/var/cache/pacman/pkg
root@archiso ~ # du -h <backup-path>/old-laptop.sqsh
3.8G	<backup-path>/old-laptop.sqsh

So it went from 16-4.5=11.5 GiB to 3.8… Nice!

It produced a slightly smaller archive than both partclone and fsarchiver (even when accounting for the non-excluded pacman cache), but it took 11.5 minutes. After testing, fsarchiver savedir took 3min08s and produced a 4.2 GiB image (with --exclude='*.pkg.tar.zst'), but then fsarchiver doesn’t produce browsable archives and doesn’t do any deduplication 😊

Rsync

One step more remote from system imaging, the ever powerful Swiss-army knife of file transfer can evidently also be used to create full system backups. Here is my personal take on it:

rsync \
    --archive \
    --hard-links \
    --acls \
    --xattrs \
    --atimes \
    --open-noatime \
    --sparse \
    --human-readable \
    --info=progress2 \
    --partial \
    --exclude={"var/cache/pacman/pkg/*","*/lost+found"} \
    /mnt/* /path/to/backup/dir

Or as a one-liner:

rsync -aHAXUUSh --info=progress2 --partial --exclude={"var/cache/pacman/pkg/*","*/lost+found"} /mnt/* /path/to/backup/dir

See its man page for details on the options used. Note that the repeated -U is not a mistake but a shortcut to enable both --atimes and --open-noatime 😉

However, while doing so I realized rsync wasn’t exactly the best solution when the destination filesystem is {ex,}FAT… Timestamps and permissions/ACLs/xattr won’t carry over correctly¹ 😒 This is what got me for the very first time interested in using…

Tar

O venerable Lord of computerized backups! Thy name is reminiscent of the glorious days of yore, when backups were mere tape archives…

Okay, but how does it fare in 2022 for complete system backup with high fidelity? After delving into its manpage this is what I came up with:

cd /mnt
tar --create \
    --preserve-permissions \
    --acls \
    --xattrs \
    --atime-preserve=system \
    --exclude={"var/cache/pacman/pkg/*","*lost+found"} \
    -I zstdmt \
    -f /path/to/backup/dir/<backup-name>.tar.zst \
    *
tar: /var/run/nscd/socket: socket ignored
...
# repeat dozens of times

Hmmm okay, apart from socket files which apparently clearly don’t matter, what about the completeness of the backup? Sections verify and compare from tar manual cover this question. A few pitfalls to be aware of however:

--verify (only for archive creation) doesn’t work with compressed archives; if you don’t mind you can first create and verify an uncompressed archive, then compress it in a second step…
--compare “ignores files in the file system that do not have corresponding members in the archive”, so if something was left over during the backup, no errors will be reported.

If you have the space and will for it, you could also just extract the archive and diff --recursive it against the source, but all in all I feel this is an aspect where the original (and still current) purpose of tar as a tape specific archiving tool, makes itself felt.

From my superficial testing everything seemed mostly alright, but I can’t shake the feelings that tar isn’t making as complete a clone as fsarchiver or rsync (see its limitations too). But that’s probably because my beard isn’t grey enough I guess… 😁

Wrap up

In conclusion, if you are in a situation where Clonezilla doesn’t work for you (most notably when using LUKS), my recommendation would be to turn to either FSArchiver, partclone or SquashFS. They seem to provide the most complete and reliable feature-set of the lot for imaging systems in a correct and efficient way. Rsync and tar being more general purpose tools, they require more options not to leave something out or otherwise shoot yourself in the foot.

Also, while laboring over this article I realized that while it was all fine and dandy to have a static archive of my old system, the next step was to make an interactive image of it that I could boot on whenever I want to see which GTK theme I used, or otherwise take a nostalgic whiff and relapse in my former computing abode…

This reminded me of the Moving an existing install into a virtual machine article, and I have an entire article coming up with the fruits of this endeavour: creating a VM image from the backups made here, testing in the process the oh-so-important restoration side of all the different tools used here (which taught me that an un-restored backup is no backup at all), making the required adjustments to be able to boot from it… Stay stuned for the second part! [edit: there it goes]

Thanks for reading and happy imaging 🙏

One workaround is to use a loop device file, i.e. create a file sufficiently large to hold your backup, partition it, activate it via losetup, then finally format it with a more amenable filesystem and mount it as the backup destination (example). ↩︎

Parallel ffmpeg stream manipulation

Fri, 29 Oct 2021 00:00:00 +0200

This post will illustrate how we can apply different operations (audio transcoding, subtitles selection, video, chapters and fonts passthrough) on multistream MKVs with ffmpeg, while optimizing for multicore systems.

More precisely, here is what we are trying to achieve:

transcode the Japanese audio track from a entire anime season, from a superfluously lossless Flac to the more efficient Opus format¹
shed the useless (to me) English audio track (also in Flac, and 5.1…), as well as the unneeded “Signs-only” subtitle tracks
pass-through the video track
retain all attachments (fonts, chapters…)
preserve file modification times
replace the original files with the new ones
do all this as fast and efficiently as possible 😎

When starting, I knew how to accomplish this in two steps using two different tools:

Identify and remove the unwanted streams with mkvmerge^(source):

mkvmerge -i *.mkv
File '01.mkv': container: Matroska
Track ID 0: video (HEVC/H.265/MPEG-H)
Track ID 1: audio (Flac)
Track ID 2: audio (Flac)
Track ID 3: subtitles (SubStationAlpha)
Track ID 4: subtitles (SubStationAlpha)
Track ID 5: subtitles (HDMV PGS)
Track ID 6: subtitles (HDMV PGS)
Attachment ID 1: type 'application/vnd.ms-opentype', size 30712 bytes, file name 'font.ttf'
(...)

for f in *.mkv; do mkvmerge -o "remuxed/$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; done

Note

Here I was able to use !eng because the audio track was labeled as such, but we could achieve the same with numerical index (just as we are doing for subs).

Use ffmpeg to transcode the remaining audio track to Opus, using a generous 192 kbps (96k per audio channel) bitrate and then do the cleanup:
```
for f in remuxed/*.mkv; do ffmpeg -i "$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done
```
Where -map 0 ports over all input streams, -c copy keeps them intact, and -c:a libopus -b:a 192k modulates this by transcoding all audio streams (in our case, the one we kept) to 192k Opus. Finally, touch -r updates our file modtime to reflect the original one, and mv overwrite the source with the modified file (warning: make sure to always test on a copy first, otherwise data loss will incur!).

From there it was rather easy to merge it all in one fell for loop:

for f in *.mkv; do \
    mkvmerge -o "tmp_$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; \
    ffmpeg -i "tmp_$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp2_$f"; \
    touch -r "$f" "tmp2_$f"; \
    mv "tmp2_$f" "$f"; \
    rm "tmp_$f"; \
done

However, now the next step was to get rid of the superfluous mkvmerge and do it all with the Ever Almighty Multimedia Swiss-Army Knife®, ffmpeg. For that I had to dig quite a bit, but here is the result:

for f in *.mkv; do ffmpeg -i "$f" -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done

As before we first select all streams via -map 0, then make use of negative mapping (notice the minus sign at the front of the subsequent -map values) to get rid of the unwanted tracks (second audio track, second and fourth sub tracks… Beware, this time we are using ffmpeg’s indexes/IDs which are relative to the category and 0-indexed). Then it is as before, we copy all remaining streams without changing them (-c copy) except audio ones which are transcoded to Opus.

Now this is all well and dandy, but this command runs sequentially, iterating over each file while queuing up the others, disregarding our multicore CPU architecture and thus slowing down the process. How can we run all this concurrently, dispatching a job to each CPU thread?

GNU Parallel² to the rescue!

parallel ffmpeg -i {} -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k tmp_{}';' touch -r {} tmp_{}';' mv tmp_{} {} ::: *.mkv

Note

Of course, overall time will also depend on your storage read/write speeds…

Et voilà, that’s how you efficiently and stylishly reduce a bloated 10.9GB series to a lean and mean 4.8GB one. Great success! 👍👏🤗

See Why you should never use FLAC (for that use case that is) ↩︎
See this Unix SE answer for why Parallel is superior to Xargs in this matter. ↩︎