Cli on Bastien Traverse

Ejecting and reattaching a USB drive from the CLI

Thu, 24 Feb 2022 00:00:00 +0100

Setup: I was booted on archlinux-2022.02.01-x86_64.iso using a multiboot USB flash drive made with Ventoy 1.0.65, and had selected the “Copy to RAM” option in the ISO menu with the idea of reusing the drive as a backup destination for a system backup.

The issue: I was unable to normally eject the drive, both eject and udisksctl power-off (after installing udisks2) gave the error Device or resource busy, and lsof | grep -e ventoy -e sdb didn’t return anything. Here is what it looked like:

root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
│ └─ventoy    254:0    0 812.3M  1 dm    <-- problematic entry
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

root@archiso ~ # eject -v /dev/sdb
eject: device name is `/dev/sdb'
eject: /dev/sdb: not mounted
eject: /dev/sdb: is whole-disk device
eject: cannot open /dev/sdb: Device or resource busy

root@archiso ~ # eject -v /dev/mapper/ventoy
eject: device name is `/dev/mapper/ventoy'
eject: /dev/mapper/ventoy: not mounted
eject: /dev/mapper/ventoy: is whole-disk device
eject: /dev/mapper/ventoy: is not hot-pluggable device

root@archiso ~ # ls -l /dev/mapper/ventoy  
lrwxrwxrwx 1 root root 7 Feb 24 20:24 /dev/mapper/ventoy -> ../dm-0

root@archiso ~ # ls -l /dev/dm-0
brw-rw---- 1 root disk 254, 0 Feb 24 20:24 /dev/dm-0

root@archiso ~ # cat /sys/class/block/dm-0/dm/name
ventoy

So, I wasn’t sure what was causing the issue, and I didn’t want to simply pull the drive out as it equates to playing Russian roulette with your devices (do you like getting the rug pulled under you or being knocked inconscious by a stealthy sucker punch? Well, neither do most filesystems).

At first I had worked around the issue by using this trick :

echo offline > /sys/block/sdb/device/state
echo 1 > /sys/block/sdb/device/delete

See this post on the Arch forums for a more thorough procedure along the same vein.

Still, I wasn’t entirely convinced: it didn’t address the root cause of the issue.

Digging a little deeper led me to dmsetup:

root@archiso ~ # dmsetup table
boot-arch: 0 516096 linear 8:1 2048
ventoy: 0 1663640 linear 8:17 59447552

root@archiso ~ # dmsetup info /dev/mapper/ventoy  
Name:              ventoy
State:             ACTIVE (READ-ONLY)
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      0
Major, minor:      254, 0
Number of targets: 1

Well, that READ-ONLY state was already good news, but checking dmsetup manpage I saw it offered a remove command:

root@archiso ~ # dmsetup remove ventoy
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

Victory, the mapping is gone! And now…

root@archiso ~ # eject /dev/sdb -v
eject: device name is `/dev/sdb'
eject: /dev/sdb: not mounted
eject: /dev/sdb: is whole-disk device
eject: /dev/sdb: trying to eject using CD-ROM eject command
eject: CD-ROM eject command failed
eject: /dev/sdb: trying to eject using SCSI commands
eject: SCSI eject succeeded

root@archiso ~ # lsblk  
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
sr0            11:0    1  1024M  0 rom

Success! I saw the LED on my thumb drive flash quickly, and then it fell silent. But wait, what is all this red in my logs?

archiso kernel: sd 6:0:0:0: [sdb] Media removed, stopped polling
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: ldm_validate_partition_table(): Disk read failed.
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel:  sdb: unable to read partition table
archiso kernel: sd 6:0:0:0: [sdb] Media removed, stopped polling
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: ldm_validate_partition_table(): Disk read failed.
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
archiso kernel: Buffer I/O error on dev sdb, logical block 0, async page read
archiso kernel:  sdb: unable to read partition table
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed
archiso kernel: sd 6:0:0:0: [sdb] tag#0 device offline or changed

Hum, looks like eject is doing something kind of dirty… Let’s move ahead for now, we’ll come back to it later.

Now, since this is a laptop laying on my desk, I certainly can manually unplug the drive before plugging it back again to make it available. But what if I am lazy, or the machine is faraway, or I just want to take extra care of that super frail USB plug and prevent a superfluous actuation cycle?

Reattaching an ejected USB device from the CLI

Here we are going to make use of the ability to bind and unbind drivers from devices manually from user space. First, get the drive Bus number:

root@archiso ~ # lsusb  
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive  <-- here is my device
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Then Port for “Bus 4 Device 3”:

root@archiso ~ # lsusb -t  
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M  <-- spot it?
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

Now that I know that my USB device is identified as 4-1 (bus-port) (this can be confirmed by checking dmesg | grep usb-storage), we can make it go through an unbinding/binding cycle:

root@archiso ~ # echo "4-1" > /sys/bus/usb/drivers/usb/unbind
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sr0            11:0    1  1024M  0 rom  
root@archiso ~ # echo "4-1" > /sys/bus/usb/drivers/usb/bind  
root@archiso ~ # lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0 657.4M  1 loop /run/archiso/airootfs
sda             8:0    0 111.8G  0 disk
├─sda1          8:1    0     1G  0 part
│ └─boot-arch 254:1    0   252M  0 lvm  
└─sda2          8:2    0   100G  0 part
sdb             8:16   0  59.6G  0 disk
├─sdb1          8:17   0  59.6G  0 part
└─sdb2          8:18   0    32M  0 part
sr0            11:0    1  1024M  0 rom

Et voilà, the flash drive is once again available for mount without having to physically remove it!

Note

Trying to bind directly after eject without unbind leads to a write error: device or resource busy . So now I am really left wondering what eject is doing to my drive…

An alternative to manually writing to /sys/bus/usb/drivers/usb/ is to use usb_modeswitch (included on the Arch installation media). In that case we only need the ID pair from lsusb:

root@archiso ~ # lsusb  
...
Bus 004 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive
                       ==== ==== ------ ---------
                                      |         |
root@archiso ~ # usb_modeswitch -v 0x0781 -p 0x5580 --reset-usb
Look for default devices ...
 Found devices in default mode (1)
Access device 003 on bus 004
Get the current device configuration ...
Current configuration number is 1
Use interface number 0
 with class 8
Warning: no switching method given. See documentation
Reset USB device .
 Device was reset
-> Run lsusb to note any changes. Bye!

Props to this AskUbuntu answer, confirmed and extended via this Stackoverflow one.

Wrap up

After a few trial and errors, I realized ejecting the drive was entirely unnecessary: I could get away with just dmsetup remove ventoy and an unbind/bind cycle.

Now if you ever have to cleanly power-off a device on the command line, here is the ultimate method:

% udisksctl power-off -h
Utilisation :
  udisksctl power-off [OPTION…]

Safely power off a drive.

Options :
  -p, --object-path         Object path for ATA device
  -b, --block-device        Device file for ATA device
  --no-user-interaction     Do not authenticate the user if needed

E.g. udisksctl power-off -b /dev/sdb.

For the life of me I don’t know why this utility isn’t included by default on the arch installation media, but in any case you are just one pacman -S udisks2 away from safely-removed device nibbana.

That’s all folks for today’s learning diary. Thanks for reading and take care of your devices!

A system imaging tale: LUKS, Clonezilla and friends

Mon, 14 Feb 2022 00:00:00 +0100

Today’s project: make a system image of my previous laptop for archival purposes before refurbishing it.

Conditions:

make a space efficient image, i.e. not a block-level copy: no need to retain encryption or logical volume setup, only their content matters. However:
all standard and extended file attributes must be correctly preserved, so that if I ever want to boot this image the system will be in a correct state.

The plan: boot up an Arch ISO, open the encrypted drives, set everything up for clonezilla – profit.

Here is how it actually went.

An unexpected obstacle to boot: Ventoy shenanigans

In order to boot into the Arch live environment, I unsheathed my very dear and trusted Sandisk Extreme multiboot USB thumb drive (not the currently sold, lower performing one but the ever chart-topping one from about 10 years ago), which had been irreproachably and impeccably fulfilling its duty in the two years since I had discovered Ventoy, the multiboot solution which surpassed them all and ended my long standing quest to find an easy to use, low-maintenance and low-friction multiboot USB solution. And yes, this was the longest sentence of the entire article 😁

Just recently I had updated it to the latest version using the provided Linux script, which completed without issue. But as soon as I booted on it, I was faced with an error: Warning! This is not a standard device and is not officially supported. Um, beg your pardon? It was working fine those past two years, so certainly it was the update which broke something…

Searching for this error, I eventually found this “kill switch” the dev introduced slightly after I used the software, to try and contain the flood of users with non-standard installs complaining to them when something got wrong.

I can’t really fault them for that, but here is the thing: mine was a pure product of the official tool from back in 2020, with no deviation from the prescribed layout/FS whatsoever. The first partition was starting at the 2048/1 MB boundary and formatted with the default exFAT, the second partition was 65536 sector large, the drive had an MBR partition table and all and all…

# fdisk -l /dev/sda
Disk /dev/sda: 59.63 GiB, 64023257088 bytes, 125045424 sectors
Disk model: Extreme  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd1a119ab

Device     Boot     Start       End   Sectors  Size Id Type
/dev/sda1  *         2048 124979887 124977840 59.6G  7 HPFS/NTFS/exFAT
/dev/sda2       124979888 125045423     65536   32M ef EFI (FAT-12/16/32)>

The official command line tool didn’t complain either while upgrading or when run manually:

# ventoy -l /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Ventoy Version in Disk: 1.0.65
Disk Partition Style  : MBR
Secure Boot Support   : NO

Indeed but nope, I got axed by the kill switch all the same. Fuck artificial restrictions and treacherous computing. And I am burned by a free software project at that… What to expect from this world.

Anyhoo, after transfering out my 24 GB of GNU/Linux ISOs (God bless USB 3.0 & quality flash memory speeds), I forced a reinstall with the -I switch:

# ventoy -I /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Disk : /dev/sda
Size : 59 GB
Style: MBR


Attention:
You will install Ventoy to /dev/sda.
All the data on the disk /dev/sda will be lost!!!

Continue? (y/n) y

All the data on the disk /dev/sda will be lost!!!
Double-check. Continue? (y/n) y

Create partitions on /dev/sda by parted in MBR style ...
Done
Wait for partitions ...
partition exist OK
create efi fat fs /dev/sda2 ...
mkfs.fat 4.2 (2021-01-31)
success
Wait for partitions ...
/dev/sda1 exist OK
/dev/sda2 exist OK
partition exist OK
Format partition 1 /dev/sda1 ...
mkexfatfs 1.3.0
Creating... done.
Flushing... done.
File system created successfully.
mkexfatfs success
writing data to disk ...
sync data ...
esp partition processing ...
Open ventoy efi file 0x610ac0
ventoy x64 efi file size 1757184 ...
Open bootx64 efi file 0x610ac0
Open ventoy ia32 efi file 0x610f10
ventoy efi file size 1183744 ...
Open bootia32 efi file 0x610ac0

Install Ventoy to /dev/sda successfully finished.

sudo ventoy -I /dev/sda  5,11s user 2,58s system 19% cpu 39,813 total

After transferring my files back and rebooting on the flash drive, everything was back in order and I was ready to rock 🤟

Optional step: setting up a comfortable environment

Being console-based, the archiso environment is somewhat austere. So the first thing I usually do is to set up SSH access so as to connect from my main machine:

# set french keymap
# the "-latin9" part is only needed if using e.g. accents
loadkeys fr-latin9
# directly connect to a known Wifi SSID
iwctl station wlan0 connect <SSID>
# Enable NTP to have accurate time
timedatectl set-timezone <Area/City>
timedatectl set-ntp true
# set root passhrase to enable SSH connection
passwd
# start SSH service
systemctl start sshd
# get IP address
iwctl station wlan0 show | grep IP | awk '{print $3}'
# or without iwd:
ip a | grep wlan | awk 'FNR == 2 {print substr($2, 1, length($2)-3)}'

Finally, connect to the live environment via SSH:

ssh -o PreferredAuthentications=password \
    -o PubkeyAuthentication=no \
    -o UserKnownHostsFile=/dev/null \
    root@<other-machine-IP>

👉️ The options used allow not to be affected by the local SSH configuration (SSH keys…), as well as well not polluting the local KnownHosts file with this one-time connection.

That’s it, now we can resume with the common procedure!

Clonezilla and LUKS/LVM

In order to efficiently image the system, we first need to open any LUKS container and make sure all the LVs are displayed correctly:

cryptsetup open --allow-discards /dev/sdb2 os
lvs

Then we create the /home/partimag directory (hardcoded backup destination in Clonezilla) and mount our drive on it:

mkdir /home/partimag
mount -o noatime /dev/sdc1 !$

Note

When booting the archiso, I selected the “Copy to RAM” option so as to free up the drive and reuse it as backup destination.

However, in order to do that I first needed to get rid of a lingering /dev/mapper/ventoy mapping on its first partition, which made eject/udisksctl throw errors despite the fact that it wasn’t mounted and lsof | grep -e ventoy -e sdb didn’t return anything. I didn’t want to pull it out on the off chance it’d corrupt something…

I wrote a separate post which goes into the details, but the gist is a simple dmsetup remove ventoy took care of the mapping, and then I was able to normally mount the drive.

And now it is on! Launch clonezilla from the command line and follow the on-screen instructions depending on your needs. Some personal recommendations:

use expert mode
avoid the 4G backup splitting by entering a large number when prompted (like 100000)
select “parallel zstd” (-z9p) as the compression algorithm

Or if you feel like keeping it CLI-only…

/usr/bin/ocs-sr \
    --use-partclone \
    --confirm \
    --clone-hidden-data \
    --zstdmt-compress
    --image-size 1000000 \
    --fsck-src-part \
    --skip-enc-ocs-img \
    --postaction choose \
    saveparts <backup-name> sdb1 sdb2

This is the long form equivalent of the

/usr/bin/ocs-sr -q2 -c -j2 -z9p -i 1000000 -fsck -senc -p choose saveparts <backup-name> sdb1 sdb

that got copied in a file under /tmp after validating my options in the TUI.

Then we just sit back and relax… Wait, what? dd mode? Why aren’t my OS logical volumes backed up by the regular free space-aware partclone mode? I explicitly decrypted the containers for that purpose, and saw the LVs being listed during the preparation screens! Although it’s true I could only select whole partitions to be backed up and not specific logical volumes, and saw a line flash by saying something like Shutting down the Volume manager… Oh come on.

One Ctrl+c later the backup of /dev/sdb2 was cancelled, but the unencrypted /dev/sdb1 with the arch LV (the only one in the boot VG) proceeded without issue. What the heck clonezilla?

Hum alright, according to this 2014 thread with the developer’s input, it seems this isn’t as straightforward as it looks. And it happened even much more recently to another user (although according to the dev’s answer it shouldn’t be the case), so anyway… Time for some manual partclone fun I guess!

System imaging, the manual way

Full disclosure: I had already stumbled upon the “manual partclone required” caveat when trying to efficiently backup LUKS-encrypted systems with Clonezilla, so I wasn’t really at a loss. All in all it’s pretty simple, you just have to use partclone like so:

partclone.$fstype --clone --source ... --output ...

Wich gave in my case:

partclone.ext4 --clone --source /dev/mapper/arch-root --output <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl.img

Tip

If you want to have the same nice ncurses-based UI as Clonezilla does, add the -N/--ncurses option (example).

Though if we really want to mimick Clonezilla and have space-efficient backups, we gotta add compression to the mix!

Compressing partclone images with zstd

Being a devout Zstandard Zealot© (this post summarizes well my feelings – no affiliation with the author), here is how to interface it with partclone for a nicely multi-threaded compression:

partclone.ext4 --clone --source /dev/mapper/arch-root | zstdmt -o <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl-img.zst

Info

From zstd man page: zstdmt is equivalent to zstd -T0, enabling auto-adjusted multi-threading.

If you are like me and have several LVs to clone and their names follow a pattern, you can use a for loop to optimize things a bit:

for i in arch-{root,home,opt,var} boot-arch; do partclone.ext4 --clone --source /dev/mapper/"${i}" | zstdmt -o <backup path>/$(date -I)-"${i}".ext4-ptcl-img.zst; done

Et voilà, space-efficient compressed images of your decrypted LVs!

Tip

Someone made a gist using somewhat older commands and touching upon restoration etc… Give it a look if you want to review the process!

Extra: comparing the process with different tools

A very precious thing in this world is diversity. Of people, ideas, food… And backup tools. Here is a rundown of the same task undertaken with four of them:

FSArchiver

The first time I realized I couldn’t use Clonezilla to properly back up my decrypted LVs, I stumbled upon a great alternative to partclone: FSArchiver.

The main reason for using it over the former? FSArchiver:

directly integrates multi-threaded compression with zstd (and recommends it over any others)
allows to specify multiple filesystems to be archived at once
also works with directories (creating a compressed and checksummed tarball of sorts)
allows to restore to a different filesystem and creating it in one go
allows to restore to a smaller partition by default, whereas partclone needs the -C option (and often just ends on a seek error)
accepts exclusion patterns to filter what’s archived/restored (the syntax is a bit tricky though)
offers to optionally encrypt data in the archive
and is generally really careful with data integrity… If that wasn’t already reason enough 😉

Its multi-threaded compression-ready invocation goes something like this:

fsarchiver --zstd=3 --jobs=$(nproc) savefs <backup path>/<archive-name>.fsa <list of devices to backup> --exclude=<exclude pattern>

--zstd=3 stands for zstd-compression at level 3
--jobs=$(nproc) works exactly like the eponymous make switch, enabling as many threads as there are logical cores on your system.

So for example:

fsarchiver -Z3 -j$(nproc) -v savefs <backup-path>/old-laptop.fsa /dev/mapper/arch-{root,home,opt,var} /dev/mapper/boot-arch --exclude="lost+found" --exclude="/var/cache/pacman/pkg/*"

The -v/--verbose option is useful to get a progress report, however it will list every file being archived and as such likely blow away your terminal history.

BTW, here is why clonezilla defaults to partclone instead of fsarchiver in the words of its author. Tldr: partclone supports pipes/stdin/stdout which are needed for other Clonezilla features, while fsarchiver doesn’t. Tough life.

Note however that FSArchiver (just like partclone) doesn’t deal with recreating a partition table on the destination, or any other form of partitioning/preparing the drive such as LUKS/LVM: you gotta do all that manually up to the point where you are ready to restore filsystems (that is to say, fsarchiver isn’t a disk cloning solution).

Also, you cannot merge the content of separate partitions created this way into one destination (e.g. if you want to consolidate separate LVs into one FS): they will overwrite each other. For that you’d rather use the savedir/restdir mode, which is akin to a tarball on steroids.

Squashfs

I discovered this method by stumbling upon this gem on the Arch wiki, “Full system backup with SquashFS”, and although the article is in dire need of a proper rewrite (which I’ve added to my todo list) I really wanted to give it a try.

In the words of TLDP, SquashFS is

“a read-only file system that lets you compress whole file systems or single directories, write them to other devices/partitions or to ordinary files, and then mount them directly (if a device) or using a loopback device (if it is a file). […] For archiving purposes, SquashFS gives you a lot more flexibility and performance speed than a tarball archive.”

Its selling points to me are:

random access & transparent decompression: no need to decompress and extract the entire image to browse its content (this is huge, especially when talking about gigabyte-sized images…)
file-level deduplication
exclusion patterns
zstd support 😊

However be aware that it doesn’t support ACLs, so if you rely on them you should look somewhere else.

After quickly perusing its manpage I gave it a go:

mount -o ro /dev/mapper/arch-root /mnt
mount -o ro /dev/mapper/boot-arch /mnt/boot
for i in home opt var; do mount -o ro /dev/mapper/arch-"${i}" /mnt/$i
cd /mnt

mksquashfs ./ <backup-path>/old-laptop.sqsh \
    -comp zstd \
    -no-exports \
    -progress \
    -mem 6G \
    -not-reproducible \
    -e var/cache/pacman/pkg

Results:

Squashfs 4.0 filesystem, zstd compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 3912057.66 Kbytes (3820.37 Mbytes)
	35.64% of uncompressed filesystem size (10975717.90 Kbytes)
Inode table size 2886293 bytes (2818.65 Kbytes)
	24.42% of uncompressed inode table size (11819496 bytes)
Directory table size 3606490 bytes (3521.96 Kbytes)
	36.39% of uncompressed directory table size (9910813 bytes)
Xattr table size 621 bytes (0.61 Kbytes)
	28.98% of uncompressed xattr table size (2143 bytes)

To put things in perspective:

root@archiso ~ # du -sh /mnt
16G	/mnt
root@archiso ~ # du -sh /mnt/var/cache/pacman/pkg
4.5G	/mnt/var/cache/pacman/pkg
root@archiso ~ # du -h <backup-path>/old-laptop.sqsh
3.8G	<backup-path>/old-laptop.sqsh

So it went from 16-4.5=11.5 GiB to 3.8… Nice!

It produced a slightly smaller archive than both partclone and fsarchiver (even when accounting for the non-excluded pacman cache), but it took 11.5 minutes. After testing, fsarchiver savedir took 3min08s and produced a 4.2 GiB image (with --exclude='*.pkg.tar.zst'), but then fsarchiver doesn’t produce browsable archives and doesn’t do any deduplication 😊

Rsync

One step more remote from system imaging, the ever powerful Swiss-army knife of file transfer can evidently also be used to create full system backups. Here is my personal take on it:

rsync \
    --archive \
    --hard-links \
    --acls \
    --xattrs \
    --atimes \
    --open-noatime \
    --sparse \
    --human-readable \
    --info=progress2 \
    --partial \
    --exclude={"var/cache/pacman/pkg/*","*/lost+found"} \
    /mnt/* /path/to/backup/dir

Or as a one-liner:

rsync -aHAXUUSh --info=progress2 --partial --exclude={"var/cache/pacman/pkg/*","*/lost+found"} /mnt/* /path/to/backup/dir

See its man page for details on the options used. Note that the repeated -U is not a mistake but a shortcut to enable both --atimes and --open-noatime 😉

However, while doing so I realized rsync wasn’t exactly the best solution when the destination filesystem is {ex,}FAT… Timestamps and permissions/ACLs/xattr won’t carry over correctly¹ 😒 This is what got me for the very first time interested in using…

Tar

O venerable Lord of computerized backups! Thy name is reminiscent of the glorious days of yore, when backups were mere tape archives…

Okay, but how does it fare in 2022 for complete system backup with high fidelity? After delving into its manpage this is what I came up with:

cd /mnt
tar --create \
    --preserve-permissions \
    --acls \
    --xattrs \
    --atime-preserve=system \
    --exclude={"var/cache/pacman/pkg/*","*lost+found"} \
    -I zstdmt \
    -f /path/to/backup/dir/<backup-name>.tar.zst \
    *
tar: /var/run/nscd/socket: socket ignored
...
# repeat dozens of times

Hmmm okay, apart from socket files which apparently clearly don’t matter, what about the completeness of the backup? Sections verify and compare from tar manual cover this question. A few pitfalls to be aware of however:

--verify (only for archive creation) doesn’t work with compressed archives; if you don’t mind you can first create and verify an uncompressed archive, then compress it in a second step…
--compare “ignores files in the file system that do not have corresponding members in the archive”, so if something was left over during the backup, no errors will be reported.

If you have the space and will for it, you could also just extract the archive and diff --recursive it against the source, but all in all I feel this is an aspect where the original (and still current) purpose of tar as a tape specific archiving tool, makes itself felt.

From my superficial testing everything seemed mostly alright, but I can’t shake the feelings that tar isn’t making as complete a clone as fsarchiver or rsync (see its limitations too). But that’s probably because my beard isn’t grey enough I guess… 😁

Wrap up

In conclusion, if you are in a situation where Clonezilla doesn’t work for you (most notably when using LUKS), my recommendation would be to turn to either FSArchiver, partclone or SquashFS. They seem to provide the most complete and reliable feature-set of the lot for imaging systems in a correct and efficient way. Rsync and tar being more general purpose tools, they require more options not to leave something out or otherwise shoot yourself in the foot.

Also, while laboring over this article I realized that while it was all fine and dandy to have a static archive of my old system, the next step was to make an interactive image of it that I could boot on whenever I want to see which GTK theme I used, or otherwise take a nostalgic whiff and relapse in my former computing abode…

This reminded me of the Moving an existing install into a virtual machine article, and I have an entire article coming up with the fruits of this endeavour: creating a VM image from the backups made here, testing in the process the oh-so-important restoration side of all the different tools used here (which taught me that an un-restored backup is no backup at all), making the required adjustments to be able to boot from it… Stay stuned for the second part! [edit: there it goes]

Thanks for reading and happy imaging 🙏

One workaround is to use a loop device file, i.e. create a file sufficiently large to hold your backup, partition it, activate it via losetup, then finally format it with a more amenable filesystem and mount it as the backup destination (example). ↩︎

Parallel ffmpeg stream manipulation

Fri, 29 Oct 2021 00:00:00 +0200

This post will illustrate how we can apply different operations (audio transcoding, subtitles selection, video, chapters and fonts passthrough) on multistream MKVs with ffmpeg, while optimizing for multicore systems.

More precisely, here is what we are trying to achieve:

transcode the Japanese audio track from a entire anime season, from a superfluously lossless Flac to the more efficient Opus format¹
shed the useless (to me) English audio track (also in Flac, and 5.1…), as well as the unneeded “Signs-only” subtitle tracks
pass-through the video track
retain all attachments (fonts, chapters…)
preserve file modification times
replace the original files with the new ones
do all this as fast and efficiently as possible 😎

When starting, I knew how to accomplish this in two steps using two different tools:

Identify and remove the unwanted streams with mkvmerge^(source):

mkvmerge -i *.mkv
File '01.mkv': container: Matroska
Track ID 0: video (HEVC/H.265/MPEG-H)
Track ID 1: audio (Flac)
Track ID 2: audio (Flac)
Track ID 3: subtitles (SubStationAlpha)
Track ID 4: subtitles (SubStationAlpha)
Track ID 5: subtitles (HDMV PGS)
Track ID 6: subtitles (HDMV PGS)
Attachment ID 1: type 'application/vnd.ms-opentype', size 30712 bytes, file name 'font.ttf'
(...)

for f in *.mkv; do mkvmerge -o "remuxed/$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; done

Note

Here I was able to use !eng because the audio track was labeled as such, but we could achieve the same with numerical index (just as we are doing for subs).

Use ffmpeg to transcode the remaining audio track to Opus, using a generous 192 kbps (96k per audio channel) bitrate and then do the cleanup:
```
for f in remuxed/*.mkv; do ffmpeg -i "$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done
```
Where -map 0 ports over all input streams, -c copy keeps them intact, and -c:a libopus -b:a 192k modulates this by transcoding all audio streams (in our case, the one we kept) to 192k Opus. Finally, touch -r updates our file modtime to reflect the original one, and mv overwrite the source with the modified file (warning: make sure to always test on a copy first, otherwise data loss will incur!).

From there it was rather easy to merge it all in one fell for loop:

for f in *.mkv; do \
    mkvmerge -o "tmp_$f" --audio-tracks '!eng' --subtitle-tracks '!4,6' "$f"; \
    ffmpeg -i "tmp_$f" -map 0 -c copy -c:a libopus -b:a 192k "tmp2_$f"; \
    touch -r "$f" "tmp2_$f"; \
    mv "tmp2_$f" "$f"; \
    rm "tmp_$f"; \
done

However, now the next step was to get rid of the superfluous mkvmerge and do it all with the Ever Almighty Multimedia Swiss-Army Knife®, ffmpeg. For that I had to dig quite a bit, but here is the result:

for f in *.mkv; do ffmpeg -i "$f" -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k "tmp_$f"; touch -r "$f" "tmp_$f"; mv "tmp_$f" "$f"; done

As before we first select all streams via -map 0, then make use of negative mapping (notice the minus sign at the front of the subsequent -map values) to get rid of the unwanted tracks (second audio track, second and fourth sub tracks… Beware, this time we are using ffmpeg’s indexes/IDs which are relative to the category and 0-indexed). Then it is as before, we copy all remaining streams without changing them (-c copy) except audio ones which are transcoded to Opus.

Now this is all well and dandy, but this command runs sequentially, iterating over each file while queuing up the others, disregarding our multicore CPU architecture and thus slowing down the process. How can we run all this concurrently, dispatching a job to each CPU thread?

GNU Parallel² to the rescue!

parallel ffmpeg -i {} -map 0 -map -0:a:1 -map -0:s:1 -map -0:s:3 -c copy -c:a libopus -b:a 192k tmp_{}';' touch -r {} tmp_{}';' mv tmp_{} {} ::: *.mkv

Note

Of course, overall time will also depend on your storage read/write speeds…

Et voilà, that’s how you efficiently and stylishly reduce a bloated 10.9GB series to a lean and mean 4.8GB one. Great success! 👍👏🤗

See Why you should never use FLAC (for that use case that is) ↩︎
See this Unix SE answer for why Parallel is superior to Xargs in this matter. ↩︎