A system imaging tale: LUKS, Clonezilla and friends

Today’s project: make a system image of my previous laptop for archival purposes before refurbishing it.

Conditions:

• make a space efficient image, i.e. not a block-level copy: no need to retain encryption or logical volume setup, only their content matters. However:
• all standard and extended file attributes must be correctly preserved, so that if I ever want to boot this image the system will be in a correct state.

The plan: boot up an Arch ISO, open the encrypted drives, set everything up for clonezilla – profit.

Here is how it actually went.

An unexpected obstacle to boot: Ventoy shenanigans

In order to boot into the Arch live environment, I unsheathed my very dear and trusted Sandisk Extreme multiboot USB thumb drive (not the currently sold, lower performing one but the ever chart-topping one from about 10 years ago), which had been irreproachably and impeccably fulfilling its duty in the two years since I had discovered Ventoy, the multiboot solution which surpassed them all and ended my long standing quest to find an easy to use, low-maintenance and low-friction multiboot USB solution. And yes, this was the longest sentence of the entire article 😁

Just recently I had updated it to the latest version using the provided Linux script, which completed without issue. But as soon as I booted on it, I was faced with an error: Warning! This is not a standard device and is not officially supported. Um, beg your pardon? It was working fine those past two years, so certainly it was the update which broke something…

Searching for this error, I eventually found this “kill switch” the dev introduced slightly after I used the software, to try and contain the flood of users with non-standard installs complaining to them when something got wrong.

I can’t really fault them for that, but here is the thing: mine was a pure product of the official tool from back in 2020, with no deviation from the prescribed layout/FS whatsoever. The first partition was starting at the 2048/1 MB boundary and formatted with the default exFAT, the second partition was 65536 sector large, the drive had an MBR partition table and all and all…

# fdisk -l /dev/sda
Disk /dev/sda: 59.63 GiB, 64023257088 bytes, 125045424 sectors
Disk model: Extreme  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd1a119ab

Device     Boot     Start       End   Sectors  Size Id Type
/dev/sda1  *         2048 124979887 124977840 59.6G  7 HPFS/NTFS/exFAT
/dev/sda2       124979888 125045423     65536   32M ef EFI (FAT-12/16/32)>

The official command line tool didn’t complain either while upgrading or when run manually:

# ventoy -l /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Ventoy Version in Disk: 1.0.65
Disk Partition Style  : MBR
Secure Boot Support   : NO

Indeed but nope, I got axed by the kill switch all the same. Fuck artificial restrictions and treacherous computing. And I am burned by a free software project at that… What to expect from this world.

Anyhoo, after transfering out my 24 GB of GNU/Linux ISOs (God bless USB 3.0 & quality flash memory speeds), I forced a reinstall with the -I switch:

# ventoy -I /dev/sda

**********************************************
      Ventoy: 1.0.65  x86_64
      longpanda admin@ventoy.net
      https://www.ventoy.net
**********************************************

Disk : /dev/sda
Size : 59 GB
Style: MBR


Attention:
You will install Ventoy to /dev/sda.
All the data on the disk /dev/sda will be lost!!!

Continue? (y/n) y

All the data on the disk /dev/sda will be lost!!!
Double-check. Continue? (y/n) y

Create partitions on /dev/sda by parted in MBR style ...
Done
Wait for partitions ...
partition exist OK
create efi fat fs /dev/sda2 ...
mkfs.fat 4.2 (2021-01-31)
success
Wait for partitions ...
/dev/sda1 exist OK
/dev/sda2 exist OK
partition exist OK
Format partition 1 /dev/sda1 ...
mkexfatfs 1.3.0
Creating... done.
Flushing... done.
File system created successfully.
mkexfatfs success
writing data to disk ...
sync data ...
esp partition processing ...
Open ventoy efi file 0x610ac0
ventoy x64 efi file size 1757184 ...
Open bootx64 efi file 0x610ac0
Open ventoy ia32 efi file 0x610f10
ventoy efi file size 1183744 ...
Open bootia32 efi file 0x610ac0

Install Ventoy to /dev/sda successfully finished.

sudo ventoy -I /dev/sda  5,11s user 2,58s system 19% cpu 39,813 total

After transferring my files back and rebooting on the flash drive, everything was back in order and I was ready to rock 🤟

Optional step: setting up a comfortable environment

Being console-based, the archiso environment is somewhat austere. So the first thing I usually do is to set up SSH access so as to connect from my main machine:

# set french keymap
# the "-latin9" part is only needed if using e.g. accents
loadkeys fr-latin9
# directly connect to a known Wifi SSID
iwctl station wlan0 connect <SSID>
# Enable NTP to have accurate time
timedatectl set-timezone <Area/City>
timedatectl set-ntp true
# set root passhrase to enable SSH connection
passwd
# start SSH service
systemctl start sshd
# get IP address
iwctl station wlan0 show | grep IP | awk '{print $3}'
# or without iwd:
ip a | grep wlan | awk 'FNR == 2 {print substr($2, 1, length($2)-3)}'

Finally, connect to the live environment via SSH:

ssh -o PreferredAuthentications=password \
    -o PubkeyAuthentication=no \
    -o UserKnownHostsFile=/dev/null \
    root@<other-machine-IP>

That’s it, now we can resume with the common procedure!

Clonezilla and LUKS/LVM

In order to efficiently image the system, we first need to open any LUKS container and make sure all the LVs are displayed correctly:

cryptsetup open --allow-discards /dev/sdb2 os
lvs

Then we create the /home/partimag directory (hardcoded backup destination in Clonezilla) and mount our drive on it:

mkdir /home/partimag
mount -o noatime /dev/sdc1 !$

And now it is on! Launch clonezilla from the command line and follow the on-screen instructions depending on your needs. Some personal recommendations:

• use expert mode
• avoid the 4G backup splitting by entering a large number when prompted (like 100000)
• select “parallel zstd” (-z9p) as the compression algorithm

Or if you feel like keeping it CLI-only…

/usr/bin/ocs-sr \
    --use-partclone \
    --confirm \
    --clone-hidden-data \
    --zstdmt-compress
    --image-size 1000000 \
    --fsck-src-part \
    --skip-enc-ocs-img \
    --postaction choose \
    saveparts <backup-name> sdb1 sdb2

This is the long form equivalent of the

/usr/bin/ocs-sr -q2 -c -j2 -z9p -i 1000000 -fsck -senc -p choose saveparts <backup-name> sdb1 sdb

that got copied in a file under /tmp after validating my options in the TUI.

Then we just sit back and relax… Wait, what? dd mode? Why aren’t my OS logical volumes backed up by the regular free space-aware partclone mode? I explicitly decrypted the containers for that purpose, and saw the LVs being listed during the preparation screens! Although it’s true I could only select whole partitions to be backed up and not specific logical volumes, and saw a line flash by saying something like Shutting down the Volume manager… Oh come on.

One Ctrl+c later the backup of /dev/sdb2 was cancelled, but the unencrypted /dev/sdb1 with the arch LV (the only one in the boot VG) proceeded without issue. What the heck clonezilla?

Hum alright, according to this 2014 thread with the developer’s input, it seems this isn’t as straightforward as it looks. And it happened even much more recently to another user (although according to the dev’s answer it shouldn’t be the case), so anyway… Time for some manual partclone fun I guess!

System imaging, the manual way

Full disclosure: I had already stumbled upon the “manual partclone required” caveat when trying to efficiently backup LUKS-encrypted systems with Clonezilla, so I wasn’t really at a loss. All in all it’s pretty simple, you just have to use partclone like so:

partclone.$fstype --clone --source ... --output ...

Wich gave in my case:

partclone.ext4 --clone --source /dev/mapper/arch-root --output <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl.img

Though if we really want to mimick Clonezilla and have space-efficient backups, we gotta add compression to the mix!

Compressing partclone images with zstd

Being a devout Zstandard Zealot© (this post summarizes well my feelings – no affiliation with the author), here is how to interface it with partclone for a nicely multi-threaded compression:

partclone.ext4 --clone --source /dev/mapper/arch-root | zstdmt -o <backup path>/$(date -I)-old-laptop-archroot.ext4-ptcl-img.zst

If you are like me and have several LVs to clone and their names follow a pattern, you can use a for loop to optimize things a bit:

for i in arch-{root,home,opt,var} boot-arch; do partclone.ext4 --clone --source /dev/mapper/"${i}" | zstdmt -o <backup path>/$(date -I)-"${i}".ext4-ptcl-img.zst; done

Et voilà, space-efficient compressed images of your decrypted LVs!

Extra: comparing the process with different tools

A very precious thing in this world is diversity. Of people, ideas, food… And backup tools. Here is a rundown of the same task undertaken with four of them:

FSArchiver

The first time I realized I couldn’t use Clonezilla to properly back up my decrypted LVs, I stumbled upon a great alternative to partclone: FSArchiver.

The main reason for using it over the former? FSArchiver:

• directly integrates multi-threaded compression with zstd (and recommends it over any others)
• allows to specify multiple filesystems to be archived at once
• also works with directories (creating a compressed and checksummed tarball of sorts)
• allows to restore to a different filesystem and creating it in one go
• allows to restore to a smaller partition by default, whereas partclone needs the -C option (and often just ends on a seek error)
• accepts exclusion patterns to filter what’s archived/restored (the syntax is a bit tricky though)
• offers to optionally encrypt data in the archive
• and is generally really careful with data integrity… If that wasn’t already reason enough 😉

Its multi-threaded compression-ready invocation goes something like this:

fsarchiver --zstd=3 --jobs=$(nproc) savefs <backup path>/<archive-name>.fsa <list of devices to backup> --exclude=<exclude pattern>

• --zstd=3 stands for zstd-compression at level 3
• --jobs=$(nproc) works exactly like the eponymous make switch, enabling as many threads as there are logical cores on your system.

So for example:

fsarchiver -Z3 -j$(nproc) -v savefs <backup-path>/old-laptop.fsa /dev/mapper/arch-{root,home,opt,var} /dev/mapper/boot-arch --exclude="lost+found" --exclude="/var/cache/pacman/pkg/*"

The -v/--verbose option is useful to get a progress report, however it will list every file being archived and as such likely blow away your terminal history.

Note however that FSArchiver (just like partclone) doesn’t deal with recreating a partition table on the destination, or any other form of partitioning/preparing the drive such as LUKS/LVM: you gotta do all that manually up to the point where you are ready to restore filsystems (that is to say, fsarchiver isn’t a disk cloning solution).

Also, you cannot merge the content of separate partitions created this way into one destination (e.g. if you want to consolidate separate LVs into one FS): they will overwrite each other. For that you’d rather use the savedir/restdir mode, which is akin to a tarball on steroids.

Squashfs

I discovered this method by stumbling upon this gem on the Arch wiki, “Full system backup with SquashFS”, and although the article is in dire need of a proper rewrite (which I’ve added to my todo list) I really wanted to give it a try.

In the words of TLDP, SquashFS is

“a read-only file system that lets you compress whole file systems or single directories, write them to other devices/partitions or to ordinary files, and then mount them directly (if a device) or using a loopback device (if it is a file). […] For archiving purposes, SquashFS gives you a lot more flexibility and performance speed than a tarball archive.”

Its selling points to me are:

• random access & transparent decompression: no need to decompress and extract the entire image to browse its content (this is huge, especially when talking about gigabyte-sized images…)
• file-level deduplication
• exclusion patterns
• zstd support 😊

However be aware that it doesn’t support ACLs, so if you rely on them you should look somewhere else.

After quickly perusing its manpage I gave it a go:

mount -o ro /dev/mapper/arch-root /mnt
mount -o ro /dev/mapper/boot-arch /mnt/boot
for i in home opt var; do mount -o ro /dev/mapper/arch-"${i}" /mnt/$i
cd /mnt

mksquashfs ./ <backup-path>/old-laptop.sqsh \
    -comp zstd \
    -no-exports \
    -progress \
    -mem 6G \
    -not-reproducible \
    -e var/cache/pacman/pkg

Results:

Squashfs 4.0 filesystem, zstd compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 3912057.66 Kbytes (3820.37 Mbytes)
	35.64% of uncompressed filesystem size (10975717.90 Kbytes)
Inode table size 2886293 bytes (2818.65 Kbytes)
	24.42% of uncompressed inode table size (11819496 bytes)
Directory table size 3606490 bytes (3521.96 Kbytes)
	36.39% of uncompressed directory table size (9910813 bytes)
Xattr table size 621 bytes (0.61 Kbytes)
	28.98% of uncompressed xattr table size (2143 bytes)

To put things in perspective:

root@archiso ~ # du -sh /mnt
16G	/mnt
root@archiso ~ # du -sh /mnt/var/cache/pacman/pkg
4.5G	/mnt/var/cache/pacman/pkg
root@archiso ~ # du -h <backup-path>/old-laptop.sqsh
3.8G	<backup-path>/old-laptop.sqsh

So it went from 16-4.5=11.5 GiB to 3.8… Nice!

It produced a slightly smaller archive than both partclone and fsarchiver (even when accounting for the non-excluded pacman cache), but it took 11.5 minutes. After testing, fsarchiver savedir took 3min08s and produced a 4.2 GiB image (with --exclude='*.pkg.tar.zst'), but then fsarchiver doesn’t produce browsable archives and doesn’t do any deduplication 😊

Rsync

One step more remote from system imaging, the ever powerful Swiss-army knife of file transfer can evidently also be used to create full system backups. Here is my personal take on it:

rsync \
    --archive \
    --hard-links \
    --acls \
    --xattrs \
    --atimes \
    --open-noatime \
    --sparse \
    --human-readable \
    --info=progress2 \
    --partial \
    --exclude={"var/cache/pacman/pkg/*","*/lost+found"} \
    /mnt/* /path/to/backup/dir

Or as a one-liner:

rsync -aHAXUUSh --info=progress2 --partial --exclude={"var/cache/pacman/pkg/*","*/lost+found"} /mnt/* /path/to/backup/dir

See its man page for details on the options used. Note that the repeated -U is not a mistake but a shortcut to enable both --atimes and --open-noatime 😉

However, while doing so I realized rsync wasn’t exactly the best solution when the destination filesystem is {ex,}FAT… Timestamps and permissions/ACLs/xattr won’t carry over correctly¹ 😒 This is what got me for the very first time interested in using…

Tar

O venerable Lord of computerized backups! Thy name is reminiscent of the glorious days of yore, when backups were mere tape archives…

Okay, but how does it fare in 2022 for complete system backup with high fidelity? After delving into its manpage this is what I came up with:

cd /mnt
tar --create \
    --preserve-permissions \
    --acls \
    --xattrs \
    --atime-preserve=system \
    --exclude={"var/cache/pacman/pkg/*","*lost+found"} \
    -I zstdmt \
    -f /path/to/backup/dir/<backup-name>.tar.zst \
    *
tar: /var/run/nscd/socket: socket ignored
...
# repeat dozens of times

Hmmm okay, apart from socket files which apparently clearly don’t matter, what about the completeness of the backup? Sections verify and compare from tar manual cover this question. A few pitfalls to be aware of however:

• --verify (only for archive creation) doesn’t work with compressed archives; if you don’t mind you can first create and verify an uncompressed archive, then compress it in a second step…
• --compare “ignores files in the file system that do not have corresponding members in the archive”, so if something was left over during the backup, no errors will be reported.

If you have the space and will for it, you could also just extract the archive and diff --recursive it against the source, but all in all I feel this is an aspect where the original (and still current) purpose of tar as a tape specific archiving tool, makes itself felt.

From my superficial testing everything seemed mostly alright, but I can’t shake the feelings that tar isn’t making as complete a clone as fsarchiver or rsync (see its limitations too). But that’s probably because my beard isn’t grey enough I guess… 😁

Wrap up

In conclusion, if you are in a situation where Clonezilla doesn’t work for you (most notably when using LUKS), my recommendation would be to turn to either FSArchiver, partclone or SquashFS. They seem to provide the most complete and reliable feature-set of the lot for imaging systems in a correct and efficient way. Rsync and tar being more general purpose tools, they require more options not to leave something out or otherwise shoot yourself in the foot.

Also, while laboring over this article I realized that while it was all fine and dandy to have a static archive of my old system, the next step was to make an interactive image of it that I could boot on whenever I want to see which GTK theme I used, or otherwise take a nostalgic whiff and relapse in my former computing abode…

This reminded me of the Moving an existing install into a virtual machine article, and I have an entire article coming up with the fruits of this endeavour: creating a VM image from the backups made here, testing in the process the oh-so-important restoration side of all the different tools used here (which taught me that an un-restored backup is no backup at all), making the required adjustments to be able to boot from it… Stay stuned for the second part! [edit: there it goes]

Thanks for reading and happy imaging 🙏