Remote rescue without a KVM
Context
I was looking for a way to get remote access to my laptop-turned-server even in the case the OS became unbootable (e.g. if I try to remotely encrypt the root and bork something), or more generally have a way to reinstall it from afar.
In other words, I wanted to emulate the functionalities of a KVM/cloud shell, but on consumer hardware with no remote management functionalities (e.g. vPro).
First pass
My first idea was to leave a USB drive containing a Live rescue system plugged into the laptop. The issue then becomes: how do I access the rescue environment?
One has to find either a Live system with sshd enabled by default and a default password (risky), or make/customize one so that either user-configured credentials are used (e.g. an SSH public key) or some kind of tunnel is opened automatically from the live system (I was thinking about unattended Tailnet joining, but as it sounded quite more convoluted than a simple preconfigured SSH key I didn’t explore this path further).
Searching for “live distro with ssh pre enabled” led me to two Q&A threads that proved quite useful:
- https://askubuntu.com/questions/821009/most-simple-way-to-create-a-bootable-usb-stick-for-headless-installation-with-ss
- https://unix.stackexchange.com/questions/43012/remote-accessible-live-distribution-aka-live-cd
Some specific Live systems were mentioned for their easy configurability:
- Plop: https://www.plop.at/en/ploplinux/live/faq.html#faq2
- grml: http://blog.grml.org/archives/367-Create-a-Grml-ISO-image-with-your-own-ssh-keys-for-password-less-login.html
However, I then discovered that my beloved Arch had cloud-init support baked into its regular ISO, which I am already familiar with and know to be very versatile, so I went for that.
The chosen path
As usual the Arch Wiki didn’t disappoint, offering short and practical instructions on how to use cloud-init to configure user-defined SSH keys and repacking the Live ISO so that I could burn it to my USB drive.
Tip
The “single ISO” method isn’t compatible with a default Ventoy configuration; it didn’t recognize the appended partition containing cloud-init data, and therefore it didn’t behave any different than the standard archiso. That’s why I went for burning the ISO to the drive.After making sure I could SSH with my SSH key into the live system from the local network, I set up port redirection on the router to forward SSH requests to this machine (which I had already set up a static DHCP lease for previously).
The last and trickiest part was: how to boot on the live system only in case it’s needed?
Deciding on the boot order
Configuring the firmware to always boot from the USB drive wouldn’t work, because unlike with a proper KVM I don’t have access to the bootloader screen to choose booting from the first drive/continuing to boot normally.
I certainly can configure the next boot to use the USB drive just once by using efibootmgr -n/bootctl on the installed system, but what if the server isnt accessible anymore? Then game over, I either need a benevolent human or a domestic robot. Or, you know, pay hundreds of €€€ to fly there myself 😂
In this case this laptop-server stays at some relatives’ place, so I can rely on them to punctually press F2 to select a different boot entry.
For those who are not so lucky, I cannot think of any other parry than going with something like a Jet/PiKVM; after all, this is exactly why server motherboard/CPUs usually come with remote management features like vPro.
Conclusion
Even though it will require a third-party intervention for abnormal/rescue situations, I am pretty happy I made this solution available on my setup. It provides some easy-to-trigger reassurance. Without it it’d feel like abandoning my brave laptop-turned-server to its own device (ha!) in a chilly closet under the stairs, like an electronic Harry Potter… This way at least, I feel like I can care for it.