Problem statement

As part of dabbling with self-hosting again, I installed CasaOS on an Oracle Cloud free Ampere instance to try it out.

After setting it aside for a few weeks, when I logged in via SSH and tried to use sudo I realized I absolutely couldn’t remember my user’s password 😅

The standard operating procedure in this case is to either reboot the machine on a live system and use that to chroot into the local install, or fiddle with GRUB rescue/kernel command line.1

But since I could still install containers through CasaOS web interface, I thought I’d use this opportunity to explore ways to recover my sudo access/change my password without rebooting (which could be nice in case it is essential to avoid downtime).

Edit: As I was finishing this post, I realized that the third constraint listed below was incorrect, and as a result a less cumbersome resolution was possible. So read on if you are interested in the learning journey, or jump straight to the simplified solution.

The constraints

  • the CLI/SSH access is “useless”: my user is not setup to interact with the Docker socket without elevated privileges (which is usually a good thing)
  • besides its App Store content, CasaOS allows to manually install containers using a Compose file or a docker run command (which is then composerized): see official video or screenshots of the process
  • its interface however doesn’t allow to docker exec or interact with containers in any other way; the installed containers must expose a web interface for us to be able to interact with them. As it turns out, CasaOS interface does offer access to containers’ console (equivalent to a docker exec -it <containe> /bin/sh), at least for apps installed from its official Store, via the app Settings > “Terminal and Logs” icon.
Access to a container shell and logs in CasaOS interface

Access to a container shell and logs in CasaOS interface

Hacking around

Because I’m pretty new to this, my first idea was to start a container which would give me a root shell with access to the docker socket, and then use that environment to start a second privileged container (Docker-in-Docker style) from which I would mount the host filesystem and use {ch,}passwd or such… Yes, I felt very smart thinking this up 😅

Unaware of my mistaken approach, I looked online for a “web terminal container” and the first result on StartPage was the web terminal GitHub repo, which did what I needed (expose a root shell in the browser) but hadn’t been updated since August 2021 😕

StartPage results for web terminal container
Aside

Docker Hub search is appalling: searching for web terminal gave me a bunch of completely unrelated results. I had to put a dash between the two words to get anywhere, but then the two first results hadn’t been updated in 2 years either…

Searching for web shell and "web shell" didn’t give better results, while web-shell and webshell led to very outdated stuff without any description. I can’t believe there isn’t a reference implementation of this stuff!!

I also looked up ttyd, the software used by web-terminal, and here the first result was fresh from a few days ago; however there was no mention of “How to use this with Docker” , so I wasn’t sure a simple docker run would achieve the desired result… Had I checked its Dockerfile, I would have seen that its ENTRYPOINT/CMD was, in fact, starting up the service 🙃

Lo and behold, I took the “risk” (after checking the image’s Dockerfile) and entered the following command in CasaOS Docker CLI interface:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 7681:7681 raonigabriel/web-terminal:latest

After setting up the access port in CasaOS interface and confirming everything was okay, I clicked “Install” and connected to my server’s port 7681 in a browser… Success, I was in!

For the next stage of my plan I ran the following command inspired by this StackExchange thread:

docker run -ti --privileged --net=host --pid=host --ipc=host --volume /:/host busybox chroot /host
/bin/sh: docker: not found

Crap, I didn’t even check that the container had docker installed 🤦 You can tell I really don’t know what I am doing 😂

Of course when I subsequently tried to install docker…

root@1d0fa4b64c55:/$ apk add docker
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
  docker (missing):
    required by: world[docker]
root@1d0fa4b64c55:/$ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 36 distinct packages available

A quick search seems to indicate that the image is simply too old. Building an updated image was out of the scope I had set for this experiment, so I paused and took time to think.


That’s when it occurred to me that this “nested container” approach was completely useless, and would most likely not work since at that point you have the first container’s virtualized filesystem namespace acting as a buffer between the host and the DinD container… Complete misdirection, backing up!

The solution

After thinking it through some more, I realized one could achieve the desired outcome by simply mounting the host / read-write as an attached volume 😁

docker run -d -v /:/host -p 7681:7681 raonigabriel/web-terminal:latest

Et voilà, I finally had a root shell from which I could chroot into the host and update the user password and/or configure password-less sudo (which amounts to the same). Or really, do (nearly) everything to the host system 😨

Root shell provided by the web-terminal container

Root shell provided by the web-terminal container

Initially I thought it would be necessary to use a privileged container, but trying it showed that wasn’t the case. I guess it’s because I was only editing files/using regular utilites, and not trying to create new devices/nodes etc.

The solution: simplified

This is what happens when you don’t know your tools enough… You miss very obvious pathways that lead to simpler solutions 😁

Indeed, there is no need to install a third-party container. It is enough to add the /:/host volume to an existing app installed from CasaOS official App Store (e.g. NextCloud), and after making sure in its settings it runs with the root UID/GID, we can use the interface to connect the container’s console!

Root shell inside nextcloud container

Root shell inside nextcloud container

And that is the Easy Way© to get a root shell on your server using containers!

Trying to refine the solution

Now that I was there, I felt like I should have been able to use a simple busybox image to execute a command non-interactively directly from docker run/Compose file, eliminating the need to find an image that exposes a web service or to connect to the container’s console.

After a bit of fiddling, I came up with the following invocation:

docker run -v /etc/sudoers:/host/etc/sudoers busybox /bin/sh -c echo '%sudo  ALL=(ALL) NOPASSWD: ALL' >> /host/etc/sudoers

But CasaOS interface constantly threw an error. Probably the redirection in the command, but no amount of quoting led me to a successful run.

At that point I threw the towel in: I had recovered access to my sudo access and learnt quite a few things along the way. Time to wrap up.

Takeways

  • Docker default security story is scary: because the daemon runs as root by default, the mere ability to run containers (even without --privileged) and mount any desired host path as a volume gives root-equivalent status
  • Due to this, access to CasaOS interface is equivalent to root access on the server

That’s all folks, thanks for reading this account I hope you enjoyed!