Selfhosting on Bastien Traverse

Using a container to sidestep a forgotten password in CasaOS

Sat, 03 Feb 2024 00:00:00 +0100

Problem statement

As part of dabbling with self-hosting again, I installed CasaOS on an Oracle Cloud free Ampere instance to try it out.

After setting it aside for a few weeks, when I logged in via SSH and tried to use sudo I realized I absolutely couldn’t remember my user’s password 😅

The standard operating procedure in this case is to either reboot the machine on a live system and use that to chroot into the local install, or fiddle with GRUB rescue/kernel command line.¹

But since I could still install containers through CasaOS web interface, I thought I’d use this opportunity to explore ways to recover my sudo access/change my password without rebooting (which could be nice in case it is essential to avoid downtime).

Edit: As I was finishing this post, I realized that the third constraint listed below was incorrect, and as a result a less cumbersome resolution was possible. So read on if you are interested in the learning journey, or jump straight to the simplified solution.

The constraints

the CLI/SSH access is “useless”: my user is not setup to interact with the Docker socket without elevated privileges (which is usually a good thing)
besides its App Store content, CasaOS allows to manually install containers using a Compose file or a docker run command (which is then composerized): see official video or screenshots of the process
~~its interface however doesn’t allow to docker exec or interact with containers in any other way; the installed containers must expose a web interface for us to be able to interact with them.~~ As it turns out, CasaOS interface does offer access to containers’ console (equivalent to a docker exec -it <containe> /bin/sh), at least for apps installed from its official Store, via the app Settings > “Terminal and Logs” icon.

Access to a container shell and logs in CasaOS interface

Hacking around

Because I’m pretty new to this, my first idea was to start a container which would give me a root shell with access to the docker socket, and then use that environment to start a second privileged container (Docker-in-Docker style) from which I would mount the host filesystem and use {ch,}passwd or such… Yes, I felt very smart thinking this up 😅

Unaware of my mistaken approach, I looked online for a “web terminal container” and the first result on StartPage was the web terminal GitHub repo, which did what I needed (expose a root shell in the browser) but hadn’t been updated since August 2021 😕

Aside

Docker Hub search is appalling: searching for web terminal gave me a bunch of completely unrelated results. I had to put a dash between the two words to get anywhere, but then the two first results hadn’t been updated in 2 years either…

Searching for web shell and "web shell" didn’t give better results, while web-shell and webshell led to very outdated stuff without any description. I can’t believe there isn’t a reference implementation of this stuff!!

I also looked up ttyd, the software used by web-terminal, and here the first result was fresh from a few days ago; however there was no mention of “How to use this with Docker” , so I wasn’t sure a simple docker run would achieve the desired result… Had I checked its Dockerfile, I would have seen that its ENTRYPOINT/CMD was, in fact, starting up the service 🙃

Lo and behold, I took the “risk” (after checking the image’s Dockerfile) and entered the following command in CasaOS Docker CLI interface:

docker run -d -v /var/run/docker.sock:/var/run/docker.sock -p 7681:7681 raonigabriel/web-terminal:latest

After setting up the access port in CasaOS interface and confirming everything was okay, I clicked “Install” and connected to my server’s port 7681 in a browser… Success, I was in!

For the next stage of my plan I ran the following command inspired by this StackExchange thread:

docker run -ti --privileged --net=host --pid=host --ipc=host --volume /:/host busybox chroot /host
/bin/sh: docker: not found

Crap, I didn’t even check that the container had docker installed 🤦 You can tell I really don’t know what I am doing 😂

Of course when I subsequently tried to install docker…

root@1d0fa4b64c55:/$ apk add docker
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
  docker (missing):
    required by: world[docker]
root@1d0fa4b64c55:/$ apk update
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/main: UNTRUSTED signature
WARNING: Ignoring APKINDEX.e37b76c2.tar.gz: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/aarch64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/edge/community: UNTRUSTED signature
WARNING: Ignoring APKINDEX.d022dfc8.tar.gz: No such file or directory
2 errors; 36 distinct packages available

A quick search seems to indicate that the image is simply too old. Building an updated image was out of the scope I had set for this experiment, so I paused and took time to think.

That’s when it occurred to me that this “nested container” approach was completely useless, and would most likely not work since at that point you have the first container’s virtualized filesystem namespace acting as a buffer between the host and the DinD container… Complete misdirection, backing up!

The solution

After thinking it through some more, I realized one could achieve the desired outcome by simply mounting the host / read-write as an attached volume 😁

docker run -d -v /:/host -p 7681:7681 raonigabriel/web-terminal:latest

Et voilà, I finally had a root shell from which I could chroot into the host and update the user password and/or configure password-less sudo (which amounts to the same). Or really, do (nearly) everything to the host system 😨

Root shell provided by the web-terminal container

Initially I thought it would be necessary to use a privileged container, but trying it showed that wasn’t the case. I guess it’s because I was only editing files/using regular utilites, and not trying to create new devices/nodes etc.

The solution: simplified

This is what happens when you don’t know your tools enough… You miss very obvious pathways that lead to simpler solutions 😁

Indeed, there is no need to install a third-party container. It is enough to add the /:/host volume to an existing app installed from CasaOS official App Store (e.g. NextCloud), and after making sure in its settings it runs with the root UID/GID, we can use the interface to connect the container’s console!

Root shell inside nextcloud container

And that is the Easy Way© to get a root shell on your server using containers!

Trying to refine the solution

Now that I was there, I felt like I should have been able to use a simple busybox image to execute a command non-interactively directly from docker run/Compose file, eliminating the need to find an image that exposes a web service or to connect to the container’s console.

After a bit of fiddling, I came up with the following invocation:

docker run -v /etc/sudoers:/host/etc/sudoers busybox /bin/sh -c echo '%sudo  ALL=(ALL) NOPASSWD: ALL' >> /host/etc/sudoers

But CasaOS interface constantly threw an error. Probably the redirection in the command, but no amount of quoting led me to a successful run.

At that point I threw the towel in: I had recovered access to my sudo access and learnt quite a few things along the way. Time to wrap up.

Takeways

Docker default security story is scary: because the daemon runs as root by default, the mere ability to run containers (even without --privileged) and mount any desired host path as a volume gives root-equivalent status
Due to this, access to CasaOS interface is equivalent to root access on the server

That’s all folks, thanks for reading this account I hope you enjoyed!

See e.g. https://gcore.com/learning/how-to-reset-password-in-linux/ ↩︎

Graphical management interfaces for Linux servers

Mon, 11 Dec 2023 00:00:00 +0100

Terminology considerations

“Server management interface”, “control panel”, “dashboard”… While investigating Free and Open Source graphical server management software, I had trouble establishing the categorical difference between the different solutions I was reading about.

My use case being a “single user homelab/server” and not a multi-tenancy reseller/web hosting solution, I initially left aside the well established category of control panels (with the exception of HestiaCP).

The projects that caught my interest were Cockpit, HestiaCP, Portainer, CapRover, CasaOS, Cloudron, and to a lesser extent, Yacht.

But try as I might, I just couldn’t wrap my head around what precisely distinguished them. Of course I could see that they had different focus: some were more sysadmin/general server management-oriented while others were more app- and Docker-oriented; but I lacked a general category theory for them.

My search-fu not yielding anything relevant, I thought I’d probe ChatGPT about it. This resulted in a most enlightening conversation, as it answered with a distinction that felt like a Eureka moment to me:

The solutions you’ve mentioned, including Portainer, HestiaCP, CapRover, CasaOS, Cloudron, and Yacht, represent a different category of software compared to the server management tools like Cockpit, Webmin, Ajenti, and others.

It then went on to provide the following breakup:

Server Management Tools (e.g. Cockpit, Webmin, Ajenti):

Purpose:

Primarily focused on server administration and configuration.

Provide a broad range of controls over system settings, users, and services.

Complexity:

Varies from beginner-friendly to advanced, catering to users with different skill levels.

Interface Type:

Often web-based but can include desktop GUI options.

Aimed at managing servers directly, not specifically for application or container orchestration.

Scalability:

Typically suitable for small to large enterprises.

Designed for overall server management, including security, networking, and system configuration.

Community Support:

Active open-source communities provide support and contribute to feature development.

Integration:

Extensive third-party module or plugin support for customization.

May include features for managing server-wide tasks and configurations.

Application and Container Orchestration Tools (e.g. Portainer, HestiaCP, CapRover, CasaOS, Cloudron, Yacht):

Purpose:

Focused on managing and orchestrating applications, containers, and services.

Streamlined for deploying and managing web applications and services in containerized environments.

Complexity:

Typically designed to be user-friendly, catering to developers and administrators with varying skill levels.

Interface Type:

Primarily web-based interfaces, emphasizing container and application management.

Often include features for easy deployment, scaling, and monitoring of applications.

Scalability:

Suited for small to medium-scale applications and services.

Designed for ease of use in deploying and managing containerized applications.

Community Support:

Active communities, often developer-focused, providing support and contributing to the development of container-related features.

Integration:

Strong integration with containerization technologies (e.g., Docker, Kubernetes).

Emphasis on managing application stacks, updating applications, and maintaining service availability.

Note that:

the content of some categories don’t really fit their header (e.g. “Interface Type”)
HestiaCP is lumped in the “Application and Container Orchestration Tools” category, despite its features being arguably more control panel-y/towards server management.

This is to just highlight the very reasonable limits of ChatGPT (3.5), in what was otherwise a super useful answer.

Coming up with a unified categorization

I kept poking at the prompt until I managed to formulate a General Category Theory of Server Management Graphical User Interfaces™, inclusive of control panels and mere dashboards. Here’s the result:

The terms “server management tools”, “control panels”, “application and container orchestration tools” and “server dashboards” are sometimes used interchangeably, but they often refer to different aspects of managing and monitoring servers. Here’s how you can distinguish between them:

Server Management Tools:

Purpose: Server management tools are software applications designed to facilitate the configuration, administration, and monitoring of servers. They offer a range of features for tasks such as user management, service configuration, and system monitoring.

Scope: Server management tools encompass a broad set of functionalities related to overall server administration, configuration, and maintenance.

Examples: Cockpit, Webmin, Ajenti and ISPConfig are examples of server management tools.

Control Panels:

Purpose: Control panels are specialized interfaces that provide centralized control over specific server functionalities, often in a web-based environment. They are designed to simplify the management of specific server aspects, such as web hosting, email services, or database management.

Scope: Control panels are more focused and may not cover all aspects of server management. They often excel in specific use cases and are user-friendly for individuals who may not have extensive server administration experience.

Examples: HestiaCP, Froxlor and Virtualmin are examples of control panels.

Application and Container Orchestration Tools:

Purpose: These tools aim to streamline application deployment, scale applications automatically to handle varying workloads, provide centralized management for containers, and automate various aspects of deployment and updates, ultimately enhancing overall efficiency.

Scope: Their scope includes container orchestration, service discovery, load balancing, health monitoring, and automated rollbacks and updates. They focus on managing and coordinating the deployment, scaling, and operation of application containers in a containerized environment, ensuring seamless and efficient operations.

Examples: Portainer, CapRover and Yacht are examples of application and container orchestration tools.

Server Dashboards:

Purpose: Server dashboards are visual representations of real-time or historical server data, allowing users to monitor key metrics and performance indicators. They provide a quick overview of the server’s health and status.

Scope: Dashboards are primarily focused on providing visibility into server performance metrics and may not offer the same depth of configuration and administration features as management tools or control panels.

Examples: Grafana, Kibana, and custom dashboards created using tools like Prometheus or Nagios are examples of server dashboards.

In summary, while there is some overlap in functionality, server management tools generally provide a comprehensive set of features for overall server administration. Control panels are more specialized, focusing on specific server functionalities. Application and container orchestration tools are geared towards managing and orchestrating applications and containers, ensuring efficient deployment, scaling, and maintenance. Server dashboards, on the other hand, offer visual representations of server performance metrics for monitoring purposes. The choice between them depends on the specific needs and preferences of the user or organization.

So, even though ChatGPT kind of threw my FOSS requirement to the wind by mentioning famous proprietary panels (that I replaced here), all in all I feel like we did a great job coming up with a good typology of graphical, general purpose GNU/Linux server management solutions (what a mouthful 😅).

Category members

Now that our categories are well defined, it is time to flesh them out by filling them with our candidates!

Without further ado, here’s a curated list with stuff somewhat beyond my initial use case:

Server Management Tools:

Cockpit Project
- License: LGPL 2.1
- Purpose: Web-based server administration and monitoring.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for both SMEs and large enterprises.
- Community Support: Growing community.
- Integration: Focus on integration with systemd and containers.
- GitHub Repository
Webmin
- License: BSD
- Purpose: Comprehensive web-based server configuration and management.
- Complexity: Varies from beginner-friendly to advanced.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs and large enterprises.
- Community Support: Active community.
- Integration: Extensive third-party module support.
- GitHub Repository
Ajenti
- License: AGPLv3
- Purpose: Web-based server administration with support for plugins.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Supports various plugins for extended functionality.
- GitHub Repository
ISPConfig
- License: BSD
- Purpose: All-in-one server management panel.
- Complexity: Intermediate to Advanced.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs and large enterprises.
- Community Support: Active community.
- Integration: Strong support for multi-server setups.
- GitHub Repository
Zentyal
- License: GPLv2
- Purpose: Linux Small Business Server.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Offers a wide range of server modules.
- GitHub Repository

Web hosting/Control Panels

HestiaCP
- License: GPLv3
- Purpose: Web hosting control panel.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages website hosting, email services, and server configurations.
- GitHub Repository
Froxlor
- License: GPL
- Purpose: Server management panel with a focus on web hosting and server administration.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages web hosting, domain configurations, and server settings.
- GitHub Repository
Virtualmin
- License: GPL
- Purpose: Web hosting control panel with a focus on managing virtual hosts.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages website hosting, email services, and server configurations.
- GitHub Repository (Note: Virtualmin is primarily available through its official website, and its source code is available on GitHub for specific components.)
CyberPanel
- License: GPL
- Purpose: Web hosting control panel with a focus on performance and security.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages website hosting, email services, and server configurations.
- GitHub Repository (Note: CyberPanel is primarily available through its official website, and its source code is available on GitHub.)
AlternC
- License: GPLv3
- Purpose: Web hosting control panel with a focus on environmental responsibility.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- GitHub Repository

Application/Container Orchestration Tools:

Portainer
- License: AGPLv3
- Purpose: Docker container management.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages Docker containers with a user-friendly interface.
- GitHub Repository
CapRover
- License: Apache License 2.0
- Purpose: Automated deployment for web apps.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Focus on automated deployment of web applications.
- GitHub Repository
CasaOS
- License: AGPLv3
- Purpose: Server management and automation.
- Complexity: Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Provides server management and automation features.
- GitHub Repository
Cloudron
- License: Proprietary
- Purpose: Self-hosted web app platform.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Manages self-hosted web applications.
- GitHub Repository
Yacht
- License: MIT
- Purpose: Docker management with a focus on simplicity.
- Complexity: Beginner to Intermediate.
- Interface Type: Web-based.
- Scalability: Suitable for SMEs.
- Community Support: Active community.
- Integration: Simplifies Docker container management.
- GitHub Repository

Now it’s experimentation time! 🥳

PS: If you need even more, here is a random article listing 30 similar solutions including many dashboards.

Happy hacking!