Outsourcing space hoggers on my Kubernetes cluster

The Issue

Because of my kubernetes cluster only being virtualized on Cloudia and thus only having 16GB root drives located on the proxmox host root drives, I encountered disk pressure and ephemeral storage issues a few times. So far they never gave me a reason to actually take care of it, because I picked the lazy route whenever this happened. The lazy route being downscaling of deployments, deleting cached image layers and other stuff that does not need to be stored on the nodes, essentially relieving the issue and making kubernetes behave and schedule pods on nodes again. I was aware of this not being the ideal solution and my need to take care of it eventually, but it worked for me for a long time. But ‘eventually’ came recently, when a friend of mine and me started to discuss a project they were developing. I got curious and wanted to try it myself, so I asked them if they could just upload the source to my GitLab, add a pipeline to it and let my instance runner handle the work until the completed images are stored in my registry. So they got to work, designed a pipeline, pushed the code, aaaand: the jobs could not be processed, due to ephemeral storage being full and thus runners not being able to be scheduled. So I had no choice but to deal with the elephant in the room and come up with a solution.

The Idea

My solution: offloading big storage hoarders onto my Ceph cluster, until I can find the nerve to set up a proper pull-through image cache registry like Harbor or registry2 to cache image layers in my local network. That would not resolve the ephemeral storage issue though, so parts of the following approach will have to stay in place.

My goal: I wanted to offload log data into a centralized location, so I can view and read the logs with my Arch desktop without having to dig through virtual machines with SSH and journalctl. Also I wanted to store image layers per node on Ceph, so storage gets consumed where plenty is available, same with the ephemeral storage, which in my case matters most for gitlab runners (think about compilation artifacts, job execution logs, compilation dependencies, and so on).

Definitions and Tradeoffs

With my requirements defined, the next question became how much stuff will be stored and where it should live. I took some wild guessing into the blue and started with the easiest of those requirements - the log data offloading. Since log files essentially boil down to being append-only file interactions, CephFS became the easiest and laziest choice. It handles the file I/O just fine while scaling with remaining storage space on the cluster and gives me the ability to read the logs from my desktop by just mounting the filesystem on it as well. For the image layers and the ephemeral storage, I started with more educated guessing and opted for 32GB RBD volumes for the image layers as well as 128GB RBD volumes for the ephemeral storage on the worker nodes. Since the control plane nodes don’t run heavy workloads or interact with large amounts of data, I chose to keep them a little more on the lean side with another 32GB. Why RBD? Because every node essentially needs to have dedicated access to the storage on their own behalf without concurrency being the one messing up everything in the long run, so either I dedicate a specific folder on a CephFS, or just go the sane route and pick block devices with mutually exclusive access by default. The latter being the easier and more performant way by giving me a block device I can format the way containerd and kubelet prefer it to and not having to fight POSIX semantics, although concurrency was the driving motivation. I had no interest in opening a can of worms on concurrent image layer access.

One thing I am not planning to touch for a long while: etcd. It stays where it is, completely untouched and ignored. That’s a rabbit hole I’ll dig into way later, or, most likely, never.

The tradeoff by doing all this: Ceph now is no longer just a peripheral piece of the stack, but becomes THE carrying leg of Kubernetes. If Ceph has issues, Kubernetes is directly affected by it. For my specific environment, where boot order is handled by Proxmox, Ceph being online before Kubernetes is even allowed to boot, that is an impactful risk I am willing to take. I don’t tend to poke around on Ceph and see what happens just for the fun of it. Just relying on boot order and ideal scenarios isn’t really feasible either, so some form of fallback mechanisms need to be in place nonetheless. If a node is unable to mount Ceph stuff, it should gracefully degrade, instead of tipping over and crying. But - as always - I still managed to create a failure scenario not accounted for.

The Solution: Staged Mounting

I started with creating two new pools on my Ceph, one for the log metadata and one for the log data and creating a CephFS named ‘logs’ on them. After trying to add a few more standby MDS daemons, I shot one to the moon by being impatient and stupid. I tried adding them with the Ceph orchestrator and missed a detail in the placement command, essentially scaling my existing five MDS daemons down to three, while having four active filesystems, resulting in some filesystems flapping due to not enough daemons being able to handle them. Fixing my mistake by rerunning the placement again - this time with seven daemons - I watched the filesystems come online again in a stable state. I also created a new user on my Ceph for the log collection, with scoped permissions to only the filesystem just created. It should store logs; it does not have any business interacting with other parts of my cluster, so it does not need more than that (least privilege at play).

When rolling out the generated keyring to the first Kubernetes VM that I chose to be the victim for my testing, I stumbled upon Ceph dependencies not being installed, as well as the ceph.conf file not being present. So I went on to run apt to install ceph-common and afterwards put the configuration into /etc/ceph/ceph.conf, so the Ceph kernel module knows how to reach my cluster. Then I created the path for the mountpoint for the logs and live-mounted the CephFS to this new mountpoint. After verifying the mount itself is working fine (with a little touch-ing a test file as well as confirming its presence with my desktop, which mounts the log filesystem as well) and does not produce any errors, I drained the node, stopped kubelet and containerd and got annoyed at having to install even more stuff, rsync in this case (why is rsync not a default on Debian yet?).

apt finished its job, so I got back to work. Beginning with rsyncing /var/log/pods over to the new location on Ceph, running diff against the mirror to confirm everything is present, healthy and the same, I made Claude be a good boy and give me a script that I can run automatically on boot as a service. The script serves the purpose of bind-mounting specified directories via a config file automatically, right in between the CephFS mount being completed and containerd as well as kubelet reporting for duty. For now, only /var/log/pods/ is automatically mounted over the local storage path, but as the need for it arises, I can just add more paths to the config and on restarting the service or rebooting the node, the service-script will just handle the mounting, essentially replacing the local paths with the CephFS log storage. My approach on this is backed by two requirements: One being my demand to not have to edit the fstab for every path I want to put on the remote (flexibility with the config file), the second being a way of gracefully falling back to the local path. I wanted to avoid having a dead - and presumably empty - mount location when Ceph is acting up, which would result in losing all log data to the void.

Right now the state ends up being me having two paths on the same VM, that should - because of the bind-mount - have the same content. One being the default local path /var/log/pods and the second one being /mnt/log/pods - the CephFS mountpoint from the fstab entry. Since I rsynced everything from local to remote and everything looked identical, I was skeptical about them truly being the same thing. I never worked with bind-mounting before, only ‘brute force overmounting via fstab’, which is an entirely different behavior and it felt so much like two different directories just having the same content, that I wanted to confirm it, truly, because everything about this approach and idea depends on it. Running mount and reading its output told me that the service successfully bound /mnt/logs/pods to /var/log/pods. But due to the nature of bind-mounting it, I still wasn’t sold on it truly being linked together and the same. Hence why I also ran some copious amounts of du and df against both paths, as well as stat-ing them to look at the inodes. They matched and this finally gave me the certainty to accept, that a better confirmation for the process working as intended would be hard to get, so I left it at this.

Cloud Nine

Fingers crossed, I restarted the containerd and kubelet services and watched the journal for anything that looked out of order, but - prayers got listened to - nothing crashed, nothing failed and after uncordoning the node (a leftover of the drain earlier) Kubernetes started scheduling stuff on it, as well as the baseline noise daemonsets came online. But for me, the biggest smile-moment came when I refreshed the mounted logs dir on my Arch desktop and saw the running pods log dirs being created and updated, while old ones were being deleted - kubelet’s garbage collector doing its job. What a nice moment to finish the day and procrastinate on doing the same task on the other five nodes.

Two days later, I repeated all the steps again on the other nodes, so:

• install ceph-common, copy ceph.conf and the keyring over
• create mount paths
• add the fstab entry
• copy over the mount config, the bind-mount script and the systemd unit
• drain the node
• stop kubelet and containerd
• rsync local dir to Ceph remote & diff check it
• enable and start the services after triggering the mount
• restart containerd and kubelet
• uncordon the node and watch for errors if they occur

The Elephant

Next up: Creating the RBD images. For my six VMs, I needed 12 images, one for the ephemeral storage and another one for the image layer caching. As already stated at the beginning of this article, I opted for 32GB images for the image layer caching on all six nodes, but splitting the ephemeral storage into two different sizes, according to the workload. The control plane nodes will not run heavy data workloads and thus should be fine with 32GB of ephemeral storage, while the worker nodes needed more room to breathe (for the pipeline of my friend for example) and thus I chose 128GB. With that idea in mind, I began setting up the storage pool on Ceph where those images should reside as well as the images themselves. The usual stuff followed after, creating a user, applying permissions, rolling out the keyring file in preparation for the new mounts, just as I did it with the CephFS for the logs.

After editing the /etc/ceph/rbdmap file to tell Ceph (or the rbdmap service to be precise) what to mount and how to authenticate, I enabled the rbdmap service unit, started it and watched for the journal output. No crashes, so at least I didn’t fuck up too badly, but rbd showmapped didn’t show my mappings, so something was off. Sure enough, I mistyped something in the rbdmap file. Fixed it, restarted the service and voila: the mappings showed up.

Filesystem Choices

Having the images mapped does not make them usable. To the VM at this point it’s just a new, empty and uninitialized block device, so formatting them is on the menu now. Claude recommended XFS for those types of workload and I never questioned it up until this point of writing this post, but the reasoning ends up being roughly this:

Container image layers when unpacked as well as kubelet’s per-pod working directories produce a lot of small files relative to their total size, so even if not being - or even becoming - a problem on my homelab scale, EXT4 would have a real limitation: its fixed inode count at format time, based on bytes-per-inode ratio, which gets chosen when running mkfs - if not configured by passing in specific parameters to mkfs. XFS on the other hand handles inodes dynamically, while also parallelizing metadata operations better because of its allocation group behavior - which helps with many small files I/O. Because ‘small file I/O’ will be exactly the type of workload that occurs when pulling and unpacking image layers as well as pod lifecycles with their configmaps, secrets, and so on, it essentially becomes the obvious choice here.

Downside of XFS in that scenario: I can’t shrink the volumes - XFS does not support that. EXT4 would have worked fine in my scenario and would have given me the option to shrink, but considering that I have built all this to expand storage, not to remove it, I assume that the Venn diagram of “storage on my Ceph getting low” and “me thinking about shrinking the storage of Kubernetes and containerd” will have very little overlap.

Or in short: XFS got picked as “the right tool for the job”, rather than “EXT4 would have collapsed or created issues”.

Next up: more fstab editing. I added the mount entries for the rbd images with their stable symlink paths rather than relying on the “/dev/rbd0” and “/dev/rbd1” mappings, for the same reasons as why anyone would rather opt for using Part-UUIDs or UUIDs over “/dev/sda1”. It depends on discovery order and can mix up mounting behavior, something I would like to avoid having issues with.

Breaking the Law

Then more of the same followed. Stopping kubelet and containerd again, rsyncing over containerd’s data to the image cache volume, verifying with diff, business as usual. After verifying everything was synced over, I edited the config.toml of containerd and changed its root path to the new mount point of the remote. This created the first violation of my premise of things degrading gracefully instead of failing, because I essentially just switched over the root path to the mount location I created and set up via the fstab. If the mount fails, containerd will not have its data where it is expecting it and thus be unable to do its job. I will have to set up the bind-mounting for containerd as well at some point, but for now I’ll just leave it as is, a known risk I have to keep in mind.

After starting containerd again, looking at the journal and verifying it does not crash, I downloaded crictl (why ever it wasn’t present on my Kubernetes nodes) and checked the known images to that node and - happy moment - they were all present as if nothing had ever changed. One step closer to the finish line. Then I deleted the original containerd folder on the root drive of the VM to get my 7GB of storage back and avoid dead data.

Following the Rules Again

For kubelet I wanted to change the root-dir in the same way, since it seemed like the easiest way to accomplish the goal, just as I did it with containerd. But after some trying and poking around and realizing that a lot more things depend on /var/lib/kubelet as the base path (Ceph CSI for persistent volume handling, for example), I eventually abandoned that idea for several reasons, the first one being kubelet not finding its configuration file, another being Ceph CSI running as a daemonset. The latter became the bigger problem, because I would either have to deploy a second daemonset with node affinity settings just to isolate the “done” nodes from the “still open for modification” nodes, or have to redeploy the entire daemonset after changing all nodes. And after realizing that both approaches were not fitting my taste, as well as me needing to cold boot the cluster anyway at this point, I bit the bullet and went with the same approach as for the logs, bind-mounting over the local path, which has the same graceful degradation as the logs have and which containerd is missing right now.

To spare myself a lot of copy-pasting later down the line, mixed with retracing what I did on which VM, I began to work in parallel on all VMs at once at this point. So I had to catch up with the other five VMs to the state of my testing victim, resulting in me adding the relevant rbdmap entries, adding the new keyring for the rbd images user, enabling the rbdmap service and so on. As I caught up, and every node had its containerd located on Ceph, I created the required mount point for the ephemeral storage, rsynced kubelet’s lib folder to the remote, and diff’ed it. Then I did even more of the same: bind-mounting it to the local path, creating the required fstab entries on all six nodes (again), and creating a new systemd unit as a oneshot task, that runs after rbdmap finishes and before kubelet starts, just like the logs setup and for the same reasons.

Having completed all this on all six nodes, I re-enabled kubelet, made sure containerd was enabled on every node as well as the services and then began to reboot the nodes one by one. After all nodes had booted up, I verified the mounts being in place where they should be, the custom services being executed and doing their job as well as containerd and kubelet not complaining. Good place to be in. Next up was bootstrapping the control plane API server again, due to a little chicken and egg issue resulting from having kube-vip as a daemonset running and providing the API server on a virtual IP address, which is only available after the daemonset is actively running. But that’s a story (and a fix) for another time.

Cloud 9.1

When the cluster came back online (I already know how to deal with the chicken and egg; for now, I just need to implement an automatic solution for it), I tasked my friend with a new pipeline trigger. The result: It just works, which is bliss after this endeavor. The only thing I had to change after all this: the image cache volumes got crammed really fast with about 25GB usage on 32GB images, just from their pipeline run. So I opted to type some more on the command line and increase the volume sizes to 64GB on the worker nodes, refreshing the RBD read-in on the VMs and then expanded the XFS to use the full 64GB. For now, that’s the stopping point of - at least - this journey.