Docker Desktop the Hard Way

I guess one of the many reasons I never considered myself to be a Real Developer™ — just "a guy who writes code to automate things" — is my habits and preferences aren't what I observe to be the norm for Real Developers™.

•  I fucking hate Apple products: I want my computing environment to be fully and wholly tailorable to my my tastes. That and Apple hardware's keyboards, mice and track-pads are trash and their WM sucks. Since I can't have a NeXTish (closest you get, nowadays, is GNUstep under WindowMaker; but back in the early 00s, used to be able to run a shell-replacement that gave you an OpenStep type of launcher) or 4Dwm (SGI) style window-manager, I'd much rather Windows' interface. While it's not my ideal, it's at least got the key-bindings I prefer. Plus, I can still easily get devices with a full US101-style keyboard (everyone's gone to "buttonless" track-pads, so I'm doomed on that front whenever I'm forced to change off). It's why I've been resisting my company's attempts to issue me a work laptop since they decided that Apple hardware was a great perk to attract new hires with …and subsequently implemented centralized device management (MDM) to meet compliance goals.
• Resultant of the immediately prior, I use
• A Dell laptop (and an HP before it)
• I run Windows 11 Pro (though, when I bought it in spring of 2020, it came with and was upgraded from Windows 10)
• I would run a Linux-based desktop, but, when I bought the current laptop, the options for laptops preloaded with Linux were quite limited (and I just didn't want to have to ass around with it). If I had the money and time, I'd try to run a hypervisor as my bare-metal operating system, then, as my default VM, have an RPM-based distro like Fedora, Alma or Rocky as its OS. Professionally, my customers are RPM-based, so, just like in my SGI and then Sun sysadmin days, I tried to minimize the cognitive-load of OS-switching by running equivalents on my personal systems. 
• That said, my choice of Windows as my OS being a matter of convenience, I don't really use Windows as much more than a hosting environment.
• I use WSL2 for my "daily-driver" working-environment. My main WLS2 instance runs Oracle Enterprise Linux 9 (at the time, it was the only free EL-style instance available and I didn't want to go the Ubuntu route). I run an OpenSSH daemon inside it so I can use PuTTY to login to it. 
• I use Hyper•V on the occasions where I need to run a full VM. Those occasions are fairly rare, thoug. Mostly, I used them when I need something that runs DBUS/systemd or needs to interact with something closer to "real" devices (e.g., when I'm having to prototype automation for services get pissed when there's no DBUS/systemd bits)
• I use PodMan, instead of Docker, inside my WSL2 or Hyper•V VMs.
• I run work-related tooling — be that browsers, tools like Teams/WebEx/etc. — either from my WSL instance or a full VM. I generally believe in keeping work stuff siloed off from my main OS

Recently, a new project popped up on my work RADAR. That's going to require using something other than either a WSL-encapsulated PodMan or a Hyper•V. Instead, for the early phases, I'll need to do at least some tasks in Docker Desktop (which, on Windows, is underpinned by Hyper•V).

I'm not really a GUI guy nor do I want to have to dick around with CMD.EXE or PowerShell to manage Docker. I much prefer to do everything from a BASH prompt and to use vi and other tools that I have two to three-and-a-half decades of finger-memory for.

 On the plus side, Docker Desktop has native integration with WSL2. You just have to enable it:

 

On the minus side, if, like me, you've installed the podman-docker RPM into your environment, it makes interacting with DockerDesktop from your WSL2 CLI session a bit more of a bother to set up. It requires:

• Knowing where DockerDesktop has created its docker command within WSL
• Knowing where DockerDesktop has place its UNIX domain-socket into WSL 

Once you know those — with Docker Desktop 4.77.0, those were "/mnt/wsl/docker-desktop/cli-tools/usr/bin" and "/mnt/wsl/docker-desktop-bind-mounts/OracleLinux_9/docker.sock", respectively — you need to tell your shell where to find them. For the former, you extend your "PATH" and for the latter you alter your "DOCKER_HOST" (shell envs).

That you have to know these things is a side effect of the podman-docker RPM wanting to manage many of the same things that the Docker Desktop insertions into the WSL 2 instance wants to take care of.

The other fun thing with the setup is that you have to keep re-defining "PATH" and "DOCKER_HOST" shell envs. That's not generally something I want to be bothered with or, more importantly, remembering. The "remembering" thing is why I write articles like this one.

Since my need for either PodMan or DockerDesktop is sporadic, I'd rather things like shell-envs be self-maintaining. A great way to do that is by using direnv. With that tool, all I have to do is drop my shell-envs into a ".envrc" file hosted in a specific directory(-tree) and, whenever I "cd" to that directory, I get the settings I desire. Podman is my default, so, for Docker Desktop, I only need to manage that in a top-level directory at the root of my "${HOME}". In my case, that directory is "DockerDesktop" (or "~/DockerDesktop" in BASH-ese). So, my "~/DockerDesktop/.envrc" ends up looking like:

PATH_add /mnt/wsl/docker-desktop/cli-tools/usr/bin
PATH_add .bin
export DOCKER_HOST=unix:///mnt/wsl/docker-desktop-bind-mounts/OracleLinux_9/docker.sock

That "PATH_add .bin" is the last bit of metaphorical "glue". It allows me to define a simple shell-wrapper at "~/DockerDesktop/.bin/docker-compose", allowing me to type "docker-compose" rather than "/mnt/wsl/docker-desktop/cli-tools/usr/local/lib/docker/cli-plugins/docker-compose" ...whenever I want to run "docker compose ..." to interact with Docker Desktop instead of PodMan. And, when I say "simple" I mean:

#!/bin/sh
exec /mnt/wsl/docker-desktop/cli-tools/usr/local/lib/docker/cli-plugins/docker-compose "$@"

I assume there's probably even cleaner and/or less-effortful approaches, but this is what I was able to most-quickly cobble together, this morning

WSL Space Recovery

Recently, the backup client I use for my laptop started popping up, "operation aborted: out of space" messages. I'd been confused because, the last time I'd looked — just a few days prior — I had over 100GiB of free space in my boot-drive. Yet, when I looked at my disk-utilization, I only had a few tens of MiB free. WHAT??

So, I began the process of tracking down what was suddenly chewing up all my disk-space. Ultimately, I found a nearly 70GiB "ext4.vhdx" buried deep within my Windows home-directory's "AppData" hierarchy.

Did a quick web search and found that what I was seeing was the virtual hard disk for my WSL2 instance. This confused me because I'd been pretty scrupulous in keeping my WSL2 instance's storage in check. In fact, when I checked, my WSL instances visible storage was only 23GiB, nowhere near the 70GiB+ of the "ext4.vhdx" file that was backing that 23GiB of visible storage-usage.

Further Googling turned up that WSL doesn't really reclaim freed storage. So, that differential between visible storage an the size of ext4.vhdx" file was effectively wastage. Presumably my virtual hard drive was significantly sparse.

Next thing I looked up was "how to reclaim wasted space in a WSL drive." Ultimately discovered that I needed to:

1. Stop my WSL instance
2. Back up my WSL instance
3. "Optimize" my instance's hard drive
4. Restart my WSL instance

The first step was dead easy: just fire up a command.exe (or PowerShell session), then issue a `wsl --shutdown`.

Second step was also fairly easy, as it was something I was doing every few months, any way: while my backup client should be backing up the virtual hard disk, I don't trust that those backups are anything beyond "crash consistent". At any rate, I took the opportunity to do a:

wsl --export <WSL_DISTRO> G:\WSL_Backups\<WSL_DISTRO>\backup-$( date '+%Y%m%d' ).tar

Once I found articles on the subject, the VHD compression was also pretty straight-forward (and, thus far, no need for the backups):

1. Open a PowerShell window (the optimization-command is a PowerShell commandlet) 
2. Navigate to the directory hosting the VHD file 
3. Execute Optimize-VHD -Path .\ext4.vhdx -Mode full.

The optimization crushed the VHD file back down to a hair larger than the (internal to the instance) disk-size. 

Restarting my WSL instance is just going into my Search box and typing in the name of my instance, then clicking on the menu-item that appears.

Situation fixed, my next problem was, "why the hell did I end up with such a huge disk-image in the first place". The answer to that didn't really come until a few days later when my VHD blew up again.

I primarily use my WSL instance for work-related stuff. Recently, I'd been using podman to do some — a bunch, actually — container work. Worse, some of that Podman-based container-work was resulting in Buildah images getting generated. Whenever I would run `podman system prune --all --volumes && podman system prune --external`, the tool would tell me I'd recovered 5-20GiB of space (particularly the --external runs). 

Those space-recovery numbers made it occurr to me, "are my podman activities blowing my disk up?" So, after a fresh `… Optimize-VHD …` run, I decided to see if I could intentionally provoke a "VHD is multiples of my visible-use" situation. And, yes, I could.

Moral of the story: while you can use WSL instances with things like Podman, doing so will likely make it so you'll need to habituate to doing more system cleanup activities. 

TIL: Podman Cleanup

Recently, I started working on a gig that uses Ansible for their build-automation tasks. While I have experience with other types of build-automation frameworks, Ansible was new to me.

Unfortunately, my customer is very early in their DevOps journey. While my customer has some privately-hosted toolchain services, they're not really fully fleshed out: their GitLab has no runners; their Jenkins is not general access; etc. In short, not a lot of ability to develop in their environment — at least not in a way that allows me to set up automated validation of my work.

Ultimately, I opted to move my initial efforts to my laptop with the goal of exporting the results. Because my customer is a RHEL environment, I set up RHEL and CentOS 7 and 8 VMs on my laptop via Hyper•V. 

Side-note: While on prior laptops I used other virtualization solutions, I'm using Hyper•V because it came with Windows 10, not because I prefer it over other options. Hypervisor selection aside…

As easy as VMs are to rebuild, I've yet to actually take the time out to automate my VMs' builds to make it less painful if I do something that renders one of them utterly FUBAR. Needless to say, I don't particularly want to crap-up my VMs, right now. So, how to provide a degree of blast-isolation within those VMs to hopefully better-avoid not-yet-automated rebuilds?

Containers can be a great approach. And, for something as simple as experimenting with Ansible and writing actual playbooks, it's more than sufficient. That said, since my VMs are all Enterpise Linux 7.8 or higher, Podman seemed the easier path than Docker ...and definitely easier than either full Kubernetes or K3S. After all, Podman is just a `yum install` away from being able to start cranking containers. Podman also means can run containers in user-space (without needing to set up Kubernetes or K3S), which further limits how hard I can bone myself.

At any rate, I've been playing around with Ansible, teaching myself how to author flexible playbooks and even starting to write some content that will eventually go into production for my customer. However, after creating and destroying dozens of containers over the past couple weeks, I happened to notice that the partition my ${HOME} is on was nearly full. I'd made the silly assumption that when I killed and removed my running containers that the associated storage was released. Instead, I found that my ${HOME}/.local/share/containers was chewing up nearly 4GiB of space. Worse, when I ran find (ahead of doing any rms), I was getting all sorts of permission denied errors. This kind of surprised me since I thought that, by running in user-space, any files that would be created would be owned by me.

So, I hit up the almighty Googs. I ended up finding Dan Walsh's blog-entry on the topic. Turns out that, because of how Podman uses name-spaces, it creates files that my non-privileged user can't actually directly access. Per the blog-entry, instead of being able to just do find ${HOME}/.local/share/containers -mtime +3 | xargs rm, I had to invoke buildah unshare and do my cleanup using that context.

So, "today I learned" ...and now I have over 3GiB of the nearly 4GiB of space back.