Docker Desktop the Even Harder Way

This post is a brief followup to the prior "Docker Desktop the Hard Way" post.

My laptop's main disk is a bit tight on space It's got enough space for my day-to-day activities. It even has enough space to run a single OpenClaw instance. However, since I intend to run multiple OpenClaw instances and especially, since I think that, at some point, I want to also try running Ollama, the amount of remaining free space on my main disk is completely insufficient.

I have number of USB-attachable SSDs. So, I opted to move my containers' storage onto one of them. Unfortunately, doing so adds further wrinkles to the "hard way" of doing things. My USB-attached SSD is an NTFS-formatted device. It's got a non-trivial amount of other data on it, meaing I don't want to reformat it. Unfortunately, bridging its storage through WSL means that the virtual stoage-drivers — the ones that cause my laptop's boot-drive to show up at "/mnt/c" — causes the external device to be similarly-presented. Since Windows sees it as "G:\", WSL makes it available at "/mnt/g". At first, I had my pods' storage configured rooted under "/mnt/g/OpenClaw".

Things worked great that way …at first. However, as I was trying to add a "skill" to my first OpenClaw instance, I began to get "EPERM" errors. Turns out, the same virtual filesystem drivers that cause my external drive to show up as "/mnt/g" were also dropping the Linux filesystem-semantics on the floor. Thus, the "EPERM" errors.

Digging in, discovered that what happens when WSL translates "G:\" to "/mnt/g" is that WSL is translating the NTFS filesystem through a "plan9" filesystem interface. You can see this by doing:

$ df -T | '(/mnt/[a-z]$|^File)'
Filesystem     Type      1K-blocks      Used  Available Use% Mounted on
C:\            9p        497396732 440031392   57365340  89% /mnt/c
G:\            9p       1953450496 426553600 1526896896  22% /mnt/g

That "9p" indicates the use of a "plan9" filesystem interface.

To resolve the "EPERM" errors, I needed the filesystem I was using as my persistent volume-claim (PVC) location to be on a Linux-native filesystem-type (preferably an EXTn filesystem-type). Like I say, I didn't want to reformat the entire external drive. That constrained my options. Instead, what I ended up doing was creating a VHDX file in the NTFS-formatted SSD, laying down an EXT4 filesystem on it and mounting that directly into my WSL namespace. Now, WSL sees it as a native Linux filesystem:

$ df -t ext4
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sdd       263112772 26584328 223090288  11% /
/dev/sde        51290592   121952  48530816   1% /mnt/wsl/openclaw-data
/dev/sdg          139283    73329     55131  58% /mnt/wsl/docker-desktop/docker-desktop-user-distro

Currently, the VHDX shows up in the WSL namespace as the "/dev/sde" device. That device is mounted at "/mnt/wsl/openclaw-data".

As you might also notice, Docker Desktop creates its own virtual EXT4 filesystem to present to WSL (when you enable Docker Desktop's WSL-integration). It created its virtual device at "/dev/sdg" and mounted it to "/mnt/wsl/docker-desktop/docker-desktop-user-distro".

Unfortunately…

1. Those block-devices' /dev-paths aren't necessarily stable. If you aren't very particular on the order you use to start up WSL and Docker Desktop, it's very likely that the block-devices' /dev-paths will change between reboots (or even just restarts of WSL and/or Docker Desktop)
2. Even if they were stable, if you restart WSL (like happens when one needs to do a patching-related reboot) that "Mounted on" location goes away.
3. Mounting the virtual EXT4 filesystem into the WSL namespace has to be done from the host operating system (using wsl.exe from either a CMD.EXE or PowerShell session).
4. Between WSL, itself, and the host operating system's User Account Control (UAC), automating the remounting of the external drive into WSL is very difficult — enough so that I have yet to sort out a reliable/non-brittle method for doing so.
5. WSL doen't really do a traditional Linux fstab …nor does it run systemd (so you can't work around things via that avenue, either).

All of which is to say that, for now, whenever I want to play with my KinD/OpenClaw setup, I need to first do:

1. Start WSL instance
2. (From a DOS or PowerShell window running as the same user that will run the WSL instance): Run `wsl --mount --name openclaw-data --vhd G:\OpenClaw\openclaw-data.vhdx`
3. (From WSL instance): Verify that device has mounted:

   $ ( lsblk ; df -PH ) | grep /mnt/wsl/openclaw-data
   sde    8:64   0    50G  0 disk /mnt/wsl/openclaw-data
     /dev/sde         53G  125M   50G   1% /mnt/wsl/openclaw-data

4. Start Docker Desktop
5. Ensure KinD "cluster" is running
6. (From a DOS or PowerShell window running as the same user that started the WSL instance): Run `docker restart kind-control-plane`
7. Bring OpenClaw pod back online (e.g., `kubectl apply -f openclaw-personal-first.yaml`)
8. Verify that webUI is working (get URL by doing `jq -r '"http://localhost:\(.gateway.port)/?token=\(.gateway.auth.token)"' /mnt/wsl/openclaw-data/…/openclaw.json`)

I hope to one day maybe figure out how to automate that nonsense…

 

Docker Desktop the Hard Way

I guess one of the many reasons I never considered myself to be a Real Developer™ — just "a guy who writes code to automate things" — is my habits and preferences aren't what I observe to be the norm for Real Developers™.

•  I fucking hate Apple products: I want my computing environment to be fully and wholly tailorable to my my tastes. That and Apple hardware's keyboards, mice and track-pads are trash and their WM sucks. Since I can't have a NeXTish (closest you get, nowadays, is GNUstep under WindowMaker; but back in the early 00s, used to be able to run a shell-replacement that gave you an OpenStep type of launcher) or 4Dwm (SGI) style window-manager, I'd much rather Windows' interface. While it's not my ideal, it's at least got the key-bindings I prefer. Plus, I can still easily get devices with a full US101-style keyboard (everyone's gone to "buttonless" track-pads, so I'm doomed on that front whenever I'm forced to change off). It's why I've been resisting my company's attempts to issue me a work laptop since they decided that Apple hardware was a great perk to attract new hires with …and subsequently implemented centralized device management (MDM) to meet compliance goals.
• Resultant of the immediately prior, I use
• A Dell laptop (and an HP before it)
• I run Windows 11 Pro (though, when I bought it in spring of 2020, it came with and was upgraded from Windows 10)
• I would run a Linux-based desktop, but, when I bought the current laptop, the options for laptops preloaded with Linux were quite limited (and I just didn't want to have to ass around with it). If I had the money and time, I'd try to run a hypervisor as my bare-metal operating system, then, as my default VM, have an RPM-based distro like Fedora, Alma or Rocky as its OS. Professionally, my customers are RPM-based, so, just like in my SGI and then Sun sysadmin days, I tried to minimize the cognitive-load of OS-switching by running equivalents on my personal systems. 
• That said, my choice of Windows as my OS being a matter of convenience, I don't really use Windows as much more than a hosting environment.
• I use WSL2 for my "daily-driver" working-environment. My main WLS2 instance runs Oracle Enterprise Linux 9 (at the time, it was the only free EL-style instance available and I didn't want to go the Ubuntu route). I run an OpenSSH daemon inside it so I can use PuTTY to login to it. 
• I use Hyper•V on the occasions where I need to run a full VM. Those occasions are fairly rare, thoug. Mostly, I used them when I need something that runs DBUS/systemd or needs to interact with something closer to "real" devices (e.g., when I'm having to prototype automation for services get pissed when there's no DBUS/systemd bits)
• I use PodMan, instead of Docker, inside my WSL2 or Hyper•V VMs.
• I run work-related tooling — be that browsers, tools like Teams/WebEx/etc. — either from my WSL instance or a full VM. I generally believe in keeping work stuff siloed off from my main OS

Recently, a new project popped up on my work RADAR. That's going to require using something other than either a WSL-encapsulated PodMan or a Hyper•V. Instead, for the early phases, I'll need to do at least some tasks in Docker Desktop (which, on Windows, is underpinned by Hyper•V).

I'm not really a GUI guy nor do I want to have to dick around with CMD.EXE or PowerShell to manage Docker. I much prefer to do everything from a BASH prompt and to use vi and other tools that I have two to three-and-a-half decades of finger-memory for.

 On the plus side, Docker Desktop has native integration with WSL2. You just have to enable it:

 

On the minus side, if, like me, you've installed the podman-docker RPM into your environment, it makes interacting with DockerDesktop from your WSL2 CLI session a bit more of a bother to set up. It requires:

• Knowing where DockerDesktop has created its docker command within WSL
• Knowing where DockerDesktop has place its UNIX domain-socket into WSL 

Once you know those — with Docker Desktop 4.77.0, those were "/mnt/wsl/docker-desktop/cli-tools/usr/bin" and "/mnt/wsl/docker-desktop-bind-mounts/OracleLinux_9/docker.sock", respectively — you need to tell your shell where to find them. For the former, you extend your "PATH" and for the latter you alter your "DOCKER_HOST" (shell envs).

That you have to know these things is a side effect of the podman-docker RPM wanting to manage many of the same things that the Docker Desktop insertions into the WSL 2 instance wants to take care of.

The other fun thing with the setup is that you have to keep re-defining "PATH" and "DOCKER_HOST" shell envs. That's not generally something I want to be bothered with or, more importantly, remembering. The "remembering" thing is why I write articles like this one.

Since my need for either PodMan or DockerDesktop is sporadic, I'd rather things like shell-envs be self-maintaining. A great way to do that is by using direnv. With that tool, all I have to do is drop my shell-envs into a ".envrc" file hosted in a specific directory(-tree) and, whenever I "cd" to that directory, I get the settings I desire. Podman is my default, so, for Docker Desktop, I only need to manage that in a top-level directory at the root of my "${HOME}". In my case, that directory is "DockerDesktop" (or "~/DockerDesktop" in BASH-ese). So, my "~/DockerDesktop/.envrc" ends up looking like:

PATH_add /mnt/wsl/docker-desktop/cli-tools/usr/bin
PATH_add .bin
export DOCKER_HOST=unix:///mnt/wsl/docker-desktop-bind-mounts/OracleLinux_9/docker.sock

That "PATH_add .bin" is the last bit of metaphorical "glue". It allows me to define a simple shell-wrapper at "~/DockerDesktop/.bin/docker-compose", allowing me to type "docker-compose" rather than "/mnt/wsl/docker-desktop/cli-tools/usr/local/lib/docker/cli-plugins/docker-compose" ...whenever I want to run "docker compose ..." to interact with Docker Desktop instead of PodMan. And, when I say "simple" I mean:

#!/bin/sh
exec /mnt/wsl/docker-desktop/cli-tools/usr/local/lib/docker/cli-plugins/docker-compose "$@"

I assume there's probably even cleaner and/or less-effortful approaches, but this is what I was able to most-quickly cobble together, this morning

Networking Hurts My Soul

Over my IT career, I've had to dig into networking-related issues far more than I would have liked. That said, some especially bad times were:

In the early to mid-90s before the networking landscape stabilized and everyone decided "TCP/IP and Ethernet-type networking are the way to go".

In the 90s, doing office LANs was a revolving cast of Banyan, NetWare, AppleTalk and others. Woe be unto you if you had to make them interoperate.

Similarly, mainframes from various manufacturers neither wanted to play nice with each other nor those new-fangled "open" systems. SNA, HIPPI etc. were all fun.

Trying to find out which host on a token-ring network had decided it wanted to just keep the token was also quite fun. There's nothing like telling everyone in an office, "everyone, yo need to shut down all your computers, printers, etc. We have to reboot every device to get everything talking, again."

LAN-to-WAN? Joy.

Getting PC-type systems to talk to Apple type systems? Joy. Fortunately(?) never had to deal with bridging the gap from AppleTalk systems to anything by PCs.

Getting PC-type systems or Apple systems to talk to "Open" (i.e., "UNIX") systems? Not usually too horrible, mostly just sorting out installing a TCP/IP stack like Trumpet WinSock for Windows systems.

Getting PC-type systems to talk to mainframes? It was usually others' repsonsibility to get the mainframes onto the TCP/IP networks. Then all the PCs needed was a 3270 emulator application ...turning the PCs into glorified dumb-terminals. The need to enable real interop was mercifully rare, but decidedly "joyful" when it was required.

More frequent than full PC/mainframe interop was  "open" systems to mainframe ...and "mainframe" was a whole gallery of pain: IBM big-iron and (while not technically "mainframes, setting up interop still wasn't exactly pain-free) A/S 400s; Burroughs; Tandem; Stratus; ...the hell-list goes on.

 At any rate, all of the above prose was mostly an excuse to re-share this XKCD.