Diagnosing Init Script Issues

Recently, I've been wrestling with some COTS software that one of my customers wanted me to automate the deployment of. Automating its deployment has been... Patience-trying.

The first time I ran the installer as a boot-time "run once" script, it bombed. Over several troubleshooting iterations, I found that one of the five sub-components was at fault. When I deployed the five sub-components individually (and to different hosts), all but that one sub-component failed. Worse, if I ran the automated installer from an interactive user's shell, the installation would succeed. Every. Damned. Time.

So, I figured, "hmm... need to see if I can adequately simulate how to run this thing from an interactive user's shell yet fool the installer into thinking it had a similar environment to what the host's init process provides." So, I commenced to Google'ing.

Commence to Googlin'

Eventually, I found reference to `setsid`. Basically, this utility allows you to spawn a subshell that's detached from a TTY ...just like an init-spawned process. So, I started out with:

     setsid bash -c '/path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

Unfortunately, while the above creates a TTY-less subshell for the script to run in, it still wasn't quite fully simulating an init-like context. The damned installer was still completing successfully. Not what I wanted since, when run from an init-spawned context, it was failing in a very specific way. So, back to the almighty Googs.

Eventually, it occurred to me, "init-spawned processes have very different environments set than interactive user shells do. And, as a shell forked off from my login shell, that `setsid` process would inherit all my interactive shell's environment settings. Once I came to that realization, the Googs quickly pointed me to `env -i`. Upon integrating this nugget:

     setsid bash -c 'env -i /path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

My installer started failing the same when launched from an interactive shell as from an init-spawned context. I had something with which I could tell the vendor, "your unattended installer is broken: here's how to simulate/reproduce the problem. FIX IT." I dunno about you, but, to me, an "unattended installer" that I have to kick off by hand from an interactive shell really isn't all that useful.