Systemd And Timing Issues

Unlike a lot of Linux people, I'm not a knee-jerk hater of systemd. My "salaried UNIX" background, up through 2008, was primarily with OSes like Solaris and AIX. With Solaris, in particular, I was used to sytemd-type init-systems due to SMF.

That said, making the switch from RHEL and CentOS 6 to RHEL and CentOS 7 hasn't been without its issues. The change from upstart to systemd is a lot more dramatic than from SysV-init to upstart.

Much of the pain with systemd comes with COTS software originally written to work on EL6. Some vendors really only due fairly cursory testing before saying something is EL7 compatible. Many — especially earlier in the EL 7 lifecycle — didn't bother creating systemd services at all. They simply relied on systemd-sysv-generator utility to do the dirty work for them.

While the systemd-sysv-generator utility does a fairly decent job, one of the places it can fall down is if the legacy-init script (files hosted in /etc/rc.d/init.d) is actually a symbolic link to someplace else in the filesystem. Even then, it's not super much a problem if "someplace else" is still within the "/" filesystem. However, if your SOPs include "segregate OS and application onto different filesystems", then "someplace else can" very much be a problem — when "someplace else" is on a different filesystem from "/".

Recently, I was asked to automate the installation of some COTS software with the "it works on EL6 so it ought to work on EL7" type of "compatibility". Not only did the software not come with systemd service files, its legacy-init files linked out to software installed in /opt. Our shop's SOPs are of the "applications on their own filesystems" variety. Thus, the /opt/<APPLICATION> directory is actually its own filesystem hosted on its own storage device. After doing the installation, I'd reboot the system. ...And when the system came back, even though there was a boot script in /etc/rc.d/init.d, the service wasn't starting. Poring over the logs, I eventually found:

systemd-sysv-generator[NNN]: stat() failed on /etc/rc.d/init.d/<script_name>
No such file or directory

This struck me odd given that the link and its destination very much did exist.

Turns out, systemd invokes the systemd-sysv-generator utility very early in the system-initialization proces. It invokes it so early, in fact, that the /opt/<APPLICATION>  filesystem has yet to be mounted when it runs. Thus, when it's looking to do the conversion the file the sym-link points to actually does not yet exist.

My first thought was, "screw it: I'll just write a systemd service file for the stupid application." Unfortunately, the application's starter was kind of a rats nest of suck and fail; complete and utter lossage. Trying to invoke it from directly via a systemd service definition resulted in the application's packaged controller-process not knowing where to find a stupid number of its sub-components. Brittle. So, I searched for other alternatives...

Eventually, my searches led me to both the nugget about when systemd invokes the systemd-sysv-generator utility and how to overcome the "sym-link to a yet-to-be-mounted-filesystem" problem. Under systemd-enabled systems, there's a new-with-systemd mount-option you can place in /etc/fstab — x-initrd.mount. You also need to make sure that your filesystem's fs_passno is set to "0" ...and if your filesystem lives on an LVM2 volume, you need to update your GRUB2 config to ensure that the LVM gets onlined prior to systemd invoking the systemd-sysv-generator utility. Fugly.

At any rate, once I implemented this fix, the systemd-sysv-generator utility became happy with the sym-linked legacy-init script ...And my vendor's crappy application was happy to restart on reboot.

Given that I'm deploying on AWS, I was able to accommodate setting these fstab options by doing:

mounts:
  - [ "/dev/nvme1n1", "/opt/<application> , "auto", "defaults,x-initrd.mount", "0", "0" ]

Within my cloud-init declaration-block. This should work in any context that allows you to use cloud-init.

I wish I could say that this was the worst problem I've run into with this particular application. But, really, this application is an all around steaming pile of technology.

Diagnosing Init Script Issues

Recently, I've been wrestling with some COTS software that one of my customers wanted me to automate the deployment of. Automating its deployment has been... Patience-trying.

The first time I ran the installer as a boot-time "run once" script, it bombed. Over several troubleshooting iterations, I found that one of the five sub-components was at fault. When I deployed the five sub-components individually (and to different hosts), all but that one sub-component failed. Worse, if I ran the automated installer from an interactive user's shell, the installation would succeed. Every. Damned. Time.

So, I figured, "hmm... need to see if I can adequately simulate how to run this thing from an interactive user's shell yet fool the installer into thinking it had a similar environment to what the host's init process provides." So, I commenced to Google'ing.

Commence to Googlin'

Eventually, I found reference to `setsid`. Basically, this utility allows you to spawn a subshell that's detached from a TTY ...just like an init-spawned process. So, I started out with:

     setsid bash -c '/path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

Unfortunately, while the above creates a TTY-less subshell for the script to run in, it still wasn't quite fully simulating an init-like context. The damned installer was still completing successfully. Not what I wanted since, when run from an init-spawned context, it was failing in a very specific way. So, back to the almighty Googs.

Eventually, it occurred to me, "init-spawned processes have very different environments set than interactive user shells do. And, as a shell forked off from my login shell, that `setsid` process would inherit all my interactive shell's environment settings. Once I came to that realization, the Googs quickly pointed me to `env -i`. Upon integrating this nugget:

     setsid bash -c 'env -i /path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

My installer started failing the same when launched from an interactive shell as from an init-spawned context. I had something with which I could tell the vendor, "your unattended installer is broken: here's how to simulate/reproduce the problem. FIX IT." I dunno about you, but, to me, an "unattended installer" that I have to kick off by hand from an interactive shell really isn't all that useful.

JMSE Query Fu

Yesterday, I posted up an article illustrating the pain of LDAP's query-language. Today, I had to dick around with JMSE reporting.

The group I work for manages a number of AWS accounts in a number of regions. We also provide support to a number of tenants and their AWS accounts.

This support is necessary because, in much the same way AWS strives to make it easy to adopt "cloud", a lot of people flocking aboard don't necessarily know how to do so cost-effectively. Using AWS can be a cost-effective way to do things, but failure to exercise adequate house-keeping can bite you right in the wallet. Unfortunately, part of the low bar to entry means that a lot of users coming into AWS don't really comprehend that housekeeping is necessary. And, even if they did, they don't necessarily understand how to implement automated housekeeping methods. So, stuff tends to build up over time ...leading to unnecessary cost-exposures.

Inevitably, this leads to a "help: our expenses are running much higher than expected" types of situation. What usually turns out to be a majar cause is disused storage (and orphaned EC2s left running 24/7/365 ...but that's another story). Frequently, this comes out to some combination of orphaned (long-detached) EBS volumes, poorly maintained S3 buckets and elderly EBS snapshots.

IDin'g orphaned EBS volumes is pretty straight forward. Not a lot of query-fu is needed. To see it.

Poorly maintained S3 buckets are a skosh harder to suss out. But, you can usually just enable bucket inventory-reporting an lifecycle policies to help automate your cost-exposure reduction.

While EBS snapshots are relatively low-cost, when enough of them build up, you start to notice the expenses. Reporting can be relatively straightforward. However, if you happen to be maintaining fleets of custom AMIs, doing a simple "what can I delete" report becomes an exercise in query-filtering. Fortunately, AMI-related snapshots are generally identifiable (excludable) with a couple of filters, leaving you a list of snapshots to futher investigate. Unfortunately, writing the query-filters means dicking with JMSE filter syntax ...which is kind of horrid. Dunno if it's more or less horrid than LDAP's.

At any rate, I ended up writing a really simple reporting script to give me a basic idea of whether stale snapshots are even a problem ...and the likely scope of said problem if it existed. While few of our tenants maintain their own AMIs, we do in the account I primarily work with. So, I wrote my script to exclude their snapshots from relevant reports:

for YEAR in $( seq 2015 2018 )
do
   for MONTH in $( seq -w 1 12 )
   do
      for REGION in us-{ea,we}st-{1,2}
      do
         printf "%s-%s (%s): " "${YEAR}" "${MONTH}" "${REGION}"
         aws --region "${REGION}" ec2 describe-snapshots --owner <ACCOUNT> \
           --filter "Name=start-time,Values=${YEAR}-${MONTH}*" --query \
           'Snapshots[?starts_with(Description, `Created by CreateImage`) == `false`]|[?starts_with(Description, `Copied for DestinationAmi`) == `false`].[StartTime,SnapshotId,VolumeSize,Description]' \
           --output text | wc -l
      done
    done
done | sed '/ 0$/d'

The first suck part with a JMSE query is it can be really long. Worse: it  really doesn't tolerate line-breaking to make scripts less "sprawly". If you're like me, you like to keep a given line of code in a script no longer than "X" characters. I usually prefer X=80. JMSE queries pretty much say, "yeah: screw all that prettiness nonsense".

At any rate to quickly explain the easier parts of  the above:

• First, this is a basic BASH wrapper (should work in other shells that are aware of Bourne-ish syntax)
• I'm using nested loops: first level is year, second level is month, third level is AWS region. This allows easy grouping of output.
• I'm using `printf` to give my output meaning
• At the end of my `aws` command, I'm using `wc` to provide a simple count of lines returned: when one uses the `--output text` argument to the `aws` command, each object's returned data is done as a single line ...allowing `wc` to provide a quick tally of lines meeting the JMSE selection-criteria
• At the end of all my looping, I'm using `sed` to suppress any lines where a given region/month/year has no snapshots found

Notice I don't have a bullet answering why I'm bothering to define an output string ...and then, essentially, throwing that string away. Simply put, I have to output something for `wc` to  count and I may as well output several useful items in case I want to `tee` it off and use the data in an extension to the script.

The fun part of the above is the JMSE horror-show:

• When you query a EBS snapshot object, it outputs a JSON document. That document's structure looks like:

  {
      "Snapshots": [
          {
              "Description": "Snapshot Description Strin",
              "Tags": [
                  {
                      "Value": "TAG1 Value",
                      "Key": "TAG1"
                  },
                  {
                      "Value": "TAG2 Value",
                      "Key": "TAG2"
                  },
                  {
                      "Value": "TAG3 Value",
                      "Key": "TAG3"
                  }
              ],
              "Encrypted": false,
              "VolumeId": "",
              "State": "completed",
              "VolumeSize": ,
              "StartTime": "--
  T::.000Z",
              "Progress": "100%",
              "OwnerId": "",
              "SnapshotId": ""
          }
      ]
  }
  

• The ".[StartTime,SnapshotId,VolumeSize,Description]" portion of the "--query" string constrains the output to containing the "StartTime", "SnapshotId", "VolumeSize" and "Description" elements from the object's JSON document. Again, not of immediate use when doing a basic element-count census but is convertible to something useful in later tasks.
• When constructing a JMSE query, you're informing the tool, which elements of the JSON node-tree you want to work on. You pretty much always have to specify the top-level document node. In this case, that's "Snapshots". Thus, the initial part of the query-string is "Snapshots[]". Not directly germane to this document, but likely worth knowing:
  • In JMSE, "Snapshots[]" basically means "give me everything from 'Snapshots' on down".
  • An equivalent to this is "Snapshots[*]".
  • If you wanted just the first object returned, you'd use  "Snapshots[0]"
  • If you wanted just a sub-range of objects returned, you'd use  "Snapshots[X:Y]"
• If you want to restrict your output based on selection criteria, JMSE provides the "[?]" construct. In this particular case, I'm using the "starts_with" query, or  "[?starts_with]". This query takes arguments: what sub-attribute you're querying; the string to match against and whether your positively or negatively selecting. Thus:
  • To eliminate snapshots with descriptions starting "Created by CreateImage", my query looks like: "?starts_with(Description, `Created by CreateImage`) == `false`"
  • To eliminate snapshots with descriptions starting "Copied for DestinationAmi", my query looks like: "?starts_with(Description, `Copied for DestinationAmi`) == `false`"
  • Were I selecting for either of these strings — rather than the current against-match — I would need to change my "false" selectors to "true".
• To do a compound query of an OR type, one does "[query1]|[query2]". Notionally, I could string together as many of these as I wanted if I needed to select for/against any arbitrary number of attribute values by adding in further "|" operators.

At the end of running my script, I ended up with output that looked like:

    2017-07 (us-east-1): 61
    2017-08 (us-east-1): 746
    2017-09 (us-east-1): 196
    2017-10 (us-east-1): 4
    2017-11 (us-east-1): 113
    2017-12 (us-east-1): 600
    2018-01 (us-east-1): 149
    2018-03 (us-east-1): 6
    2018-04 (us-east-1): 3
    2018-06 (us-east-1): 302
    2018-07 (us-east-1): 1620
    2018-08 (us-east-1): 206
    2018-08 (us-west-2): 3

Obviously, there's some cleanup to be done in the queried account: it's unlikely that any snapshots older than 30 days — stuff that's from 2017 is almost a lock to be the result of "bad housekeeping".

The Pain of LDAP

Recently, we've had a push to enable 2FA on all of the internet-facing services we offer. A manager asked, "how many users do we have to worry about?" We have three basic types of users:

• Those who only have access to the Remote Desktops
• Those who only have access to the various applications
• Those who have access to both the Remote Desktops and to the various appliations

All such access is unprivileged. All accesses are managed via Active Directory group memberships. 

• Our RDSH users are in a "remote desktops" group.
• Our application users are in a mix of groups. Fortunately, we long ago implemented standard naming for our groups. All group-names for one application-class start with "T_"; all group-names for the other application-class start with "E_"

Fortunately, the above means that I could get an answer for the manager by way of LDAP queries. A user's membership in an Active Directory is conveyed via the user-object's meberOf attribute. This means that the basic query-objects I needed to consider were:

• RDSH Users: In LDAP-speak, this works out to CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>
• "T" Application-group: In LDAP-speak, this works out to CN=T_*
• "E" Application group: In LDAP-speak, this works out to CN=E_*

LDAP queries generally operate on object-sets. Sets are denoted by parentheses. Using the previously-mentioned groups, a single-element set for each of my group-memberships would be:

• (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• (memberof=CN=T_*)
• (memberof=CN=E_*)

Where things get really freaking ugly is when you need to combine these into appropriate queries. Combining query-objects is a matter of using the booleans:

• "&": the logical AND operator
• "|": the logical OR operator
• "!": the logical NOT operator
• If you're familiar with logical operators, you'll notice that there isn't a built in eXclusive OR type of operator. 

Why I say things are ugly is that you apply a given operator within a set and the syntax is a little non-intutive:

• ANDing sets: Uses the syntax (&(SET1)(SET2)(...)(SET_N)). Basically, this says, "AND all of the following sets within this AND-set. An AND-set can consist of 2+ sets for evaluation
• ORing sets: Similarly, this uses the syntax (|(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this OR-set. An OR-set can consist of 2+ sets for evaluation
• NOTin sets: Similarly, this uses the syntax (!(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this NOT-set. An NOT-set can consist of 2+ sets for evaluation

Where it gets really ugly is when a logical operation acually requires two or more sub operations. For example:

• When you want to tell if a user is a member of one group but not another, you need to do something like: (&(GROUP1)(!(GROUP2))
• Similarly, when you want to tell if a user is a member of one group but not a member of both of two other groups, you would do something like (&(GROUP1)(!(&(GROUP2)(GROUP3))
• Similarly, when you want to tell if a user is a member of one group but not a member of either of two other groups, you would do something like (&(GROUP1)(!(|(GROUP2)(GROUP3))

As you can probably tell, as the number of selectors goes up, so does the ugliness.

At any rate, back to my original queries:

• Getting a list of all people with RDSH access, my query looks like: (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• Getting a list of people with membership in either of the application groups, my query looks like: (|(memberof=CN=T_*)(memberof=CN=T_*))
• Getting a list of people with RDSH access and with membership in either of the application groups, my query looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(|(memberof=CN=T_*)(memberof=CN=T_*)))
• Getting a list of people with RDSH access but not access to either of the other groups , my query now looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(!(|(memberof=CN=T_*)(memberof=CN=T_*))))
• Conversely, getting a list of people without RDSH access but with access to either of the other groups, my query now looks like: (&(!(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>))(|(memberof=CN=T_*)(memberof=CN=T_*)))

All that ugliness aside, I was able to get the manager the answer to his question and was able to do it quickly. This post is mostly a result of him asking "what the hell does all that gobbledygook mean" when I sent him the queries I ran to get him the answer. I'd sent that as part of the thread for the same reason I'm writing this: so I have the queries the next time the information is asked for again (and don't have to re-work it all out in my head again).