Crib-Notes: Tuning MTU-size on EL7-based DHCP Clients

Because:

• One of the networks I deal with is broken
• I have only about a 15% hit-rate of what I actually need whenever I try to re-find the info
• most of my Google hits seem to think that using `ip link set...` is a fine, long-term solution and the ones that point to persistency referenc either nmcli or the /etc/sysconfig/network-scripts file

Need to save this here (maybe by doing so, Google will at least return this page when I need to re-search)...

To override the default MTU (in my case turning off Jumbo frames), edit the /etc/dhcp/dhclient-eth0.conf file like so:

interface eth0 {
    supersede interface-mtu 1350;
}

Reboot and everything should be fine (actually able to vi a file, use man pages, etc., without locking up my SSH session!).

Shutdown In a Hurry

Last year, I put together an automated deployment of a CI tool for a customer. The customer was using it to provide a CI service for different developer-groups within their company. Each developer group acts like an individual tenant of my customer's CI service.

Recently, as more of their developer-groups have started using the service, space-consumption has exploded. Initially, they tried setting up  job within their CI system to automate cleanups of some of their tenants more-poorly architected CI jobs — ones that didn't appropriately clean up after themselves. Unfortunately, once the CI systems' filesystems fill up, jobs stop working ...including the cleanup jobs.

When I'd written the initial automation, I'd written automation to create two different end-states for their CI system: one that is basically just a cloud-hosted version of a standard, standalone deployment of the CI service; the other being a cloud-enabled version that's designed to automatically rebuild itself on a regular basis or in the event of service failure. They chose to deploy the former because it's the type of deployment they were most familiar with.

With their tenants frequently causing service-components to go offline and their cleaning jobs not being reliable, they asked me to investigate things and come up with a work around. I pointed them to the original set of auto-rebuilding tools I'd provided them, noting that, with a small change, those tools could detect the filesystem-full state and initiate emergency actions. In this case, the proposed emergency action being a system-suicide that caused the automation to rebuild the service back to a healthy state.

Initially, I was going to patch the automation so that, upon detecting a disk-full state, it would trigger a graceful shutdown. Then it struck me, "I don't care about these systems' integrity, I care about them going away quickly," since the quicker they go away, the quicker the automation will notice and start taking steps to re-establish the service. What I wanted was, instead of an `init 6` style shutdown, to have a "yank the power cord out of the wall" style of shutdown.

In my pre-Linux days, the commercial UNIX systems I dealt with each had methods for forcing a system to immediately stop dead. Do not pass go. Do not collect $200. So, started digging around for analogues.

Prior to many Linux distributions — including the one the CI service was deployed onto — moving to systemd, you could halt the system by doing:

# kill -SEGV 1

Under systemd, however, the above will cause systemd to dump a core file but not halt the system. Instead, systemd instantly respawns:

Broadcast message from systemd-journald@test02.lab (Tue 2018-11-06 18:35:52 UTC):

systemd[1]: Caught <segv>, dumped core as pid 23769.


Broadcast message from systemd-journald@test02.lab (Tue 2018-11-06 18:35:52 UTC):

systemd[1]: Freezing execution.


Message from syslogd@test02 at Nov  6 18:35:52 ...
 systemd:Caught <segv>, dumped core as pid 23769.

Message from syslogd@test02 at Nov  6 18:35:52 ...
 systemd:Freezing execution.

[root@test02 ~]# who -r
         run-level 5  2018-11-06 00:16

So, I did a bit more digging around. In doing so, I found that Linux has a functional analog to hitting the SysRq key on a 80s or 90s vintage system. This is an optional functionality that is enabled in Enterprise Linux. For a list of things that used to be doable with a SysRq key, I'd point you to this "Magic SysRq Key" article.

So, I tested it out by doing:

# echo o >/proc/sysrq-trigger

Which pretty much instantly causes an SSH connection to the remote system to "hang". It's not so much that the connection has hung as doing the above causes the remote system to immediately halt. It can take a few seconds, but, eventually the SSH client will decide that the endpoint has dropped, resulting in the "hung" SSH session exiting similarly to:

# Connection reset by 10.6.142.20

This bit of knowledge verified, I had my suicide-method for my automation. Basically, I set up a cron job to run every five minutes to test the "fullness" of ${APPLICATION_HOME}:

#!/bin/bash
#
##################################################
PROGNAME="$(basename ${0})"
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
APPDIR=$(awk -F= '/APP_WORKDIR/{print $2}' /etc/cfn/Deployment.envs)
RAWBLOCKS="$(( $( stat -f --format='%b*%S' ${APPDIR} ) ))"
USEBLOCKS="$(( $( stat -f --format='%a*%S' ${APPDIR} ) ))"
PRCNTFREE=$( printf '%.0f' $(awk "BEGIN { print ( ${USEBLOCKS} / ${RAWBLOCKS} ) * 100 }") )

if [[ ${PRCNTFREE} -gt 5 ]]
then
   echo "Found enough free space" > /dev/null
else
   # Put a bullet in systemd's brain
   echo o > /proc/sysrq-trigger
fi

Basically, the above finds the number of available blocks in the target filesystem and divides by the number of raw blocks and converts it into a percentage (specifically, it checks the number of blocks available to a non-privileged application rather than the root user). If the percentage drops to or below "5", it sends a "hard stop" signal via the SysRq interface.

The external (re)build automation takes things from there. The preliminary benchmarked recovery time from exceeding the disk-full threshold to being back in business on a replacement node is approximately 15 minutes (though will be faster in future iterations by better optimizing the build-time hardening routines).

S3 And Impacts of Using an IAM Role Instead of an IAM User

I work for a group that manages a number of AWS accounts. To keep things somewhat simpler from a user-management perspective, we use IAM roles to access the various accounts rather than IAM users. This means that we can manage users via Active Directory such that, as new team members are added, existing team members' responsibilities change or team members leave, all we have to do is update their AD user-objects and the breadth and depth of their access-privileges are changed.

However, the use of IAM Roles isn't without it's limitations. Role-based users' accesses are granted by way of ephemeral tokens. Tokens last, at most, 3600 seconds. If you're a GUI user, it's kind of annoying because it means you that across an 8+ hour work-session, you'll be prompted to refresh your tokens at least seven times (on top of the initial login). If you're working mostly via the CLI, you won't necessarily notice things as each command you run silently refreshes your token.

I use the caveat "won't necessarily notice" because there are places where using an IAM Role can bite you in the ass. Worse, it will bite you in the ass in a way that may leave you scratching your head wondering "WTF isn't this working as I expected".

The most recent adventure in "WTF isn't this working as I expected" was in the context of trying to use S3 pre-signed URLs. If you haven't used them before, pre-singed URLs allow you to provide temporary access to S3-hosted objects that you otherwise want to not make anonymously-available. In the grand scheme of things, one can do one of the following to provide access to S3-hosted data for automated tools:

• Public-read object hosted in a public S3 bucket: this is a great way to end up in new stories about accidental data leaks
• Public-read object hosted in a private S3 bucket: you're still providing unfettered access to a specific S3-hosted object/object-set, but some rando can't simply explore the hosting S3 bucket for interesting data. You can still end up in the newspapers, but, the scope of the damage is likely to be much smaller
• Private-read object hosted in a private S3 bucket: the least-likely to get you in the newspapers, but requires some additional "magic" to allow your automation access to S3-hosted files:
• IAM User Credentials stored on the EC2 instance requiring access to the S3-hosted files: a good method, right until someone compromises that EC2 instance and steals the credentials. Then, the attacker has access to everything those credentials had access to (until such time that you discover the breach, and have deactivated or changed the credentials)
• IAM Instance-role: a good way of providing broad-scope access to S3 buckets to an instance or group of instances sharing a role. Note that, absent some additional configuration trickery, every process running on an enabled instance has access to everything that the Instance-role provides access to. Thus, probably not a good choice for systems that allow interactive logins or that run more than one, attackable service.
• Pre-signed URLs: a way to provide fine-grained, temporary access to S3-hosted objects. Primary down-fall is there's significant overhead in setting up access to a large collection of files or providing continuing access to said files. Thus, likely suitable for providing basic CloudFormation EC2 access to configuration files, but not if the EC2s are going to need ongoing access to said files (as they would in, say, an AutoScaling Group type of use-case)

There's likely more access methods one can play with - each with their own security, granularity and overhead tradeoffs. The above are simply the ones I've used.

The automation I write tends to include a degree of user-customizable content. Which is to say, I write my automation to take care of 90% or so of a given service's configuration, then hand the automation off to others to use as they see fit. To help prevent the need for significant code-divergence in these specific use-cases, my automation generally allows a user to specify the ingestion of configuration-specification files and/or secondary configuration scripts. These files or scripts often contain data that you wouldn't want to put in a public GitHub repository. Thus, I generally recommend to the automation's users "put that data someplace safe - here's methods for doing it relatively safely via S3".

Circling back so that the title of this post makes sense... Typically, I recommend the use of pre-signed URLs for automation-users that want to provide secure access to these once-in-an-instance-lifecycle files. Pre-signed URLs' access-granting can be as little as a couple seconds to as much as seven days. Without specifying a desired time, the granted-access is granted for two hours.

However, that degree of flexibility depends greatly on what type of IAM object is creating the ephemeral access-grant. A standard IAM user can grant with all of the previously mentioned time-flexibility. An IAM role, however, is constrained to however long their currently-active token is good for. Thus, if executing from the CLI using an IAM role the grantable lifetime for a pre-signed is 0-3600 seconds.

If you're confused whether the presigned URL you've created/were given was from an IAM user or Role, look for the presence of X-Amz-* in the URL. If you see any such elements, it was generate by an IAM Role and will only last up to 3600 seconds.

So You Created a Regression?

Sometimes, when you're fixing up files in git-managed files, you'll create a regression by nuking a line (or whole blocks) of code. If you can remember something that was in that nuked chunk of code, you can write a quick script to find all prior commits that referenced that chunk of code.

In this particular case, one of my peers was working on writing some Jenkins pipeline-definitions. The pipeline needed to automagically create S3 pre-signed URLs. At some point, the routine for doing so got nuked. Because coming up with the requisite interpolation-protections had been kind of a pain in the ass, we really didn't want to have to go through the pain of reinventing that particular wheel.

So, how to make git help us find the missing snippet. `git log`, horse to a quick loop-iteration, can do the trick:

for FILE in $( find <PROJECT_ROOT_DIR> -name "*.groovy" )
do
   echo $FILE
   git log --pretty="   %H" -Spresign $FILE
done | grep ^commit

In the above:

• We're executing within the directory created by the original `git clone` invocation. To limit the search-scope, you can also run it from a subdirectory of the project.
• Since, in this example case, we know that all the Jenkins pipeline definitions end with the `.groovy` extension, we limit our search to just those file-types.
• The `-Spresign` is used to tell `git log` to look for the string `presign`.
• The `--pretty=" %H"` suppresses all the other output from the `git log` command's output - ensuring that only the commit-ID is print. The leading spaces in the quoted string provide a bit of indenting to make the output-groupings a bit easier to intuit.

This quick loop provides us a nice list of commit-IDs like so:

./Deployment/Jenkins/agent/agent-instance.groovy
   64e2039d593f653f75fd1776ca94bdf556166710
   619a44054a6732bacfacc51305b353f8a7e5ebf6
   1e8d5e40c7db2963671457afaf2d16e80e42951f
   bb7af6fd6ed54aeca54b084627e7e98f54025c85
./Deployment/Jenkins/master/Jenkins_master_infra.groovy
./Deployment/Jenkins/master/Jenkins_S3-MigrationHelper.groovy
./Deployment/Jenkins/master/Master-Ec2-Instance.groovy
./Deployment/Jenkins/master/Master-Elbv1.groovy
./Deployment/Jenkins/master/parent-instance.groovy

In the above, only the file `Deployment/Jenkins/agent/agent-instance.groovy` contains our searched-for string. The first-listed commit-ID (indented under the filename) contains the subtraction-diff for the targeted string. Similary, the second commit-ID contains the code snippet we're actually after. The remaining commit-IDs contain the original "invention of the wheel",

In this particular case, we couldn't simply revert to the specific commit as there were a lot of other changes that were desired. However, it did let the developer use `git show` so that he could copy out the full snippet he wanted back in the current version(s) of his pipelines.

Exposed By Time and Use

Last year, a project with both high complexity and high urgency was dropped in my lap. Worse, the project was, to be charitable, not well specified. It was basically, "we need you to automate the deployment of these six services and we need to have something demo-able within thirty days".

Naturally, the six services were ones that I'd never worked with from the standpoint of installing or administering. Thus, there as a bit of a learning curve around how best to automate things that wasn't aided by the paucity of "here's the desired end-state" or other related details. All they really knew was:

• Automate the deployment into AWS
• Make sure it works on our customized/hardened build
• Make sure that it backs itself up in case things blow up
• GO!

I took my shot at the problem. I met the deadline. Obviously, the results were not exactly "optimal" — especially from the "turn it over to others to maintain standpoint. Naturally, after I turned over the initial capability to the requester, they were radio silent for a couple weeks.

When they finally replied, it was to let me know that the deadline had been extended by several months. So, I opted to use that time to make the automation a bit friendlier for the uninitiated to use. That's mostly irrelevant here — just more "background".

At any rate, we're now nearly a year-and-a-half removed from that initial rush-job. And, while I've improved the ease of use for the automation (it's been turned over to others for daily care-and-feeding), much of the underlying logic hasn't been revisited.

Over that kind of span, time tends to expose a given solution's shortcomings. Recently, they were attempting to do a parallel upgrade of one of the services and found that the data-move portion of the solution was resulting in build-timeouts. Turns out the size of the dataset being backed up (and recovered from as part of the automated migration process) had exploded. I'd set up the backups to operate incrementally, so, the increase in raw transfer times had been hidden.

The incremental backups were only taking a few minutes; however, restores of the dataset were taking upwards of 35 minutes. The build-automation was set to time out at 15 minutes (early in the service-deployment, a similar opration took 3-7 minutes) So, step one was to adjust the automation's timeouts to make allowances for the new restore-time realities. Second step was to investigate why the restores were so slow.

The quick-n-dirty backup method I'd opted for was a simple `s3 sync --delete /<APPLICATION_HOME_DIR>/ s3://<BUCKET>/<FOLDER>`. It was a dead-simple way to "sweep" the contents of the  directory /<APPLICATION_HOME_DIR>/ to S3. And, because S3's `sync` method defaults to incremental, the cron-managed backups were taking the same couple minutes each day that they were a year-plus ago.

Fun fact about S3 and its transfer performance: if the objects you're uploading have keys with high degrees of commonality, transfer performance will become abysmal.

You may be asking why I mention "keys" since I've not mentioned encryption. S3, being an object-based filesystem, doesn't have the hierarchical layout of legacy, host-oriented storage. If I take a file from a host-oriented storage and use the S3 CLI utility to copy that file via its fully-qualified pathname to S3, the object created in S3 will look like:

<FULLY>/<QUALIFIED>/<PATH>/<FILE>

Of the above, "<FILE>" is the object name stored in S3 while "<FULLY>/<QUALIFIED>/<PATH>" is the key for that file. If you have a few thousand objects with the same or sufficiently-similar "<FULLY>/<QUALIFIED>/<PATH>" values, you'll run into that "transfer performance will become abysmal" issue mentioned earlier.

We very definitely did run into that problem. HARD. The dataset in question is (currently) a skosh less than 11GiB in size. The instance being backed up has an expected througput of about 0.45Gbps of sustained network throughput. So, we were expecting that dataset to take only a couple minutes to transfer. However, as noted above, it was taking 35+ minutes to do so.

So, how to fix? One of the easier methods is to stream your backups to a single file. I quick series of benchmarking-runs showed that doing so cut that transfer from over 35 minutes to under five minutes. Similarly, were one to iterate over all the files in the dataset, and individually copying the files into S3 using either randomize filenames (and setting the "real" fully-qualified-path as an attribute/tag of the file) or simply reversing the path-name (doing something like `S3NAME=$( echo "${REAL_PATHNAME} | perl -lne 'print join "/", reverse split/\//;')`) and storing that then your performance goes up dramatically.

I'll likely end up doing one of those three methods ...once I have enough contiguous time to allocate to re-engineering the backup and restore/rebuild methods.

Systemd And Timing Issues

Unlike a lot of Linux people, I'm not a knee-jerk hater of systemd. My "salaried UNIX" background, up through 2008, was primarily with OSes like Solaris and AIX. With Solaris, in particular, I was used to sytemd-type init-systems due to SMF.

That said, making the switch from RHEL and CentOS 6 to RHEL and CentOS 7 hasn't been without its issues. The change from upstart to systemd is a lot more dramatic than from SysV-init to upstart.

Much of the pain with systemd comes with COTS software originally written to work on EL6. Some vendors really only due fairly cursory testing before saying something is EL7 compatible. Many — especially earlier in the EL 7 lifecycle — didn't bother creating systemd services at all. They simply relied on systemd-sysv-generator utility to do the dirty work for them.

While the systemd-sysv-generator utility does a fairly decent job, one of the places it can fall down is if the legacy-init script (files hosted in /etc/rc.d/init.d) is actually a symbolic link to someplace else in the filesystem. Even then, it's not super much a problem if "someplace else" is still within the "/" filesystem. However, if your SOPs include "segregate OS and application onto different filesystems", then "someplace else can" very much be a problem — when "someplace else" is on a different filesystem from "/".

Recently, I was asked to automate the installation of some COTS software with the "it works on EL6 so it ought to work on EL7" type of "compatibility". Not only did the software not come with systemd service files, its legacy-init files linked out to software installed in /opt. Our shop's SOPs are of the "applications on their own filesystems" variety. Thus, the /opt/<APPLICATION> directory is actually its own filesystem hosted on its own storage device. After doing the installation, I'd reboot the system. ...And when the system came back, even though there was a boot script in /etc/rc.d/init.d, the service wasn't starting. Poring over the logs, I eventually found:

systemd-sysv-generator[NNN]: stat() failed on /etc/rc.d/init.d/<script_name>
No such file or directory

This struck me odd given that the link and its destination very much did exist.

Turns out, systemd invokes the systemd-sysv-generator utility very early in the system-initialization proces. It invokes it so early, in fact, that the /opt/<APPLICATION>  filesystem has yet to be mounted when it runs. Thus, when it's looking to do the conversion the file the sym-link points to actually does not yet exist.

My first thought was, "screw it: I'll just write a systemd service file for the stupid application." Unfortunately, the application's starter was kind of a rats nest of suck and fail; complete and utter lossage. Trying to invoke it from directly via a systemd service definition resulted in the application's packaged controller-process not knowing where to find a stupid number of its sub-components. Brittle. So, I searched for other alternatives...

Eventually, my searches led me to both the nugget about when systemd invokes the systemd-sysv-generator utility and how to overcome the "sym-link to a yet-to-be-mounted-filesystem" problem. Under systemd-enabled systems, there's a new-with-systemd mount-option you can place in /etc/fstab — x-initrd.mount. You also need to make sure that your filesystem's fs_passno is set to "0" ...and if your filesystem lives on an LVM2 volume, you need to update your GRUB2 config to ensure that the LVM gets onlined prior to systemd invoking the systemd-sysv-generator utility. Fugly.

At any rate, once I implemented this fix, the systemd-sysv-generator utility became happy with the sym-linked legacy-init script ...And my vendor's crappy application was happy to restart on reboot.

Given that I'm deploying on AWS, I was able to accommodate setting these fstab options by doing:

mounts:
  - [ "/dev/nvme1n1", "/opt/<application> , "auto", "defaults,x-initrd.mount", "0", "0" ]

Within my cloud-init declaration-block. This should work in any context that allows you to use cloud-init.

I wish I could say that this was the worst problem I've run into with this particular application. But, really, this application is an all around steaming pile of technology.

Diagnosing Init Script Issues

Recently, I've been wrestling with some COTS software that one of my customers wanted me to automate the deployment of. Automating its deployment has been... Patience-trying.

The first time I ran the installer as a boot-time "run once" script, it bombed. Over several troubleshooting iterations, I found that one of the five sub-components was at fault. When I deployed the five sub-components individually (and to different hosts), all but that one sub-component failed. Worse, if I ran the automated installer from an interactive user's shell, the installation would succeed. Every. Damned. Time.

So, I figured, "hmm... need to see if I can adequately simulate how to run this thing from an interactive user's shell yet fool the installer into thinking it had a similar environment to what the host's init process provides." So, I commenced to Google'ing.

Commence to Googlin'

Eventually, I found reference to `setsid`. Basically, this utility allows you to spawn a subshell that's detached from a TTY ...just like an init-spawned process. So, I started out with:

     setsid bash -c '/path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

Unfortunately, while the above creates a TTY-less subshell for the script to run in, it still wasn't quite fully simulating an init-like context. The damned installer was still completing successfully. Not what I wanted since, when run from an init-spawned context, it was failing in a very specific way. So, back to the almighty Googs.

Eventually, it occurred to me, "init-spawned processes have very different environments set than interactive user shells do. And, as a shell forked off from my login shell, that `setsid` process would inherit all my interactive shell's environment settings. Once I came to that realization, the Googs quickly pointed me to `env -i`. Upon integrating this nugget:

     setsid bash -c 'env -i /path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

My installer started failing the same when launched from an interactive shell as from an init-spawned context. I had something with which I could tell the vendor, "your unattended installer is broken: here's how to simulate/reproduce the problem. FIX IT." I dunno about you, but, to me, an "unattended installer" that I have to kick off by hand from an interactive shell really isn't all that useful.

JMSE Query Fu

Yesterday, I posted up an article illustrating the pain of LDAP's query-language. Today, I had to dick around with JMSE reporting.

The group I work for manages a number of AWS accounts in a number of regions. We also provide support to a number of tenants and their AWS accounts.

This support is necessary because, in much the same way AWS strives to make it easy to adopt "cloud", a lot of people flocking aboard don't necessarily know how to do so cost-effectively. Using AWS can be a cost-effective way to do things, but failure to exercise adequate house-keeping can bite you right in the wallet. Unfortunately, part of the low bar to entry means that a lot of users coming into AWS don't really comprehend that housekeeping is necessary. And, even if they did, they don't necessarily understand how to implement automated housekeeping methods. So, stuff tends to build up over time ...leading to unnecessary cost-exposures.

Inevitably, this leads to a "help: our expenses are running much higher than expected" types of situation. What usually turns out to be a majar cause is disused storage (and orphaned EC2s left running 24/7/365 ...but that's another story). Frequently, this comes out to some combination of orphaned (long-detached) EBS volumes, poorly maintained S3 buckets and elderly EBS snapshots.

IDin'g orphaned EBS volumes is pretty straight forward. Not a lot of query-fu is needed. To see it.

Poorly maintained S3 buckets are a skosh harder to suss out. But, you can usually just enable bucket inventory-reporting an lifecycle policies to help automate your cost-exposure reduction.

While EBS snapshots are relatively low-cost, when enough of them build up, you start to notice the expenses. Reporting can be relatively straightforward. However, if you happen to be maintaining fleets of custom AMIs, doing a simple "what can I delete" report becomes an exercise in query-filtering. Fortunately, AMI-related snapshots are generally identifiable (excludable) with a couple of filters, leaving you a list of snapshots to futher investigate. Unfortunately, writing the query-filters means dicking with JMSE filter syntax ...which is kind of horrid. Dunno if it's more or less horrid than LDAP's.

At any rate, I ended up writing a really simple reporting script to give me a basic idea of whether stale snapshots are even a problem ...and the likely scope of said problem if it existed. While few of our tenants maintain their own AMIs, we do in the account I primarily work with. So, I wrote my script to exclude their snapshots from relevant reports:

for YEAR in $( seq 2015 2018 )
do
   for MONTH in $( seq -w 1 12 )
   do
      for REGION in us-{ea,we}st-{1,2}
      do
         printf "%s-%s (%s): " "${YEAR}" "${MONTH}" "${REGION}"
         aws --region "${REGION}" ec2 describe-snapshots --owner <ACCOUNT> \
           --filter "Name=start-time,Values=${YEAR}-${MONTH}*" --query \
           'Snapshots[?starts_with(Description, `Created by CreateImage`) == `false`]|[?starts_with(Description, `Copied for DestinationAmi`) == `false`].[StartTime,SnapshotId,VolumeSize,Description]' \
           --output text | wc -l
      done
    done
done | sed '/ 0$/d'

The first suck part with a JMSE query is it can be really long. Worse: it  really doesn't tolerate line-breaking to make scripts less "sprawly". If you're like me, you like to keep a given line of code in a script no longer than "X" characters. I usually prefer X=80. JMSE queries pretty much say, "yeah: screw all that prettiness nonsense".

At any rate to quickly explain the easier parts of  the above:

• First, this is a basic BASH wrapper (should work in other shells that are aware of Bourne-ish syntax)
• I'm using nested loops: first level is year, second level is month, third level is AWS region. This allows easy grouping of output.
• I'm using `printf` to give my output meaning
• At the end of my `aws` command, I'm using `wc` to provide a simple count of lines returned: when one uses the `--output text` argument to the `aws` command, each object's returned data is done as a single line ...allowing `wc` to provide a quick tally of lines meeting the JMSE selection-criteria
• At the end of all my looping, I'm using `sed` to suppress any lines where a given region/month/year has no snapshots found

Notice I don't have a bullet answering why I'm bothering to define an output string ...and then, essentially, throwing that string away. Simply put, I have to output something for `wc` to  count and I may as well output several useful items in case I want to `tee` it off and use the data in an extension to the script.

The fun part of the above is the JMSE horror-show:

• When you query a EBS snapshot object, it outputs a JSON document. That document's structure looks like:

  {
      "Snapshots": [
          {
              "Description": "Snapshot Description Strin",
              "Tags": [
                  {
                      "Value": "TAG1 Value",
                      "Key": "TAG1"
                  },
                  {
                      "Value": "TAG2 Value",
                      "Key": "TAG2"
                  },
                  {
                      "Value": "TAG3 Value",
                      "Key": "TAG3"
                  }
              ],
              "Encrypted": false,
              "VolumeId": "",
              "State": "completed",
              "VolumeSize": ,
              "StartTime": "--
  T::.000Z",
              "Progress": "100%",
              "OwnerId": "",
              "SnapshotId": ""
          }
      ]
  }
  

• The ".[StartTime,SnapshotId,VolumeSize,Description]" portion of the "--query" string constrains the output to containing the "StartTime", "SnapshotId", "VolumeSize" and "Description" elements from the object's JSON document. Again, not of immediate use when doing a basic element-count census but is convertible to something useful in later tasks.
• When constructing a JMSE query, you're informing the tool, which elements of the JSON node-tree you want to work on. You pretty much always have to specify the top-level document node. In this case, that's "Snapshots". Thus, the initial part of the query-string is "Snapshots[]". Not directly germane to this document, but likely worth knowing:
  • In JMSE, "Snapshots[]" basically means "give me everything from 'Snapshots' on down".
  • An equivalent to this is "Snapshots[*]".
  • If you wanted just the first object returned, you'd use  "Snapshots[0]"
  • If you wanted just a sub-range of objects returned, you'd use  "Snapshots[X:Y]"
• If you want to restrict your output based on selection criteria, JMSE provides the "[?]" construct. In this particular case, I'm using the "starts_with" query, or  "[?starts_with]". This query takes arguments: what sub-attribute you're querying; the string to match against and whether your positively or negatively selecting. Thus:
  • To eliminate snapshots with descriptions starting "Created by CreateImage", my query looks like: "?starts_with(Description, `Created by CreateImage`) == `false`"
  • To eliminate snapshots with descriptions starting "Copied for DestinationAmi", my query looks like: "?starts_with(Description, `Copied for DestinationAmi`) == `false`"
  • Were I selecting for either of these strings — rather than the current against-match — I would need to change my "false" selectors to "true".
• To do a compound query of an OR type, one does "[query1]|[query2]". Notionally, I could string together as many of these as I wanted if I needed to select for/against any arbitrary number of attribute values by adding in further "|" operators.

At the end of running my script, I ended up with output that looked like:

    2017-07 (us-east-1): 61
    2017-08 (us-east-1): 746
    2017-09 (us-east-1): 196
    2017-10 (us-east-1): 4
    2017-11 (us-east-1): 113
    2017-12 (us-east-1): 600
    2018-01 (us-east-1): 149
    2018-03 (us-east-1): 6
    2018-04 (us-east-1): 3
    2018-06 (us-east-1): 302
    2018-07 (us-east-1): 1620
    2018-08 (us-east-1): 206
    2018-08 (us-west-2): 3

Obviously, there's some cleanup to be done in the queried account: it's unlikely that any snapshots older than 30 days — stuff that's from 2017 is almost a lock to be the result of "bad housekeeping".

The Pain of LDAP

Recently, we've had a push to enable 2FA on all of the internet-facing services we offer. A manager asked, "how many users do we have to worry about?" We have three basic types of users:

• Those who only have access to the Remote Desktops
• Those who only have access to the various applications
• Those who have access to both the Remote Desktops and to the various appliations

All such access is unprivileged. All accesses are managed via Active Directory group memberships. 

• Our RDSH users are in a "remote desktops" group.
• Our application users are in a mix of groups. Fortunately, we long ago implemented standard naming for our groups. All group-names for one application-class start with "T_"; all group-names for the other application-class start with "E_"

Fortunately, the above means that I could get an answer for the manager by way of LDAP queries. A user's membership in an Active Directory is conveyed via the user-object's meberOf attribute. This means that the basic query-objects I needed to consider were:

• RDSH Users: In LDAP-speak, this works out to CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>
• "T" Application-group: In LDAP-speak, this works out to CN=T_*
• "E" Application group: In LDAP-speak, this works out to CN=E_*

LDAP queries generally operate on object-sets. Sets are denoted by parentheses. Using the previously-mentioned groups, a single-element set for each of my group-memberships would be:

• (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• (memberof=CN=T_*)
• (memberof=CN=E_*)

Where things get really freaking ugly is when you need to combine these into appropriate queries. Combining query-objects is a matter of using the booleans:

• "&": the logical AND operator
• "|": the logical OR operator
• "!": the logical NOT operator
• If you're familiar with logical operators, you'll notice that there isn't a built in eXclusive OR type of operator. 

Why I say things are ugly is that you apply a given operator within a set and the syntax is a little non-intutive:

• ANDing sets: Uses the syntax (&(SET1)(SET2)(...)(SET_N)). Basically, this says, "AND all of the following sets within this AND-set. An AND-set can consist of 2+ sets for evaluation
• ORing sets: Similarly, this uses the syntax (|(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this OR-set. An OR-set can consist of 2+ sets for evaluation
• NOTin sets: Similarly, this uses the syntax (!(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this NOT-set. An NOT-set can consist of 2+ sets for evaluation

Where it gets really ugly is when a logical operation acually requires two or more sub operations. For example:

• When you want to tell if a user is a member of one group but not another, you need to do something like: (&(GROUP1)(!(GROUP2))
• Similarly, when you want to tell if a user is a member of one group but not a member of both of two other groups, you would do something like (&(GROUP1)(!(&(GROUP2)(GROUP3))
• Similarly, when you want to tell if a user is a member of one group but not a member of either of two other groups, you would do something like (&(GROUP1)(!(|(GROUP2)(GROUP3))

As you can probably tell, as the number of selectors goes up, so does the ugliness.

At any rate, back to my original queries:

• Getting a list of all people with RDSH access, my query looks like: (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• Getting a list of people with membership in either of the application groups, my query looks like: (|(memberof=CN=T_*)(memberof=CN=T_*))
• Getting a list of people with RDSH access and with membership in either of the application groups, my query looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(|(memberof=CN=T_*)(memberof=CN=T_*)))
• Getting a list of people with RDSH access but not access to either of the other groups , my query now looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(!(|(memberof=CN=T_*)(memberof=CN=T_*))))
• Conversely, getting a list of people without RDSH access but with access to either of the other groups, my query now looks like: (&(!(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>))(|(memberof=CN=T_*)(memberof=CN=T_*)))

All that ugliness aside, I was able to get the manager the answer to his question and was able to do it quickly. This post is mostly a result of him asking "what the hell does all that gobbledygook mean" when I sent him the queries I ran to get him the answer. I'd sent that as part of the thread for the same reason I'm writing this: so I have the queries the next time the information is asked for again (and don't have to re-work it all out in my head again).

2FA and the Heartbreak of SMS

So, a good long while ago, I started enabling all of my Internet-accessible accounts that supported it with two-factor authentication (a.k.a., "2FA"). It has been a somewhat 'fraught' road.

Typically, sites offer you the capability of using token-based — hardware or, thankfully, "virtual"/software — devices or SMS. I chose SMS because several of my customers make use of hardware-based tokens (like the YubiKeys I wanted to use) not practicable. Then, a couple years in, someone discovered "SS7 has a vulnerability that has the side-effect of rendering SMS-based 2FA no longer secure" and, not long after, a documented exploitation was discovered. Warnings were sent out saying "don't use SMS for 2FA".

However, when your work circumstances are like mine sometimes are, SMS-based 2FA was the only meaningful option, so you opted to accept the risk and keep going as is. Afterall, some form of 2FA - even if potentially exploitable - is better than no 2FA at all ...if only in that it made attacking you harder than someone that didn't have even that meager second level of protection. I mean, you lock the doors to your car and to your house even though you know that an adequately determined and/or skillful thief can still break-in, right?

At any rate, those warnings started coming out even before "in the wild" exploits were discovered. However, prior to those discoveries, the warnings were mostly advisory. Then, last year, AWS decided "we're sunsetting SMS-based 2FA: it was a beta capability, any way." That left me kind of "stuck". When I'd first set up my AWS accounts with 2FA, the only non-SMS options were either hardware keys or cellphone-based soft-keys. As mentioned previously, the hardware keys were not an option. Similarly, so were cellphone-based soft-keys. While there were other software-based key options, few of them had desktop-based clients. The few of those that had desktop-based options, most seemed to assume that you only ever used one desktop (such that, if you, say, set up your bank account for 2FA using such an app on your home computer, you no longer had the ability to use your bank's website at work without using one of the "backup" access codes). I thought I was going to be stuck going back to 2FA-less for my AWS accounts.

Fortunately, it seems like there's more options for 2FA, some of which have included the design-assumption, "many people don't have the ability to rely on just one token (virtual) device". Or, maybe they simply realized that no one was going to carry around an inch-thick deck of filing-cards, each with a given 2FA-enabled site's backup codes printed on it. Don't really care on the why, simply care that now I can "2FA all the things" (that support 2FA at all).

This morning, it was a dead day at work (a lot of people in the US take off the day following the Fourth of July). So, I decided to see whether/how to change my AWS account to use a 2FA protection that wasn't reliant on SMS. I looked at AWS's list of supported soft-keys — Google Authenticator (and presumably ones that operate compatibly – like LastPass's claims to) and Authy — it made my choice easy: the Google Authenticator only works on cellphones, thus, I could eliminate it. So, I opted to download and install Authy to see if I could make it work (surprisingly, they don't yet seem to have options for Linux desktop users). Also wanted to see how hard the setup was so I could make an informed "we should require all our AWS users to 2FA-enable their accounts" (or not) recommendation to our security-policy team. So, after downloading and installing Authy:

1. Not wanting to lock myself out of my AWS account, I created a testing console-user with console login rights.
2. I logged out of the account I have that's enabled for IAM-user creation
3. Logged in to my test account using standard user/password credentials
4. Went to the IAM console for my test-user
5. Clicked on the "Security Credentials" tab
6. Clicked on the (edit) pencil-icon next to the "Assigned MFA device" row
7. Selected "A virtual MFA device" from the first dialog-popup
8. Clicked "Next" till I got to the "Manage MFA Device" screen
9. Clicked on the "Show secret key for manual configuration" button to expose the manual-configuration token-string (since, my computer had no ability to scan a QR code)
10. Clicked on the "Tokens" icon in Authy, and then the "+" icon to add a new account-binding
11. Copied the secret key from the AWS page to my cut-buffer
12. Pasted the AWS secret key into the "Enter Code given by the website" text-box in Authy and hit the "Add Account" button
13. Chose a meaningful account name and icon and hit the "Save" button
14. Then copied the first and second six-charater tokens into the "Authentication Code 1" and "Authentication Code 2" fields, then clicked the "Activate Virtual MFA" button
15. After AWS popped up its "The MFA device was successfully associated with your account.
    " message, I clicked the "Finish" button to exit the MFA setup tool
16. I then logged into the account using a different browser. After entering the test user's normal user/password credentials, it prompted me for my Authy-generated string. I entered that and I was into the AWS console just like normal.
    
17. I quickly logged out from my test account and then nuked it via my IAM-administrator enabled account.

I repeated the above with my other AWS accounts (I have several - each with different priv-sets). Each was a breeze to set up and use.

Now to see how many of my other 2FA-via-SMS accounts I can convert to Authy. I'd really rather have "one ring", but some sites want you to use particular 2FA tools.

Streamed Backups to S3

Introduction/Background

Many would-be users of AWS come to AWS from a legacy hosting background. Often times, when moving to AWS, the question, "how do I back my stuff up when I no longer have access to my enterprise backup tools," is asked. If not, it's a question that would-be AWS users should be asking.

AWS provides a number of storage options. Each option has use-cases that it is optimized for. Each also has a combination of performance, feature and pricing tradeoffs (see my document for a quick summary of these tradeoffs). The lowest-cost - and therefore most attractive for data-retention use-cases typical of backups-related activities - is S3. Further, within S3, there are pricing/capability tiers that are appropriate to different types of backup needs (the following list is organized by price, highest to lowest):

• If there is a need to perform frequent full or partial recoveries, the S3 Standard tier is probably the best option
• If recovery-frequency is pretty much "never" — but needs to be quick if there actually is a need to perform recoveries — and the policies governing backups mandates up to a thirty-day recoverability window, the best option is likely the S3 Infrequent Access (IA) tier.
• If there's generally no need for recovery beyond legal compliance capabilities, or the recovery-time objectives (RTO) for backups will tolerate a multi-hour wait for data to become unavailable, the S3 glacier layer is probable the best option.

Further, if projects backup needs span the usage profiles of the previous list, data lifecycle policies can be created that will move data from a higher-cost layer to a lower-cost layer based on time thresholds. And, to prevent being billed for data that has no further utility, the lifecycle policies can include an expiration-age at which AWS will simply delete and stop charging for the backed up data.

There are a couple of ways to get backup data into S3:

• Copy: The easiest — and likely most well known — is to simply copy the data from a host into an S3 bucket. Every file on disk that's copied to S3 exists as an individually downloadable file in S3. Copy operations can be iterative or recursive. If the copy operation takes the form of a recursive-copy, basic location relationship between files is preserved (though, things like hard- or soft-links get converted into multiple copies of a given file). While this method is easy, it includes a loss of filesystem metadata — not just the previously-mentioned loss of link-style file-data but ownerships, permissions, MAC-tags, etc.
• Sync: Similarly easy is the "sync" method. Like the basic copy Every file on disk that's copied to S3 exists as an individually downloadable file in S3. The sync operation is inherently recursive. Further, if an identical copy of a file exists within S3 at a given location, the sync operation will only overwrite the S3-hosted file if the to-be-copied file is different. This provides good support for incremental style backups. As with the basic copy-to-S3 method, this method results in the loss of file-link and other filesystem metadata.
  
  
  Note: if using this method, it is probably a good idea to turn on bucket-versioning to ensure that each version of an uploaded file is kept. This allows a recovery operation to recover a given point-in-time's version of the backed-up file.
• Streaming copy: This method is the least well-known. However, this method can be leveraged to overcome the problem of loss filesystem metadata. If the stream-to-S3 operation includes an inlined data-encapsulation operation (e.g., piping the stream through the tar utility), filesystem metadata will be preserved.
  
  
  Note: the cost of preserving metadata via encapsulation is that the encapsulated object is opaque to S3. As such, there's no (direct) means by which to emulate an incremental backup operation.

Technical Implementation

As the title of this article suggests, the technical-implementation focus of this article is on streamed backups to S3.

Most users of S3 are aware of is static file-copy options. That is copying a file from an EC2 instance directly to S3. Most such users, when they want to store files in EC2 and need to retain filesystem metadata either look to things like s3fs or do staged encapsulation.

The former allows you to treat S3 as though it were a local filesystem. However, for various reasons, many organizations are not comfortable using FUSE-based filesystem-implementstions - particularly opensource project ones (usually due to fears about support if something goes awry)

The latter means using an archiving tool to create a pre-packaged copy of the data first staged to disk as a complete file and then copying that file to S3. Common archiving tools include the Linux Tape ARchive utility (`tar`), cpio or even `mkisofs`/`genisoimage`. However, if the archiving tool supports reading from STDIN and/or writing to STDOUT, the tool can be used to create an archive directly within S3 using S3's streaming-copy capabilities.

Best practices for backups is to ensure that the target data-set is in a consistent state. Generally, this means that the data to be archived is non-changing. This can be done by quiescing a filesystem ...or snapshotting a filesystem and backing up the snapshot. Use of LVM snapshots will be used to illustrate how to a consistent backup of a live filesystem (like those used to host the operating system.)

Note: this illustration assumes that the filesystem to be  backed up is built on top of LVM. If the filesystem is built on a bare (EBS-provided) device, the filesystem will need to be stopped before it can be consistently streamed to S3.

The high-level procedure is as follows:

1. Create a snapshot of the logical volume hosting the filesystem to be backed up (note that LVM issues an `fsfreeze` operation before to creating the snapshot: this flushes all pending I/Os before making the snapshot, ensuring that the resultant snapshot is in a consistent state). Thin or static-sized snapshots may be selected (thin snapshots are especially useful when snapshotting multiple volumes within the same volume-group as one has less need to worry about getting the snapshot volume's size-specification correct).
2. Mount the snapshot
3. Use the archiving-tool to stream the filesystem data to standard output
4. Pipe the stream to S3's `cp` tool, specifying to read from a stream and to write to object-name in S3
5. Unmount the snapshot
6. Delete the snapshot
7. Validate the backup by using S3's `cp` tool, specifying to write to a stream and then read the stream using the original archiving tool's capability to read from standard input. If the archiving tool has a "test" mode, use that; if it does not, it is likely possible to specify /dev/null as its output destination.

For a basic, automated implementation of the above, see the linked-to tool. Note that this tool is "dumb": it assumes that all logical volumes hosting a filesystem should be backed up. The only argument it takes is the name of the S3 bucket to upload to. The script does only very basic "pre-flight" checking:

• Ensure that the AWS CLI is found within the script's inherited PATH env.
• Ensure that either an AWS IAM instance-role is attached to the instance or that an IAM user-role is defined in the script's execution environment (${HOME}/.aws/credential files not currently supported). No attempt is made to ensure the instance- or IAM user-role has sufficient permissions to write to the selected S3 bucket
• Ensure that a bucket-name has been passed, but not checked for validity.

Once the pre-flights pass: the script will attept to snapshot all volumes hosting a filesystem; mount the snapshots under the /mnt hierarchy — recreating the original volumes' mount-locations, but rooted in /mnt; use the `tar` utility to encapsulate and stream the to-be-archived data to the s3 utility; use the S3 cp utility to write tar's streamed, encapsulated output to the named S3 bucket's "/Backups/" folder. Once the S3 cp utility closes the stream without errors, the script will then dismount and delete the snapshots.

Alternatives

As mentioned previously, it's possible to do similar actions to the above for filesystems that do not reside on LVM2 logical volumes. However, doing so will either require different methods for creating a consistent state for the backup-set or backing up postentially inconsistent data (and possibly even wholly missing "in flight" data).

EBS has the native ability to create copy-on-write snapshots. However, the EBS volume's snapshot capability is generally decoupled from the OS'es ability to "pause" a filesystem. One can use a tool — like those in the LxEBSbackups project — to coordinate the pausing of the filesystem so that the EBS snapshot can create a consistent copy of the data (and then unpause the filesystem as soon as the EBS snapshot has been started).

One can leave the data "as is" in the EBS snapshot or one can then mount the snapshot to the EC2 and execute a streamed archive operation to S3. The former has the value of being low effort. The latter has the benefit of storing the data to lower-priced tiers (even S3 standard is cheaper than snapshots of EBS volumes) and allowing the backed up data to be placed under S3 lifecycle policies.

"How Big Is It," You Ask?

For one of the projects I'm supporting, they want to deploy into an isolated network. However, they want to be able to flexibly support systems in that network being able to fetch feature and patch RPMs (and other stuff) as needed. Being on an isolated network, they don't want to overwhelm the security gateways by having a bunch of systems fetching directly from the internet. Result: a desire to host a controlled mirror of the relevant upstream software repositories.

This begged the question, "what all do we need to mirror." I stress the word "need" because this customer is used to thinking in terms of manual rather than automated efforts (working on that with them, too). As a result of this manual mind-set, they want to keep data-sets around tasks small.

Unfortunately, their idea of small is also a bit out of date. To them, 100GiB — at least for software repositories — falls into the "not small" category. To me, if I can fit something on my phone, it's a small amount of data. We'll ignore the fact that I'm biased by the fact that I could jam a 512GiB micro SD card into my phone. At any rate, they were wanting to minimize the number of repositories and channels they were going to need to mirror so that they could keep the copy-jobs small. I pointed out "you want these systems to have a similar degree of fetching-functionality to what they'd have if they weren't being blocked from downloading directly from the Internet: that means you need <insert exhaustive list of repositories and channels>". Naturally, they balked. The questions "how do you know they'll actually need all of that" and "how much space is all of that going to take" were asked (with a certain degree of overwroughtness).

I pointed out that I couldn't meaningfully answer the former question because I wasn't part of the design-groups for the systems that were to be deployed in the isolated network. I also pointed out that they'd probably be better asking the owners of the prospective systems what they'd anticipate needing (knowing full well that, usually, such questions' answers are somewhere in the "I'm not sure, yet" neighborhood). As such, my exhaustive list was a hedge: better to have and not need than to need and not have. Given the stupid-cheapness of storage and that it can actually be easier to sync all of the channels in a given repository vice syncing a subset, I didn't see a a benefit to not throw storage at the problem.

To the second question, I pointed out, "I'm sitting at this meeting, not in front of my computer. I can get you a full, channel-by-channel breakdown once I get back to my desk.

One of the nice things about yum repositories (where the feather and patch RPMs would come from) is that they're easily queryble for both item-counts and aggregate sizes. Only down side is that the OS-included tools for doing so are more for human-centric, ad hoc queries rather than something that can be jammed into an Excel spreadsheet and =SUM formulae being run. In other words, sizes are put into "friendly" units: if you have 100KiB of data, the number is reported in KiB; if you have 1055KiB of data, the number is reported in MiB; and so on. So, I needed to "wrap" the native tools output to put everything into consistent units (which Excel prefers for =SUMing and other mathematical actions). Because it was a "quick" task, I did it in BASH. In retrospect, using another language likely would have been far less ugly. However, what I came up with worked for creating a CSV:

#!/bin/bash

for YUM in $(
   yum repolist all | awk '/abled/{ print $1}' | \
      sed -e '{
         /-test/d
         /-debuginfo/d
         /-source/d
         /-media/d
      }'  | sed 's/\/.*$//'
   )
do
IFS=$'
'
   REPOSTRUCT=($(
      yum --disablerepo=* --enablerepo=${YUM} repolist -v | \
      grep ^Repo- | grep -E "(id|-name|pkgs|size) *:" | \
      sed 's/ *: /:/'
   ))
   unset IFS

   REPSZ=($(echo "${REPOSTRUCT[3]}" | sed 's/^Repo-size://'))

   if [[ $( echo "${REPSZ[1]}" ) = M ]]
   then
      SIZE=$(echo "${REPSZ[0]} * 1024" | bc)
   elif [[ $( echo "${REPSZ[1]}" ) = G ]]
   then
      SIZE=$(echo "${REPSZ[0]} * 1024 * 1024 " | bc)
   else
      SIZE="${REPSZ[0]}"
   fi
 
   for CT in 0 1 2
   do
      printf "%s;" "${REPOSTRUCT[${CT}]}"
   done
   echo ${SIZE}
done | sed 's/Repo-[a-z]*://'

Yeah... hideous and likely far from optimal ...but not worth my time (even if I had it) to revisit. It gets the job done and there's a certain joy to writing hideous code to solve a problem you didn't want to be asked to solve in the first place.

At any rate, checking against all of the repos that we'd want to mirror for the project, the initial-sync data-set would fit on my phone (without having to upgrade to one of  top-end beasties). Pointing out that the only the initial sync would be "large" and that only a couple of the channels updated with anything resembling regularity (the rest being essentially static), the monthly delta-sync would be vastly smaller and trivial to babysit. So, we'll see whether that assuages their anxieties or not.

Head-Smashy Goodness

One of the joys in the life of being an automation engineer is when you have to "spiff up" the automation put together by other. In general, I consider myself a crap programmer. This assessment is in spite of seeing the garbage written by people that actually consider themselves to be good programmers. It's a matter of what you compare yourself to, I guess: some people compare themselves to where they started; I compare myself to some idea of where I perceive "expert" to be.

At any rate, my most recent project has me "improving" on the automation that another group slapped together and never had time to improve. I look at the code they left me to "improve" and can't help but suspect that, given all the time in the universe, they wouln't meaningfully improve. Oh well: it's giving me an excuse to learn and use a new tool-set.

My first assigned task was improving the deployment automation for their Sonarqube software. I'm still a very long way from where I'd like the improvements to be, but I finally figured out why the hell their RDS deployments from snapshots were taking so mind-bendingly long. You see, when I write RDS deployment-automation, I specify everything. I'm a control freak, so that's just how I am. Sadly, I sort of assume a similar approach will be taken by others. Bad assumption to make.

In trying to debug why it was taking the best part of ninety minutes to deploy a small (100GiB) database from an RDS snapshot, I discovered that what was going on was that RDS was taking forever and a day to convert an instance launched as a "standard" ("magnetic") disk-type instance to one that would run as a GP2. Apparently, when the original template was written, they hadn't specified the RDS instance type. This failure to specify means that RDS uses its default-type ...which is "standard".

Subsequent to their having deployed from that template, they'd (apparently) manually updated the RDS instance type to GP2. I'm assuming this to be the case because the automation around the RDS had never been modified - no "stack update" had been done at any point in time. Indeed, there was nothing in their stack-definition that even allowed the subsequent re-selection of the instance's storage-type.

Interestingly enough, when one launches a new RDS from a snapshot taken of another RDS, AWS attempts to ensure that the new RDS is of the same storage-type. I the snapshot was taken of a GP2-enabled instance, it wants to make the new RDS use GP2. This is all well and good: it prevents you from having unexpectedly different performance-characteristics between the snap-source and the new RDS instantiation.

Unfortunately, in a case where the templates don't override defaults, you get into a HORRIBLE situation when you deploy from that snapshot. Specifically, CloudFormation will create an RDS instance-type with the default, "magnetic" storage. Then, once the instance is created and the data recovered, RDS then attempts to convert the new instance to use storage-type that matched the snap-source. You sit there, staring at the RDS output wondering "what the hell are you 'modifying' and why is it taking so freaking long???". Staring unseeingly at the screen long enough, you might notice that the pending modification task is "storage". If that registered to your wait-numbed brain, you'll have an "oh shit" moment and hand-crank a new RDS from the same snapshot, being careful to match the storage-types of the new instance and the snapshot. Then, you'll watch with some degree of incredulity while the hand-cranked RDS reaches a ready state long before the automated deployment reaches its'.

Oh well, at least I now know why it was so balls-slow an have "improved" the templates accordingly. Looks like I'll be waiting maybe ten minutes for a snap-sourced RDS rather than the 90+ that it was taking.

So You Wanna Run Sonarqube on (Hardened) RHEL/CentOS 7

As developers become more concerned with injecting security-awareness into their processes, tools like Sonarqube become more popular. As that popularity increases, the desire to run that software on more than just the Linux distro(s) it was originally built to run on increases.

Most of my customers only allow the use of Red Hat (and, increasingly, CentOS) on their production networks. As these customers hire developers that cut their teeth not on Red Hat, the pressure for deploying their preferred-tools onto Enterprise Linux increases.

Sonarqube has two relevant installation-methods listed on their installation documentation. One is to explode a ZIP archive and install the files wherever desired and set up requisite automated startup as appropriate. The other is to install a "semi-official" RPM-packaging. Initially, I chose the former, but subsequently changed to the latter. While the former is fine if you're installing very infrequently (and installing by hand), it can fall down if you're doing automated deployments by way of service-frameworks like Amazon's CloudFormation (CFn) and/or AutoScaling.

My preferred installation method is to use CFn to create AutoScaling Groups (ASGs). I don't run Sonarqube clustered: the community edition that my developers lack of funding allows me to deploy doesnt support clustering.  Using ASGs means that I can increase the service's availability by having the Sonarqube service simply rebuild itself if the service ever becomes unexpectedly offline. Rebuilds only take a couple minutes and generally don't require any administrator intervention.

This method is/was compatible with the ZIP-based installation method for the first month or so of running the service. Then, one night, it stopped working. When I investigated to figure out why it stop, I found that the URL the deployment-automation was pointing to no longer existed. I'm not really sure what happened because the Sonarqube maintaiers seem to keep back versions of the software around indefinitely.

/shrug

At any rate, it caused me to return to the installation documentation. This time, I noticed that there was an RPM option. So, I went down that path. For the most part, it made things dead easy vice using the ZIP packaging. There were only a few gotchas with the RPM:

• The current packaging, in addition to being a "noarch" packaging, is neither EL version-specific nor is it cross-version. Effectively, it's EL6-oriented: it includes a legacy init script but not a systemd service definition. The RPM maintainer likely needs to either make an EL6 and an EL7 packaging or needs to update the packaging to include files for both init-types and then use a platform-selector to install and activate the appropriate one
• The current packaging creates the service account 'sonar' and installs the packaged files as user:group 'sonar' ...but doesn't include start-logic to ensure that the Sonarqube runs as 'sonar'. This will manifest in Sonarqube's es.log file similarly to:
  

  [2018-02-21T15:45:51,714][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
  org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
          at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
  Caused by: java.lang.RuntimeException: can not run elasticsearch as root
          at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:100) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:176) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
          ... 6 more

• The embedded ElastiCache service requires that the file-descriptors ulimit be at a specific minimum value: if the deployment is on a hardened system, it is reasonably likely the default ulimits are too low. If too low, this will manifest in Sonarqube's es.log file similarly to:
  

  [2018-02-21T15:47:23,787][ERROR][bootstrap                ] Exception
  java.lang.RuntimeException: max file descriptors [65535] for elasticsearch process likely too low, increase to at least [65536]
      at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
      at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
      at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:188)
      at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:264)
      at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
      at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
      at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
      at org.elasticsearch.cli.Command.main(Command.java:53)
      at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
      at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)

• The embedded ElastiCache service requires that the kernel's vm.max_map_count runtime parameter be tweaked. If not adequately tweaked, this will manifest in Sonarqube's es.log file similarly to:

  max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

• The embedded ElastiCache service's JVM won't start if the /tmp directory's had the noexec mount-option applied as part of the host's hardening. This will manifest in Sonarqube's es.log file similarly to:
  

  WARN  es[][o.e.b.Natives] cannot check if running as root because JNA is not available 
  WARN  es[][o.e.b.Natives] cannot install system call filter because JNA is not available 
  WARN  es[][o.e.b.Natives] cannot register console handler because JNA is not available 
  WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_NPROC because JNA is not available
  WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_AS because JNA is not available
  WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_FSIZE because JNA is not available

• External users won't be able to communicate with the service if the host-based firewall hasn't been appropriately configured.

The first three items are all addressable via a properly-written systemd service definition. The following is what I put together for my deployments:

[Unit]
Description=SonarQube
After=network.target network-online.target
Wants=network-online.target

[Service]
ExecStart=/opt/sonar/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonar/bin/linux-x86-64/sonar.sh stop
ExecReload=/opt/sonar/bin/linux-x86-64/sonar.sh restart
Group=sonar
LimitNOFILE=65536
PIDFile=/opt/sonar/bin/linux-x86-64/SonarQube.pid
Restart=always
RestartSec=30
User=sonar
Type=forking

[Install]
WantedBy=multi-user.target

I put this into GitHub project I set up as part of automating my deployments for AWS. The 'User=sonar' option causes systemd to start the process as the sonar user. The 'Group=sonar' directive ensures that the process runs under the sonar. The 'LimitNOFILE=65536' ensures that the service's file-descriptor's ulimit is set sufficiently high.

The fourth item is addressed by creating an /etc/sysctl.d/sonarqube file containing:

vm.max_map_count = 262144

I take care of this with my automation but the RPM maintainer could obviate the need to do so by including an /etc/sysctl.d/sonarqube file as part of the RPM.

The fifth item is addressable through the sonar.properties file. Adding a line like:

sonar.search.javaAdditionalOpts=-Djava.io.tmpdir=/var/tmp/elasticsearch

Will cause Sonarqube to point ElasticSearch's temp-directory to /var/tmp/elasticsearch. Note that this is an example directory: pick a location in which the sonar user can create a directory and that is on a filesystem that does not have the noexec mount-option set. Without making this change, ElasticSearch will fail to start because JNI cannot be started.

The final item is adressable by adding a port-exception to the firewalld service. This can be done as simply as executing `firewall-cmd --permanent --service=sonarqube --add-port=9000/tcp` or adding a /etc/firewalld/services/sonarqube.xml file containing:

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>Sonarqube service ports</short>
  <description>Firewalld options supporting Sonarqube deployments</description>
  <port protocol="tcp" port="9000" />
  <port protocol="tcp" port="9001" />
</service>

Either is suitable for inclusion in an RPM: the former by way of a %post script; the latter as a config-file packaged in the RPM.

Once all of the bulleted-items are accounted for, the RPM-included service should start right up and function correctly on a hardened EL7 system. Surprisingly enough, no other tweaks are typically needed to make Sonarqube run on a hardened system. Sonarqube will typically function just fine with both FIPS-mode and SELinux active.

It's likely that once Sonarqube is deployed, functionality beyond the defaults will be desired. There are quite a number of functionality-extending plugins available. Download the ones you want and install them to <SONAR_INSTALL_ROOT>/extensions/plugins and restart the service. Most plugins behavior is configurable via the Sonarqube administration GUI.

Note: while much of the 'fixes' following the gotcha list are scattered throughout the Sonarqube documentation, on an unhardened EL 7 system, the system defaults are loose enough that accounting for them was not necessary for either the ZIP- or RPM-based installation-methods.

Extra: If you want the above taken care of for you and you're trying to deploy Sonarqube onto AWS, I put together a set of template-driven CloudFormation templates and helper scripts. They can be found on GitHub.

Why I Love The UNIX/Linux CLI

There's many reasons, really. Probably too many to list in any one sitting.

That said, some days you pull an old trick out of your bag and you're reminded all over why you love something. Today was such a day.

All day, we've been having sporadic performance issues with AWS's CloudFormation functionality. Some jobs will run in (a few tens of) minutes, others will take an hour or more. Now, this would be understandable if they were markedly different tasks. But that's not the case. It's literally "run this template 'A' and wait" followed by "re-run template 'A' and wait" where the first run goes in about the time you'd expect and the second run of the same automation-template takes 10x as long.

Fun thing with cloudformation is that, when you launch from the command line, you can pass all of your parameters via a file. Some templates, however, don't like it when there's two active copies that contain the same parameter values. The way around this is to generalize your parameters file and then "process" that parameter file for each running. The UNIX/Linux CLI means that you can do all of this inline, like so:

aws --region us-east-1 cloudformation create-stack --stack-name RedMine07 \
  --disable-rollback --capabilities CAPABILITY_NAMED_IAM \
  --template-url https://s3.amazonaws.com/my-automation-bucket/Templates/RedMine.tmplt.json \
  --parameters file://<(sed 's/__NUM__/07/g' RedMine.parms.json)

The nifty part here is in the last line of that command. When you wrap a shell command in "<()", it runs the command within the parentheses and encapsulates it into a file-handle. Thus, if your main command requires an input-file be specified, the output of your encapsulated command gets treated just like it was being read from an on-disk file rather than as an inline-command.

Slick.

In my case, I'd generalized my parameters file to include a substitutable token, "__NUM__". Each time I need to run the command, all I have to do is change that "07" to a new value and bingo, new stack with unique values where needed.

Another fun thing is the shell's editable history function. If I want to run that exact same command, again, changing my values — both those passed as stack-parameters and the name of the resultant stack — I can do:

!!:gs/07/08

Which causes the prior stanza to be re-run as so:

aws --region us-east-1 cloudformation create-stack --stack-name RedMine08 \
  --disable-rollback --capabilities CAPABILITY_NAMED_IAM \
  --template-url https://s3.amazonaws.com/my-automation-bucket/Templates/RedMine.tmplt.json \
  --parameters file://<(sed 's/__NUM__/08/g' RedMine.parms.json)

Similarly, if the command I wanted to change was the 66th in my history rather than the one I just ran, I could do:

!66:gs/07/08

And achieve the same results.

The Joys of SCL (or, "So You Want to Host RedMine on Red Hat/CentOS")

Recently, I needed to go back and try to re-engineer my company's RedMine deployment. Previously, I had rushed through a quick-standup of the service using the generic CentOS AMI furnished by CentOS.Org. All of the supporting automation was very slap-dash and had more things hard-coded than I really would have liked. However, the original time-scale the automation needed to be delivered under meant that "stupid but works ain't stupid" ruled the day.

That ad hoc implementation was over two years ago, at this point. Since the initial implementation, I had never had the chance to start to revisit it. The lack of a contiguous chunk of time to re-implement changed over the holidays (hooray for everyone being gone over the holiday weeks!). At any rate, there were a couple things I wanted to accomplish with the re-build:

1. Change from using the generic CentOS AMI to using our customized AMI
   • Our AMI has a specific partitioning-scheme for our OS disks that we prefer to be used across all of our services. Wanted to make our RedMine deployment no longer be an "outlier".
   • Our AMIs are updated monthly, whereas the CentOS.Org ones are typically updated when there's a new 7.X. Using our AMIs means not having to wait nearly as long for launch-time patch-up to complete before things can become "live".
2. I wanted to make use of our standard, provisioning-time OS-hardening framework — both so that the instance is better configured to be Internet-facing and so it is more in accordance with our other service-hosts
3. I wanted to be able to use a vendor-packaged Ruby rather than the self-packaged/self-managed Ruby. I had previously gone the custom Ruby route both because the vendor's default Ruby is too old for RedMine to use and because SCL was not quite as mature an offering back in 2015. In updating my automation to make use of SCL, I was figuring that a vendor-packaged RPM is more likely to be kept up to date without potentially introducing breakage that self-packaged RPMs can be prone to.
4. I wanted to switch from using UserData to using CloudFormation (CFn) for launch automation: 
   • Relatively speaking, CFn automation includes a lot more ability to catch errors/deployment-breakage with lower-effort. Typically, it's easier to avoid the "I thought it deployed correctly" happenstances.
   • Using CFn automation exposes less information via instance-metadata either via the management UIs/CLIs or by reading metadata from the instance.
   • Using CFn automation makes it easier to coordinate the provisioning of non-EC2 solution components (e.g., security groups, IAM roles, S3 buckets, EFS shares, RDS databases, load-balancers etc.)
5. I wanted to clean up the overall automation-scripts so I could push more stuff into (public) GitHub (projects) and have to pull less information from private repositories (like S3 buckets).

Ultimately, the biggest challenge turned out to be dealing with SCL. I can probably account for the problems with SCL being their focus. Specifically, they seem to be designed more for use by developers in interactive environments than they are to underpin services. If you're a developer, all you really need to do to have a "more up-to-date than vendor-standard" language packages is:

• Enable the SCL repos
• Install the SCL-hosted language package you're looking for
• Update either your user's shell-inits or the system-wide shell inits
• Log-out and back in
• Go to town with your new SCL-furnished capabilities.

If, on the other hand, you want to use an SCL-managed language to back a service that can't run with the standard vendor-packaged languages (etc)., things are a little more involved.

A little bit of background:

The SCL-managed utilities are typically rooted wholly in "/opt/rh". This means binaries, libraries, documentation. Because of this, your shell's PATH, MANPATH and linker-paths will need to be updated. Fortunately, each SCL-managed package ships with an `enable` utility. You can do something as simple as `scl_enabled <SCL_PACKAGE_NAME> bash`, do the equivalent in your user's "${HOME}/.profile" and/or "${HOME}/.bash_profile" files, or you can do something by way of a file in "/etc/profile.d/". Since I assume "everyone on the system wants to use it", I enable by way of a file in "/etc/profile.d/". Typically, I create a file named "enable_<PACKAGE>.sh" and create contents similar to:

#!/bin/bash
source /opt/rh/<RPM_NAME>/enable
export X_SCLS="$(scl enable <RPM_NAME> 'echo $X_SCLS')"

Within it. Each time an interactive-user logs in, they get the SCL-managed package configured for their shell. If multiple SCL-managed packages are installed, they all "stack" nicely (failure to do the above can result in one SCL-managed package interfering with another).

Caveats:

Unfortunately, while this works well for interactive shell users, it's not generally sufficient for the non-interactive environments. This includes actual running services or stuff that's provisioned at launch-time via scripts kicked off by way of `systemd` (or `init`). Such processes do not use the contents of the "/etc/profile.d" directory (or other user-level initialization scripts). This creates a "what to do" scenario.

Linux affords a couple methods for handling this:

• You can modify the system-wide PATH (etc.). To make the run-time linker find libraries in non-standard paths, you can create files in "/etc/ld.conf.d" that enumerate the desired linker search paths and then run `ldconfig` (the vendor-packaging of MariaDB and a few others do this). However, this is traditionally been considered not to be "good form" and frowned upon.
• Use under systemd:
  • You can modify the scripts invoked via systemd/init; however, if those scripts are RPM-managed, such modifications can be lost when the RPM gets updated via a system patch-up
  • Under systemd, you can create customized service-definitions that contain the necessary environmentals. Typically, this can be quickly done by copying the service-definition found in the "/usr/lib/systemd/system" directory to the "/etc/systemd/system" directory and modifying as necessary.
• Cron jobs: If you need to automate the running of tasks via cron, you can either:
  • Write action-wrappers that include the requisite environmental settings
  • Make use of the ability to set environmentals in jobs defined in /etc/cron.d
• Some applications (such as Apache) allow you to extend library and other search directives via run-time configuration directives. In the case of RedMine, Apache calls the Passenger framework which, in turn, runs the Ruby-based RedMine code. To make Apache happy, you create an "/etc/httpd/conf.d/Passenger.conf" that looks something like:
  

  LoadModule passenger_module /opt/rh/rh-ruby24/root/usr/local/share/gems/gems/passenger-5.1.12/buildout/apache2/mod_passenger.so
  
  <IfModule mod_passenger.c>
          PassengerRoot /opt/rh/rh-ruby24/root/usr/local/share/gems/gems/passenger-5.1.12
          PassengerDefaultRuby /opt/rh/rh-ruby24/root/usr/bin/ruby
          SetEnv LD_LIBRARY_PATH /opt/rh/rh-ruby24/root/usr/lib64
  </IfModule>
  

  The directives:
  • "LoadModule" directive binds the module-name to the fully-qualified path to the Passenger Apache-plugin (shared-library). This file is created by the Passenger installer.
  • "PassengerRoot" tells Apache where to look for further, related gems and libraries.
  • "PassengerDefaultRuby" tells Apache which Ruby to use (useful if there's more than one Ruby version installed or, as in the case of SCL-managed packages, Ruby is not found in the standard system search paths).
  • "SetEnv LD_LIBRARY_PATH" tells Apachels run-time linker where else to look for shared library files.
  Without these, Apache won't find RedMine or all of the associate (SCL-managed) Ruby dependencies.

The above list should be viewed in "worst-to-first" ascending-order of preference. The second systemd and cron methods and the application-configuration method (notionally) have the narrowest scope (area of effect, blast-radius, etc.) — only effecting where the desired executions look for non-standard components — and are the generally preferred methods.

Note On Provisioning Automation Tools:

Because automation tools like CloudFormation (and cloud-init) are not interactive processes, they too will not process the contents of the "/etc/profile.d/" directory. One can use one of the above enumerated methods for modifying the invocation of these tools or include explicit sourcing of the relevant "/etc/profile.d/" files within the scripts executed by these frameworks.