Diagnosing Init Script Issues

Recently, I've been wrestling with some COTS software that one of my customers wanted me to automate the deployment of. Automating its deployment has been... Patience-trying.

The first time I ran the installer as a boot-time "run once" script, it bombed. Over several troubleshooting iterations, I found that one of the five sub-components was at fault. When I deployed the five sub-components individually (and to different hosts), all but that one sub-component failed. Worse, if I ran the automated installer from an interactive user's shell, the installation would succeed. Every. Damned. Time.

So, I figured, "hmm... need to see if I can adequately simulate how to run this thing from an interactive user's shell yet fool the installer into thinking it had a similar environment to what the host's init process provides." So, I commenced to Google'ing.

Commence to Googlin'

Eventually, I found reference to `setsid`. Basically, this utility allows you to spawn a subshell that's detached from a TTY ...just like an init-spawned process. So, I started out with:

     setsid bash -c '/path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

Unfortunately, while the above creates a TTY-less subshell for the script to run in, it still wasn't quite fully simulating an init-like context. The damned installer was still completing successfully. Not what I wanted since, when run from an init-spawned context, it was failing in a very specific way. So, back to the almighty Googs.

Eventually, it occurred to me, "init-spawned processes have very different environments set than interactive user shells do. And, as a shell forked off from my login shell, that `setsid` process would inherit all my interactive shell's environment settings. Once I came to that realization, the Googs quickly pointed me to `env -i`. Upon integrating this nugget:

     setsid bash -c 'env -i /path/to/installer -- \
       -c /path/to/answer.json' < /dev/null 2>&1 \
       | tee /root/log.noshell

My installer started failing the same when launched from an interactive shell as from an init-spawned context. I had something with which I could tell the vendor, "your unattended installer is broken: here's how to simulate/reproduce the problem. FIX IT." I dunno about you, but, to me, an "unattended installer" that I have to kick off by hand from an interactive shell really isn't all that useful.

JMSE Query Fu

Yesterday, I posted up an article illustrating the pain of LDAP's query-language. Today, I had to dick around with JMSE reporting.

The group I work for manages a number of AWS accounts in a number of regions. We also provide support to a number of tenants and their AWS accounts.

This support is necessary because, in much the same way AWS strives to make it easy to adopt "cloud", a lot of people flocking aboard don't necessarily know how to do so cost-effectively. Using AWS can be a cost-effective way to do things, but failure to exercise adequate house-keeping can bite you right in the wallet. Unfortunately, part of the low bar to entry means that a lot of users coming into AWS don't really comprehend that housekeeping is necessary. And, even if they did, they don't necessarily understand how to implement automated housekeeping methods. So, stuff tends to build up over time ...leading to unnecessary cost-exposures.

Inevitably, this leads to a "help: our expenses are running much higher than expected" types of situation. What usually turns out to be a majar cause is disused storage (and orphaned EC2s left running 24/7/365 ...but that's another story). Frequently, this comes out to some combination of orphaned (long-detached) EBS volumes, poorly maintained S3 buckets and elderly EBS snapshots.

IDin'g orphaned EBS volumes is pretty straight forward. Not a lot of query-fu is needed. To see it.

Poorly maintained S3 buckets are a skosh harder to suss out. But, you can usually just enable bucket inventory-reporting an lifecycle policies to help automate your cost-exposure reduction.

While EBS snapshots are relatively low-cost, when enough of them build up, you start to notice the expenses. Reporting can be relatively straightforward. However, if you happen to be maintaining fleets of custom AMIs, doing a simple "what can I delete" report becomes an exercise in query-filtering. Fortunately, AMI-related snapshots are generally identifiable (excludable) with a couple of filters, leaving you a list of snapshots to futher investigate. Unfortunately, writing the query-filters means dicking with JMSE filter syntax ...which is kind of horrid. Dunno if it's more or less horrid than LDAP's.

At any rate, I ended up writing a really simple reporting script to give me a basic idea of whether stale snapshots are even a problem ...and the likely scope of said problem if it existed. While few of our tenants maintain their own AMIs, we do in the account I primarily work with. So, I wrote my script to exclude their snapshots from relevant reports:

for YEAR in $( seq 2015 2018 )
do
   for MONTH in $( seq -w 1 12 )
   do
      for REGION in us-{ea,we}st-{1,2}
      do
         printf "%s-%s (%s): " "${YEAR}" "${MONTH}" "${REGION}"
         aws --region "${REGION}" ec2 describe-snapshots --owner <ACCOUNT> \
           --filter "Name=start-time,Values=${YEAR}-${MONTH}*" --query \
           'Snapshots[?starts_with(Description, `Created by CreateImage`) == `false`]|[?starts_with(Description, `Copied for DestinationAmi`) == `false`].[StartTime,SnapshotId,VolumeSize,Description]' \
           --output text | wc -l
      done
    done
done | sed '/ 0$/d'

The first suck part with a JMSE query is it can be really long. Worse: it  really doesn't tolerate line-breaking to make scripts less "sprawly". If you're like me, you like to keep a given line of code in a script no longer than "X" characters. I usually prefer X=80. JMSE queries pretty much say, "yeah: screw all that prettiness nonsense".

At any rate to quickly explain the easier parts of  the above:

• First, this is a basic BASH wrapper (should work in other shells that are aware of Bourne-ish syntax)
• I'm using nested loops: first level is year, second level is month, third level is AWS region. This allows easy grouping of output.
• I'm using `printf` to give my output meaning
• At the end of my `aws` command, I'm using `wc` to provide a simple count of lines returned: when one uses the `--output text` argument to the `aws` command, each object's returned data is done as a single line ...allowing `wc` to provide a quick tally of lines meeting the JMSE selection-criteria
• At the end of all my looping, I'm using `sed` to suppress any lines where a given region/month/year has no snapshots found

Notice I don't have a bullet answering why I'm bothering to define an output string ...and then, essentially, throwing that string away. Simply put, I have to output something for `wc` to  count and I may as well output several useful items in case I want to `tee` it off and use the data in an extension to the script.

The fun part of the above is the JMSE horror-show:

• When you query a EBS snapshot object, it outputs a JSON document. That document's structure looks like:

  {
      "Snapshots": [
          {
              "Description": "Snapshot Description Strin",
              "Tags": [
                  {
                      "Value": "TAG1 Value",
                      "Key": "TAG1"
                  },
                  {
                      "Value": "TAG2 Value",
                      "Key": "TAG2"
                  },
                  {
                      "Value": "TAG3 Value",
                      "Key": "TAG3"
                  }
              ],
              "Encrypted": false,
              "VolumeId": "",
              "State": "completed",
              "VolumeSize": ,
              "StartTime": "--
  T::.000Z",
              "Progress": "100%",
              "OwnerId": "",
              "SnapshotId": ""
          }
      ]
  }
  

• The ".[StartTime,SnapshotId,VolumeSize,Description]" portion of the "--query" string constrains the output to containing the "StartTime", "SnapshotId", "VolumeSize" and "Description" elements from the object's JSON document. Again, not of immediate use when doing a basic element-count census but is convertible to something useful in later tasks.
• When constructing a JMSE query, you're informing the tool, which elements of the JSON node-tree you want to work on. You pretty much always have to specify the top-level document node. In this case, that's "Snapshots". Thus, the initial part of the query-string is "Snapshots[]". Not directly germane to this document, but likely worth knowing:
  • In JMSE, "Snapshots[]" basically means "give me everything from 'Snapshots' on down".
  • An equivalent to this is "Snapshots[*]".
  • If you wanted just the first object returned, you'd use  "Snapshots[0]"
  • If you wanted just a sub-range of objects returned, you'd use  "Snapshots[X:Y]"
• If you want to restrict your output based on selection criteria, JMSE provides the "[?]" construct. In this particular case, I'm using the "starts_with" query, or  "[?starts_with]". This query takes arguments: what sub-attribute you're querying; the string to match against and whether your positively or negatively selecting. Thus:
  • To eliminate snapshots with descriptions starting "Created by CreateImage", my query looks like: "?starts_with(Description, `Created by CreateImage`) == `false`"
  • To eliminate snapshots with descriptions starting "Copied for DestinationAmi", my query looks like: "?starts_with(Description, `Copied for DestinationAmi`) == `false`"
  • Were I selecting for either of these strings — rather than the current against-match — I would need to change my "false" selectors to "true".
• To do a compound query of an OR type, one does "[query1]|[query2]". Notionally, I could string together as many of these as I wanted if I needed to select for/against any arbitrary number of attribute values by adding in further "|" operators.

At the end of running my script, I ended up with output that looked like:

    2017-07 (us-east-1): 61
    2017-08 (us-east-1): 746
    2017-09 (us-east-1): 196
    2017-10 (us-east-1): 4
    2017-11 (us-east-1): 113
    2017-12 (us-east-1): 600
    2018-01 (us-east-1): 149
    2018-03 (us-east-1): 6
    2018-04 (us-east-1): 3
    2018-06 (us-east-1): 302
    2018-07 (us-east-1): 1620
    2018-08 (us-east-1): 206
    2018-08 (us-west-2): 3

Obviously, there's some cleanup to be done in the queried account: it's unlikely that any snapshots older than 30 days — stuff that's from 2017 is almost a lock to be the result of "bad housekeeping".

The Pain of LDAP

Recently, we've had a push to enable 2FA on all of the internet-facing services we offer. A manager asked, "how many users do we have to worry about?" We have three basic types of users:

• Those who only have access to the Remote Desktops
• Those who only have access to the various applications
• Those who have access to both the Remote Desktops and to the various appliations

All such access is unprivileged. All accesses are managed via Active Directory group memberships. 

• Our RDSH users are in a "remote desktops" group.
• Our application users are in a mix of groups. Fortunately, we long ago implemented standard naming for our groups. All group-names for one application-class start with "T_"; all group-names for the other application-class start with "E_"

Fortunately, the above means that I could get an answer for the manager by way of LDAP queries. A user's membership in an Active Directory is conveyed via the user-object's meberOf attribute. This means that the basic query-objects I needed to consider were:

• RDSH Users: In LDAP-speak, this works out to CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>
• "T" Application-group: In LDAP-speak, this works out to CN=T_*
• "E" Application group: In LDAP-speak, this works out to CN=E_*

LDAP queries generally operate on object-sets. Sets are denoted by parentheses. Using the previously-mentioned groups, a single-element set for each of my group-memberships would be:

• (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• (memberof=CN=T_*)
• (memberof=CN=E_*)

Where things get really freaking ugly is when you need to combine these into appropriate queries. Combining query-objects is a matter of using the booleans:

• "&": the logical AND operator
• "|": the logical OR operator
• "!": the logical NOT operator
• If you're familiar with logical operators, you'll notice that there isn't a built in eXclusive OR type of operator. 

Why I say things are ugly is that you apply a given operator within a set and the syntax is a little non-intutive:

• ANDing sets: Uses the syntax (&(SET1)(SET2)(...)(SET_N)). Basically, this says, "AND all of the following sets within this AND-set. An AND-set can consist of 2+ sets for evaluation
• ORing sets: Similarly, this uses the syntax (|(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this OR-set. An OR-set can consist of 2+ sets for evaluation
• NOTin sets: Similarly, this uses the syntax (!(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this NOT-set. An NOT-set can consist of 2+ sets for evaluation

Where it gets really ugly is when a logical operation acually requires two or more sub operations. For example:

• When you want to tell if a user is a member of one group but not another, you need to do something like: (&(GROUP1)(!(GROUP2))
• Similarly, when you want to tell if a user is a member of one group but not a member of both of two other groups, you would do something like (&(GROUP1)(!(&(GROUP2)(GROUP3))
• Similarly, when you want to tell if a user is a member of one group but not a member of either of two other groups, you would do something like (&(GROUP1)(!(|(GROUP2)(GROUP3))

As you can probably tell, as the number of selectors goes up, so does the ugliness.

At any rate, back to my original queries:

• Getting a list of all people with RDSH access, my query looks like: (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• Getting a list of people with membership in either of the application groups, my query looks like: (|(memberof=CN=T_*)(memberof=CN=T_*))
• Getting a list of people with RDSH access and with membership in either of the application groups, my query looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(|(memberof=CN=T_*)(memberof=CN=T_*)))
• Getting a list of people with RDSH access but not access to either of the other groups , my query now looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(!(|(memberof=CN=T_*)(memberof=CN=T_*))))
• Conversely, getting a list of people without RDSH access but with access to either of the other groups, my query now looks like: (&(!(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>))(|(memberof=CN=T_*)(memberof=CN=T_*)))

All that ugliness aside, I was able to get the manager the answer to his question and was able to do it quickly. This post is mostly a result of him asking "what the hell does all that gobbledygook mean" when I sent him the queries I ran to get him the answer. I'd sent that as part of the thread for the same reason I'm writing this: so I have the queries the next time the information is asked for again (and don't have to re-work it all out in my head again).

2FA and the Heartbreak of SMS

So, a good long while ago, I started enabling all of my Internet-accessible accounts that supported it with two-factor authentication (a.k.a., "2FA"). It has been a somewhat 'fraught' road.

Typically, sites offer you the capability of using token-based — hardware or, thankfully, "virtual"/software — devices or SMS. I chose SMS because several of my customers make use of hardware-based tokens (like the YubiKeys I wanted to use) not practicable. Then, a couple years in, someone discovered "SS7 has a vulnerability that has the side-effect of rendering SMS-based 2FA no longer secure" and, not long after, a documented exploitation was discovered. Warnings were sent out saying "don't use SMS for 2FA".

However, when your work circumstances are like mine sometimes are, SMS-based 2FA was the only meaningful option, so you opted to accept the risk and keep going as is. Afterall, some form of 2FA - even if potentially exploitable - is better than no 2FA at all ...if only in that it made attacking you harder than someone that didn't have even that meager second level of protection. I mean, you lock the doors to your car and to your house even though you know that an adequately determined and/or skillful thief can still break-in, right?

At any rate, those warnings started coming out even before "in the wild" exploits were discovered. However, prior to those discoveries, the warnings were mostly advisory. Then, last year, AWS decided "we're sunsetting SMS-based 2FA: it was a beta capability, any way." That left me kind of "stuck". When I'd first set up my AWS accounts with 2FA, the only non-SMS options were either hardware keys or cellphone-based soft-keys. As mentioned previously, the hardware keys were not an option. Similarly, so were cellphone-based soft-keys. While there were other software-based key options, few of them had desktop-based clients. The few of those that had desktop-based options, most seemed to assume that you only ever used one desktop (such that, if you, say, set up your bank account for 2FA using such an app on your home computer, you no longer had the ability to use your bank's website at work without using one of the "backup" access codes). I thought I was going to be stuck going back to 2FA-less for my AWS accounts.

Fortunately, it seems like there's more options for 2FA, some of which have included the design-assumption, "many people don't have the ability to rely on just one token (virtual) device". Or, maybe they simply realized that no one was going to carry around an inch-thick deck of filing-cards, each with a given 2FA-enabled site's backup codes printed on it. Don't really care on the why, simply care that now I can "2FA all the things" (that support 2FA at all).

This morning, it was a dead day at work (a lot of people in the US take off the day following the Fourth of July). So, I decided to see whether/how to change my AWS account to use a 2FA protection that wasn't reliant on SMS. I looked at AWS's list of supported soft-keys — Google Authenticator (and presumably ones that operate compatibly – like LastPass's claims to) and Authy — it made my choice easy: the Google Authenticator only works on cellphones, thus, I could eliminate it. So, I opted to download and install Authy to see if I could make it work (surprisingly, they don't yet seem to have options for Linux desktop users). Also wanted to see how hard the setup was so I could make an informed "we should require all our AWS users to 2FA-enable their accounts" (or not) recommendation to our security-policy team. So, after downloading and installing Authy:

1. Not wanting to lock myself out of my AWS account, I created a testing console-user with console login rights.
2. I logged out of the account I have that's enabled for IAM-user creation
3. Logged in to my test account using standard user/password credentials
4. Went to the IAM console for my test-user
5. Clicked on the "Security Credentials" tab
6. Clicked on the (edit) pencil-icon next to the "Assigned MFA device" row
7. Selected "A virtual MFA device" from the first dialog-popup
8. Clicked "Next" till I got to the "Manage MFA Device" screen
9. Clicked on the "Show secret key for manual configuration" button to expose the manual-configuration token-string (since, my computer had no ability to scan a QR code)
10. Clicked on the "Tokens" icon in Authy, and then the "+" icon to add a new account-binding
11. Copied the secret key from the AWS page to my cut-buffer
12. Pasted the AWS secret key into the "Enter Code given by the website" text-box in Authy and hit the "Add Account" button
13. Chose a meaningful account name and icon and hit the "Save" button
14. Then copied the first and second six-charater tokens into the "Authentication Code 1" and "Authentication Code 2" fields, then clicked the "Activate Virtual MFA" button
15. After AWS popped up its "The MFA device was successfully associated with your account.
    " message, I clicked the "Finish" button to exit the MFA setup tool
16. I then logged into the account using a different browser. After entering the test user's normal user/password credentials, it prompted me for my Authy-generated string. I entered that and I was into the AWS console just like normal.
    
17. I quickly logged out from my test account and then nuked it via my IAM-administrator enabled account.

I repeated the above with my other AWS accounts (I have several - each with different priv-sets). Each was a breeze to set up and use.

Now to see how many of my other 2FA-via-SMS accounts I can convert to Authy. I'd really rather have "one ring", but some sites want you to use particular 2FA tools.

Streamed Backups to S3

Introduction/Background

Many would-be users of AWS come to AWS from a legacy hosting background. Often times, when moving to AWS, the question, "how do I back my stuff up when I no longer have access to my enterprise backup tools," is asked. If not, it's a question that would-be AWS users should be asking.

AWS provides a number of storage options. Each option has use-cases that it is optimized for. Each also has a combination of performance, feature and pricing tradeoffs (see my document for a quick summary of these tradeoffs). The lowest-cost - and therefore most attractive for data-retention use-cases typical of backups-related activities - is S3. Further, within S3, there are pricing/capability tiers that are appropriate to different types of backup needs (the following list is organized by price, highest to lowest):

• If there is a need to perform frequent full or partial recoveries, the S3 Standard tier is probably the best option
• If recovery-frequency is pretty much "never" — but needs to be quick if there actually is a need to perform recoveries — and the policies governing backups mandates up to a thirty-day recoverability window, the best option is likely the S3 Infrequent Access (IA) tier.
• If there's generally no need for recovery beyond legal compliance capabilities, or the recovery-time objectives (RTO) for backups will tolerate a multi-hour wait for data to become unavailable, the S3 glacier layer is probable the best option.

Further, if projects backup needs span the usage profiles of the previous list, data lifecycle policies can be created that will move data from a higher-cost layer to a lower-cost layer based on time thresholds. And, to prevent being billed for data that has no further utility, the lifecycle policies can include an expiration-age at which AWS will simply delete and stop charging for the backed up data.

There are a couple of ways to get backup data into S3:

• Copy: The easiest — and likely most well known — is to simply copy the data from a host into an S3 bucket. Every file on disk that's copied to S3 exists as an individually downloadable file in S3. Copy operations can be iterative or recursive. If the copy operation takes the form of a recursive-copy, basic location relationship between files is preserved (though, things like hard- or soft-links get converted into multiple copies of a given file). While this method is easy, it includes a loss of filesystem metadata — not just the previously-mentioned loss of link-style file-data but ownerships, permissions, MAC-tags, etc.
• Sync: Similarly easy is the "sync" method. Like the basic copy Every file on disk that's copied to S3 exists as an individually downloadable file in S3. The sync operation is inherently recursive. Further, if an identical copy of a file exists within S3 at a given location, the sync operation will only overwrite the S3-hosted file if the to-be-copied file is different. This provides good support for incremental style backups. As with the basic copy-to-S3 method, this method results in the loss of file-link and other filesystem metadata.
  
  
  Note: if using this method, it is probably a good idea to turn on bucket-versioning to ensure that each version of an uploaded file is kept. This allows a recovery operation to recover a given point-in-time's version of the backed-up file.
• Streaming copy: This method is the least well-known. However, this method can be leveraged to overcome the problem of loss filesystem metadata. If the stream-to-S3 operation includes an inlined data-encapsulation operation (e.g., piping the stream through the tar utility), filesystem metadata will be preserved.
  
  
  Note: the cost of preserving metadata via encapsulation is that the encapsulated object is opaque to S3. As such, there's no (direct) means by which to emulate an incremental backup operation.

Technical Implementation

As the title of this article suggests, the technical-implementation focus of this article is on streamed backups to S3.

Most users of S3 are aware of is static file-copy options. That is copying a file from an EC2 instance directly to S3. Most such users, when they want to store files in EC2 and need to retain filesystem metadata either look to things like s3fs or do staged encapsulation.

The former allows you to treat S3 as though it were a local filesystem. However, for various reasons, many organizations are not comfortable using FUSE-based filesystem-implementstions - particularly opensource project ones (usually due to fears about support if something goes awry)

The latter means using an archiving tool to create a pre-packaged copy of the data first staged to disk as a complete file and then copying that file to S3. Common archiving tools include the Linux Tape ARchive utility (`tar`), cpio or even `mkisofs`/`genisoimage`. However, if the archiving tool supports reading from STDIN and/or writing to STDOUT, the tool can be used to create an archive directly within S3 using S3's streaming-copy capabilities.

Best practices for backups is to ensure that the target data-set is in a consistent state. Generally, this means that the data to be archived is non-changing. This can be done by quiescing a filesystem ...or snapshotting a filesystem and backing up the snapshot. Use of LVM snapshots will be used to illustrate how to a consistent backup of a live filesystem (like those used to host the operating system.)

Note: this illustration assumes that the filesystem to be  backed up is built on top of LVM. If the filesystem is built on a bare (EBS-provided) device, the filesystem will need to be stopped before it can be consistently streamed to S3.

The high-level procedure is as follows:

1. Create a snapshot of the logical volume hosting the filesystem to be backed up (note that LVM issues an `fsfreeze` operation before to creating the snapshot: this flushes all pending I/Os before making the snapshot, ensuring that the resultant snapshot is in a consistent state). Thin or static-sized snapshots may be selected (thin snapshots are especially useful when snapshotting multiple volumes within the same volume-group as one has less need to worry about getting the snapshot volume's size-specification correct).
2. Mount the snapshot
3. Use the archiving-tool to stream the filesystem data to standard output
4. Pipe the stream to S3's `cp` tool, specifying to read from a stream and to write to object-name in S3
5. Unmount the snapshot
6. Delete the snapshot
7. Validate the backup by using S3's `cp` tool, specifying to write to a stream and then read the stream using the original archiving tool's capability to read from standard input. If the archiving tool has a "test" mode, use that; if it does not, it is likely possible to specify /dev/null as its output destination.

For a basic, automated implementation of the above, see the linked-to tool. Note that this tool is "dumb": it assumes that all logical volumes hosting a filesystem should be backed up. The only argument it takes is the name of the S3 bucket to upload to. The script does only very basic "pre-flight" checking:

• Ensure that the AWS CLI is found within the script's inherited PATH env.
• Ensure that either an AWS IAM instance-role is attached to the instance or that an IAM user-role is defined in the script's execution environment (${HOME}/.aws/credential files not currently supported). No attempt is made to ensure the instance- or IAM user-role has sufficient permissions to write to the selected S3 bucket
• Ensure that a bucket-name has been passed, but not checked for validity.

Once the pre-flights pass: the script will attept to snapshot all volumes hosting a filesystem; mount the snapshots under the /mnt hierarchy — recreating the original volumes' mount-locations, but rooted in /mnt; use the `tar` utility to encapsulate and stream the to-be-archived data to the s3 utility; use the S3 cp utility to write tar's streamed, encapsulated output to the named S3 bucket's "/Backups/" folder. Once the S3 cp utility closes the stream without errors, the script will then dismount and delete the snapshots.

Alternatives

As mentioned previously, it's possible to do similar actions to the above for filesystems that do not reside on LVM2 logical volumes. However, doing so will either require different methods for creating a consistent state for the backup-set or backing up postentially inconsistent data (and possibly even wholly missing "in flight" data).

EBS has the native ability to create copy-on-write snapshots. However, the EBS volume's snapshot capability is generally decoupled from the OS'es ability to "pause" a filesystem. One can use a tool — like those in the LxEBSbackups project — to coordinate the pausing of the filesystem so that the EBS snapshot can create a consistent copy of the data (and then unpause the filesystem as soon as the EBS snapshot has been started).

One can leave the data "as is" in the EBS snapshot or one can then mount the snapshot to the EC2 and execute a streamed archive operation to S3. The former has the value of being low effort. The latter has the benefit of storing the data to lower-priced tiers (even S3 standard is cheaper than snapshots of EBS volumes) and allowing the backed up data to be placed under S3 lifecycle policies.

"How Big Is It," You Ask?

For one of the projects I'm supporting, they want to deploy into an isolated network. However, they want to be able to flexibly support systems in that network being able to fetch feature and patch RPMs (and other stuff) as needed. Being on an isolated network, they don't want to overwhelm the security gateways by having a bunch of systems fetching directly from the internet. Result: a desire to host a controlled mirror of the relevant upstream software repositories.

This begged the question, "what all do we need to mirror." I stress the word "need" because this customer is used to thinking in terms of manual rather than automated efforts (working on that with them, too). As a result of this manual mind-set, they want to keep data-sets around tasks small.

Unfortunately, their idea of small is also a bit out of date. To them, 100GiB — at least for software repositories — falls into the "not small" category. To me, if I can fit something on my phone, it's a small amount of data. We'll ignore the fact that I'm biased by the fact that I could jam a 512GiB micro SD card into my phone. At any rate, they were wanting to minimize the number of repositories and channels they were going to need to mirror so that they could keep the copy-jobs small. I pointed out "you want these systems to have a similar degree of fetching-functionality to what they'd have if they weren't being blocked from downloading directly from the Internet: that means you need <insert exhaustive list of repositories and channels>". Naturally, they balked. The questions "how do you know they'll actually need all of that" and "how much space is all of that going to take" were asked (with a certain degree of overwroughtness).

I pointed out that I couldn't meaningfully answer the former question because I wasn't part of the design-groups for the systems that were to be deployed in the isolated network. I also pointed out that they'd probably be better asking the owners of the prospective systems what they'd anticipate needing (knowing full well that, usually, such questions' answers are somewhere in the "I'm not sure, yet" neighborhood). As such, my exhaustive list was a hedge: better to have and not need than to need and not have. Given the stupid-cheapness of storage and that it can actually be easier to sync all of the channels in a given repository vice syncing a subset, I didn't see a a benefit to not throw storage at the problem.

To the second question, I pointed out, "I'm sitting at this meeting, not in front of my computer. I can get you a full, channel-by-channel breakdown once I get back to my desk.

One of the nice things about yum repositories (where the feather and patch RPMs would come from) is that they're easily queryble for both item-counts and aggregate sizes. Only down side is that the OS-included tools for doing so are more for human-centric, ad hoc queries rather than something that can be jammed into an Excel spreadsheet and =SUM formulae being run. In other words, sizes are put into "friendly" units: if you have 100KiB of data, the number is reported in KiB; if you have 1055KiB of data, the number is reported in MiB; and so on. So, I needed to "wrap" the native tools output to put everything into consistent units (which Excel prefers for =SUMing and other mathematical actions). Because it was a "quick" task, I did it in BASH. In retrospect, using another language likely would have been far less ugly. However, what I came up with worked for creating a CSV:

#!/bin/bash

for YUM in $(
   yum repolist all | awk '/abled/{ print $1}' | \
      sed -e '{
         /-test/d
         /-debuginfo/d
         /-source/d
         /-media/d
      }'  | sed 's/\/.*$//'
   )
do
IFS=$'
'
   REPOSTRUCT=($(
      yum --disablerepo=* --enablerepo=${YUM} repolist -v | \
      grep ^Repo- | grep -E "(id|-name|pkgs|size) *:" | \
      sed 's/ *: /:/'
   ))
   unset IFS

   REPSZ=($(echo "${REPOSTRUCT[3]}" | sed 's/^Repo-size://'))

   if [[ $( echo "${REPSZ[1]}" ) = M ]]
   then
      SIZE=$(echo "${REPSZ[0]} * 1024" | bc)
   elif [[ $( echo "${REPSZ[1]}" ) = G ]]
   then
      SIZE=$(echo "${REPSZ[0]} * 1024 * 1024 " | bc)
   else
      SIZE="${REPSZ[0]}"
   fi
 
   for CT in 0 1 2
   do
      printf "%s;" "${REPOSTRUCT[${CT}]}"
   done
   echo ${SIZE}
done | sed 's/Repo-[a-z]*://'

Yeah... hideous and likely far from optimal ...but not worth my time (even if I had it) to revisit. It gets the job done and there's a certain joy to writing hideous code to solve a problem you didn't want to be asked to solve in the first place.

At any rate, checking against all of the repos that we'd want to mirror for the project, the initial-sync data-set would fit on my phone (without having to upgrade to one of  top-end beasties). Pointing out that the only the initial sync would be "large" and that only a couple of the channels updated with anything resembling regularity (the rest being essentially static), the monthly delta-sync would be vastly smaller and trivial to babysit. So, we'll see whether that assuages their anxieties or not.

Head-Smashy Goodness

One of the joys in the life of being an automation engineer is when you have to "spiff up" the automation put together by other. In general, I consider myself a crap programmer. This assessment is in spite of seeing the garbage written by people that actually consider themselves to be good programmers. It's a matter of what you compare yourself to, I guess: some people compare themselves to where they started; I compare myself to some idea of where I perceive "expert" to be.

At any rate, my most recent project has me "improving" on the automation that another group slapped together and never had time to improve. I look at the code they left me to "improve" and can't help but suspect that, given all the time in the universe, they wouln't meaningfully improve. Oh well: it's giving me an excuse to learn and use a new tool-set.

My first assigned task was improving the deployment automation for their Sonarqube software. I'm still a very long way from where I'd like the improvements to be, but I finally figured out why the hell their RDS deployments from snapshots were taking so mind-bendingly long. You see, when I write RDS deployment-automation, I specify everything. I'm a control freak, so that's just how I am. Sadly, I sort of assume a similar approach will be taken by others. Bad assumption to make.

In trying to debug why it was taking the best part of ninety minutes to deploy a small (100GiB) database from an RDS snapshot, I discovered that what was going on was that RDS was taking forever and a day to convert an instance launched as a "standard" ("magnetic") disk-type instance to one that would run as a GP2. Apparently, when the original template was written, they hadn't specified the RDS instance type. This failure to specify means that RDS uses its default-type ...which is "standard".

Subsequent to their having deployed from that template, they'd (apparently) manually updated the RDS instance type to GP2. I'm assuming this to be the case because the automation around the RDS had never been modified - no "stack update" had been done at any point in time. Indeed, there was nothing in their stack-definition that even allowed the subsequent re-selection of the instance's storage-type.

Interestingly enough, when one launches a new RDS from a snapshot taken of another RDS, AWS attempts to ensure that the new RDS is of the same storage-type. I the snapshot was taken of a GP2-enabled instance, it wants to make the new RDS use GP2. This is all well and good: it prevents you from having unexpectedly different performance-characteristics between the snap-source and the new RDS instantiation.

Unfortunately, in a case where the templates don't override defaults, you get into a HORRIBLE situation when you deploy from that snapshot. Specifically, CloudFormation will create an RDS instance-type with the default, "magnetic" storage. Then, once the instance is created and the data recovered, RDS then attempts to convert the new instance to use storage-type that matched the snap-source. You sit there, staring at the RDS output wondering "what the hell are you 'modifying' and why is it taking so freaking long???". Staring unseeingly at the screen long enough, you might notice that the pending modification task is "storage". If that registered to your wait-numbed brain, you'll have an "oh shit" moment and hand-crank a new RDS from the same snapshot, being careful to match the storage-types of the new instance and the snapshot. Then, you'll watch with some degree of incredulity while the hand-cranked RDS reaches a ready state long before the automated deployment reaches its'.

Oh well, at least I now know why it was so balls-slow an have "improved" the templates accordingly. Looks like I'll be waiting maybe ten minutes for a snap-sourced RDS rather than the 90+ that it was taking.

So You Wanna Run Sonarqube on (Hardened) RHEL/CentOS 7

As developers become more concerned with injecting security-awareness into their processes, tools like Sonarqube become more popular. As that popularity increases, the desire to run that software on more than just the Linux distro(s) it was originally built to run on increases.

Most of my customers only allow the use of Red Hat (and, increasingly, CentOS) on their production networks. As these customers hire developers that cut their teeth not on Red Hat, the pressure for deploying their preferred-tools onto Enterprise Linux increases.

Sonarqube has two relevant installation-methods listed on their installation documentation. One is to explode a ZIP archive and install the files wherever desired and set up requisite automated startup as appropriate. The other is to install a "semi-official" RPM-packaging. Initially, I chose the former, but subsequently changed to the latter. While the former is fine if you're installing very infrequently (and installing by hand), it can fall down if you're doing automated deployments by way of service-frameworks like Amazon's CloudFormation (CFn) and/or AutoScaling.

My preferred installation method is to use CFn to create AutoScaling Groups (ASGs). I don't run Sonarqube clustered: the community edition that my developers lack of funding allows me to deploy doesnt support clustering.  Using ASGs means that I can increase the service's availability by having the Sonarqube service simply rebuild itself if the service ever becomes unexpectedly offline. Rebuilds only take a couple minutes and generally don't require any administrator intervention.

This method is/was compatible with the ZIP-based installation method for the first month or so of running the service. Then, one night, it stopped working. When I investigated to figure out why it stop, I found that the URL the deployment-automation was pointing to no longer existed. I'm not really sure what happened because the Sonarqube maintaiers seem to keep back versions of the software around indefinitely.

/shrug

At any rate, it caused me to return to the installation documentation. This time, I noticed that there was an RPM option. So, I went down that path. For the most part, it made things dead easy vice using the ZIP packaging. There were only a few gotchas with the RPM:

• The current packaging, in addition to being a "noarch" packaging, is neither EL version-specific nor is it cross-version. Effectively, it's EL6-oriented: it includes a legacy init script but not a systemd service definition. The RPM maintainer likely needs to either make an EL6 and an EL7 packaging or needs to update the packaging to include files for both init-types and then use a platform-selector to install and activate the appropriate one
• The current packaging creates the service account 'sonar' and installs the packaged files as user:group 'sonar' ...but doesn't include start-logic to ensure that the Sonarqube runs as 'sonar'. This will manifest in Sonarqube's es.log file similarly to:
  

  [2018-02-21T15:45:51,714][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
  org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
          at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.1.jar:5.1.1]
  Caused by: java.lang.RuntimeException: can not run elasticsearch as root
          at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:100) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:176) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:306) ~[elasticsearch-5.1.1.jar:5.1.1]
          at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.1.jar:5.1.1]
          ... 6 more

• The embedded ElastiCache service requires that the file-descriptors ulimit be at a specific minimum value: if the deployment is on a hardened system, it is reasonably likely the default ulimits are too low. If too low, this will manifest in Sonarqube's es.log file similarly to:
  

  [2018-02-21T15:47:23,787][ERROR][bootstrap                ] Exception
  java.lang.RuntimeException: max file descriptors [65535] for elasticsearch process likely too low, increase to at least [65536]
      at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
      at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
      at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:188)
      at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:264)
      at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
      at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
      at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
      at org.elasticsearch.cli.Command.main(Command.java:53)
      at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
      at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)

• The embedded ElastiCache service requires that the kernel's vm.max_map_count runtime parameter be tweaked. If not adequately tweaked, this will manifest in Sonarqube's es.log file similarly to:

  max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

• The embedded ElastiCache service's JVM won't start if the /tmp directory's had the noexec mount-option applied as part of the host's hardening. This will manifest in Sonarqube's es.log file similarly to:
  

  WARN  es[][o.e.b.Natives] cannot check if running as root because JNA is not available 
  WARN  es[][o.e.b.Natives] cannot install system call filter because JNA is not available 
  WARN  es[][o.e.b.Natives] cannot register console handler because JNA is not available 
  WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_NPROC because JNA is not available
  WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_AS because JNA is not available
  WARN  es[][o.e.b.Natives] cannot getrlimit RLIMIT_FSIZE because JNA is not available

• External users won't be able to communicate with the service if the host-based firewall hasn't been appropriately configured.

The first three items are all addressable via a properly-written systemd service definition. The following is what I put together for my deployments:

[Unit]
Description=SonarQube
After=network.target network-online.target
Wants=network-online.target

[Service]
ExecStart=/opt/sonar/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonar/bin/linux-x86-64/sonar.sh stop
ExecReload=/opt/sonar/bin/linux-x86-64/sonar.sh restart
Group=sonar
LimitNOFILE=65536
PIDFile=/opt/sonar/bin/linux-x86-64/SonarQube.pid
Restart=always
RestartSec=30
User=sonar
Type=forking

[Install]
WantedBy=multi-user.target

I put this into GitHub project I set up as part of automating my deployments for AWS. The 'User=sonar' option causes systemd to start the process as the sonar user. The 'Group=sonar' directive ensures that the process runs under the sonar. The 'LimitNOFILE=65536' ensures that the service's file-descriptor's ulimit is set sufficiently high.

The fourth item is addressed by creating an /etc/sysctl.d/sonarqube file containing:

vm.max_map_count = 262144

I take care of this with my automation but the RPM maintainer could obviate the need to do so by including an /etc/sysctl.d/sonarqube file as part of the RPM.

The fifth item is addressable through the sonar.properties file. Adding a line like:

sonar.search.javaAdditionalOpts=-Djava.io.tmpdir=/var/tmp/elasticsearch

Will cause Sonarqube to point ElasticSearch's temp-directory to /var/tmp/elasticsearch. Note that this is an example directory: pick a location in which the sonar user can create a directory and that is on a filesystem that does not have the noexec mount-option set. Without making this change, ElasticSearch will fail to start because JNI cannot be started.

The final item is adressable by adding a port-exception to the firewalld service. This can be done as simply as executing `firewall-cmd --permanent --service=sonarqube --add-port=9000/tcp` or adding a /etc/firewalld/services/sonarqube.xml file containing:

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>Sonarqube service ports</short>
  <description>Firewalld options supporting Sonarqube deployments</description>
  <port protocol="tcp" port="9000" />
  <port protocol="tcp" port="9001" />
</service>

Either is suitable for inclusion in an RPM: the former by way of a %post script; the latter as a config-file packaged in the RPM.

Once all of the bulleted-items are accounted for, the RPM-included service should start right up and function correctly on a hardened EL7 system. Surprisingly enough, no other tweaks are typically needed to make Sonarqube run on a hardened system. Sonarqube will typically function just fine with both FIPS-mode and SELinux active.

It's likely that once Sonarqube is deployed, functionality beyond the defaults will be desired. There are quite a number of functionality-extending plugins available. Download the ones you want and install them to <SONAR_INSTALL_ROOT>/extensions/plugins and restart the service. Most plugins behavior is configurable via the Sonarqube administration GUI.

Note: while much of the 'fixes' following the gotcha list are scattered throughout the Sonarqube documentation, on an unhardened EL 7 system, the system defaults are loose enough that accounting for them was not necessary for either the ZIP- or RPM-based installation-methods.

Extra: If you want the above taken care of for you and you're trying to deploy Sonarqube onto AWS, I put together a set of template-driven CloudFormation templates and helper scripts. They can be found on GitHub.

Why I Love The UNIX/Linux CLI

There's many reasons, really. Probably too many to list in any one sitting.

That said, some days you pull an old trick out of your bag and you're reminded all over why you love something. Today was such a day.

All day, we've been having sporadic performance issues with AWS's CloudFormation functionality. Some jobs will run in (a few tens of) minutes, others will take an hour or more. Now, this would be understandable if they were markedly different tasks. But that's not the case. It's literally "run this template 'A' and wait" followed by "re-run template 'A' and wait" where the first run goes in about the time you'd expect and the second run of the same automation-template takes 10x as long.

Fun thing with cloudformation is that, when you launch from the command line, you can pass all of your parameters via a file. Some templates, however, don't like it when there's two active copies that contain the same parameter values. The way around this is to generalize your parameters file and then "process" that parameter file for each running. The UNIX/Linux CLI means that you can do all of this inline, like so:

aws --region us-east-1 cloudformation create-stack --stack-name RedMine07 \
  --disable-rollback --capabilities CAPABILITY_NAMED_IAM \
  --template-url https://s3.amazonaws.com/my-automation-bucket/Templates/RedMine.tmplt.json \
  --parameters file://<(sed 's/__NUM__/07/g' RedMine.parms.json)

The nifty part here is in the last line of that command. When you wrap a shell command in "<()", it runs the command within the parentheses and encapsulates it into a file-handle. Thus, if your main command requires an input-file be specified, the output of your encapsulated command gets treated just like it was being read from an on-disk file rather than as an inline-command.

Slick.

In my case, I'd generalized my parameters file to include a substitutable token, "__NUM__". Each time I need to run the command, all I have to do is change that "07" to a new value and bingo, new stack with unique values where needed.

Another fun thing is the shell's editable history function. If I want to run that exact same command, again, changing my values — both those passed as stack-parameters and the name of the resultant stack — I can do:

!!:gs/07/08

Which causes the prior stanza to be re-run as so:

aws --region us-east-1 cloudformation create-stack --stack-name RedMine08 \
  --disable-rollback --capabilities CAPABILITY_NAMED_IAM \
  --template-url https://s3.amazonaws.com/my-automation-bucket/Templates/RedMine.tmplt.json \
  --parameters file://<(sed 's/__NUM__/08/g' RedMine.parms.json)

Similarly, if the command I wanted to change was the 66th in my history rather than the one I just ran, I could do:

!66:gs/07/08

And achieve the same results.

The Joys of SCL (or, "So You Want to Host RedMine on Red Hat/CentOS")

Recently, I needed to go back and try to re-engineer my company's RedMine deployment. Previously, I had rushed through a quick-standup of the service using the generic CentOS AMI furnished by CentOS.Org. All of the supporting automation was very slap-dash and had more things hard-coded than I really would have liked. However, the original time-scale the automation needed to be delivered under meant that "stupid but works ain't stupid" ruled the day.

That ad hoc implementation was over two years ago, at this point. Since the initial implementation, I had never had the chance to start to revisit it. The lack of a contiguous chunk of time to re-implement changed over the holidays (hooray for everyone being gone over the holiday weeks!). At any rate, there were a couple things I wanted to accomplish with the re-build:

1. Change from using the generic CentOS AMI to using our customized AMI
   • Our AMI has a specific partitioning-scheme for our OS disks that we prefer to be used across all of our services. Wanted to make our RedMine deployment no longer be an "outlier".
   • Our AMIs are updated monthly, whereas the CentOS.Org ones are typically updated when there's a new 7.X. Using our AMIs means not having to wait nearly as long for launch-time patch-up to complete before things can become "live".
2. I wanted to make use of our standard, provisioning-time OS-hardening framework — both so that the instance is better configured to be Internet-facing and so it is more in accordance with our other service-hosts
3. I wanted to be able to use a vendor-packaged Ruby rather than the self-packaged/self-managed Ruby. I had previously gone the custom Ruby route both because the vendor's default Ruby is too old for RedMine to use and because SCL was not quite as mature an offering back in 2015. In updating my automation to make use of SCL, I was figuring that a vendor-packaged RPM is more likely to be kept up to date without potentially introducing breakage that self-packaged RPMs can be prone to.
4. I wanted to switch from using UserData to using CloudFormation (CFn) for launch automation: 
   • Relatively speaking, CFn automation includes a lot more ability to catch errors/deployment-breakage with lower-effort. Typically, it's easier to avoid the "I thought it deployed correctly" happenstances.
   • Using CFn automation exposes less information via instance-metadata either via the management UIs/CLIs or by reading metadata from the instance.
   • Using CFn automation makes it easier to coordinate the provisioning of non-EC2 solution components (e.g., security groups, IAM roles, S3 buckets, EFS shares, RDS databases, load-balancers etc.)
5. I wanted to clean up the overall automation-scripts so I could push more stuff into (public) GitHub (projects) and have to pull less information from private repositories (like S3 buckets).

Ultimately, the biggest challenge turned out to be dealing with SCL. I can probably account for the problems with SCL being their focus. Specifically, they seem to be designed more for use by developers in interactive environments than they are to underpin services. If you're a developer, all you really need to do to have a "more up-to-date than vendor-standard" language packages is:

• Enable the SCL repos
• Install the SCL-hosted language package you're looking for
• Update either your user's shell-inits or the system-wide shell inits
• Log-out and back in
• Go to town with your new SCL-furnished capabilities.

If, on the other hand, you want to use an SCL-managed language to back a service that can't run with the standard vendor-packaged languages (etc)., things are a little more involved.

A little bit of background:

The SCL-managed utilities are typically rooted wholly in "/opt/rh". This means binaries, libraries, documentation. Because of this, your shell's PATH, MANPATH and linker-paths will need to be updated. Fortunately, each SCL-managed package ships with an `enable` utility. You can do something as simple as `scl_enabled <SCL_PACKAGE_NAME> bash`, do the equivalent in your user's "${HOME}/.profile" and/or "${HOME}/.bash_profile" files, or you can do something by way of a file in "/etc/profile.d/". Since I assume "everyone on the system wants to use it", I enable by way of a file in "/etc/profile.d/". Typically, I create a file named "enable_<PACKAGE>.sh" and create contents similar to:

#!/bin/bash
source /opt/rh/<RPM_NAME>/enable
export X_SCLS="$(scl enable <RPM_NAME> 'echo $X_SCLS')"

Within it. Each time an interactive-user logs in, they get the SCL-managed package configured for their shell. If multiple SCL-managed packages are installed, they all "stack" nicely (failure to do the above can result in one SCL-managed package interfering with another).

Caveats:

Unfortunately, while this works well for interactive shell users, it's not generally sufficient for the non-interactive environments. This includes actual running services or stuff that's provisioned at launch-time via scripts kicked off by way of `systemd` (or `init`). Such processes do not use the contents of the "/etc/profile.d" directory (or other user-level initialization scripts). This creates a "what to do" scenario.

Linux affords a couple methods for handling this:

• You can modify the system-wide PATH (etc.). To make the run-time linker find libraries in non-standard paths, you can create files in "/etc/ld.conf.d" that enumerate the desired linker search paths and then run `ldconfig` (the vendor-packaging of MariaDB and a few others do this). However, this is traditionally been considered not to be "good form" and frowned upon.
• Use under systemd:
  • You can modify the scripts invoked via systemd/init; however, if those scripts are RPM-managed, such modifications can be lost when the RPM gets updated via a system patch-up
  • Under systemd, you can create customized service-definitions that contain the necessary environmentals. Typically, this can be quickly done by copying the service-definition found in the "/usr/lib/systemd/system" directory to the "/etc/systemd/system" directory and modifying as necessary.
• Cron jobs: If you need to automate the running of tasks via cron, you can either:
  • Write action-wrappers that include the requisite environmental settings
  • Make use of the ability to set environmentals in jobs defined in /etc/cron.d
• Some applications (such as Apache) allow you to extend library and other search directives via run-time configuration directives. In the case of RedMine, Apache calls the Passenger framework which, in turn, runs the Ruby-based RedMine code. To make Apache happy, you create an "/etc/httpd/conf.d/Passenger.conf" that looks something like:
  

  LoadModule passenger_module /opt/rh/rh-ruby24/root/usr/local/share/gems/gems/passenger-5.1.12/buildout/apache2/mod_passenger.so
  
  <IfModule mod_passenger.c>
          PassengerRoot /opt/rh/rh-ruby24/root/usr/local/share/gems/gems/passenger-5.1.12
          PassengerDefaultRuby /opt/rh/rh-ruby24/root/usr/bin/ruby
          SetEnv LD_LIBRARY_PATH /opt/rh/rh-ruby24/root/usr/lib64
  </IfModule>
  

  The directives:
  • "LoadModule" directive binds the module-name to the fully-qualified path to the Passenger Apache-plugin (shared-library). This file is created by the Passenger installer.
  • "PassengerRoot" tells Apache where to look for further, related gems and libraries.
  • "PassengerDefaultRuby" tells Apache which Ruby to use (useful if there's more than one Ruby version installed or, as in the case of SCL-managed packages, Ruby is not found in the standard system search paths).
  • "SetEnv LD_LIBRARY_PATH" tells Apachels run-time linker where else to look for shared library files.
  Without these, Apache won't find RedMine or all of the associate (SCL-managed) Ruby dependencies.

The above list should be viewed in "worst-to-first" ascending-order of preference. The second systemd and cron methods and the application-configuration method (notionally) have the narrowest scope (area of effect, blast-radius, etc.) — only effecting where the desired executions look for non-standard components — and are the generally preferred methods.

Note On Provisioning Automation Tools:

Because automation tools like CloudFormation (and cloud-init) are not interactive processes, they too will not process the contents of the "/etc/profile.d/" directory. One can use one of the above enumerated methods for modifying the invocation of these tools or include explicit sourcing of the relevant "/etc/profile.d/" files within the scripts executed by these frameworks.