The Joys of SCL (or, "So You Want to Host RedMine on Red Hat/CentOS")

Recently, I needed to go back and try to re-engineer my company's RedMine deployment. Previously, I had rushed through a quick-standup of the service using the generic CentOS AMI furnished by CentOS.Org. All of the supporting automation was very slap-dash and had more things hard-coded than I really would have liked. However, the original time-scale the automation needed to be delivered under meant that "stupid but works ain't stupid" ruled the day.

That ad hoc implementation was over two years ago, at this point. Since the initial implementation, I had never had the chance to start to revisit it. The lack of a contiguous chunk of time to re-implement changed over the holidays (hooray for everyone being gone over the holiday weeks!). At any rate, there were a couple things I wanted to accomplish with the re-build:

1. Change from using the generic CentOS AMI to using our customized AMI
   • Our AMI has a specific partitioning-scheme for our OS disks that we prefer to be used across all of our services. Wanted to make our RedMine deployment no longer be an "outlier".
   • Our AMIs are updated monthly, whereas the CentOS.Org ones are typically updated when there's a new 7.X. Using our AMIs means not having to wait nearly as long for launch-time patch-up to complete before things can become "live".
2. I wanted to make use of our standard, provisioning-time OS-hardening framework — both so that the instance is better configured to be Internet-facing and so it is more in accordance with our other service-hosts
3. I wanted to be able to use a vendor-packaged Ruby rather than the self-packaged/self-managed Ruby. I had previously gone the custom Ruby route both because the vendor's default Ruby is too old for RedMine to use and because SCL was not quite as mature an offering back in 2015. In updating my automation to make use of SCL, I was figuring that a vendor-packaged RPM is more likely to be kept up to date without potentially introducing breakage that self-packaged RPMs can be prone to.
4. I wanted to switch from using UserData to using CloudFormation (CFn) for launch automation: 
   • Relatively speaking, CFn automation includes a lot more ability to catch errors/deployment-breakage with lower-effort. Typically, it's easier to avoid the "I thought it deployed correctly" happenstances.
   • Using CFn automation exposes less information via instance-metadata either via the management UIs/CLIs or by reading metadata from the instance.
   • Using CFn automation makes it easier to coordinate the provisioning of non-EC2 solution components (e.g., security groups, IAM roles, S3 buckets, EFS shares, RDS databases, load-balancers etc.)
5. I wanted to clean up the overall automation-scripts so I could push more stuff into (public) GitHub (projects) and have to pull less information from private repositories (like S3 buckets).

Ultimately, the biggest challenge turned out to be dealing with SCL. I can probably account for the problems with SCL being their focus. Specifically, they seem to be designed more for use by developers in interactive environments than they are to underpin services. If you're a developer, all you really need to do to have a "more up-to-date than vendor-standard" language packages is:

• Enable the SCL repos
• Install the SCL-hosted language package you're looking for
• Update either your user's shell-inits or the system-wide shell inits
• Log-out and back in
• Go to town with your new SCL-furnished capabilities.

If, on the other hand, you want to use an SCL-managed language to back a service that can't run with the standard vendor-packaged languages (etc)., things are a little more involved.

A little bit of background:

The SCL-managed utilities are typically rooted wholly in "/opt/rh". This means binaries, libraries, documentation. Because of this, your shell's PATH, MANPATH and linker-paths will need to be updated. Fortunately, each SCL-managed package ships with an `enable` utility. You can do something as simple as `scl_enabled <SCL_PACKAGE_NAME> bash`, do the equivalent in your user's "${HOME}/.profile" and/or "${HOME}/.bash_profile" files, or you can do something by way of a file in "/etc/profile.d/". Since I assume "everyone on the system wants to use it", I enable by way of a file in "/etc/profile.d/". Typically, I create a file named "enable_<PACKAGE>.sh" and create contents similar to:

#!/bin/bash
source /opt/rh/<RPM_NAME>/enable
export X_SCLS="$(scl enable <RPM_NAME> 'echo $X_SCLS')"

Within it. Each time an interactive-user logs in, they get the SCL-managed package configured for their shell. If multiple SCL-managed packages are installed, they all "stack" nicely (failure to do the above can result in one SCL-managed package interfering with another).

Caveats:

Unfortunately, while this works well for interactive shell users, it's not generally sufficient for the non-interactive environments. This includes actual running services or stuff that's provisioned at launch-time via scripts kicked off by way of `systemd` (or `init`). Such processes do not use the contents of the "/etc/profile.d" directory (or other user-level initialization scripts). This creates a "what to do" scenario.

Linux affords a couple methods for handling this:

• You can modify the system-wide PATH (etc.). To make the run-time linker find libraries in non-standard paths, you can create files in "/etc/ld.conf.d" that enumerate the desired linker search paths and then run `ldconfig` (the vendor-packaging of MariaDB and a few others do this). However, this is traditionally been considered not to be "good form" and frowned upon.
• Use under systemd:
  • You can modify the scripts invoked via systemd/init; however, if those scripts are RPM-managed, such modifications can be lost when the RPM gets updated via a system patch-up
  • Under systemd, you can create customized service-definitions that contain the necessary environmentals. Typically, this can be quickly done by copying the service-definition found in the "/usr/lib/systemd/system" directory to the "/etc/systemd/system" directory and modifying as necessary.
• Cron jobs: If you need to automate the running of tasks via cron, you can either:
  • Write action-wrappers that include the requisite environmental settings
  • Make use of the ability to set environmentals in jobs defined in /etc/cron.d
• Some applications (such as Apache) allow you to extend library and other search directives via run-time configuration directives. In the case of RedMine, Apache calls the Passenger framework which, in turn, runs the Ruby-based RedMine code. To make Apache happy, you create an "/etc/httpd/conf.d/Passenger.conf" that looks something like:
  

  LoadModule passenger_module /opt/rh/rh-ruby24/root/usr/local/share/gems/gems/passenger-5.1.12/buildout/apache2/mod_passenger.so
  
  <IfModule mod_passenger.c>
          PassengerRoot /opt/rh/rh-ruby24/root/usr/local/share/gems/gems/passenger-5.1.12
          PassengerDefaultRuby /opt/rh/rh-ruby24/root/usr/bin/ruby
          SetEnv LD_LIBRARY_PATH /opt/rh/rh-ruby24/root/usr/lib64
  </IfModule>
  

  The directives:
  • "LoadModule" directive binds the module-name to the fully-qualified path to the Passenger Apache-plugin (shared-library). This file is created by the Passenger installer.
  • "PassengerRoot" tells Apache where to look for further, related gems and libraries.
  • "PassengerDefaultRuby" tells Apache which Ruby to use (useful if there's more than one Ruby version installed or, as in the case of SCL-managed packages, Ruby is not found in the standard system search paths).
  • "SetEnv LD_LIBRARY_PATH" tells Apachels run-time linker where else to look for shared library files.
  Without these, Apache won't find RedMine or all of the associate (SCL-managed) Ruby dependencies.

The above list should be viewed in "worst-to-first" ascending-order of preference. The second systemd and cron methods and the application-configuration method (notionally) have the narrowest scope (area of effect, blast-radius, etc.) — only effecting where the desired executions look for non-standard components — and are the generally preferred methods.

Note On Provisioning Automation Tools:

Because automation tools like CloudFormation (and cloud-init) are not interactive processes, they too will not process the contents of the "/etc/profile.d/" directory. One can use one of the above enumerated methods for modifying the invocation of these tools or include explicit sourcing of the relevant "/etc/profile.d/" files within the scripts executed by these frameworks.