Seriously, CentOS?

One of my (many) duties in our shop is doing cross-platform maintenance of RPMs. Previously, when I was maintaining them for EL5 and EL6, things were fairly straight-forward. You got or made a SPEC file to packages your SOURCES into RPMS and you were pretty much good to go. Imagine my surprise when I went to start porting things to EL7 and all my freaking packages had el7.centos in their damned names. WTF?? These RPMs are for use on all of our EL7 systems, not just CentOS: why the hell are you dropping the implementation into my %{dist}-string?? So, now I have to make sure that my ${HOME}/.rpmmacros file has a line in it that looks like:

%dist .el7

If I don't want my %{dist}-string get crapped-up.

Screw you, CentOS. Stuff was just fine on CentOS 5 and CentOS 6. This change is not an improvement.

SSH Via GateOne

If you're like me, you occasionally work from locations that have restrictive firewall rules. Those rules may even be so restrictive that you're unable to SSH to remote hosts that you need to manage or do development work on.

The solution to this scenario is to use an SSH-over-HTML proxy service. In my case, I use GateOne. It's a light-weight service that's quick and easy to configure. The software maintainers even make Docker-ready bundles for it.

In my case, I create a GateOne-enabled SSH-proxy in AWS. Doing so is as simple as picking an EL6-compatible AMI (CentOS, RHEL, Scientific Linux, Amazon Linux, etc.) and giving the AMI launch-tool the following user-data:

#!/bin/sh

yum update -y
yum install -y git
pip install --upgrade pip
pip install --upgrade tornado
git clone https://github.com/liftoff/GateOne.git /tmp/GateOne
(cd /tmp/GateOne ; python setup.py install)
chkconfig gateone on
service gateone start
printf "Sleeping for 15s..."
sleep 15
echo "Done!"
pkill gateone
sed -i -e '/"https_redirect"/s/: .*$/: true,/' \
    -e '/"origins":/s/:.*$/: ["*"],/' \
    $(readlink -f /etc/gateone/conf.d/10server.conf)
sed -i '/"auth"/s/: .*$/: "pam",/' \
    $(readlink -f /etc/gateone/conf.d/20authentication.conf)
service gateone restart

When installed from its Git repository, the default configuration of GateOne only allows connection from an instances local interfaces. The `sed` operation against the "/etc/gateone/conf.d/10server.conf" configuration-file opens up this restriction. Setting the "origins" definition to '"*"' will allow browser connections from anywhere (if this is unacceptable, you can use AWS security-groups to lock things back down a bit).

Because I like to think of myself as lazy, I also enable a port 80 → 443 automatic redirect. Saves me from having to type "https://" into my browser (every keystroke saved counts!). This redirect is created by setting the "/etc/gateone/conf.d/10server.conf" configuration-file's "https_redirect" configuration-option to "true".

Because I prefer to do key-based logins and I don't want to make my SSH keys easily snarfable, I disable anonymous logins. This also has the side-effect of making it so it's not as easy for randos to discover your SSH proxy and use it for their own purposes. Disabling anonymous logins is done by setting the "/etc/gateone/conf.d/20authentication.conf" file's "auth" parameter. By setting it to "pam", as above, you instruct GateOne to allow local user-credentials to be used for logins.

When GateOne's authentication is set to "pam", it will be necessary to set up a valid user/password at the OS layer to be able to login via GateOne. You can do this via userdata or by logging into the instance and manually creating the GateOne user-credentials. I prefer the userdata method, as it means I can fully-automate the whole GateOne deployment process (e.g., via a CLI-script, CloudFormation template, etc.).

Note on the sleep statement: GateOne's configuration files are not present until the first time the service starts. Without the sleep, the `sed` operations will fail with a "missing file" error.

But I Don't Like That Username

One of the clients I do work for is in the process of adopting commercial cloud solutions. Early in the process, they had a private network that they were doing executing virtualization and private-cloud efforts. The maintainers of those two environments had created standard builds for use within those environments. For better or worse, my client has a number of developer teams they've contracted out to whose primary efforts are conducted either in-house (to the contractor) or within AWS, Azure or Google Compute Engine.

The group I work for has been tasked with standardizing and automating the deployment of enterprise components across all of the various environments and providing stewardship of the other contracted-out development efforts. When we were first given this task, the customer's build engineers would not provide any methods to replicate the production build - not even sufficient documentation that would allow us to accurately mimic it well enough to enable the other developers with a seamless dev -> test -> prod workflow.

Early in our involvement, I ended up creating my own build. Eventually, in the previously-described vacuum, others decided to adopt my build. Now, there's enough different groups using that build that it's pressuring the maintainers of the internal build to abandon theirs and adopt mine.

Fun part of DevOps is that when tends to be consensus-driven. If there's a slow or unresponsive link in the chain, the old "top down" approach frequently becomes a casualty when critical mass is achieved bottom up..

At any rate, we were recently in discussions with the enterprise build maintainers to show them how to adopt the consensus build. One of their pushbacks was "but that build doesn't include the 'break-glass' account that the legacy builds do." They wanted to know if we could modify the build to be compliant with that user-account.

This struck me odd, since, our group's (and other dev-groups') approach to such issues is "modify it in code". This isn't an approach familiar to the enterprise team. They're very "golden image" oriented. So, I provideded them a quick set of instructions on how to take the incoming standard build and make it compatible with their tools' expectations

#cloud-config
system_info:
  default_user:
    name: ent-adm

For enterprise components that will be migrated from private virtualization and cloud solutions to commercial cloud offerings, the above allows them to take not just my build, but any build that's enabled for automated provisioning and inject their account. Instead of launching a system with whatever the default-user is that's baked in, the above allows them to reset any system's initial username to be whatever they want (the above's 'ent-adm' is just an example - I dunno what their preferred account name is - I've simply used the above whenever I'm deploying instance-templates and don't want to remember "instance X uses userid A; instance Y uses userid B; and instance Z uses userid C").

Even more fun if they don't like any of the attributes associated with the default user's account, they can override it with any user-parameter available in cloud-init.

Per-Sender SMTP SASL Authentication

One of the customers I do work for runs a multi-tenant environment. One of the issues this customer has been having is "how do we notify tenants that their systems want patching". For their Linux systems, the immediate answer was to use the `yum-cron` facility to handle it.

Unfortunately, the system that would receive these types of notification emails is different than the ones that handle generic SMTP relaying. Instead of being able to set up each Linux system's Postfix service to use a single Smart-relay, we needed to be able to have just the account that's used to send the ready-to-patch notifications relay into through the notification gateway while all other emails get directed through a broader-scope relay.

It took several search iterations to finally uncover the "trick" for allowing this (credit to zmwangx for his GitHub Gist-posting). Overall, it's pretty straight-forward. Only thing that was not immediately obvious was which tokens mapped-through (what was the common key). In summary, to create the solution one needs to do three things:

1. Modify the Postfix configuration:
   1. Do the standard tasks around defining a "smart" relay
   2. Do the standard tasks around enabling Postfix to act as a SASL-authenticated client
   3. Enable per-sender authentication
   4. Define a per-sender relay-map
2. Create a file to map a local sender-address to a SASL-credential
3. Create a file to map a local sender-address to a specific relay-host

One that's done, it's simply a matter of verifying that things are working the way you think they should be.

Modify Postfix: Define Default Smart Relay
As with much of my experimentation the past year or two, I did my testing using Amazon Web Services resources. In this case, I used AWS's Simple Email Service (SES). Configuring Postfix to relay is trivial. Since my testing was designed to ensure that everything except my wanted sender-address to deny relaying, I configured my test system to point to its local SES relay without actually configuring an SES account to enable that relaying. For postfix, this was simply a matter of doing:

postconf -e "relayhost = [email-smtp.us-west-2.amazonaws.com]:587"

Appends creates line to your /etc/postfix/main.cf that looks like:

relayhost = [email-smtp.us-west-2.amazonaws.com]:587

Note: The AWS SES relays are currently only available within a few regions (as of this writing, NoVA/us-east-1, Oregon/us-west-2 and Ireland/eu-west-1). Each relay requires a SASL credential be created to allow relaying. So, no big deal publicizing the relay's name if spammers don't have such credentials.

At this point, any mail sent via Postfix will attempt to relay through the listed relay-host. Further, because SASL client-credentials are not yet set up, those relay attempts will fail.

Modify Postfix: Define Default SASL-Client Credentials
Add a block similar to the following to begin to turn Postfix into an SMTP SASL-client:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_mechanism_filter = plain
smtp_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt
smtp_use_tls = yes
smtp_tls_security_level = encrypt

Two critical items above are the "smtp_sasl_password_maps" and "smtp_tls_CAfile" parameters:

1. The former instructs Postfix where to look for SASL credentials.
2. The latter tells postfix where to look for root Certificate Authorities so it can verify that the relay host's SSL certificates are valid. The path listed in the above is the default for Red Hat-derived distributions: alter to fit your distribution's locations

The rest of the parameters instruct Postfix to use SASL-authentication over TLS-encrypted channels (required when connecting to an SMTP-relay via port 587) and to use the "plain" mechanism for sending credentials to the relay-host.

Modify Postfix: Enable Per-Sender Authentication

Use the command `postconf -e "smtp_sender_dependent_authentication = yes"` to enable Postfix's per-sender authentication modules. This will add a line to the /etc/postfix/main.cf that looks like:

smtp_sender_dependent_authentication = yes

Modify Postfix: Define SASL-Sender Map

Once per-sender authentication is enabled, Postfix needs to be instructed where to find mappings of senders to credentials. Use the command,  `postconf -e "sender_dependent_relayhost_maps = hash:/etc/postfix/sender_relay"` to enable Postfix's per-sender authentication modules. This will add a line to the /etc/postfix/main.cf that looks like:

sender_dependent_relayhost_maps = hash:/etc/postfix/sender_relay

Create a Sender-to-Credential Map

Edit/create the sender-to-credential mapping file /etc/postfix/sender_passwd. Its contents should be similar to the following:

# Sender Address                                 <userid>:<password>
patch-alert@ses-test.cloudlab.xanthia.com        AKIAICAOGQX5UA0ACDSVJ:pDHM1n4uYLGN4BQOnzGcTSeQXSRDjcKCy6VkmQk+CoBV

Postfix will use the "patch-alert@ses-test.cloudlab.xanthia.com" sender-address as a common key with the value in the relay-map (following).


Create a Sender-to-Relay Map

Edit/create the sender-to-credential mapping file /etc/postfix/sender_relay. Its contents should be similar to the following:

# Sender Address                            [relay-host]:port
patch-alert@ses-test.cloudlab.xanthia.com        [email-smtp.us-west-2.amazonaws.com]:587

Postfix will use "patch-alert@ses-test.cloudlab.xanthia.com" sender-address as a common key with the value in the credential-map (preceding).

Verification

A quick verification test is to send an email from a mapped user-address and a non-mapped user address:

# sendmail -f patch-alert@ses-test.cloudlab.xanthia.com -t <<EOF
To: fubar@cloudlab.xanthia.com
Subject: Per-User SASL Test
Content-type: text/html

If this arrived, things are probably set up correctly
EOF
# sendmail -f unmapped-user@ses-test.cloudlab.xanthia.com -t <<EOF
To: fubar@cloudlab.xanthia.com
Subject: Per-User SASL Test
Content-type: text/html

If this bounced, things are probably set up correctly
EOF

This should result in an SMTP log-snippet that resembles the following:

Feb 16 18:08:09 ses-test maintuser: MARK == MARK == MARK
Feb 16 18:08:22 ses-test postfix/pickup[5484]: 2B7D244AB: uid=0 from=<patch-alert@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:22 ses-test postfix/cleanup[5583]: 2B7D244AB: message-id=<20160216180822.2B7D244AB@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:22 ses-test postfix/qmgr[5485]: 2B7D244AB: from=<patch-alert@ses-test.cloudlab.xanthia.com>, size=403, nrcpt=1 (queue active)
Feb 16 18:08:22 ses-test postfix/smtp[5585]: 2B7D244AB: to=<thjones2@gmail.com>, relay=email-smtp.us-west-2.amazonaws.com[54.187.123.10]:587, delay=0.37, delays=0.02/0.03/0.19/0.13, dsn=2.0.0, status=sent (250 Ok 00000152eb44d396-408494a9-93f0-4f21-8985-460c057537bf-000000)
Feb 16 18:08:22 ses-test postfix/qmgr[5485]: 2B7D244AB: removed
Feb 16 18:08:32 ses-test postfix/pickup[5484]: A339E44AB: uid=0 from=<bad-sender@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:32 ses-test postfix/cleanup[5583]: A339E44AB: message-id=<20160216180832.A339E44AB@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:32 ses-test postfix/qmgr[5485]: A339E44AB: from=<bad-sender@ses-test.cloudlab.xanthia.com>, size=408, nrcpt=1 (queue active)
Feb 16 18:08:32 ses-test postfix/smtp[5585]: A339E44AB: to=<thjones2@gmail.com>, relay=email-smtp.us-west-2.amazonaws.com[54.69.81.169]:587, delay=0.09, delays=0.01/0/0.08/0, dsn=5.0.0, status=bounced (host email-smtp.us-west-2.amazonaws.com[54.69.81.169] said: 530 Authentication required (in reply to MAIL FROM command))

As can be seen in the snippet, the first message (from the mapped sender) was relayed while the second message (from the unmapped sender) was rejected.

Solving Root VG Collisions in LVM-Enabled Virtualization Templates

The nature of template-based Linux OS deployments means that, if a template uses LVM for its root filesystems, any system built from that template will have non-unique volume group (VG) names. In most situations, non-unique VG names are not a problem. However, if you encounter a situation where you need to a broken instance by correcting a problem within the instance's root filesystems, non-unique VG names can make that task more difficult.

To avoid this eventuality, the template user can easily modify each launched template by executing steps similar to the following:

#!/bin/sh

DEFIF=$(ip route show | awk '/^default/{print $5}')
BASEIP=$(printf '%02X' \
         $(ip addr show ${DEFIF} | \
           awk '/inet /{print $2}' | \
           sed -e 's#/.*$##' -e 's/\./ /g' \
          ))

vgrename -v VolGroup00 VolGroup00_${BASEIP}
sed -i 's/VolGroup00/&_'${BASEIP}'/' /etc/fstab
sed -i 's/VolGroup00/&_'${BASEIP}'/g' /boot/grub/grub.conf

for KRNL in $(awk '/initrd/{print $2}' /boot/grub/grub.conf | \
              sed -e 's/^.*initramfs-//' -e 's/\.img$//')
do
   mkinitrd -f -v /boot/initramfs-${KRNL}.img ${KRNL}
done

init 6

Note that the above script assumes that the current root VG name is "VolGroup00". If your current root VG name is different, change the value in the script above as appropriate.

This script may be executed either at instance launch-time or anywhere in the life-cycle of an an instance. The above script takes the existing root VG name and tacks on a uniqueness component. In this case, the uniqueness is achieved by taking the IP address of the instance's primary interface and converting it to a hexadecimal string. So long as a group of systems does not contain any repeated primary IP addresses, this should provide a sufficient level of uniqueness for a group of deployed systems.

Note: renaming the root VG will _not_ solve the problems caused by PV UUID non-uniqueness. Currently, there is no known-good solution to this issue. The general recommendation is to avoid that problem by using a different template to build your recovery-host than used to build your broken host.

Simple Guacamole

If the enterprise you work for is like mine, access through the corporate firewall is tightly-controlled. You may find that only we-related protocols are left through (mostly) unfettered. When you're trying to work with a serivce like AWS, this can make management of Linux- and/or Windows-based resources problematic.

A decent solution to such a situation is the use of HTML-based remote connection gateway services. If all you're looking to do is SSH, the GateOne SSH-over-HTTP gateway is a quick and easy to setup solution. If you need to manage instances via graphical desktops - most typically Windows but some people like it for Linux as well - a better solution is Guacamole.

Guacamole is an extensible, HTTP-based solution. It runs as a Java servlet under a Unix hosted service like Tomcat. If you're like me, you may also prefer to encapsulate/broker the Tomcat service through a generic HTTP service like Apache or Nginx. My preference has been Apache - but mostly because I've been using Apache since not long after it was formally forked off of the NCSA project. I also tend to favor Apache because it's historically been part of the core repositories of my Linux of choice, Red Hat/CentOS.

Guacamole gives you HTTP-tunneling options for SSH, Telnet, RDP and VNC. this walk through is designed to get you quickly running Guacamole as an web-based SSH front end. Once you've got the SSH component running, adding other management protocols is easy. This procedure is also designed to be doable even if you don't yet actually have the ability to SSH to a AWS-hosted instance.

1. Start the AWS web console's "launch instance" wizard.
2. Select an appropriate EL6-based AMI.
3. Select an appropriate instance type (the free tier instances are suitable for a basic SSH proxy)
4. On the "Configure Instance" page, expand the "Advanced Details" section.
5. In the now-available text box, paste in the contents of this script. Note that this script is flexible enough that, if the version of Guacamole hosted via the EPEL project is updated, the script should continue to work. With a slight bit of massaging, the script could also be made to work with EL 7 and associated Tomcat and EPEL-hosted RPMs.
6. If the AMI you've picked does not provide the option of password-based logins for the default SSH user, add steps (in the "Advanced Details" text box) for creating an interactive SSH user with a password. Ensure that the user also has the ability to use `sudo` to get root privileges.
7. Finish up the rest of the process for deploying an instance.

Once the instance finishes deploying, you should be able to set your browser to the public hostname shown for the instance in the AWS console. Add "/guacamole/" after the hostname. Assuming all went well, you will be presented with a Guacamole login prompt. Enter the credentials:
Note that these credentials can be changing the:

printf "\t<authorize username=\"admin\" password=\"PASSWORD\">\n"

Line of the pasted-in script. Once you've authenticated to Guacamole, you'll be able to login to the hosting-instance via SSH using the instance's no-privileged user's credentials. Once logged in, you can escalate privileges and then configure additional authentication mechanisms and connection destinations and protocols.

Note: Guacamole doesn't currently support key-based login mechanisms. If key-based logins are a must make use of GateOne, instead.

EXTn and the Tyranny of AWS

One of the organizations I provide consulting services to opted to start migrating from an in-house, VMware-based virtualization solution to an Amazon-hosted cloud solution. The transition has been somewhat fraught, here and there - possibly moreso than the prior transition from physical (primarily Solaris-based) servers to virtualized (Linux) servers.

One of the huge "problems" is that the organization's various sub-units have habits formed of decade or longer lifecycles. Staff are particularly used to being able to get on console for various things (using GUI-based software-installers, graphical IDEs as well as tasks that actually require console access - like recovering a system stuck in its startup sequence).

For all that AWS offers, console access isn't one of them. Amazon's model for how customers should deploy and manage systems means that they don't figure that console access is strictly necessary.

In a wholly self-service model, this is probably an ok assumption. Unfortunately, the IT model that the organization in question is moving to doesn't offer instance-owners true self-service. They're essentially trying to transport an on-premises, managed multi-tenancy model into AWS. The model they're moving from didn't have self-service, so they're not interested in enabling self-service (at least not during phase one of the move). Their tenants not only don't have console access in the new environment, they don't have the ability to execute the AWS-style recovery-methods you'd use in the absence of console access. The tenants are impatient and the group supporting them is small, so it's a tough situation.

The migration has been ongoing for a sufficiently long period of time that the default `mkfs` behavior for EXT-based filesystems is starting to rear its head. Being well beyond the 180 day mark since the inception of the migration, tenants are finding that, when their Linux instances reboot, they're not coming back as quickly as they did towards the beginning of the migration ...because their builds still leave autofsck enabled.

If you're reading this, you may have run into similar issues.

The solution to this, while still maintaining the spirit of the "fsck every 180 days or so" best practices for EXTn-based filesystems is fairly straight forward:

1. Disable the autofsck settings on your instances' EXTn filesystems: use `tune2efs -i 0 -c -1 /dev/<DEVNODE>`
2. Schedule periodic fsck "simulations". This can be done either by running fsck in "dryrun" mode or by doing an fsck of a filesystem metadata image.

The "dryrun" method is fairly straight forward: just run fsck with the "-N" option. I'm not super much a fan of this as it doesn't feel like it gives me the info I'm looking for to feel good about the state of my filesystems.

The "fsck of a filesystem metadata image" is pretty straight forward, automatable and provides a bit more on the "warm fuzzies" side of thing. To do it:

1. Create a metadata image file using `e2image -fr /dev/<DEVNODE> /IMAGE/FILE/PATH` (e.g. `e2image -fr /dev/RootVG/auditVol /tmp/auditVol.img`)
2. Use `losetup` to create an fsck'able block-device from the image file (e.g., `losetup /dev/loop0 /tmp/auditVol.img`)
3. Execute an fsck against the loopback device (e.g., `fsck /dev/loop0`). Output will look similar to the following:

   # fsck /dev/loop0
   fsck from util-linux-ng 2.17.2
   e2fsck 1.41.12 (17-May-2010)
   /dev/loop0: recovering journal
   /dev/loop0: clean, 13/297184 files, 56066/1187840 blocks

4. If the output indicates anything other than good health, schedule an outage to do a proper repair of your live filesystem(s)

Granted, if you find you need to do the full check of the real filesystem(s), you're still potentially stuck with the "no console" issue. Even that is potentially surmountable:

1. Create a "/forcefsck" file
2. Create a "/fsckoptions" file with the contents "-sy"
3. Schedule your reboot

When the reboot happens, depending how long the system takes to boot, the EC2 launch monitors may time out: just be patient. If you can't be patient, just monitor the boot logs (either in the AWS console or using the AWS CLI's equivalent option).

Could You Make It Just A Little More Difficult?

Every so often, you need to share out an HTML file via a public web-server. Back in the pre-cloud days, this meant that you'd toss it up on your vanity web server and call it a day. In the cloud era, you have other options. Amazon's S3 was the first one that I used, but other cloud-storage services can be leveraged in a similar manner for static content.

One of those others is Google's "Drive" service. Unfortunately, Google doesn't exactly seem to want to make it a straight forward affair to share static web content straight from drive. It's easy if you want viewers to see the raw HTML, but not so great if you want them to see rendered HTML.

At any rate, as of the writing of this document (warning: judging by my Google results, the method seems to change over time and even Google's own help pages aren't really kept up to date), this was what I had to do:

1. Create a new folder in Google Drive (optional: if you're willing to make your top-level gDrive folder publicly-browsable or already have a folder that's set - or you're willing to set - as publicly-browsable, you can skip this step)
2. Edit the sharing on the folder, setting the permission to "Public View"
3. Navigate into the folder. Take note of its path. It will look something like:
   

   https://drive.google.com/drive/#folders/0F3SA-qkPpztNflU1bUtyekYYC091a2ttHZJpMElwTm9UcFNqN1pNMlf3iUlTUkJ0UU5PUVk

4. Take the left part of the URL, up to and including "/#folders/" and nuke it
5. Replace the deleted part of the original URL and replace it with:
   

   http://www.googledrive.com/host/

   The browsable URL to your publicly-viewable folder will now look like:
   

   http://www.googledrive.com/host/0F3SA-qkPpztNflU1bUtyekYYC091a2ttHZJpMElwTm9UcFNqN1pNMlf3iUlTUkJ0UU5PUVk

6. Clicking on  that link will take you to the folder holding your static web content. To get the sharable URL for your file(s), click on the link to the file.
7. When the file opens in your browser, copy the URL for the file, then send it out (for the sake of your recipients' sanity, you might want to pump it through a URL-shorter service, first - like maybe goo.gl)

In my case, I was trying to collaborate on a system-hardening toolset. I'd run my system through a security scanner that had flagged a number of false findings (actually, all the "fail" findings turned out to be bogus). I wanted to share the report with him and the rules files that the security tool had reported against. So, I sorted out the above so I could post links into our collaboration tool.

Maybe one day Google will make sharing static web content from Drive as (relatively) easy as Amazon has with S3. A "share as web page" button sure would be nice.

So You Don't Want to BYOL

The Amazon Web Services MarketPlace is pretty awesome. There's  oodles of pre-made machine templates to choose some. Even in the face of all that choice, it's not unusual to find that, of all the choices you have, none quite fit your needs. That's the scenario I found myself in.

Right now, I'm supporting a customer that's a heavy user of Linux for their business support systems. They're in the process of migrating from our legacy hosting environment to hosting things on AWS. During their development phase, use of CentOS was sufficient for their needs. As they move to production, however, they want "real" Red Hat Enterprise Linux.

Go up on the MarketPlace and there's plenty of options to choose from. However, my customer doesn't want to deal with buying a stand-alone entitlement to patch-support for their AWS-hosted systems. This requirement considerably cuts down on the useful choices in the MarketPlace. There's still "license included" Red Hat options to choose from.

Unfortunately, my customer also has fairly specific partitioning requirements that are not met by the "license included" AMIs. When using CentOS, this wan't a problem - CentOS's patch repos are open-access. Creating an AMI with suitable partitioning and access to those public repos is about a 20 minute process. While some of that process is leveragable for creating a Red Hat AMI, making the resultant AMI be "license included" is a bit more challenging.

When I tried to simply re-use my CentOS process, supplemented by the Amazon repo RPMs, I ended up with a system that, when I did a yum-query, got me 401 errors. I was missing something.

Google searches weren't terribly helpful in solving my problem. I found a lot of "how do I do this" posts, but damned few that actually included the answer. Ultimately, what it turns out to be is that if you generate your AMI from an EBS snapshot, instances launched from that AMI don't have an entitlement key to access the Amazon yum repos. You can see this by looking at your launched instance's metadata:

# curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "717243568699",
  "architecture" : "x86_64",
  "availabilityZone" : "us-west-2b",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "imageId" : "ami-9df0ec7a",
  "instanceId" : "i-51825ba7",
  "instanceType" : "t1.micro",
  "kernelId" : "aki-fc8f11cc",
  "pendingTime" : "2015-03-25T19:04:51Z",
  "privateIp" : "172.31.19.148",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2010-08-31"
}

Specifically, what you want to look at is the value for "billingProducts". If it's "null", your yum isn't going to be able to access the Amazon RPM repositories. Where I came up close to empty on my Google searches was "how to make this attribute persist across images".

I found a small note in a community forum post indicating that AMIs generated from an EBS snapshot will always have "billingProducts" set to "null". This is due to a limitation in the tool used to register an image from a snapshot.

To get around this limitation, one has to create an AMI from a instance of an entitled AMI. Basically, after you've created the EBS you've readied to make a custom AMI, you do a disk-swap with a properly-entitled instance. You then use the "create image" option from that instance. Once you launch AMI you created via the EBS-swap, your instance's metadata will now look something like:

# curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "717243568699",
  "architecture" : "x86_64",
  "availabilityZone" : "us-west-2b",
  "billingProducts" : [ "bp-6fa54006" ],
  "devpayProductCodes" : null,
  "imageId" : "ami-9df0ec7a",
  "instanceId" : "i-51825ba7",
  "instanceType" : "t1.micro",
  "kernelId" : "aki-fc8f11cc",
  "pendingTime" : "2015-03-25T19:04:51Z",
  "privateIp" : "172.31.19.148",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2010-08-31"
}

Once that "billingProducts" is set, the cloud-init related first-boot scripts will take that "billingProducts" and use it to register the system with the Amazon yum repos. Voilà: you  now have a fully custom AMI that uses Amazon-billed access to Red Hat updates.

Note on Compatibility: the Red Hat provided PVM AMIs do not yield well to this method. The Red Hat provided PVM AMIs are all designed with their boot/root device set to /dev/sda1. To date, attempts to leverage the above techniques for PVM AMIs that require their boot/root device set to /dev/sda (used when using a single, partitioned EBS to host a bare /boot partition and LVM-managed root partitions) have not met with success.

Attack of the Clones

One of the clients I do work for has a fairly significant Linux footprint. However, in these times of greater fiscal responsibility/austerity, my client is looking at even cheaper alternatives. This means that, for systems whose applications don't require  Red Hat for warranty-support, CentOS is being subbed into their environment. This is particularly true for their testing environments. There've even been arguments for doing it in production, applications' vendor-support be damned, because "CentOS is the same as Red Hat"

I've previously argued, "they're very, very similar, but they're not truly identical". In particular, Red Hat handles CVEs and errata somewhat differently than CentOS does (Red Hat backports many fixes to prior EL releases, CentOS's stance is generally "upgrade it").

Today, I got bit by one place where CentOS hews far too closely to "the same as Red Hat Enterprise Linux". Specifically, I was using the `oscap` security tool to do a security audit of a test system. I should say, "I was struggling to use the `oscap` security tool...". With later versions of EL6, Red Hat, and as a derivative, CentOS, implement the CPE system for Linux.

This is all fine and good, except where the tools you use rely on the correctness of CPE-related definitions. By the standard of CPE, Red Hat and CentOS are very much not "the same". Because the security-auditing tool I was using (`oscap`) leverages CPEs and because the CentOS maintainers simply repackage the Red Hat furnished security profiles without updating the CPE call-outs, first, the security tool fails horribly. Every test comes back as "notapplicable".

To fix this situation, a bit of `sed`-fu is required:

mv /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml-DIST && \
cp /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml-DIST \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml && \
sed -i '{
   s#Red Hat Enterprise Linux 6#CentOS 6##g
   s#cpe:/o:redhat:enterprise_linux:6#cpe:/o:centos:centos:6##g
}' /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml


mv /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml-DIST && \
cp /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml-DIST \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml && \
sed -i \
   's#cpe:/o:redhat:enterprise_linux#cpe:/o:centos:centos##g' \
/usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml

Once the above is done, running `oscap` actually produces useful results.

NOTE: Ironically, doing the above edits will cause the various SCAP profiles to flag an error when running the tests that verify that RPMs have been unaltered. I've submitted a bug to the CentOS group so these fixes are included in future versions of the CentOS OpenSCAP RPMs, but, until then, you just need to be aware that the `oscap` tool  will flag the above two files.

...And if you found this page because you're trying to figure out how to run `oscap` to get results, here's a sample invocation that should act as a starting-point:

oscap xccdf eval --profile common --report \
   /var/tmp/oscap-report_`date "+%Y%m%d%H%M"`.html \
   --results /var/tmp/oscap-results_`date "+%Y%m%d%H%M"`.xml\
   --cpe /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-dictionary.xml \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml