So You Don't Want to BYOL

The Amazon Web Services MarketPlace is pretty awesome. There's  oodles of pre-made machine templates to choose some. Even in the face of all that choice, it's not unusual to find that, of all the choices you have, none quite fit your needs. That's the scenario I found myself in.

Right now, I'm supporting a customer that's a heavy user of Linux for their business support systems. They're in the process of migrating from our legacy hosting environment to hosting things on AWS. During their development phase, use of CentOS was sufficient for their needs. As they move to production, however, they want "real" Red Hat Enterprise Linux.

Go up on the MarketPlace and there's plenty of options to choose from. However, my customer doesn't want to deal with buying a stand-alone entitlement to patch-support for their AWS-hosted systems. This requirement considerably cuts down on the useful choices in the MarketPlace. There's still "license included" Red Hat options to choose from.

Unfortunately, my customer also has fairly specific partitioning requirements that are not met by the "license included" AMIs. When using CentOS, this wan't a problem - CentOS's patch repos are open-access. Creating an AMI with suitable partitioning and access to those public repos is about a 20 minute process. While some of that process is leveragable for creating a Red Hat AMI, making the resultant AMI be "license included" is a bit more challenging.

When I tried to simply re-use my CentOS process, supplemented by the Amazon repo RPMs, I ended up with a system that, when I did a yum-query, got me 401 errors. I was missing something.

Google searches weren't terribly helpful in solving my problem. I found a lot of "how do I do this" posts, but damned few that actually included the answer. Ultimately, what it turns out to be is that if you generate your AMI from an EBS snapshot, instances launched from that AMI don't have an entitlement key to access the Amazon yum repos. You can see this by looking at your launched instance's metadata:

# curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "717243568699",
  "architecture" : "x86_64",
  "availabilityZone" : "us-west-2b",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "imageId" : "ami-9df0ec7a",
  "instanceId" : "i-51825ba7",
  "instanceType" : "t1.micro",
  "kernelId" : "aki-fc8f11cc",
  "pendingTime" : "2015-03-25T19:04:51Z",
  "privateIp" : "172.31.19.148",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2010-08-31"
}

Specifically, what you want to look at is the value for "billingProducts". If it's "null", your yum isn't going to be able to access the Amazon RPM repositories. Where I came up close to empty on my Google searches was "how to make this attribute persist across images".

I found a small note in a community forum post indicating that AMIs generated from an EBS snapshot will always have "billingProducts" set to "null". This is due to a limitation in the tool used to register an image from a snapshot.

To get around this limitation, one has to create an AMI from a instance of an entitled AMI. Basically, after you've created the EBS you've readied to make a custom AMI, you do a disk-swap with a properly-entitled instance. You then use the "create image" option from that instance. Once you launch AMI you created via the EBS-swap, your instance's metadata will now look something like:

# curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "717243568699",
  "architecture" : "x86_64",
  "availabilityZone" : "us-west-2b",
  "billingProducts" : [ "bp-6fa54006" ],
  "devpayProductCodes" : null,
  "imageId" : "ami-9df0ec7a",
  "instanceId" : "i-51825ba7",
  "instanceType" : "t1.micro",
  "kernelId" : "aki-fc8f11cc",
  "pendingTime" : "2015-03-25T19:04:51Z",
  "privateIp" : "172.31.19.148",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2010-08-31"
}

Once that "billingProducts" is set, the cloud-init related first-boot scripts will take that "billingProducts" and use it to register the system with the Amazon yum repos. Voilà: you  now have a fully custom AMI that uses Amazon-billed access to Red Hat updates.

Note on Compatibility: the Red Hat provided PVM AMIs do not yield well to this method. The Red Hat provided PVM AMIs are all designed with their boot/root device set to /dev/sda1. To date, attempts to leverage the above techniques for PVM AMIs that require their boot/root device set to /dev/sda (used when using a single, partitioned EBS to host a bare /boot partition and LVM-managed root partitions) have not met with success.

Quick-n-Dirty Yum Repo via HTTP

Recently, we had a bit of a SNAFU in the year-end renewal of our RedHat support. As a result, all of the RHN accounts tied to the previous contract lost access to RHN's software download repositories. This meant that things like being able to yum-down RPMs on rhn_register'ed systems no longer worked and, we couldn't log into RHN and do a manual download, either.

Fortunately, because we're on moderately decent terms with RedHat and they know that the contract eventually will get sorted out, they were willing to help us get through our current access issues. Moving RHN accounts from one permanent contract to another, after first associating them with some kind of temporary entitlement is a paperwork-hassle for all parties involved and is apt to get your account(s) mis-associated down the road. Since all parties knew that this was a temporary problem but needed an immediate fix, our RedHat representative furnished us with the requisite physical media necessary to get us through till the contracts could get sorted out.

Knowing that I wasn't the only one that might need the software and that I might need to execute some burndown-rebuilds on a particular project I was working on, I wanted to make it easy to pull packages to my test systems. We're in an ESX environment, so, I spun up a small VM (only 1GB of virtual RAM, 1GHz of virtual CPU, a couple Gigabytes of virtual disk for a root volume and about 20GB of virtual disk to stage software onto and build an RPM repository on) to act as a yum repository server.

After spinning this basic VM, I had to sort out what to do as far as getting that physical media turned into a repo. I'm not a big fan of copying CDs as a stream of discrete files (been burned, too many times, by over-the-wire corruption, permission issues and the like). So, I took the DVD and made an ISO from it. I then took that ISO and scp'ed it up to the VM.

Once I had the ISO file copied up to the VM, did a quick mount of it (`mount -t iso9660 -o loop,ro /tmp/RHEL5.7_x86_64.iso /mnt/DVD` for those of you playing at home). Once I mounted it, I did a quick copy of its contents to the filesystem I'd set aside for it. I'm kind of a fan of cpio for things like this, so I cd'ed into the root of the mounted ISO and did a `find . -print | cpio -pmd /RepoDir` to create a good copy of my ISO data into a "real" filesystem (note, you'll want to make sure you do a `umask 022` first to ensure that the permission structures from the mounted ISO get copied, intact, along with the files, themselves).

With all of the DVD's files copied to the repo-server and into a writeable filesystem, it's necessary to create all the repo structures and references to support use by yum. Our standard build doesn't include the createrepo tool, so, first I had to locate its RPM in the repo filessytem and then install it onto my repo-server. Doing a quick `find . -name "*createrepo*rpm"` while cd'ed into repo fileystem turned up the path to the requisite RPM. I then did an `rpm -Uh [PathFromFind]` to install the  createrepo tool's RPM files.

The createrepo tool is a nifty little tool. You just cd into the root of the directory where you copied your media to, do a `createrepo .`, and it scans the directory structures to find all the RPMs and XMLs and other files and creates the requisite data structures and pointers that allow yum to know how to pull the appropriate RPMs from the filesystem.

Once that's done, if all you care about is local access to the RPMs, you can create a basic .repo file in /etc/yum.repos.d that uses a "baseurl=file:///Path/To/Media" directive in it.

In my case, I wanted to make my new repo available to other hosts at the lab. Easiest way to make the repo available over the network is to do so via HTTP. Our standard build doesn't include the standard RedHat HTTP server, by default. So, I manually installed the requisite RPMs from the repo's filesystem. I modified the stock /etc/httpd/conf/httpd.conf and added the folowing stanzas to it:

Alias /Repo/ "/RepoDir/"
<Directory "/RepoDir">
   Options Indexes MultiViews
   AllowOverride None
   Order allow,deny
   Allow from all
</Directory>

[Note: this is probably a looser configuration than I'd have in place if I was making this a permanent solution, but this was just meant as a quick-n-dirty workaround for a temporary problem.]

I made sure to do a `chkconfig httpd on` and then did a `service httpd start` to activate the web server. I then took my web browser and made sure that the repo filesystem's contents were visable via web client. It wasn't: I forgot that our standard build has port 80 blocked by default. I did the requisite juju to add an exception to iptables for port 80 and all was good to go.

With my RPMs (etc.) now visable via HTTP, I logged into the VM that I was actually needing to install RPMs to via yum. I escalated privileges to root and created an /etc/yum.repos.d/LAB.repo file that looked similar to the following:

[lab-repo]
name=RHEL 5.7
baseurl=http://repovm.domain.name/Repo
enabled=1
gpgcheck=0

I did a quick cleanup of the consuming VM's yum repo information with a `yum clean all` and then verified taht my consuming VM was able to properly see the repos's data by doing a `yum list`. All was good to go. Depending on how temporary this actually ends up being, I'll go back and make my consuming VM's .repo file a bit more "complete" and more properly layout the repo-server's filesystem and HTTP config.

Automating Yum Setup

Recently, as part of my employer's "employee development" programs, I took Red Hat's RH255 class so that I could prep for getting my RHCSA and RHCE (my opinions on the experience are grist for another post - and I don't know, yet, whether that post should be in this blog or one of my personal blogs). The RH255 class I took was based on RHEL 6. In the US, they moved to training based on RHEL 6.0 about six months ago. One of the interesting things (I thought) my instructor said was that RHEL 6.0 included a function that tried to automatically configure `yum` to take mounted CDROMs and ISOs and treat them as installation repositories.

I may have misheard or misinterpreted what he said. It may also be a case that, since my instructor is in the RHEL 6.1 beta program, he was referring to a feature in RHEL 6.1 and not RHEL 6.0. Whatever the case may be, popping an RHEL 6.0 DVD (or ISO) into and RHEL 6.0 machine does not (yet) cause that DVD to be automatically included in `yum`'s repository search. Fortunately, the RHEL 6.x media has been set up (prior RHELs may also have been, it's just been so long since I've built an RHEL 5 system from other than an automated-build system) so that it's easy enough to automatically set up the DVD or ISO for inclusion in `yum`'s seaches. All of the yum repository is already on the media. So, you needn't muck about with doing the createrepo stuff (and the time that running createrepo against a multi-gigabyte DVD can take). Below is the script I wrote to take this already-present data and make it available to `yum`:

#!/bin/sh

MNTDIR=${1:-UNDEF}

# Make sure we passed a location to check

if [ ${MNTDIR} == "UNDEF" ]

then

   echo "Usage: $0 <ISO Mount Location>"

   exit

fi

# Make sure it's actually a directory

if [ ! -d ${MNTDIR} ]

then

   echo "No such directory"

   exit

fi

#ID directories containing repository data

REPODIRS=$(find ${MNTDIR} -type d -name repodata)

if [ "${REPODIRS}" == "" ]

then

   echo "No repository data found"

   exit

fi

for DIR in ${REPODIRS}

to

   DIRNAME=$(echo ${DIR} | sed -e 's#'${MNTDIR}'/##' -e 's#/repodata##')

   BASEURL=$(dirname ${DIR})

   echo "[${DIRNAME}]"

   echo "baseurl=${BASEURL}"

   echo "enabled=1"

   echo

done

The above script takes a directory location (presumably where you mounted your DVD or ISO) and produces yum repository configuration output. I opted to have it output as a capturable stream rather than as a file. That way, I'd have to option of either capturing to a temp location or  directly into /etc/yum.repos.d. Doing so gave more flexibility in what to name a file (so that I didn't clobber any existing files). Granted, I could have parameterized the name and location of the output file, but chose not to. If you want to use the above and would rather have the output go directly to an output file in /etc/yum.repos.d (or elsewhere), it's a trivial modification to the above. Have at it.

At any rate, once you run the above and take the resultant output and move it into a file in /etc/yum.repos.d, subsequent invocations of the `yum` command should now include your DVD or ISO. If it fails to do so (or errors result) it's probably because the new .repo file contains section IDs that collide with exiting ones. Just edit your new file to eliminate any collisions.

While I've done some basic error-checking in the file (specifically: ensuring the user passed a location and that the location is a valid directory), I didn't write it to gracefully handle repository directories that have spaces in their path-names. It's also fairly trivial to fix the script to accommodate it. However, it's probably even easier to just make sure you don't mount your DVD or ISO with spaces in the path-names. I'm lazy, so, I'm choosing the latter option and avoiding the coding exercise. If you find you need to have spaces in your path-names and want to use my script, feel free to make the necessary modifications to accommodate your use of space-containing path names.