Why I Default to The Old Ways

I work with a growing team of automation engineers. Most are purely dev types. Those that have lived in the Operations world, at all, skew heavily towards Windows or only had to very lightly deal with UNIX or Linux.

I, on the other hand, have been using UNIX flavors since 1989. My first Linux system was the result of downloading a distribution from the MIT mirrors in 1992. Result, I have a lot of old habits (seriously: some of my habits are older than some of my teammates). And, because I've had to get deep into the weeds with all of those operating systems many, many, many times, over the years, those habits are pretty entrenched ("learned with blood" and all that rot).

A year or so ago, I'd submitted a PR that included some regex-heavy shell scripts. The person that reviewed the PR had asked "why are you using '[<space><TAB>]*' in your regexes rather than just '\s'?". At the time, I think my response was a semi-glib, "A) old habits die hard; and, B) I know that the former method always works".

That said, I am a lazy-typist. Typing "\s" is a lot fewer keystrokes than is "[<space><TAB>]*". Similarly, "\s" takes up a lot less in the way of column-width than does "[<space><TAB>]*" (and I/we generally like to code to fairly standard page-widths). So, for both laziness reasons and column-conservation reasons, I started to move more towards using "\s" and away from using "[<space><TAB>]*".  I think in the last 12 months, I've moved almost exclusively to  "\s".

Today, that move bit me in the ass. Well, yesterday, actually, because that's when I started receiving reports that the tool I'd authored on EL7 wasn't working when installed/used on EL6. Ultimately, I traced the problem to an `awk` invocation. Specifically, I had a chunk of code (filtering DNS output) that looked like:

awk '/\sIN SRV\s/{ printf("%s;%s\n",$7,$8)}'

Which worked a treat on EL7 but on EL6, "not so much." When I altered it to the older-style invocation:

awk '/[  ]*IN[  ]*SRV[  ]*/{ printf("%s;%s\n",$7,$8)}'

It worked fine on both EL7 and EL6. Turns out the ancient version of `awk` (3.1.7) on EL6 didn't know how to properly interpret the "\s" token. Oddly (my recollection from writing other tooling) is that EL6's version of `grep` understands the "\s" token just fine.

When I Slacked the person I'd had the original conversation with a link to the PR with a "see: this is why" note, he replied, "oh: I never really used awk, so never ran into it".

Update to EL6 and the Pain of 10Gbps Networking in AWS

In a previous post, EL6 and the Pain of 10Gbps Networking in AWS, EL6 and the Pain of 10Gbps Networking in AWS, I discussed how to enable using third-party ixgbevf drivers to support 10Gbps networking in AWS-hostd RHEL 6 and CentOS 6 instances. Under further testing, it turns out that this is overkill. The native drivers may be used, instead, with minimal hassle.

It turns out that, in my quest to ensure that my CentOS and RHEL AMIs contained as many of the AWS utilities present in the Amazon Linux AMIs, that I included two RPMS - ec2-net and ec2-net-util - that were preventing use of the native drivers. Skipping these two RPMs (and possibly sacrificing ENI hot-plug capabilities) allows a much more low-effort support of 10Gpbs networking in AWS-hosted EL6 instances.

Absent those RPMS, 10Gbps support becomes a simple matter of:

1. Add add_drivers+="ixgbe ixgbevf" to the AMI's /etc/dracut.conf file
2. Use dracut to regenerate the AMI's initramfs.
3. Ensure that there are no persistent network device mapping entries in /etc/udev/rules.d.
4. Ensure that there are no ixgbe or ixgbevf config directives in /etc/modprobe.d/* files.
5. Enable sr-iov support in the instance-to-be-registered
6. Register the AMI 

The resulting AMIs (and instances spawned from them) should support 10Gbps networking and be compatible with M4-generation instance-types.

Using DKMS to maintain driver modules

In my prior post, I noted that maintaining custom drivers for the the kernel in RHEL and CentOS hosts can be a bit painful (and prone to leaving you with an unreachable or even unbootable system). One way to take some of the pain out of owning a system with custom drivers is to leverage DKMS. In general, DKMS is the recommended way to ensure that, as kernels are updated, required kernel modules are also (automatically) updated.

Unfortunately, use of the DKMS method will require that developer tools (i.e., the GNU C-compiler) be present on the system - either in perpetuity or just any time kernel updates are applied. It is very likely that your security team will object to - or even prohibit - this. If the objection/prohibition cannot be overridden, use of the DKMS method will not be possible.

Steps

1. Set an appropriate version string into the shell-environment:
   

   export VERSION=3.2.2

2. Make sure that appropriate header files for the running-kernel are installed
   

   yum install -y kernel-devel-$(uname -r)

3. Ensure that the dkms utilities are installed:
   

   yum --enablerepo=epel install dkms

4. Download the driver sources and unarchive into the /usr/src directory:
   

   wget https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/${VERSION}/ixgbevf-${VERSION}.tar.gz/download \
       -O /tmp/ixgbevf-${VERSION}.tar.gz && \
      ( cd /usr/src && \
         tar zxf /tmp/ixgbevf-${VERSION}.tar.gz )

5. Create an appropriate DKMS configuration file for the driver:
   

   cat > /usr/src/ixgbevf-${VERSION}/dkms.conf << EOF
   PACKAGE_NAME="ixgbevf"
   PACKAGE_VERSION="${VERSION}"
   CLEAN="cd src/; make clean"
   MAKE="cd src/; make BUILD_KERNEL=\${kernelver}"
   BUILT_MODULE_LOCATION[0]="src/"
   BUILT_MODULE_NAME[0]="ixgbevf"
   DEST_MODULE_LOCATION[0]="/updates"
   DEST_MODULE_NAME[0]="ixgbevf"
   AUTOINSTALL="yes"
   EOF

6. Register the module to the DKMS-managed kernel tree:
   

   dkms add -m ixgbevf -v ${VERSION}

7. Build the module against the currently-running kernel:
   

   dkms build ixgbevf/${VERSION}

Verification

The easiest way to verify the correct functioning of DKMS is to:

1. Perform a `yum update -y`
2. Check that the new drivers were created by executing `find /lib/modules -name ixgbevf.ko`. Output should be similar to the following:
   

   find /lib/modules -name ixgbevf.ko | grep extra
   /lib/modules/2.6.32-642.1.1.el6.x86_64/extra/ixgbevf.ko
   /lib/modules/2.6.32-642.6.1.el6.x86_64/extra/ixgbevf.ko

   There should be at least two output-lines: one for the currently-running kernel and one for the kernel update. If more kernels are installed, there may be more than just two output-lines
    
   
3. Reboot the system, then check what version is active:
   

   modinfo ixgbevf | grep extra
   filename:       /lib/modules/2.6.32-642.1.1.el6.x86_64/extra/ixgbevf.ko

   If the output is null, DKMS didn't build the new module.

EL6 and the Pain of 10Gbps Networking in AWS

AWS-hosted instances with optimized networking-support enabled see the 10Gbps interface as an Intel 10Gbps ethernet adapter (`lspci | grep Ethernet` will display a string similar to Intel Corporation 82599 Virtual Function). This interface makes use the ixgbevf network-driver. Enterprise Linux 6 and 7 bundle the version 2.12.x version of the driver into the kernel RPM. Per the Enabling Enhanced Networking with the Intel 82599 VF Interface on Linux Instances in a VPC document, AWS enhanced networking recommends version 2.14.2 or higher of the ixgbevf network-driver. To meet AWS's recommendations for 10Gbps support within an EL6 instance, it will be necessary to update the ixgbevf driver to at least version 2.14.2.

The ixgbevf network-driver source-code can be found on SourceForge. It should be noted that not every AWS-compatible version will successfully compile on EL 6. Version 3.2.2 is known to successfully compile, without intervention, on EL6.

Notes:

>>>CRITICAL ITEM<<<

It is necessary to recompile the ixgbevf driver and inject it into the kernel each time the kernel version changes. This needs to be done between changing the kernel version and rebooting into the new kernel version. Failure to update the driver each time the kernel changes will result in the instance failing to return to the network after a reboot event.

Step #10 from the implementation procedure:

rpm -qa kernel | sed 's/^kernel-//' | xargs -I {} dracut -v -f /boot/initramfs-{}.img {}

Is the easiest way to ensure any available kernels are properly linked against the ixgbevf driver.

>>>CRITICAL ITEM<<<

It is possible that the above process can be avoided by installing DKMS and letting it coordinate the insertion of the ixgbevf driver modules into updated kernels.

Procedure:

The following assumes the instance-owner has privileged access to the instance OS and can make AWS-level configuration changes to the instance-configuration:

1. Login to the instance
2. Escalate privileges to root
3. Install the up-to-date ixgbevf driver. This can be installed either by compiling from source or using pre-compiled binaries.
4. Delete any `*persistent-net*.rules` files found in the /etc/udev/rules.d directory (one or both of 70-persistent-net.rules and 75-persistent-net-generator.rules may be present)
5. Ensure that an `/etc/modprobe.d` file with the following minimum contents exists:
   

   alias eth0 ixgbevf
   

   Recommend creating/placing in /etc/modprobe.d/ifaliases.conf
6. Unload any ixgbevf drivers that may be in the running kernel:
   

   modprobe -rv ixgbevf
   

7. Load the updated ixgbevf driver into the running kernel:
   

   modprobe -v ixgbevf
   

8. Ensure that the /etc/modprobe.d/ixgbevf.conf file exists. Its contents should resemble:
   

   options ixgbevf InterruptThrottleRate=1
   

9. Update the /etc/dracut.conf. Ensure that the add_drivers+="" directive is uncommented and contains reference to the ixgbevf modules (i.e., `add_drivers+="ixgbevf"`)
10. Recompile all installed kernels:
    

    rpm -qa kernel | sed 's/^kernel-//'  | xargs -I {} dracut -v -f /boot/initramfs-{}.img {}
    

11. Shut down the instance
12. When the instance has stopped, use the AWS CLI tool to enable optimized networking support:
    

    aws ec2 --region  modify-instance-attribute --instance-id  --sriov-net-support simple
    

13. Power the instance back on
14. Verify that 10Gbps capability is available:
    
    1. Check that the ixgbevf module is loaded
       

       $ sudo lsmod
       Module                  Size  Used by
       ipv6                  336282  46
       ixgbevf                63414  0
       i2c_piix4              11232  0
       i2c_core               29132  1 i2c_piix4
       ext4                  379559  6
       jbd2                   93252  1 ext4
       mbcache                 8193  1 ext4
       xen_blkfront           21998  3
       pata_acpi               3701  0
       ata_generic             3837  0
       ata_piix               24409  0
       dm_mirror              14864  0
       dm_region_hash         12085  1 dm_mirror
       dm_log                  9930  2 dm_mirror,dm_region_hash
       dm_mod                102467  20 dm_mirror,dm_log
       

    2. Check that `ethtool` is showing that the default interface (typicall "`eth0`") is using the ixgbevf driver:
       

       $ sudo ethtool -i eth0
       driver: ixgbevf
       version: 3.2.2
       firmware-version: N/A
       bus-info: 0000:00:03.0
       supports-statistics: yes
       supports-test: yes
       supports-eeprom-access: no
       supports-register-dump: yes
       supports-priv-flags: no
       

    3. Verify that the interface is listed as supporting a link mode of `10000baseT/Full` and a speed of `10000Mb/s`:
       

       $ sudo ethtool eth0
       Settings for eth0:
               Supported ports: [ ]
               Supported link modes:   10000baseT/Full
               Supported pause frame use: No
               Supports auto-negotiation: No
               Advertised link modes:  Not reported
               Advertised pause frame use: No
               Advertised auto-negotiation: No
               Speed: 10000Mb/s
               Duplex: Full
               Port: Other
               PHYAD: 0
               Transceiver: Unknown!
               Auto-negotiation: off
               Current message level: 0x00000007 (7)
                                      drv probe link
               Link detected: yes
       

Use the Force, LUKS

Not like there aren't a bunch of LUKS guides out there already ...mostly posting this one for myself.

Today, was working on turning the (attrocious - other than a long-past deadline, DISA, do you even care what you're publishing?) RHEL 7 V0R2 STIGs specifications into configuration management elements for our enterprise CM system. Got to the STIG item for "ensure that data-at-rest is encrypted as appropriate". This particular element is only semi-automatable ...since it's one of those "context" rules that has a "if local policy requires it" back-biting element to it. At any rate, this particular STIG-item prescribes the use of LUKs.

As I set about to write the code for this security-element, it occurred to me, "we typically use array-based storage encryption - or things like KMS in cloud deployments - that I can't remember how to cofigure LUKS ...least of all configure it so it doesn't require human intervention to mount volumes." So, like any good Linux-tech, I petitioned the gods of Google. Lo, there were many results — most falling into either the "here's how you encrypt a device" or the "here's how you take an encrypted device and make the OS automatically remount it at boot" camps. I was looking to do both so that my test-rig could be rebooted and just have the volume there. I was worried about testing whether devices were encrypted, not whether leaving keys on a system was adequately secure.

At any rate, at least for testing purposes (and in case I need to remember these later), here's what I synthesized from my Google searches.

1. Create a directory for storing encryption key-files. Ensure that directory is readable only by the root user:
   

   install -d -m 0700 -o root -g root /etc/crypt.d

2. Create a 4KB key from randomized data (stronger encryption key than typical, password-based unlock mechanisms):
   

   # dd if=/dev/urandom of=/etc/crypt.d/cryptFS.key bs=1024 count=4

   ...writing the key to the previously-created, protected directory. Up the key-length by increasing the value of the count parameter.
    
3. Use the key to create an encrypted raw device:
   

   # cryptsetup --key-file /etc/crypt.d/cryptFS.key \
   --cipher aes-cbc-essiv:sha256 luksFormat /dev/CryptVG/CryptVol

4. Activate/open the encrypted device for writing:
   

   # cryptsetup luksOpen --key-file /etc/crypt.d/cryptFS.key \
   /dev/CryptVG/CryptVol CryptVol_crypt

   Pass the location of the encryption-key using the --key-file parameter.
    
5. Add a mapping to the crypttab file:
   

   # ( printf "CryptVol_crypt\t/dev/CryptVG/CryptVol\t" ;
      printf "/etc/crypt.d/cryptFS.key\tluks\n" ) >> /etc/crypttab

   The OS will use this mapping-file at boot-time to open the encrypted device and ready it for mounting. The four column-values to the map are:
   1. Device-mapper Node: this is the name of the writable block-device used for creating filesystem structures and for mounting. The value is relative. When the device is activated, it will be assigned the device name /dev/mapper/<key_value>
   2. Hosting-Device: The physical device that hosts the encrypted psuedo-device. This can be a basic hard disk, a partition on a disk or an LVM volume.
   3. Key Location: Where the device's decryption-key is stored.
   4. Encryption Type: What encryption-method was used to encrypt the device (typically "luks")
    
6. Create a filesystem on the opened encrypted device:
   

   # mkfs -t ext4 /dev/mapper/CryptVol_crypt

7. Add the encrypted device's mount-information to the host's /etc/fstab file:
   

   # ( printf "/dev/mapper/CryptVol_crypt\t/cryptfs\text4" ;
      printf "defaults\t0 0\n" ) >> /etc/fstab

8. Verify that everything works by hand-mounting the device (`mount -a`)
9. Reboot the system (`init 6`) to verify that the encrypted device(s) automatically mount at boot-time

Keys and mappings in place, the system will reboot with the LUKSed devices opened and mounted. The above method's also good if you wanted to give each LUKS-protected device its own, device-specific key-file.

Note: You will really want to back up these key files. If you somehow lose the host OS but not the encrypted devices, the only way you'll be able to re-open those devices if you're able to restore the key-files to the new system. Absent those keys, you better have good backups of the unencrypted data - becuase you're starting from scratch.

Supporting Dynamic root-disk in LVM-enabled Templates

One of the main customers I support has undertaken adoption of cloud-based services. This customer's IA team also requires that the OS drive be carved up to keep logging and audit activities separate from the rest of the OS disks. Previous to adoption of cloud-based services, this was a non-problem.

Since moving to the cloud — and using a build-method that generates launch-templates directly in the cloud (EL6 and EL7) — the use of LVM has proven problematic - particularly with EL6. Out-of-th-box, EL6 does not support dynamic resizing of the boot disk. This means that specifying a larger-than-default root-disk when launching a template is pointless if using a "stock" EL6 template. This can be overcome by creating a custom launch-template and that uses the dracut-modules-growroot from EPEL in that template.

Unfortunately, this EPEL RPM is only part of the picture. The downloaded dracut-modules-growroot RPM only supports growing "/" partition to the size of the larger-than-default disk if the template's root disk is either wholly unpartitioned or the disk is partitioned such that the "/" partition is the last partition on disk. It does not support a case where the "/" filesystem is hosted within an LVM2 volume-group. To get around this, it is necessary to patch the growroot.sh script that the dracut-modules-growroot RPM installs:

--- /usr/share/dracut/modules.d/50growroot/growroot.sh  2013-11-22 13:32:42.000000000 +0000
+++ growroot.sh 2016-08-02 15:56:54.308094011 +0000
@@ -18,8 +18,20 @@
 }

 _growroot() {
-       # Remove 'block:' prefix and find the root device
-       rootdev=$(readlink -f "${root#block:}")
+       # Compute root-device
+       if [ -z "${root##*mapper*}" ]
+       then
+               set -- "${root##*mapper/}"
+               VOLGRP=${1%-*}
+               ROOTVOL=${1#*-}
+               rootdev=$(readlink -f $(pvs --noheadings | awk '/'${VOLGRP}'/{print $1}'))
+               _info "'/' is hosted on an LVM2 volume: setting \$rootdev to ${rootdev}"
+       else
+               # Remove 'block:' prefix and find the root device
+               rootdev=$(readlink -f "${root#block:}")
+       fi
+
+       # root arg was nulled at some point...
        if [ -z "${rootdev}" ] ; then
                _warning "unable to find root device"
                return

Once the growroot.sh script has been patched, it will be necessary to generate and update the template's initramfs with the grow functionality enabled. If the template has multiple kernels installed, it will be desirable to ensure that each is functionally-enabled. A quick way to ensure that all of the initramfs files in the template are properly-enabled is to execute:

rpm -qa kernel | sed 's/^kernel-//' | \
   xargs -I {} dracut -f /boot/initramfs-{}.img

Note¹: The above will likely put data into the template's /var/log/dracut.log file. It is likely desirable to null-out this file (along with all other log files) prior to sealing the template.

Note²: Patching the growroot.sh script will cause RPM-verification to fail in VMs launched from the template. This can either be handled as a known/expected exception or can be averted by performing a `yum reinstall dracut-modules-growroot` in the template or in the VMs launched from the template.

Credit: The above is an extension to a solution that I found at Backslasher.Net (my Google-fu was strong the day that I wanted to solve this problem!)