The Punishment of NFS on Hardened EL7 Systems

All of the customers I currently serve operate under two, main requirements:

• If using Linux, it has to be Enterprise Linux (RHEL or CentOS)
• All such systems must be hardened to meet organizational specifications

The latter means that IPv6 support has to be disabled on all ELx deployments.

On the plus side of current customer-trends, most are (finally) making the effort to migrate from EL6 to EL7. Unfortunately, recent releases of EL7 included an update to the RPC subsystem. Further unfortunately, this update can cause the RPC subsystem to break on a system that's been hardened to disable IPv6.

With later updates to the RPC subsystems, it will attempt to perform an IPv6 network-bind. It makes the determination of whether to attempt this based on whether the IPv6 components are available/enabled in the initramfs boot-kernel.

With typical hardening-routines, IPv6 disablement happens after the initramfs boot-kernel has loaded. This is done when the boot processes read the /etc/sysctl.conf file and files within /etc/sysctl.d. Unfortunately, if the system-owner hasn't ensured that the /etc/sysctl.conf file packaged within the initramfs looks like the one in the booted system's root filesystem and the root filesystem's /etc/sysctl.conf file disables IPv6, bad times ensue. The RPC subsystem assumes that IPv6 is available. Then, when systemd attempts to start the rpcbind.socket unit, it fails. All the other systemd units that depend on the rpcbind.socket unit then also fail. This means no RPC service and no NFS server or client services.

In this scenario, the general fix-process is:

1. Uninstall the dracut-config-generic RPM (`yum erase -y dracut-config-generic`)
2. Rebuild the kernel (`dracut -v -f`)
3. Reboot the system

Once the system comes back from the reboot, all of the RPC components – and services that rely on them – should function as expected.

...but that's only the first hurdle. When using default NFS mount options, NFS clients will attempt to perform an NFS v4.1 mount of the NFS server's shares. If NFS hasn't been explicitly configured for GSS-protected mounts, the mount of the filesystem typically takes around two minutes to occur (while the GSS-related subsystems try to negotiate the session before ultimately timing-out and reverting to "sys" security-mode). One can either force the use of NFSv3,  explicitly request the "sys" security-mode or wholly disable the rpc-gssd service-components. Explicitly requesting the sys security-model (using the sec=sys mount-option) halves the amount of time needed to negotiate the initial mount-request. Requesting NFSv3 (using the vers=3 mount-option) avoids the security-related negotiations, altogether, making the mount-action almost instantaneous. Similarly, disabling the rpc-gssd service-components (using systemctl's mask command for the rpc-gssd and/or nfs-secure services) avoids the GSS-related negotiation-components, making the mount-action almost spontaneous.

Once those bits are out of the way, then it's usually just a matter of configuring appropriate SELinux elements to allow the sharing-out of the desired filesystems and setting up the export-definitions.

Why I Default to The Old Ways

I work with a growing team of automation engineers. Most are purely dev types. Those that have lived in the Operations world, at all, skew heavily towards Windows or only had to very lightly deal with UNIX or Linux.

I, on the other hand, have been using UNIX flavors since 1989. My first Linux system was the result of downloading a distribution from the MIT mirrors in 1992. Result, I have a lot of old habits (seriously: some of my habits are older than some of my teammates). And, because I've had to get deep into the weeds with all of those operating systems many, many, many times, over the years, those habits are pretty entrenched ("learned with blood" and all that rot).

A year or so ago, I'd submitted a PR that included some regex-heavy shell scripts. The person that reviewed the PR had asked "why are you using '[<space><TAB>]*' in your regexes rather than just '\s'?". At the time, I think my response was a semi-glib, "A) old habits die hard; and, B) I know that the former method always works".

That said, I am a lazy-typist. Typing "\s" is a lot fewer keystrokes than is "[<space><TAB>]*". Similarly, "\s" takes up a lot less in the way of column-width than does "[<space><TAB>]*" (and I/we generally like to code to fairly standard page-widths). So, for both laziness reasons and column-conservation reasons, I started to move more towards using "\s" and away from using "[<space><TAB>]*".  I think in the last 12 months, I've moved almost exclusively to  "\s".

Today, that move bit me in the ass. Well, yesterday, actually, because that's when I started receiving reports that the tool I'd authored on EL7 wasn't working when installed/used on EL6. Ultimately, I traced the problem to an `awk` invocation. Specifically, I had a chunk of code (filtering DNS output) that looked like:

awk '/\sIN SRV\s/{ printf("%s;%s\n",$7,$8)}'

Which worked a treat on EL7 but on EL6, "not so much." When I altered it to the older-style invocation:

awk '/[  ]*IN[  ]*SRV[  ]*/{ printf("%s;%s\n",$7,$8)}'

It worked fine on both EL7 and EL6. Turns out the ancient version of `awk` (3.1.7) on EL6 didn't know how to properly interpret the "\s" token. Oddly (my recollection from writing other tooling) is that EL6's version of `grep` understands the "\s" token just fine.

When I Slacked the person I'd had the original conversation with a link to the PR with a "see: this is why" note, he replied, "oh: I never really used awk, so never ran into it".

Supporting Dynamic root-disk in LVM-enabled Templates - EL7 Edition

In my previous article,  Supporting Dynamic root-disk in LVM-enabled Templates, I discussed the challenges around supporting LVM-enabled VM-templates in cloud-based deployments of Enterprise Linux 6 VMs. The kernel used for Enterprise Linux 7 distributions makes template-based deployment of LVM-enabled VMs a bit easier. Instead of having to add an RPM from EPEL and then hack that RPM to make it support LVM2-encapsulated root volumes/filesystems, one need only ensure that the cloud-utils-growpart RPM is installed and do same launch-time massaging via cloud-init. By way of example:

#cloud-config
runcmd:
  - /usr/bin/growpart /dev/xvda 2
  - pvresize /dev/xvda2
  - lvresize -r -L +2G VolGroup00/logVol
  - lvresize -r -L +2G VolGroup00/auditVol

Will cause the launched instance to:

1. Grow the second partition on the boot disk to the end of the disk
2. Instruct LVM to resize the PV to match the new partition-size
3. Instruct LVM to grow the VolGroup00/logVol volume — and the filesystem on top of it — by 2GiB
4. Instruct LVM to grow the VolGroup00/auditVol volume — and the filesystem on top of it — by 2GiB

Upon login, the above launch-time configuration-actions can be verified by using `vgdisplay -s` and `lvs --segments -o +devices`:

# vgdisplay -s
  "VolGroup00" 29.53 GiB [23.53 GiB used / 6.00 GiB free]
# lvs --segments -o +devices
  LV       VG         Attr       #Str Type   SSize Devices
  auditVol VolGroup00 -wi-ao----    1 linear 8.53g /dev/xvda2(2816)
  auditVol VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(5512)
  homeVol  VolGroup00 -wi-ao----    1 linear 1.00g /dev/xvda2(1536)
  logVol   VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(2304)
  logVol   VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(5000)
  rootVol  VolGroup00 -wi-ao----    1 linear 4.00g /dev/xvda2(0)
  swapVol  VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(1024)
  varVol   VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(1792)