Travis, EL7 and Docker-Based Testing

As noted in a prior post, lot of my customer-oriented activities support deployment within networks that are either partially- or wholly-isolated from the public Internet. Yesterday, as part of supporting one such customer, I was stood up a new project to help automate the creation of yum repository configuration RPMs for private networks. I've had to hand-jam such files twice, now, and there's unwanted deltas between the two jam-sessions (in defense, they were separated from each other by nearly a three-year time-span). So, I figured it was time to standardize and automate things.

Usually, when I stand up a project, I like to include tests of the content that I wish to deliver. Since most of my projects are done in public GitHub repositories, I typically use TravisCI to automate my testing. Prior to this project, however, I wasn't trying to automate the validity-testing of RPM recipes via Travis. Typically, when automating creation of RPMs I wish to retain or deliver, I set up a Jenkins job that takes the resultant RPMs and stores them in Artifactory – both privately-hosted services. Most of my prior Travis jobs were simple, syntax-checkers (using tools like shellcheck, JSON validators, CFn validators, etc.) rather than functionality-checkers.

This time, however, I was trying to deliver a functionality (RPM spec files that would be used to generate source files from templates and package the results). So, I needed to be able to test that a set of spec files and source-templates could be reliably-used to generate RPMs. This meant, I needed my TravisCI job to generate "throwaway" RPMs from the project-files.

The TravisCI system's test-hosts are Ubuntu-based rather than RHEL or CentOS based.While there are some tools that will allow you to generate RPMs on Ubuntu, there've been some historical caveats on their reliability and/or representativeness. So, my preference was to be able to use a RHEL or CentOS-based context for my test-packagings. Fortunately, TravisCI does offer the ability to use Docker on their test-hosts.

In general, setting up a Docker-oriented job is relatively straight forward. Where things get "fun" is that the version of `rpmbuild` that comes with Enterprise Linux 7 gets kind of annoyed if it's not able to resolve the UIDs and GIDs of the files it's trying to build from (never mind that the build-user inside the running Docker-container is "root" ...and has unlimited access within that container). If it can't resolve them, the rpmbuild tasks fail with a multitude of not terribly helpful "error: Bad owner/group: /path/to/repository/file" messages.

After googling about, I ultimately found that I needed to ensure that the UIDs and GIDs of the project-content need to exist within the Docker-container's /etc/passwd and /etc/group files, respectively. Note: most of the "top" search results Google returned to me indicated that the project files needed to be `chown`ed. However, simply being mappable proved to be sufficient.

Rounding the home stretch...

To resolve the problem, I needed to determine what UIDs and GIDs the project-content had inside my Docker-container. That meant pushing a Travis job that included a (temporary) diagnostic-block to stat the relevant files and return me their UIDs and GIDs. Once the UIDs and GIDs were determined, I needed to update my Travis job to add relevant groupadd and useradd statements to my container-preparation steps. What I ended up with was.

    sudo docker exec centos-${OS_VERSION} groupadd -g 2000 rpmbuilder
    sudo docker exec centos-${OS_VERSION} adduser -g 2000 -u 2000 rpmbuilder

It was late in the day, by this point, so I simply assumed that the IDs were stable. I ran about a dozen iterations of my test, and they stayed stable, but that may have just been "luck". If I run into future "owner/group" errors, I'll update my Travis job-definition to scan the repository-contents for their current UIDs and GIDs and then set them based on those. But, for now, my test harness works: I'm able to know that updates to existing specs/templates or additional specs/templates will create working RPMs when they're taken to where they need to be used.

Linux Multipath Path-Failure Simulation

Previously, I've discussed how to set up the RedHat Linux storage multipathing software to manage fibrechannel-based storage devices. I didn't, however, cover how one tests that such a configuration is working as intended.
When it comes to testing resilient configurations, one typically has a number of testing options. In a fibrechannel fabric situation, one can do any of:

• Offline a LUN within the array
• Down an array's storage processors and/or HBAs
• Pull the fibrechannel connection between the array and the fibrechannel switching infrastructure
• Shut off a switch (or switches) in a fabric
• Pull connections between switches in a fabric
• Pull the fibrechannel connection between the fibrechannel switching infrastructure and the storage-consuming host system
• Disable paths/ports within a fibrechannel switch
• Disable HBAs on the storage-consuming host systems
• Disable particular storage targets within the storage-consuming host systems

Generally, I favor approaches that limit the impact of the tested scenario as much as possible. I favor approaches that limit the likelihood of introducing actual/lasting breakage into the tested configuration.
I also tend to favor approaches where I have as much control of the testing scenario as possible. I'm an impatient person and having to coordinate outages and administrative events with other infrastructure groups and various "stakeholders" can be a tiresome, protracted chore. Some would say that indicates I'm not a team-player: I like to think that I just prefer to get things done efficiently and as quickly as possible. Tomayto/tomahto.
Through most of my IT career, I've worked primarily the server side of the house (Solaris, AIX, Linux ...and even - *ech* - Windows) - whether as a systems administrator or as an integrator. So, my testing approaches tend to be oriented from the storage-consumer's view of the world. If I don't want to have to coordinate outside of a server's ownership/management team, I'm pretty much limited to the last three items on the above list: yanking cables from the server's HBAs, disabling the server's HBAs and disabling storage targets within the server.
Going back to the avoidance of "introducing actual/lasting breakage", I tend to like to avoid yanking cables. At the end of the day, you never know if the person doing the monkey-work of pulling the cable is going to do it like a surgeon or like a gorilla. I've, unfortunately, been burned by run-ins with more than a few gorillas. So, if I don't have to have cables physically disconnected, I avoid it.
Being able to logically disable an HBA is a nice test scenario. It effects the kind of path-failure scenario that you're hoping to test. Unfortunately, not all HBA manufacturers seem to include the ability to logically disable the HBA from within their management utilities. Within commercial UNIX variants - like Solaris or AIX - this hasn't often proven to be a problem. Under Linux, however, the abilty to logically disable HBAs from within their management utilities seems to be a bit "spotty".
Luckily, where the HBA manufacturers sometimes leave me in the lurch, RedHat Linux leaves me some alternatives. In the spirit of the Linux DIYism, those alternatives aren't really all that fun to deal with ...until you write tools, for yourself, that removes some of the pain. I wrote two tools to help myself in this area: one is a tool which offlines designated storage paths and one is a tool which attempts to restore those downed storage paths.
Linux makes it possible to change the system-perceived state of a given device path by writing the term "offline" to the file location, /sys/block/${DEV}/device/state. Thus, were one to want to make the OS think that the path to /dev/sdg was down, one would execute the command, `echo "offline" > /sys/block/sdg/device/state`. All that my path-downing script is makes it so you can down a given /dev/sdX device by executing `pathdown.sh <DEVICE>` (e.g., `pathdown sdg`). There's minimal logic built in to verify that the named /dev/sdX device is a real, downable device and it provides a post-action status of that device, but, other than that, it's a pretty simple script.
To decide which path one wants to down, it's expected that the tester will look at the multipather's view of its managed devices using `multipath -l <DEVICE>` (e.g., `multipath -l mpath0`). This command will produce output similar to the following:

mpath0 (360a9800043346d364a4a2f41592d5849) dm-7 NETAPP,LUN
[size=20G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 0:0:0:1 sda 8:0   [active][undef]
 \_ 0:0:1:1 sdb 8:16  [active][undef]
 \_ 1:0:0:1 sdc 8:32  [active][undef]
 \_ 1:0:1:1 sdd 8:48  [active][undef]

Thus, if one wanted to deactivate one of the channels in the mpath0 multipathing group, one might issue the command  `pathdown sdb`. This would result in the path associated with /dev/sdb being taken offline. After taking this action, the output of `multipath -l mpath0` would change to:

mpath0 (360a9800043346d364a4a2f41592d5849) dm-7 NETAPP,LUN
[size=20G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 0:0:0:1 sda 8:0   [active][undef]
 \_ 0:0:1:1 sdb 8:16  [failed][faulty]
 \_ 1:0:0:1 sdc 8:32  [active][undef]
 \_ 1:0:1:1 sdd 8:48  [active][undef]

Typically, when doing such testing activities, one would be performing a large file operation to the disk device (preferably a write operation). My test sequence is typically to

1. Start an `iostat` job, grepping for the devices I'm interested in, and capturing the output to a file
2. start up a file transfer (or even just a `dd` operation) into the device.
3. Start downing paths as the transfer occurs
4. Wait for the transfer to complete, then kill the `iostat` job
5. Review the captured output from the `iostat` job to ensure that the I/O behaviors I expected to see actually occurred

In the testing environment I had available when I wrote this page, I was using a NetApp filer presenting blockmode storage via fibrechannel. The NetApp multipathing plugin supports concurrent, multi-channel operations to the target LUN. Thus, the output from my `iostat` job will show uniform I/Os across all paths to the LUN, and then show outputs drop to zero on each path that I offline. Were I using an array that only supported Active/Passive I/O operations, I would expect to see the traffic move from the downed path to one of the failover paths, instead.
So, great: you've tested that your multipathing system behaves as expected. However, once you've completed that testing, all of the paths that you've offlined have stayed offline. What to do about it?
The simplest method is to reboot the system. However, I abhor knocking my systems' `uptime` if I don't absolutely have to. Fortunately, much as Linux provides the means to offline paths, it provides the means for reviving them (well, to be more accurate, to tell it "hey, go check these paths and see if they're online again"). As with offlining paths, the methods for doing so aren't currently built into any OS-provided utilities. What you have to do is:

1. Tell the OS to delete the device paths
2. Tell the OS to rescan the HBAs for devices it doesn't currently know about
3. Tell the multipath service to look for changes to paths associated with managed devices

The way you tell the OS to (gracefully) delete device paths is to write a value to a file. Specifically, one writes the value "`1" to the file /sys/block/${DEV}/device/delete. Thus, if one is trying to get the system to clean up for the downed device path, /dev/sdb, one would issue the command `echo "1" > /sys/block/sdb/device/delete`.
The way you tell the OS to rescan the HBAs is to issue the command `echo "- - -" >  /sys/class/scsi_host/${HBA}/scan`. In Linux, the HBAs are numbered in the order found and named "hostN" (i.e., "host0", "host1", etc.). Thus, to rescan HBA 0, one would issue the command `echo "- - -" >  /sys/class/scsi_host/host0/scan` (for good measure, rescan all the HBAs).
The way to tell the multipath service to look for changes to paths associated with managed devices is to issue the command `multipath` (I acutally use `multipath -v2` because I like the more verbose output that tells me what did or didn't happen as a result of the command). Granted, the multipath service periodically rescans the devices it manages to find state-change information, but I don't like to wait for systems to "get around to it".
All that my path fixing script does is rolls up the above, three steps into one, easy to remember and use command.
Depending on how you have logging configured on your system, the results of all the path offlining and restoration will be logged. Both the SCSI subsystem and the multipathing daemon should log events. Thus, you can verify the results of your activities by looking in your system logs.
That said, if the system you're testing is hooked up to an enterprise monitoring system, you will want to let your monitoring groups know that they need to ignore the red flags you'll be generating on their monitoring dashboards.