Update to EL6 and the Pain of 10Gbps Networking in AWS

In a previous post, EL6 and the Pain of 10Gbps Networking in AWS, EL6 and the Pain of 10Gbps Networking in AWS, I discussed how to enable using third-party ixgbevf drivers to support 10Gbps networking in AWS-hostd RHEL 6 and CentOS 6 instances. Under further testing, it turns out that this is overkill. The native drivers may be used, instead, with minimal hassle.

It turns out that, in my quest to ensure that my CentOS and RHEL AMIs contained as many of the AWS utilities present in the Amazon Linux AMIs, that I included two RPMS - ec2-net and ec2-net-util - that were preventing use of the native drivers. Skipping these two RPMs (and possibly sacrificing ENI hot-plug capabilities) allows a much more low-effort support of 10Gpbs networking in AWS-hosted EL6 instances.

Absent those RPMS, 10Gbps support becomes a simple matter of:

1. Add add_drivers+="ixgbe ixgbevf" to the AMI's /etc/dracut.conf file
2. Use dracut to regenerate the AMI's initramfs.
3. Ensure that there are no persistent network device mapping entries in /etc/udev/rules.d.
4. Ensure that there are no ixgbe or ixgbevf config directives in /etc/modprobe.d/* files.
5. Enable sr-iov support in the instance-to-be-registered
6. Register the AMI 

The resulting AMIs (and instances spawned from them) should support 10Gbps networking and be compatible with M4-generation instance-types.

Using DKMS to maintain driver modules

In my prior post, I noted that maintaining custom drivers for the the kernel in RHEL and CentOS hosts can be a bit painful (and prone to leaving you with an unreachable or even unbootable system). One way to take some of the pain out of owning a system with custom drivers is to leverage DKMS. In general, DKMS is the recommended way to ensure that, as kernels are updated, required kernel modules are also (automatically) updated.

Unfortunately, use of the DKMS method will require that developer tools (i.e., the GNU C-compiler) be present on the system - either in perpetuity or just any time kernel updates are applied. It is very likely that your security team will object to - or even prohibit - this. If the objection/prohibition cannot be overridden, use of the DKMS method will not be possible.

Steps

1. Set an appropriate version string into the shell-environment:
   

   export VERSION=3.2.2

2. Make sure that appropriate header files for the running-kernel are installed
   

   yum install -y kernel-devel-$(uname -r)

3. Ensure that the dkms utilities are installed:
   

   yum --enablerepo=epel install dkms

4. Download the driver sources and unarchive into the /usr/src directory:
   

   wget https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/${VERSION}/ixgbevf-${VERSION}.tar.gz/download \
       -O /tmp/ixgbevf-${VERSION}.tar.gz && \
      ( cd /usr/src && \
         tar zxf /tmp/ixgbevf-${VERSION}.tar.gz )

5. Create an appropriate DKMS configuration file for the driver:
   

   cat > /usr/src/ixgbevf-${VERSION}/dkms.conf << EOF
   PACKAGE_NAME="ixgbevf"
   PACKAGE_VERSION="${VERSION}"
   CLEAN="cd src/; make clean"
   MAKE="cd src/; make BUILD_KERNEL=\${kernelver}"
   BUILT_MODULE_LOCATION[0]="src/"
   BUILT_MODULE_NAME[0]="ixgbevf"
   DEST_MODULE_LOCATION[0]="/updates"
   DEST_MODULE_NAME[0]="ixgbevf"
   AUTOINSTALL="yes"
   EOF

6. Register the module to the DKMS-managed kernel tree:
   

   dkms add -m ixgbevf -v ${VERSION}

7. Build the module against the currently-running kernel:
   

   dkms build ixgbevf/${VERSION}

Verification

The easiest way to verify the correct functioning of DKMS is to:

1. Perform a `yum update -y`
2. Check that the new drivers were created by executing `find /lib/modules -name ixgbevf.ko`. Output should be similar to the following:
   

   find /lib/modules -name ixgbevf.ko | grep extra
   /lib/modules/2.6.32-642.1.1.el6.x86_64/extra/ixgbevf.ko
   /lib/modules/2.6.32-642.6.1.el6.x86_64/extra/ixgbevf.ko

   There should be at least two output-lines: one for the currently-running kernel and one for the kernel update. If more kernels are installed, there may be more than just two output-lines
    
   
3. Reboot the system, then check what version is active:
   

   modinfo ixgbevf | grep extra
   filename:       /lib/modules/2.6.32-642.1.1.el6.x86_64/extra/ixgbevf.ko

   If the output is null, DKMS didn't build the new module.

EL6 and the Pain of 10Gbps Networking in AWS

AWS-hosted instances with optimized networking-support enabled see the 10Gbps interface as an Intel 10Gbps ethernet adapter (`lspci | grep Ethernet` will display a string similar to Intel Corporation 82599 Virtual Function). This interface makes use the ixgbevf network-driver. Enterprise Linux 6 and 7 bundle the version 2.12.x version of the driver into the kernel RPM. Per the Enabling Enhanced Networking with the Intel 82599 VF Interface on Linux Instances in a VPC document, AWS enhanced networking recommends version 2.14.2 or higher of the ixgbevf network-driver. To meet AWS's recommendations for 10Gbps support within an EL6 instance, it will be necessary to update the ixgbevf driver to at least version 2.14.2.

The ixgbevf network-driver source-code can be found on SourceForge. It should be noted that not every AWS-compatible version will successfully compile on EL 6. Version 3.2.2 is known to successfully compile, without intervention, on EL6.

Notes:

>>>CRITICAL ITEM<<<

It is necessary to recompile the ixgbevf driver and inject it into the kernel each time the kernel version changes. This needs to be done between changing the kernel version and rebooting into the new kernel version. Failure to update the driver each time the kernel changes will result in the instance failing to return to the network after a reboot event.

Step #10 from the implementation procedure:

rpm -qa kernel | sed 's/^kernel-//' | xargs -I {} dracut -v -f /boot/initramfs-{}.img {}

Is the easiest way to ensure any available kernels are properly linked against the ixgbevf driver.

>>>CRITICAL ITEM<<<

It is possible that the above process can be avoided by installing DKMS and letting it coordinate the insertion of the ixgbevf driver modules into updated kernels.

Procedure:

The following assumes the instance-owner has privileged access to the instance OS and can make AWS-level configuration changes to the instance-configuration:

1. Login to the instance
2. Escalate privileges to root
3. Install the up-to-date ixgbevf driver. This can be installed either by compiling from source or using pre-compiled binaries.
4. Delete any `*persistent-net*.rules` files found in the /etc/udev/rules.d directory (one or both of 70-persistent-net.rules and 75-persistent-net-generator.rules may be present)
5. Ensure that an `/etc/modprobe.d` file with the following minimum contents exists:
   

   alias eth0 ixgbevf
   

   Recommend creating/placing in /etc/modprobe.d/ifaliases.conf
6. Unload any ixgbevf drivers that may be in the running kernel:
   

   modprobe -rv ixgbevf
   

7. Load the updated ixgbevf driver into the running kernel:
   

   modprobe -v ixgbevf
   

8. Ensure that the /etc/modprobe.d/ixgbevf.conf file exists. Its contents should resemble:
   

   options ixgbevf InterruptThrottleRate=1
   

9. Update the /etc/dracut.conf. Ensure that the add_drivers+="" directive is uncommented and contains reference to the ixgbevf modules (i.e., `add_drivers+="ixgbevf"`)
10. Recompile all installed kernels:
    

    rpm -qa kernel | sed 's/^kernel-//'  | xargs -I {} dracut -v -f /boot/initramfs-{}.img {}
    

11. Shut down the instance
12. When the instance has stopped, use the AWS CLI tool to enable optimized networking support:
    

    aws ec2 --region  modify-instance-attribute --instance-id  --sriov-net-support simple
    

13. Power the instance back on
14. Verify that 10Gbps capability is available:
    
    1. Check that the ixgbevf module is loaded
       

       $ sudo lsmod
       Module                  Size  Used by
       ipv6                  336282  46
       ixgbevf                63414  0
       i2c_piix4              11232  0
       i2c_core               29132  1 i2c_piix4
       ext4                  379559  6
       jbd2                   93252  1 ext4
       mbcache                 8193  1 ext4
       xen_blkfront           21998  3
       pata_acpi               3701  0
       ata_generic             3837  0
       ata_piix               24409  0
       dm_mirror              14864  0
       dm_region_hash         12085  1 dm_mirror
       dm_log                  9930  2 dm_mirror,dm_region_hash
       dm_mod                102467  20 dm_mirror,dm_log
       

    2. Check that `ethtool` is showing that the default interface (typicall "`eth0`") is using the ixgbevf driver:
       

       $ sudo ethtool -i eth0
       driver: ixgbevf
       version: 3.2.2
       firmware-version: N/A
       bus-info: 0000:00:03.0
       supports-statistics: yes
       supports-test: yes
       supports-eeprom-access: no
       supports-register-dump: yes
       supports-priv-flags: no
       

    3. Verify that the interface is listed as supporting a link mode of `10000baseT/Full` and a speed of `10000Mb/s`:
       

       $ sudo ethtool eth0
       Settings for eth0:
               Supported ports: [ ]
               Supported link modes:   10000baseT/Full
               Supported pause frame use: No
               Supports auto-negotiation: No
               Advertised link modes:  Not reported
               Advertised pause frame use: No
               Advertised auto-negotiation: No
               Speed: 10000Mb/s
               Duplex: Full
               Port: Other
               PHYAD: 0
               Transceiver: Unknown!
               Auto-negotiation: off
               Current message level: 0x00000007 (7)
                                      drv probe link
               Link detected: yes