Punishing Network Performance

Recently, I took delivery of a new laptop. My old laptop was rolling up on five years old, was still running Windows 7 and had less than 1GiB of free disk space. So, "it was time".

At any rate, the new laptop runs Windows 10 Pro. One of the nice things about this is that, because MicroSoft makes Hyper-V available for free for my OS, I no longer have to deal with VirtualBox (or any add-on virtualization solution, for that matter). It means I'm free of the endless parade of "you must reboot to install new VirtualBox drivers" that pretty much caused me to stop with local virtualization on my prior laptop.

One of the nice things with virtualization is that it lets me run Linux, locally. While WSL/WSL2 show promise, they aren't quite there yet. Similarly, I'm not super chuffed that Docker Desktop for Windows requires me to run Docker as root. So, my Linux VM is an EL8 host that I am able to run PodMan on (and able to run my containers as an unprivileged user within that VM).

That said, most of the "how to run Linux on Hyper-V on Windows 10" guides I found resulted in my networking getting completely and totally jacked up. When not jacked up, my Internet connection looks like:

However, after initially booting my CentOS VM, I noticed that my yum downloads were running stoopid-slow. I assumed the problem was constrained to the VM. However, being detail-oriented, I checked my OS's network speed. It was, uh, "not good". I was getting less than 4Mbps down and about 0.2Mbps up. I also found that, if I rebooted my system, my download speeds would come back, but my uploads still sucked. Lastly, I found that if I wholly de-configured Hyper-V, my networking returned to normal.

Sleuthing time.

I hit the Googs and everyone is saying "this is a known issue: you can fix it by changing your NIC settings." Unfortunately, the mentioned settings weren't available in my NIC. Dunno if it's because I'm using WiFi or what. All I know is that it was looking like my choices were going to be "have good Internet speeds or have Linux on Hyper-V".

I'm not real keen on such choices. So, I kept digging – eventually getting to the point where I went beyond the first page of results. Ultimately, I found an answer on one of the StackExchange forums. Interestingly, the answer I found wasn't even marked as the "best answer". Had I not read through all the responses on one thread, I wouldn't even have found it.

At any rate, the fix was to not use my NIC in bridged mode at all. Instead, I needed to ignore all the top-match guides' instructions to use an "External" interface (which puts your NIC into bridged mode) and, instead, use an "Internal" interface ...and then set up connection-sharing, allowing my private vSwitch to share (rather than bridge) through my WiFi adapter. As soon as I made that change, my speeds came back.

Crib-Notes: Tuning MTU-size on EL7-based DHCP Clients

Because:

• One of the networks I deal with is broken
• I have only about a 15% hit-rate of what I actually need whenever I try to re-find the info
• most of my Google hits seem to think that using `ip link set...` is a fine, long-term solution and the ones that point to persistency referenc either nmcli or the /etc/sysconfig/network-scripts file

Need to save this here (maybe by doing so, Google will at least return this page when I need to re-search)...

To override the default MTU (in my case turning off Jumbo frames), edit the /etc/dhcp/dhclient-eth0.conf file like so:

interface eth0 {
    supersede interface-mtu 1350;
}

Reboot and everything should be fine (actually able to vi a file, use man pages, etc., without locking up my SSH session!).

EL6 and the Pain of 10Gbps Networking in AWS

AWS-hosted instances with optimized networking-support enabled see the 10Gbps interface as an Intel 10Gbps ethernet adapter (`lspci | grep Ethernet` will display a string similar to Intel Corporation 82599 Virtual Function). This interface makes use the ixgbevf network-driver. Enterprise Linux 6 and 7 bundle the version 2.12.x version of the driver into the kernel RPM. Per the Enabling Enhanced Networking with the Intel 82599 VF Interface on Linux Instances in a VPC document, AWS enhanced networking recommends version 2.14.2 or higher of the ixgbevf network-driver. To meet AWS's recommendations for 10Gbps support within an EL6 instance, it will be necessary to update the ixgbevf driver to at least version 2.14.2.

The ixgbevf network-driver source-code can be found on SourceForge. It should be noted that not every AWS-compatible version will successfully compile on EL 6. Version 3.2.2 is known to successfully compile, without intervention, on EL6.

Notes:

>>>CRITICAL ITEM<<<

It is necessary to recompile the ixgbevf driver and inject it into the kernel each time the kernel version changes. This needs to be done between changing the kernel version and rebooting into the new kernel version. Failure to update the driver each time the kernel changes will result in the instance failing to return to the network after a reboot event.

Step #10 from the implementation procedure:

rpm -qa kernel | sed 's/^kernel-//' | xargs -I {} dracut -v -f /boot/initramfs-{}.img {}

Is the easiest way to ensure any available kernels are properly linked against the ixgbevf driver.

>>>CRITICAL ITEM<<<

It is possible that the above process can be avoided by installing DKMS and letting it coordinate the insertion of the ixgbevf driver modules into updated kernels.

Procedure:

The following assumes the instance-owner has privileged access to the instance OS and can make AWS-level configuration changes to the instance-configuration:

1. Login to the instance
2. Escalate privileges to root
3. Install the up-to-date ixgbevf driver. This can be installed either by compiling from source or using pre-compiled binaries.
4. Delete any `*persistent-net*.rules` files found in the /etc/udev/rules.d directory (one or both of 70-persistent-net.rules and 75-persistent-net-generator.rules may be present)
5. Ensure that an `/etc/modprobe.d` file with the following minimum contents exists:
   

   alias eth0 ixgbevf
   

   Recommend creating/placing in /etc/modprobe.d/ifaliases.conf
6. Unload any ixgbevf drivers that may be in the running kernel:
   

   modprobe -rv ixgbevf
   

7. Load the updated ixgbevf driver into the running kernel:
   

   modprobe -v ixgbevf
   

8. Ensure that the /etc/modprobe.d/ixgbevf.conf file exists. Its contents should resemble:
   

   options ixgbevf InterruptThrottleRate=1
   

9. Update the /etc/dracut.conf. Ensure that the add_drivers+="" directive is uncommented and contains reference to the ixgbevf modules (i.e., `add_drivers+="ixgbevf"`)
10. Recompile all installed kernels:
    

    rpm -qa kernel | sed 's/^kernel-//'  | xargs -I {} dracut -v -f /boot/initramfs-{}.img {}
    

11. Shut down the instance
12. When the instance has stopped, use the AWS CLI tool to enable optimized networking support:
    

    aws ec2 --region  modify-instance-attribute --instance-id  --sriov-net-support simple
    

13. Power the instance back on
14. Verify that 10Gbps capability is available:
    
    1. Check that the ixgbevf module is loaded
       

       $ sudo lsmod
       Module                  Size  Used by
       ipv6                  336282  46
       ixgbevf                63414  0
       i2c_piix4              11232  0
       i2c_core               29132  1 i2c_piix4
       ext4                  379559  6
       jbd2                   93252  1 ext4
       mbcache                 8193  1 ext4
       xen_blkfront           21998  3
       pata_acpi               3701  0
       ata_generic             3837  0
       ata_piix               24409  0
       dm_mirror              14864  0
       dm_region_hash         12085  1 dm_mirror
       dm_log                  9930  2 dm_mirror,dm_region_hash
       dm_mod                102467  20 dm_mirror,dm_log
       

    2. Check that `ethtool` is showing that the default interface (typicall "`eth0`") is using the ixgbevf driver:
       

       $ sudo ethtool -i eth0
       driver: ixgbevf
       version: 3.2.2
       firmware-version: N/A
       bus-info: 0000:00:03.0
       supports-statistics: yes
       supports-test: yes
       supports-eeprom-access: no
       supports-register-dump: yes
       supports-priv-flags: no
       

    3. Verify that the interface is listed as supporting a link mode of `10000baseT/Full` and a speed of `10000Mb/s`:
       

       $ sudo ethtool eth0
       Settings for eth0:
               Supported ports: [ ]
               Supported link modes:   10000baseT/Full
               Supported pause frame use: No
               Supports auto-negotiation: No
               Advertised link modes:  Not reported
               Advertised pause frame use: No
               Advertised auto-negotiation: No
               Speed: 10000Mb/s
               Duplex: Full
               Port: Other
               PHYAD: 0
               Transceiver: Unknown!
               Auto-negotiation: off
               Current message level: 0x00000007 (7)
                                      drv probe link
               Link detected: yes
       

Asymmetrical NIC-bonding

Currently, the organization I am working for is making the transition from 1Gbps networking infrastructure to 10Gbps infrastructure. The initial goal had been to first migrate all of the high network-IO servers that were using trunked 1Gbps interfaces to using 10Gbps Active/Passive configurations.
Given the current high per-port cost of 10Gbps networking, it was requested that a way be found to not waste 10Gbps ports. Having a 10Gbps port sitting idle "in case" the active port became unavailable was seen as financially wasteful. A a result, we opted to pursue the use of asymmetrical A/P bonds that used our new 10Gbps links for the active/primary path and reused our 1Gbps infrastructure for the passive/failover path.

Setting up bonding on Linux can be fairly trivial. However, when you start to do asymmetrical bonding, you want to ensure that your fastest paths are also your active paths. This requires some additional configuration of the bonded pairs beyond just the basic declaration of the bond memberships.
In a basic bonding setup, you'll have three primary files in the /etc/sysconfig/network-scripts directory: ifcfg-ethX, ifcfg-ethY and ifcfg-bondZ. The ifcfg-ethX and ifcfg-ethY files are basically identical but for their DEVICE and HWADDR parameters. At their most basic, they'll each look (roughly) like:

DEVICE=ethN
HWADDR=AA:BB:CC:DD:EE:FF
ONBOOT=yes
BOOTPROTO=none
MASTER=bondZ
SLAVE=yes

And the (basic) ifcfg-bondZ file will look like:

DEVICE=bondZ
ONBOOT=yes
BOOTPROTO=static
NETMASK=XXX.XXX.XXX.XXX
IPADDR=WWW.XXX.YYY.ZZZ
MASTER=yes
BONDING_OPTS="mode=1 miimon=100"

This type of configuration may produce the results you're looking for, but it's not guaranteed to. If you want to absolutely ensure that your faster NIC will be selected as the primary NIC (and that it will fail back to that NIC in the event that the faster NIC goes offline and then back online), you need to be a bit more explicit with your ifcfg-bondZ file. To do this, you'll mostly want to modify your BONDING_OPTS directive. I also tend to add some BONDING_SLAVEn directives, but that might be overkill. Your new ifcfg-bondZ file that forces the fastest path will look like:

DEVICE=bondZ

ONBOOT=yes

BOOTPROTO=static

NETMASK=XXX.XXX.XXX.XXX

IPADDR=WWW.XXX.YYY.ZZZ

MASTER=yes

BONDING_OPTS="mode=1 miimon=100 primary=ethX primary_reselect=1"

The primary= tells the bonding driver to set the ethX device as primary when the bonding-group first onlines. The primary_reselect= tells it to use a interface selection policy of "best".

Note: The default policy is "0". This policy simply says "return to interface declared as primary". I choose to override with policy "1" as a hedge against the primary interface coming back in some kind of degraded state (while most of our 10Gbps media is 10Gbps-only, some of the newer ones are 100/1000/1000). I only want to fail back to the 10Gbps interface if it's still running at 10Gbps and hasn't for some reason, negotiated down to some slower speed.

When using the more explicit bonding configuration, the resultant configuration will resemble something like:

Ethernet Channel Bonding Driver: v3.4.0-1 (Octobber 7, 2008)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: ethX (primary_reselect better)

Currently Active Slave: ethX

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: ethX

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: AA:BB:CC:DD:EE:FF

Slave Interface: ethY

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: FF:EE:DD:CC:BB:AA

Linux (Networking): Oh How I Hate You

Working with most commercial UNIX systems (Solaris, AIX, etc.) and even Windows, you take certain things for granted. Easy networking setup is one of those things. It seems to be particularly so for systems that are designed to work in modern, redundant networks. Setting up things like multi-homed hosts is relatively trivial. I dunno. It may just be that I'm so used to how commercial OSes do things that, when I have to deal with The Linux Way™ it seems hopelessly archaic and picayune.

If I take a Solaris host that's got more than one network address on it, routing's pretty simple. I can declare one default route or I can declare a default route per interface/network. At the end of the day, Solaris's internal routing mechanisms just get it right. The only time there's really any mucking about is if I want/need to set up multiple static routes (or use dynamic routing protocols).

Linux... Well, perhaps it's just the configuration I had to make work. Someone wanted me to get a system with multiple bonded interfaces set up with VLAN tagging to route properly. Having the commercial UNIX mindset, I figured "just declare a GATEWAY in each of the bonded interface's /etc/sysconfig/network-scripts file" and that would be the end of the day.

Nope. It seems like Linux has a "last default route declared is the default route" design. Ok. I can deal with that. I mean, I used to have to deal with that with commercial UNIXes. So, I figured, "alright, only declare a default route in one interface's scriptfile". And, that sorta worked. I always got that one default route as my interface. Unfortunately, Linux's network routing philosophy didn't allow that to fully work as experience with other OSes might lead one to expect.

On the system I was asked to configure, one of the interfaces happened to be on the same network as the host I was administering it from. It should be noted that this interface is a secondary interface. The host's canonical name points to an IP on a different LAN segment. Prior to configuring the secondary interface on this host, I was able to log into that primary interface with no problems. Unfortunately, adding that secondary interface that was on the same LAN segment as my administration host cause problems. The Linux routing saw to it that I could only connect to the secondary interface. I was knocked out of trying to get into the primary interface.

This seemed odd. So, I started to dig around on the Linux host to figure out what the heck was going on. First up, a gander at the routing tables:

# netstat -rnv
Kernel IP routing table
Destination     Gateway          Genmask         Flags   MSS Window  irtt Iface
192.168.2.0     0.0.0.0          255.255.255.0   U         0 0          0 bond1.1002
192.168.33.0    0.0.0.0          255.255.255.0   U         0 0          0 bond0.1033
169.254.0.0     0.0.0.0          255.255.0.0     U         0 0          0 bond1.1002
0.0.0.0         192.168.33.254   0.0.0.0         UG        0 0          0 bond0.1033

Hmm... Not quite what I'm used to seeing. On a Solaris system, I'd expect something more along the lines of:

IRE Table: IPv4
  Destination             Mask           Gateway          Device   Mxfrg Rtt   Ref Flg  Out  In/Fwd
-------------------- ---------------  -------------------- ------  ----- ----- --- --- ----- ------
default              0.0.0.0          192.168.8.254                1500*     0   1 UG    1836      0
192.168.8.0          255.255.255.0    192.168.8.77         ce1     1500*     0   1 U      620      0
192.168.11.0         255.255.255.0    192.168.11.222       ce0     1500*     0   1 U        0      0
224.0.0.0            240.0.0.0        192.168.8.77         ce1     1500*     0   1 U        0      0
127.0.0.1            255.255.255.255  127.0.0.1            lo0     8232*     0   1 UH   13292      0

Yeah yeah, not identically formatted output, but similar enough that things on the Linux host don't look right if what you're used to seeing is the Solaris system's way of setting up routing. On a Solaris host, network destinations (i.e., "192.168.2.0", "192.168.33.0", "192.168.8.0" and "192.168.11.0" in the above examples) get routed through an IP address on a NIC. On Linux, however, it seems like all of the network routes were configured to go through whatever the default route was.

Now, what `netstat -rnv` is showing for interface network routes may not be strictly representative of what Linux is actually doing, but, both what Linux is doing and how its presented is wrong - particularly if there's firewalls between you and the multi-homed Linux hosts. The above output is kind of a sloppy representation of Linux's symmetrical routing philosphy. Unfortunately, because of the way Linux routes, If I have a configuration where the multi-homed Linux host has an two IP addresses - 192.168.2.123 and 192.168.33.123 - and I'm connecting from a host with an address of 192.168.2.76 but am trying to connect to the Linux host's 192.168.33.123 address, my connection attempt times out. While Linux may, in fact, receive my connection request at the 192.168.33.123 address, its default routing behavior seems to be to send it back out through its 192.168.2.123 address - ostensibly because the host I'm connecting from is on the same segment as the Linux host's 192.168.2.123 address. 

Given my background, my first thought is "make the Linux routing table look like the routing tables you're more used to.". Linux is nice enough to let me do what seem to be the right `route add` statements. However, it doesn't allow me to nuke the seemingly bogus network routes pointing at 0.0.0.0.

Ok, apparently I'm in for some kind of fight. I've gotta sort out the routing philsophy differences between my experience and what the writers of the Linux networking stack are. Fortunately, I have a good friend (named "Google") who's generally pretty good at getting my back. It's with Google's help that I discover that this kind of routing problem is handled through Linux's "advanced routing" functionality. I don't really quibble about what's so "advanced" about sending a response packet back out the same interface that the request packet came in on. I just kinda shrug and chalk it up to differences in implementation philosophy. It does, however, leave me with the question of, "how do I solve this difference of philosphy?"

Apparently, I have to muck about with files that I don't have to on either single-homed Linux systems or multi-homed commercial UNIX systems. I have to configure additional routing tables so that I can set up policy routing. Ok, so, I'm starting to scratch my head here. By itself, this isn't inherently horrible. However, it's not one of those topics that seems to come up a lot. It's neither well-documented in Linux nor do many relevant hits get returned by Google. Thus, I'm left to take what hits I do find and start experimenting. Ultimately, I found that I had to set up five files (in addition to the normal /etc/sysconfig/network-scripts/ifcfg-* files) to get thinks working as I think they ought:

/etc/iproute2/rt_tables
/etc/sysconfig/network-scripts/route-bond0.1033
/etc/sysconfig/network-scripts/route-bond1.1002
/etc/sysconfig/network-scripts/rule-bond0.1033
/etc/sysconfig/network-scripts/rule-bond1.1002

Using the "/etc/iproute2/rt_tables" file is kind of optional. Mostly, it lets me assign logical/friendly names to the extra routing tables I need to set up. I like friendly names. They're easier to remember and can be more self-documenting than plain-Jane numeric IDs. So, I edit the "/etc/iproute2/rt_tables" and add two lines:

2         net1002
33        net1033

I should probably note that what I actually wanted to add was:

1002      net1002
1033      net1033

I wanted these aliases as they would be reflective of what my 802.1q VLAN IDs were. Unfortunately, Linux seems to limit the range of numbers you can allocate table IDs out of. Worse, there are reserved and semi-reserved IDs in that limited range, further limiting your pool of table ID candidates. So, creating an enterprise-deployable standard config file might not be practical on a network with lots of VLANs, subnets, etc. Friendly names set up, I then had to set up the per "NIC" routing and rules files.

I set up the "/etc/sysconfig/network-scripts/route-${IF}.{$VLANID}"  with two routing statements:

table ${TABLENAME} to ${NETBASE}/${CIDR} dev ${DEV}.${VLAN}
table ${TABLENAME} to default via ${GATEWAY} dev ${DEV}.${VLAN}

This might actually be overkill. I might only need one of those lines. However, it was late in the day and I was tired of experimenting. They worked, so, that was good enough. Sometimes, a sledgehammer's a fine substitute for a scalpel.

Lastly, I set up the "/etc/sysconfig/network-scripts/rule-${IF}.{$VLANID}" files with a single rule, each:

from ${NETBASE}/${CIDR} table ${TABLENAME} priority ${NUM}

Again, the priority value's I picked may be suboptimal (I set the net1002 priority to "2" and the net1033 priority to "33"). But, since they worked, I left them at those values.

I did a `service network restart` and was able to access my multi-homed Linux host by either IP address from my workstation. Just to be super-safe, I bounced the host (to shake out anything that might have been left in place by my experiments). When the box came back from the reboot, I was still able to access it through either IP address from my workstation.