Recently, I was tasked with a project that, as part of the preparation-phase, required patching up a whole bunch of servers to the same patch-level. In total, I patched about 20 systems that were dual-homed and equipped with asymmetrical, 10/1 active/passive bonds. All but the last system went aces. The last system ...was weird.
After patching the final system, my secondary network-pair was no longer able to talk on the network. While diagnosing, attempts to ping out to any hosts on the local LAN segment resulted in "host unreachable" errors. Any hosts that I did try to ping, ended up in the afflicted host's ARP table with a missing ("<incomplete>") MAC address entry.
Our builds are normally fairly locked down. That means, many troubleshooting tools (such as tcpdump) are not loaded. At my last straw, I opted to temporarily load tcpdump to see what, if anything, the afflicted bond (and sub-interfaces) were seeing on the network. Interestingly, as soon as I started snooping either the parent bond or the active interface, networking activity became "normal" and the previously "<incomplete>" ARP table entries populated the other hosts' MAC addresses. As soon as I stopped my tcpdump runs, networking reverted to its broken state and the other hosts' MAC address entries in the ARP table returned to "<incomplete>".
Still don't have a fix - this is a "work in progress" article, at the moment.What I ended up doing as a workaround - since this server is a critical infrastructure component - is did a `ifconfig bond1 promisc` and updated its /etc/sysconfig/network-scripts file to preserve the state should the system reboot before I find a more suitable fix. So, for right now, in order to get this one bond (of two on the system) to work, I need to leave it in promiscuous mode.
Obviously, I have our networking guys looking at the switches to see if there's a difference between how the ports for bond0 and bond1 are configured. I figure, it has to be the network, since: A) one bond works but the other doesn't; and, B) no promiscuous-mode changes were required for any of the other hosts that were patched.
At any rate, if you happen to stumble on this article before I get it beyond a "work in progress" state, please feel free to comment if you know a likely fix.
Wednesday, September 25, 2013
Tuesday, September 3, 2013
Password Encryption Methods
As a systems administrator, there are times where you have to find programatic ways to update passwords for accounts. On some operating systems, your user account modification tools don't allow you to easily set passwords in a programmatic fashion. Solaris used to be kind of a pain, in this regard, when it came time to do an operations-wide password reset. Fortunately, Linux is a bit nicer about this.
The Linux `usermod` utility allows you to (relatively easily) specify password in a programmatic fashion. The one "gotcha" of the utility is the requirement to use hashed password-strings rather than cleartext. The question becomes, "how best to generate those hashes".
The answer will likely depend on your security requirements. If MD5 hashes are acceptable, then you can use OpenSSL or the `grub-md5-crypt` utilities to generate them. If, however, your security requirements require SHA256- or even SHA512-based hashes neither of those utilities will work for you.
Newer Linux distributions (e.g. RHEL 6+) essentially replace the `grub-md5-crypt` utility with the `grub-crypt` utility. This utility supports not only the older MD5 that its predecessor supported, but also SHA256 and SHA512.
However, what do you do when `grub-crypt` is missing (e.g., you're running RedHat 5.x) or you just want one method that will work across different Linux versions (e.g., your operations environment consists of a mix of RHEL 5 and RHEL 6 systems)? While you can use a tool like `openssl` to do the dirty work, if your security requirements dictate an SHA-based hashing algorithm, it's not currently up to the task. If you want the SHAs in a cross-distribution manner, you have to leverage more generic tools like Perl or Python.
The following examples will show you how to create a hashed password-string from the cleartext password "Sm4<kT*TheFace". Some tools (e.g., OpenSSL's "passwd" command) allowed you to choose to use a fixed-salt or a random-salt. From the standpoint of being able to tell "did I generate this script", using a fixed-salt can be useful; however, using a random-salt may be marginally more secure. The Perl and Python methods pretty much demand the specification of a salt. In the examples below, the salt I'm using is "Ay4p":
The Linux `usermod` utility allows you to (relatively easily) specify password in a programmatic fashion. The one "gotcha" of the utility is the requirement to use hashed password-strings rather than cleartext. The question becomes, "how best to generate those hashes".
The answer will likely depend on your security requirements. If MD5 hashes are acceptable, then you can use OpenSSL or the `grub-md5-crypt` utilities to generate them. If, however, your security requirements require SHA256- or even SHA512-based hashes neither of those utilities will work for you.
Newer Linux distributions (e.g. RHEL 6+) essentially replace the `grub-md5-crypt` utility with the `grub-crypt` utility. This utility supports not only the older MD5 that its predecessor supported, but also SHA256 and SHA512.
However, what do you do when `grub-crypt` is missing (e.g., you're running RedHat 5.x) or you just want one method that will work across different Linux versions (e.g., your operations environment consists of a mix of RHEL 5 and RHEL 6 systems)? While you can use a tool like `openssl` to do the dirty work, if your security requirements dictate an SHA-based hashing algorithm, it's not currently up to the task. If you want the SHAs in a cross-distribution manner, you have to leverage more generic tools like Perl or Python.
The following examples will show you how to create a hashed password-string from the cleartext password "Sm4<kT*TheFace". Some tools (e.g., OpenSSL's "passwd" command) allowed you to choose to use a fixed-salt or a random-salt. From the standpoint of being able to tell "did I generate this script", using a fixed-salt can be useful; however, using a random-salt may be marginally more secure. The Perl and Python methods pretty much demand the specification of a salt. In the examples below, the salt I'm using is "Ay4p":
- Perl (SHA512) Method: `perl -e 'print crypt("Sm4<kT*TheFace", "\$6\$Ay4p\$");'`
- Python (SHA512) Method: `python -c 'import crypt; print crypt.crypt(""Sm4<kT*TheFace", "$6$Ay4p$")'
- 1 = MD5
- 2a = BlowFish (not present in all Linux distributions)
- 5 = SHA256 (Linux with GlibC 2.7+)
- 6 = SHA512 (Linux with GlibC 2.7+)
Subscribe to:
Posts (Atom)