Password Encryption Methods

As a systems administrator, there are times where you have to find programatic ways to update passwords for accounts. On some operating systems, your user account modification tools don't allow you to easily set passwords in a programmatic fashion. Solaris used to be kind of a pain, in this regard, when it came time to do an operations-wide password reset. Fortunately, Linux is a bit nicer about this.

The Linux `usermod` utility allows you to (relatively easily) specify password in a programmatic fashion. The one "gotcha" of the utility is the requirement to use hashed password-strings rather than cleartext. The question becomes, "how best to generate those hashes".

The answer will likely depend on your security requirements. If MD5 hashes are acceptable, then you can use OpenSSL or the `grub-md5-crypt` utilities to generate them. If, however, your security requirements require SHA256- or even SHA512-based hashes neither of those utilities will work for you.

Newer Linux distributions (e.g. RHEL 6+) essentially replace the `grub-md5-crypt` utility with the `grub-crypt` utility. This utility supports not only the older MD5 that its predecessor supported, but also SHA256 and SHA512.

However, what do you do when `grub-crypt` is missing (e.g., you're running RedHat 5.x) or you just want one method that will work across different Linux versions (e.g., your operations environment consists of a mix of RHEL 5 and RHEL 6 systems)? While you can use a tool like `openssl` to do the dirty work, if your security requirements dictate an SHA-based hashing algorithm, it's not currently up to the task. If you want the SHAs in a cross-distribution manner, you have to leverage more generic tools like Perl or Python.

The following examples will show you how to create a hashed password-string from the cleartext password "Sm4<kT*TheFace". Some tools (e.g., OpenSSL's "passwd" command) allowed you to choose to use a fixed-salt or a random-salt. From the standpoint of being able to tell "did I generate this script", using a fixed-salt can be useful; however, using a random-salt may be marginally more secure. The Perl and Python methods pretty much demand the specification of a salt. In the examples below, the salt I'm using is "Ay4p":

• Perl (SHA512) Method: `perl -e 'print crypt("Sm4<kT*TheFace", "\$6\$Ay4p\$");'`
• Python (SHA512) Method: `python -c 'import crypt; print crypt.crypt(""Sm4<kT*TheFace", "$6$Ay4p$")'

Note that you specify the encryption-type used by specifying an numerical representation of the standard encryption-types. The standard encryption types for Linux operating systems (from the crypt() manpage):

• 1  = MD5
• 2a = BlowFish (not present in all Linux distributions)
• 5  = SHA256 (Linux with GlibC 2.7+)
• 6  = SHA512 (Linux with GlibC 2.7+) 

Fixing CIFS ACLs On a DataDomain

On my current project, our customer makes use of DataDomain storage to provide nearline backups of system data. Backups to these devices is primarily done through tools like Veeam (for ESX-hosted virtual machines and associated application data) and NetBackup.

However, a small percentage of the tenants my customer hosts are "self backup" tenants. In this case, "self backup" means that, rather than leveraging the enterprise backup frameworks, the tenants are given an NFS or CIFS share directly off of the DataDomain that is: A) closest to their system(s); and, B) has the most free space to accommodate their data.

"Self backup" tenants that use NFS shares are minimally problematic. Most of the backup problems come from the fact that DataDomains weren't really designed for multi-tenancy. Things like quota controls are fairly lacking. So, it's possible for a tennants of a shared DataDomain to screw each other over by either soaking up all of the device's bandwidth or soaking up all the space.

Still, those problems aside, providing NFS service to tenants is fairly straight-forward. You create a directory, you go into the NFS share-export interface, create the share and access controls and call it a day. CIFS shares, on the other hand...

While we'd assumed that providing CIFS service would be on a par to providing NFS service, it's proven to be otherwise. While the DataDomains provide an NTFS-style ACL capability in their filesystem, it hasn't proven to work quite as one might expect.

The interface for creating shares allows you to set share-level access controls based on both calling-host as well as assigned users and/or groups. One would reasonably assume that this would mean that the correct way to set up a share is to export it with appropriate client-allow lists and user/group-allow lists and that the shares would be set with appropriate filesystem permissions automagically. This isn't exactly how it's turned out to work.

What we've discovered is that you pretty much have to set the shares up as being universally accessible from all CIFS clients and that you grant global "full control" access to the top-level share-folder. Normally, this would be a nightmare, but, once created, you can lock the shares down. You just have to manage the NTFS attributes from a Windows-based host. Basically, you create the share, present it to a Windows-based administrative host, then use the Windows folder security tools to modify the permissions on the share (e.g., remove all the "Everyone" rights, then manually assign appropriate appropriate ownerships and posix gropus to the folder and set up the correct DACLs.

From an engineering perspective, it means that you have to document the hell out of things and try your best to train the ops folks on how to do things The Right Way™. Then, with frequent turnovers in Operations and other "shit happens" kind of things, you have to go back and periodically audit configurations for correctness and repair the brokenness that has crept in.

Unfortunately, one of the biggest sources of brokenness that creeps in is broken permissions structures. When doing the initial folder-setup, it's absolutely critical that the person setting up the folder remembers to click the "Replace all child object permissions with inheritable permissions from this object" checkbox (accessed by clicking on the "Change Permissions" button within the "Advanced Security Settings" section for the folder). Failure to do so makes it so that each folder, subfolder and file created (by tenants) in the share have their own, tenant-created permissions structures. What this results in is a share whose permissions are not easily maintainable by the array-operators. Ultimately, it results in trouble tickets opened by tenants whose applications and/or operational folks eventually break access for themselves

Once those tickets come in, there's not much that can be easily done if the person who "owns" the share has left the organization. If you find yourself needing to fix such a situation, you need to either involve DataDomain's support staff to fix it (assuming your environment is reachable via an WebEx-type of support session) or get someone to slip you instructions on how to access the array's "Engineering Mode"

There's actually two engineering modes: there's the regular SE shell and the BASH shell. The SE shell is basically a super-set of the regular system administration CLI. The BASH shell is basically a Linux BASH shell with DataDomain-specific management commands enabled. For the most part, the two modes are interchangable. However, if you need the ability to do mass modifications or script on your array, you'll need to access the DataDomain's BASH shell mode to do it. See my prior article on accessing the DataDomain's BASH shell mode.

Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed. In addition to the standard Linux commands will be some DataDomain-specific commands. For the purposes of fixing your NTFS ACL mess, you'll be wanting to use the "dd_xcacls" command:

• Use "dd_xcacls -O '[DomainObject]' [Object]" to set the Ownership of an object. For example, to set the ownership attribute to your AD domain account, issue the command "dd_xcacls -O 'MDOMAIN\MYUSER' /data/col1/backup/ShareName".
• Use "dd_xcacls -G '[DomainObject]' [Object]" to set the POSIX group of an object.  For example, to set the POSIX group attribute to your AD domain group, issue the command "dd_xcacls -O 'MDOMAIN\MYUSER' /data/col1/backup/ShareName".
• Use "dd_xcacls -D '[ActiveDirectorySID]:[Setting]/[ScopeMask]/[RightsList]' [OBJECT]" to set the POSIX group of an object. For example, to give "Full Control" rights to your domain account, issue the command "dd_xcacls -D 'MDOMAIN\MYUSER:ALLOW/4/FullControl' /data/col1/backup/ShareName".

A couple of notes apply to the immediately preceding:

1. While the "dd_xcacls" command can notionally set rights-inheritance, I've discovered that this isn't 100% reliable in the DDOS 5.1.x family. It will likely be necessary that once you've placed the desired DACLs on the filesystem objects, you'll need to use a Windows system to set/force inheritance onto objects lower in the filesystem hierarchy.
2. When you set a DACL with "dd_xcacls -D", it replaces whatever DACLS are in place. Any permissions previously on the filesystem object will be removed. If you want more than one user/group DACL applied to the filesystem-object, you need to apply them all at once. Use the ";" token to separate DACLs within the quoted text-argument to the "-D" flag

Because you'll need to fix all of your permissions, one at a time, from this mode, you'll want to use the Linux `find` command to power your use of the  "dd_xcacls" command. On a normal Linux system, when dealing with filesystems that have spaces in directory or file object-names, you'd do something like `find [DIRECTORY] -print0 | xargs -0 [ACTION]` to more efficiently handle this. However, that doesn't seem to work exactly like on a generic Linux system, at least not on the DDOS 5.x systems I've used. Instead, you'll need to use a `find [Directory] -exec [dd_xcacls command-string] {} \;`. This is very slow and resource intensive. On a directory structure with thousands of files, this can take hours to run. Further, because of how resource-intensive using this method is, you won't be able to run more than one such job at a time. Attempting to do so will result in SEGFAULTs - and the more you attempt to run concurrently, the more frequent the SEGFAULTs will be. These SEGFAULTs will cause individual "dd_xcacls" iterations to fail, potentially leaving random filesystem objects permissions unmodified.

DataDomain Bash Shell (or: So You Want to Wreck Your DataDomain)

When dealing with the care and feeding of DataDomain arrays, there are occasions where it helps to know how to access the array's "Engineering Mode". In actuality, there are two levels of engineering mode for DataDomains:

• SE Shell: the SE (system engineer) shell mode is a superset of the normal system administration shell. It includes all of the management commands of the normal administration shell plus some powerful utilities for doing lower-level maintenance tasks on your DataDomain. These include things like fixing ACLs on your CIFS shares, changing networking settings (e.g., timeouts related to OST sessions) and other knobs that are nice to be able to twizzle
• BASH Shells: While the SE shell mode gives you more utilities for managing the array, they're still wrapped in the overall DDOS command-shell construct. The BASH shell mode is pretty much just like a normal root shell on a Linux system: you're able to script tasks in it, use tools like `find`, etc. Take all the damage you can do in the SE mode and add on the capability of doing those tasks on a massive, automated scale.

While enabling SE mode can likened to enabling you to shoot your foot off with a .22, the BASH mode could be likened to enabling you to shoot your foot off with a howitzer. Where SE mode is merely dangerous, I can't really begin to characterize the level of risk you expose yourself to when you start taking full advantage of the DataDomain's BASH shell.

Since accessing either of these modes isn't well-documented (though there's a decent number of Google searches that will turn up the basic "SE" mode) and I use this site as a personal-reminder on how to do things. I'm going to put the procedures here.

Please note: use of engineering mode allows you to do major amounts of damage to your data with a frightening degree of ease and rapidity. Don't try to access engineering mode unless you're fully prepared to have to re-install your DataDomain - inclusive of destroying what's left of the data on it.

Accessing SE Mode:

1. SSH to the DataDomain.
2. Login with an account that has system administrator privileges (this may be one of the default accounts your array was installed with, a local account you've set up for the purpose or an Active Directory managed account that has been placed into a Active Directory security-group that has been granted the system administrator role on the DataDomain
3. Get the array's serial number. The easiest way to do this is type `system show serialno` at the default command prompt
4. Access SE mode by typing `priv set se`. You will be prompted for a password - the password is the serial number from the prior step.

At this point, your command prompt will change to "SE@<ARRAYNAME>" where "<ARRAYNAME>" will be the nodename of your DataDomain. While in this mode, an additional command-set will be enabled. These commands are accessed by typing "se". You can get a further listing of the "se" sub-commands in much the same way you can get help at the normal system administration shell (in this particular case: by typing "se ?").


Accessing the SE BASH Shell:
Once you're in SE mode, the following command-sequence will allow you to access the engineering mode's BASH shell:

1. Type "fi st"
2. Type "df"
3. Type <CTRL>-C three times
4. Type "shell-escape"

At this point, a warning banner will come up to remind you of the jeopardy you've put your configuration in. The prompt will also change to include a warning. This is DataDomain's way of reminding you, at every step, the danger of the access-level you've entered.

Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed.

In addition to the standard Linux commands will be some DataDomain-specific commands. These are the commands that are accessible from the "se" command and its subcommands. The primary use-case for exercising these commands in BASH mode is that the BASH mode is pretty much as fully-scriptable as a root prompt on a normal Linux host. In other words, take all the danger and power of SE mode and wrap it in the sweaty-dynamite of an automated script (you can do a lot of modifications/damage by horsing the se sub-commands to a BASH `find` command or script).

Posterous Alternatives?

So, Posterous is shutting down at the end of the month. It was a great meta-poster. Anyone know any suitable alternatives. Really liked having it for replicating my technical ramblings across multiple sites (really helpful when you work at multiple job-sites, each with their own web-filter rule-sets).

Looking for any recommendations - maybe even paid-for services.

The "Which LUN Is That" Game

One of the fun things about SAN storage has to do with its tendency to be multipathed. Different operating systems handle this differently - but most of the ones I've worked with tend to do what I refer to as "ghosting".

"Ghosting" is, by no means, a technical term. It's mostly meant to convey that, when an OS sees a device down multiple paths, it sees the device multiple times. While you may only have one actual chunk of storage presented from your SAN to your host, you might see that chunk 2, 3, 4 or more times in your storage listings. I call these "extra" presences in the output "ghosts".

At any rate, one of the joys associated with ghosting is determining, "am I seeing all of the LUNs I expect to see and am I seeing each of them an appropriate number of times. If you're only presenting one chunk of storage down four paths and you see four copies of the LUN show up, it's easy enough to say, "yep: things look right". It's even easy(ish) when you have multiple chunks of different-sized chunks of storage to determine if things are right: 1) you have PATHsxLUNs number of storage chunks visible; 2) each set of chunks is an identifiable size. However, if you're presenting multiple chunks that are the same size or multiple chunks with differing levels of pathing-redundancy, things get a bit trickier.

One of the things I'd sorta liked about Solaris's STMS drivers were, once you'd activated the software for a given LUN, the ghosts disappeared into a single meta-device. For better or worse, not everyone used STMS - particularly not on older Solaris releases or configurations that used things like EMC's PowerPath or VERITAS's DMP software. PowerPath actually kinda made the problem worse, as, in addition to the normal ghosts, you added a PowerPath meta-device for each group of LUN-paths. This made the output from the `format` command even longer.

All of that aside, how do you easily identify which disks are ghosts of each other and which ones are completely separate LUNS? The most reliable method I've found is looking for the LUNs' serial numbers. If you have eight storge chunks visible, four of which have one serial number and four of which have a different serial number, you know that you've got two LUNs presented to your host and that each is visible down four paths. But how do you get the serial numbers?

Disks' and LUNs' serial numbers are generally found in their SCSI inquiry responses in what's referred to as "code page 83". How you get to that information is highly OS dependent.

• On Solaris - at least prior to Solaris 10 - you didn't generally have good utilities for pulling serial numbers from LUNs. If you wanted to pull that info, you'd have to fire up the `format` utility in "expert" mode, issue the command-sequence "scsi → inquire". By default, this dumps out code page 83 as part of the response. It dumps this info in two parts: a big, multi-line block of hex-codes and a smaller multi-line block of ASCII text. Your disk/LUN serial number is found by ignorg the big, multiline block of hex values and looking at the third line from the smaller ASCII block.
• On Linux, they provide you a nice little tool that allows you to directly dump out the target SCSI inquiry code-page. Its default behavior is pretty much to dump just the serial number (actually, the serial number is embedded in a longer string, but if you send that string over to your SAN guys, they'll generally recognize the relevant substring and match it up to what they've presented to your host). The way you dump out that string is to use the command `scsi_id -ugs /block/sdX` (where "sdX" is something like "sda", "sdh", etc.).

At any rate, multi-pathing software and associated utilities aside, once you've determined which serial numbers correspond to which disk device-nodes, it because a trivial exercise to determine "am I seeing all of the LUNs I expect to see" and "are my LUNs presented down the expected number of SAN-fabric paths".

Note: if you're running Solaris 10 or an earlier Solaris release with appropriate storage device management packages installed, you may have access to tools like `prtpicl`, `luxadm` and `fcinfo` with which to pull similarly-useful pathing information.

LVM Online Relayout

Prior to coming to the Linux world, most of my complex, software-based storage taskings were performed under the Veritas Storage Foundation framework. In recent years, working primarily in virtualized environments, most storage tasks are done "behind the scenes" - either at the storage array level or within the context of VMware. Up until today, I had no cause to worry about converting filesystems from using one underlying RAID-type to another.

Today, someone wanted to know, "how do I convert from a three-disk RAID-0 set to a six-disk RAID-10 set". Under Storage Foundation, this is just an online relayout operation - converting from a simple volume to a layered volume. Until I dug into it, I wasn't aware that LVM was capable of layered volumes, letalone online-conversion from one volume-type to another.

At first, I thought I was going to have to tell the person (since Storage Foundation wasn't an option for them), "create your RAID-0 sets with `mdadm` and then layer RAID-1 on top of those MD-sets with LVM". Turns out, you can do it in LVM (I spun up a VM in our lab and worked through it).

Basically the procedure assumes that you'd previously:

1. Attached your first set of disks/LUNs to your host
2. Used the usual LVM tools to create your volumegroup and LVM objects (in my testing scenario, I set up a three-disk RAID-0 with a 64KB stripe-width)
3. Created and mounted your filesystem.
4. Gone about your business.

Having done the above, your underlying LVM configuration will look something like:

# vgdisplay AppVG
  --- Volume group ---
  VG Name               AppVG
  System ID             
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               444.00 MB
  PE Size               4.00 MB
  Total PE              111
  Alloc PE / Size       102 / 408.00 MB
  Free  PE / Size       9 / 36.00 MB
  VG UUID               raOK8i-b0r5-zlcG-TEqE-uCcl-VM3L-RelQgX
# lvdisplay /dev/AppVG/AppVol 
  --- Logical volume ---
  LV Name                /dev/AppVG/AppVol
  VG Name                AppVG
  LV UUID                6QuQSv-rklG-pPv6-Tq6I-TuI0-N50T-UdQ4lu
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                408.00 MB
  Current LE             102
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:7

Take special note that there are free PEs available in the volumegroup. In order for the eventual relayout to work, you have to leave space in the volume group for LVM to do its reorganizing magic. I've found that a 10% set-aside has been safe in testing scenarios - possibly even overly generous. In a large, production configuration, that set-aside may not be enough.

When you're ready to do the conversion from RAID, add second set of identically-sized disks to the system. Format the new devices and use `vgextend` to add the new disks to the volumegroup.

Note: Realistically, so long as you increase the number of available blocks in the volumegroup by at least 100%, it likely doesn't matter whether you add the same number/composition of disks to the volumegroup. Differences in mirror compositions will mostly be a performance rather than an allowed-configuration issue.

Once the volumegroup has been sufficiently-grown, use the command `lvconvert -m 1 /dev/<VolGroupName>/<VolName>` to change the RAID-0 set to a RAID-10 set. The `lvconvert` works with the filesystem mounted and in operation - technically, there's no requirement to take an outage window to do the operation. As the `lvconvert` runs, it will generate progress information similar to the following:

AppVG/AppVol: Converted: 0.0%
AppVG/AppVol: Converted: 55.9%
AppVG/AppVol: Converted: 100.0%

Larger volumes will take a longer period of time to convert (activity on the volume will also increase the time required for conversion). Output is generated at regular intervals. The longer the operation takes, the more lines of status output that will be generated.

Once the conversion has completed, you can verify that your RAID-0 set is now a RAID-10 set with the `lvdisplay` tool:

lvdisplay /dev/AppVG/AppVol 
  --- Logical volume ---
  LV Name                /dev/AppVG/AppVol
  VG Name                AppVG
  LV UUID                6QuQSv-rklG-pPv6-Tq6I-TuI0-N50T-UdQ4lu
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                408.00 MB
  Current LE             102
  Mirrored volumes       2
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:7

The addition of the "Mirrored Volumes" line indicates that the logical volume is now a mirrored RAID-set.

No-Post Reboots

One of the things that's always sorta bugged me about Linux was reboots generally required a full reset of the system. That is, if you did an `init 6`, the default bahavior cause your system to drop back down to the boot PROM and go through its POST routines. While on a virtualized system, this is mostly an inconvenience as virtual hardware POST is fairly speedy. However, when you're running on physical hardware, it can be a genuine hardship (the HP-based servers I typically work with can take upwards of ten to fifteen minutes to run its POST routines).
At any rate, a few months back I was just dinking around online and found a nifty method for doing a quick, no-BIOS reboot:

# BOOTOPTS=`cat /proc/cmdline` ; KERNEL=`uname -r` ; \
kexec -l /boot/vmlinuz-$KERNEL --initrd=/boot/initrd-"{$KERNEL}".img \
--append=${BOOTOPTS} ; reboot

Basically, the above:

1. Reads your /proc/cmdline file to get the boot arguments of your most recent bootup and stuffs the value into the BOOTOPTS environmental
2. Grabs your system's currently running kernel release (in case you've got multiple kernels installed and want to boot back into the current one) and stuffs the value into the KERNEL environmental
3. Calls the `kexec` command (a nifty utility for directly booting into a new kernel), leveraging the previously set environmentals to tell `kexec` what to do
4. Finishes with a reboot to the `kexec`-defined kernel (and options)

In testing it on a physical server that normally takes about 15 minutes to reboot (10+ minutes of POST routines) and speeded it up by over 66%. On a VM, it only saves maybe a minute (though that will depend on your VM's configuration settings).

Why So Big

Recently, while working on getting a software suite ready for deployment, I had to find space in our certification testing environment (where our security guys scan hosts/apps and decide what needs to be fixed for them to be safe to deploy). Our CTA environment is unfortunately-tight on resources. The particular app I have to get certified wants 16GB or RAM to run in but will accept as little as 12GB (less than that and the installer utility aborts).

When I went to submit my server (VM, actually) requirements to our CTA team so they could prep me an appropriate install host, they freaked. "Why does it take so much memory" was the cry. So, I dug through the application stack.

The application includes an embedded Oracle instance that wants to reserve about 9GB for its SGA and other set-asides. It's going on a 64bit RedHat server and RedHat typically wants 1GB of memory to function acceptably (can go down to half that, but you won't normally be terribly happy). That accounted for 10GB of the 12GB minimum the vendor was recommending.

Unfortunately, the non-Oracle components of the application stack didn't seem to have a single file that described memory set asides. It looked like it was spinning up two Java processes with an aggregate heap size of about 1GB.

Added to the prior totals, the aggregated heap sizes put me at about 11GB of the vendor-specified 12GB. That still left an unaccounted for 1GB. Now, it could have been the vendor was requesting 12GB because it was a "nice round number" or they could have been adding some slop to their equations to give the app a bit more wiggle-room. 

I could have left it there, but decided, "well, the stack is running, lets see how much it really uses". So, I fired up top. Noticed that the Oracle DB ran under one userid and that the rest of the app-stack ran under a different one. I set top to look only at the userid used by the rest of the app-stack. The output was too long to fit on one screen and I was too lazy to want to add up the RSS numbers, myself. Figured since top wasn't a good avenue, I might be able to use ps (since the command supports the Berkeley-style output options).

Time to hit the man pages...

After digging through the man pages and a bit of cheating (Google is your friend) I found the invocation of ps that I wanted:

`ps -u <appuser> -U <appuser> -orss=`.

Horse that to a nice `awk '{ sum += 1 } END { print sum}' and I had a quick method of divining how much resident memory the application was actually eating up. What I found was that the app-stack had 52 processes (!) that had about 1.7GB of resident memory tied up. Mystery solved.

Why Google's Two-Factor Authentication Is Junk

To be fair, I understand the goals that Google was trying to achieve. And, they're starting down a good path. However, there are some serious flaws (as I see it) with how they've decided to treat services that don't support two-factor authentication.

• Google advertises that you can set per-service passwords for each application. That is to say, if you use third-party mail clients such as Thunderbird, third-party calendaring clients such as Lightning, and third-party chat clients such as Trillian, you can set up a password for each service. Conceivably, one could set one password for IMAP, one password for SMTP, one password for iCAL and yet another password for GoogleTalk. However, Google doesn't actually sandbox the passwords. By "sandbox" I mean restrict a given password to a specific protocol.  If I generate four passwords with the intention of using each password once for each service - as Google's per-application passwords would logically be inferred to work - one actually weakens the security of the associated services. Instead of each service having its own password, each of the four, generated passwords can be used with any of the four targeted services. Instead of having one guessable password, there are now four guessable passwords.
• Google's "per-application" passwords do not allow you to set your password strings. You have to use their password generator. While I can give Google credit for generating 16-character passwords, the strength of the generated passwords is abysmally low. Google's generated passwords are comprised solely of lower case characters. When you go to a "how strong is my password site", Google's generated passwords are ridiculously easy. The Google password is rated at "very weak" - a massive cracking array would take 14 years to break it. By contrast, the password I used on my work systems, last December, is estimated take the best part of 16,000 centuries. For the record, my work password from last year is two characters shorter than the ones Google generates.

So, what you end up with is X number of services that are each authenticatable against with X number of incredibly weak passwords.

All in all, I'd have to rate Google's efforts, at this point, pretty damned close to #FAIL: you have all the inconvenience of two-factor authentication and you actually broaden your attack surfaces if you use anything that's not HTTP/HTTPS based.

Resources:

1. GRC Password Cracker Estimator
2. PasswordMeter Password Strength Checker

Finding Patterns

I know that most of my posts are of the nature "if you're trying to accomplish 'X' here's a way that you can do it". Unfortunately, this time, my post is more of a, "I've got yet to be solved problems going on". So, there's no magic bullet fix currently available for those with similar problems who found this article. That said, if you are suffering similar problmes, know that you're not alone. Hopefully that's small consolation and what follows may even help you in investigating your own problem.

Oh: if you have suggestions, I'm all ears. Given the nature of my configuration, there's not much in the way of useful information I've yet found via Google. Any help would be much appreciated...

---

The shop I work for uses Symantec's Veritas NetBackup product to perform backups of physical servers. As part of our effort to make more of the infrastructure tools we use more enterprise-friendly, I opted to leverage NetBackup 7.1's NetBackup Access Control (NBAC) subsystem. On its own, it provides fine-grained rights-delegation and role-based access control. Horse it to Active Directory and you're able to roll out a global backup system with centralized authentication and rights-management. That is, you have all that when things work.

For the past couple months, we've been having issues with one of the dozen NetBackup domains we've deployed into our global enterprise. When I first began trougleshooting the NBAC issues, the authentication and/or authorization failures had always been associated with a corruption of LikeWise's sqlite cachedb files. At the time the issues first cropped up, these corruptions always seemed to coincide with DRS moving the NBU master server from one ESX host to another. It seemed like, when under sufficiently heavy load - the kind of load that would trigger a DRS event - LikeWise didn't respond well to having the system paused and moved. Probably something to do with the sudden apparent time-jump that happens when a VM is paused for the last parts of the DRS action. My "solution" to the problem was to disable automated-relocation for my VM.

This seemed to stabilize things. LikeWise was no longer getting corrupted and seems like I'd been able to stabilize NBAC's authentication and authorization issues. Well, they stabilized for a few weeks.

Unfortunately, the issues have begun to manifest themselves, again, in recent weeks. We've now had enough errors that some patterns are starting to emerge. Basically, it looks like something is horribly bogging the system down around the time that the nbazd crashes are happening. I'd located all the instances of nbazd crashing from its log files ("ACI" events are loged to the /usr/openv/netbackup/logs/nbazd logfiles), and then began to try correlating them with system load shown by the host's sadc collections. I found two things: 1) I probably need to increase my sample frequency - it's currently at the default 10-minute interval - if I want to more-thoroughly pin-down and/or profile the events; 2) when the crashes have happened within a minute or two of an sadc poll, I've found that the corresponding poll was either delayed by a few seconds to a couple minutes or was completely missing. So, something is causing the server to grind to a standstill and nbazd is a casualty of it.

For the sake of thoroughness (and what's likely to have matched on a Google-search and brought you here), what I've found in our logs are messages similar to the following:

/usr/openv/netbackup/logs/nbazd/vxazd.log

07/28/2012 05:11:48 AM VxSS-vxazd ERROR V-18-3004 Error encountered during ACI repository operation.
07/28/2012 05:11:48 AM VxSS-vxazd ERROR V-18-3078 Fatal error encountered. (txn.c:964)
07/28/2012 05:11:48 AM VxSS-vxazd LOG V-18-4204 Server is stopped.
07/30/2012 01:13:31 PM VxSS-vxazd LOG V-18-4201 Server is starting.

/usr/openv/netbackup/logs/nbazd/debug/vxazd_debug.log

07/28/2012 05:11:48 AM Unable to set transaction mode. error = (-1)
07/28/2012 05:11:48 AM SQL error S1000 -- [Sybase][ODBC Driver][SQL Anywhere] Connection was terminated
07/28/2012 05:11:48 AM Database fatal error in transaction, error (-1)
07/30/2012 01:13:31 PM _authd_config.c(205) Conf file path: /usr/openv/netbackup/sec/az/bin/VRTSaz.conf
07/30/2012 01:22:40 PM _authd_config.c(205) Conf file path: /usr/openv/netbackup/sec/az/bin/VRTSaz.conf

Our NBU master servers are hosted on virtual machines. It's a supported configuration and adds a lot of flexibility and resiliency to the overall enterprise-design. It also means that I have some additional metrics available to me to check. Unfortunately, when I checked those metrics, while I saw utilization spikes on the VM, those spikes corresponded to healthy operations of the VM. There weren't any major spikes (or troughs) during the grind-downs. So, to ESX, the VM appeared to be healthy.

At any rate, I've requested our ESX folks see if there might be anything going on on the physical systems hosting my VM that aren't showing up in my VM's individual statistics. I'd previously had to disable automated DRS actions to keep LikeWise from eating itself - those DRS actions wouldn't have been happening had the hosting ESX system not been experiencing loading issues - perhaps whatever was causing those DRS actions is still afflicting this VM.

I've also tagged one of our senior NBU operators to start picking through NBU's logs. I've asked him to look to see if there are any jobs (or combinations of jobs) that are always running during the bog-downs. If it's a scheduling issue (i.e., we're to blame for our problems), we can always reschedule jobs to exert less loading or we can scale up the VM's memory and/or CPU reservations to accommodate such problem jobs.

For now, it's a waiting-game. At least there's an investigation path, now. It's all in finding the patterns.