Wednesday, November 26, 2014

Converting to EL7: Solving The "Your Favorite Service Isn't Systemd-Enabled" Problem

Having finally gotten off my butt to knock out getting my RHCE (for EL6) before the 2014-12-19 drop-dead date, I'm finally ready to start focusing on migrating my personal systems to EL7-based distros.

My personal VPS is currently running CentOS 6.6. I use my VPS to host a couple of personal websites and email for family and a few friends. Yes, I realize that it would probably be easier to offload all of this to providers like Google. However, while Google is very good at SPAM-stomping, and provides me a very generous amount of space for archiving emails, one area that they do lack for is email aliases: whenever I have to register to a new web-site, I use a custom email address to do so. At my last pruning, I still had 300+, per-site, aliases. So, for me, number of available aliases ("unlimited" is best) and ease of creating them trumps all other considerations.

Since I don't have Google handling my mail for me, I have to run my own A/V and anti-spam engines. Being a good Internet Citizen, I also like to make use of Sender Policy Framework (via OpenSPF) and DomainKeys (currently via DKIMproxy).

I'm only just into the process of sorting out what I need to do to make the transition as quick and as painless (more for my family and friends than me) a process as possible. I hate outages. And, with a week off for the Thanskgiving holidays, I've got time to do things in a fairly orderly fashion.

At any rate, one of the things I discovered is that my current DomainKeys solution hasn't been updated to "just work" within the systemd framework used within EL7. This isn't terribly surprising, as it appears that the DKIMproxy SourceForge project may have gone dormant, in 2013 (so, I'll have to see if there's alternatives that have the appearance of still being a going concern - in the mean time...) Fortunately, the DKIMproxy source code does come with a `chkconfig` compatible SysV-init script. Even more fortunately, converting from SysV-init to a systemd-compatible service control is a bit more straight forward than when I was dealing with moving from Solaris 9's legacy init to Solaris 10's SMF.

If you've already got a `chkconfig` style init script, moving to systemd-managed is fairly trivial. Your `chkconfig` script can be copied, pretty much "as is" into "/usr/lib/systemd". My (current) preference is to create a "scripts" subdirectory and put it in there. Haven't read deeply enough into systemd to see if this is the "Best Practices" method, however. Also, where I work has no established conventions ...because they only started migrating to EL6 in fall of 2013 - so, I can't exactly crib anything EL7-related from how we do it at work.

Once you have your SysV-init style script placed where it's going to live (e.g., "/usr/lib/systemd/scripts"), you need to create associated service definition files. In my particular case, I had to create two as the DKIMproxy software actually has an inbound and an outbound funtion. Launched from normal SysV-init, it all gets handled as one piece. However, one of the nice things about systemd is it's not only a launcher framework, it's a service monitor framework, as well. To take full advantage, I wanted one monitor for the inbound service and one for the outbound service. The legacy init script that DKIMproxy ships with makes this easy enough as, in addition to the normal "[start|stop|restart|status]" arguments, it had per-direction subcommand (e.g., "start-in" and "stop-out"). The service-definition for my "dkim-in.service" looks like:
[Unit]
     Description=Manage the inbound DKIM service
     After=postfix.service


     [Service]
     Type=forking
     PIDFile=/usr/local/dkimproxy/var/run/dkimproxy_in.pid
     ExecStart=/usr/lib/systemd/scripts/dkim start-in
     ExecStop=/usr/lib/systemd/scripts/dkim stop-in


     [Install]
     WantedBy=multi-user.target

To break down the above:

  • The "Unit" stanza tells systemd a bit about your new service:
    • The "Description" line is just ASCII text that allows you to provide a short, meaningful of what the service does. You can see your service's description field by typing `systemctl -p Description show <SERVICENAME>`
    • The "After" parameter is a space-separated list of other services that you want to have successfully started before systemd attempts to start your new service. In my case, since DKIMproxy is an extension to Postfix, it doesn't make sense to try to have DKIMproxy running until/unless Postfix is running.
  • The "Service" stanza is where you really define how your service should be managed. This is where you tell systemd how to start, stop, or reload your service and what PID it should look for so it knows that the service is still notionally running. The following parameters are the minimum ones you'd need to get your service working. Other parameters are available to provide additional functionality:
    • The "Type" parameter tells systemd what type of service it's managing. Valid types are: simpleforking, oneshot, dbus, notify or idle. The systemd.service man page more-fully defines what each option is best used for. However, for a traditional daemonized service, you're most likely to want "forking".
    • The "PIDFile" parameter tells systemd where to find a file containing the parent PID for your service. It will then use this to do a basic check to monitor whether your service is still running (note that this only checks for presence, not actual functionality).
    • The "ExecStart" parameter tells systemd how to start your service. In the case of a SysV-init script, you point it to the fully-qualified path you installed your script to and then any arguments necessary to make that script act as a service-starter. If you don't have a single, chkconfig-style script that handles both stop and start functions, you'd simply give the path to whatever starts your service. Notice that there are no quotations surrounding the parameter's value-section. If you put quotes - in the mistaken belief that the starter-command and it's argument need to be grouped, you'll get a path error when you go to start your service the first time.
    • The "ExecStop" parameter tells systemd how to stop your service. As with the "ExecStart" parameter, if you're leveraging a fully-featured SysV-init script, you point it to the fully-qualified path you installed your script to and then any arguments necessary to make that script act as a service-stopper. Also, the same rules about white-space and quotation-marks apply to the "ExecStop" parameter as do the "ExecStart" parameter.
  • The "Install" stanza is where you tell systemd the main part of the service dependency-tree to put your service. You have two main dependency-specifiers to choose: "WantedBy" and "RequiredBy". The former is a soft-dependency while the latter is a hard-dependency. If you use the "RequiredBy" parameter, then the service unit-group (e.g., "mult-user.target") enumerated with the "RequiredBy" parameter will only be considered to have successfully onlined if the defined service has successfully launched and stayed running.  If you use the "WantedBy" parameter, then the service unit-group (e.g., "mult-user.target") enumerated with the "WantedBy" parameter will still be considered to have successfully onlined whether the defined service has successfully launched or stayed running. It's most likely you'll want to use "WantedBy" rather than "RequiredBy" as you typically won't want systemd to back off the entire unit-group just because your service failed to start or stay running (e.g., you don't want to stop all of the multi-user mode related processes just because one network service has failed.)

Tuesday, June 17, 2014

UDEV Friendly-Names to Support ASM Under VMware-hosted Linux Guest

This past month or so, we've been setting up a new vSphere hosting environment for a new customer. Our first guinnea-pig tenant is being brought into the virtualized hosting-environment. This first tenant has a mix of Windows and Linux systems running a multi-layer data-processing system based on a back-end Oracle database.

As part of our tenancy, process, we'd come up with a standard build request form. In general, we prefer a model that separates application data from OS data. In addition to the usual "how much RAM and CPU do you need" information, the form includes configuration-capture items for storage for applications hosted on the VMs.  The table has inputs for both the requested supplemental storage sizes and where/how to mount those chunks.

This first tenant simply filled in a sum of their total additional storage request with no indication as to how they expected to use it. After several iterations of "if you have specific requirements, we need you to detail them" emails, I sent a final "absent the requested configuration specifications, the storage will be added but left unconfigured". It was finally at this point that the tenant responded back saying "there's a setup guide at this URL - please read that and configure accordingly".

Normally, this is not how we do things. The solution we offer tends to be more of a extended IaaS model: in addition to providing a VM container, we provide a basic, hardened OS configuration (installing an OS and patching it to a target-state) configure basic networking and name-resolution and perform basic storage configuration tasks.

This first tenant was coming from a physical Windows/Red Hat environment and were testing the waters of virtualization. As a result, most of their configuration expectations were based on physical servers (SAN based storage with native multipathing-support). The reference documents they pointed us to were designed for implementing Oracle on a physical system using ASM on top of Linux dm-multipath storage objects ...not something normally done within an ESX-hosted Red Hat Linux configuration.

We weren't going to layer-on dm-multipath support, but the tenant still had the expectation of using "friendly" storage object names for ASM. The easy "friendly" storage object name path is to use LVM. However, Oracle generally recommends against using ASM in conjunction with third-party logical volume management systems. So, LVM was off the table. How best to give the desired storage configs?

I opted to let udev do the work for me. Unfortunately, because we weren't anticipating this particular requirements-set, the VM templates we'd created didn't have some of the hooks available that would allow udev to do its thing. Specifically, no UUIDs were being presented into the Linux guests. Further complicating things is the fact that, with the hardened Linux build we furnish, most of the udev tools and the various hardware information tools are not present. Down side is that it made things more difficult than they probably absolutely needed to be. The up side is the following procedures should be portable across a fairly wide variety of Linux implementations:
  1. To have VMware provide serial number information - from which UUIDs can be generated by the guest operating system, it's necessary to make a modification to the VM's advance configuration options. Ensure that the “disk.EnabledUUID” has been created for the VM and the value set to “TRUE”. Specific method for doing so varies depending on whether you use the vSphere web UI or the VPX client (or even the vmcli or direct editing of config files) to do your configuration tasks. Google for the specifics of your preferred management method.
  2. If the you had to create/change the value in the prior step, reboot the VM so that the config changes take effect
  3. Present the disks to be used by ASM to the Linux guest – if adding SCSI controllers, this step will need to be done while guest is powered off.
  4. Verify that VM is able to see new VMDKs. If suplemental disk presentation was done while the VM was running, initiate a SCSI-bus rescan (e.g., `echo "- - -" > /sys/class/scsi_host/host1/rescan`)
  5. Lay :down an aligned, full-disk partition with the `parted` utility for each presented VMDK/disk. For example, if one of the newly-presented VMDKs was seen by the Linux OS as /dev/sdb:

    # parted –s /dev/sdb -- mklabel msdos mkpart primary ext3 1024s 100%

    Specifying an explicit starting-block (at 1024 or 2048 blocks) and using the relative ending-location, as above, will help ensure that your partition is created on an even storage-boundary. Google around for discussions on storage alignment and positive impact on virtualization environments for details on why the immediately-prior is usually a Good Thing™.
  6. Ensure that the “options=” line in the “/etc/scsi_id.config” file contains the “-g” option
  7. For each newly-presented disk, execute the command `/sbin/scsi_id -g -s /block/{sd_device}` and capture the output.
  8. Copy each disk’s serial number (obtained in the prior step) is copied into the “/etc/udev.d/rules.d/99-oracle-udev.rules” file
  9. Edit the “/etc/udev.d/rules.d/99-oracle-udev.rules” file, ensuring that each serial number has an entry similar to:

    KERNEL=="sd*",BUS=="scsi",ENV{ID_SERIAL}=="{scsi_id}", NAME="ASM/disk1", OWNER="oracle", GROUP="oinstall", MODE="660"

    The "{scsi_id}" shown above is a variable name: substitute with the values previously captured via the `/sbin/scsi_id` command. The "NAME=" field should be similarly edited to suite and should be unique for each SCSI serial number.

    Note: If attempting to make per disk friendly-names (e.g., “/dev/db1p1”, “/dev/db2p1”, “/dev/frap1”, etc.) it will be necessary to match LUNs by size to appropriate ‘NAME=’ entries

  10. Reboot the system so that the udev service can process the new rule entries
  11. Verify that the desired /dev/ASM/<NAME>” entries exist
  12. Configure storage-consumers (e.g., “ASM”) to reference the aligned udev-defined device-nodes.
If your Linux system has various hardware information tools, udev management interfaces and the sg3tools installed, some tasks for finding information are made much easier and some of the reboot-steps specified in this document become unnecessary.

Thursday, June 12, 2014

Template-Deployed VMs and the "When Was I Built" Problem

For the past number of years, I have been supporting Linux systems hosted within various virtualization environments. Most of these environments have made used of template-based VM deployment.

In a large, dynamic, enterprise-scale environment, the question often comes up, "when was this host built". In such environments, there may be a number of methods to derive such information - hypervisor management server logs, service-automation engine logs, etc. However, such data can also be somewhat ephemeral due to things as small as log-truncation up through replacement of service-automation and configuration-management tools/fraemeworks.

Fortunately, the Enterprise Linux family of Linux distributions (Red Hat, CentOS Scientific Linux, etc.), offers a fairly stable method for determining when a system was first provisioned. Whenever you first build an ELx-based system, one of the files that gets installed - and then never gets updated - is the "basesystem" RPM. So, if you look at the install date for this RPM (and the system time was correctly-set at its installation time), you will have an accurate representation of when the system was built.

That said, it had previously-occurred to me (a while ago, actually) that the “deploy from template” method of building Linux VMs precludes using the rpm database from determining system build time. Unlike with a KickStarted system - where you can always run `rpm -q --qf '%{installtime:date}\n' basesystem` and it will give you the install-date for the system - doing so on a template-built system will mislead you. When deployed from a template, that method returns when the template VM was built, not when the running VM was deployed from that template.

This had been bugging me for several years now. I'd even posed the question of how to solve it on a few forums to no avail (a number of respondents hadn't been aware of the "use basesystem to show my system install-date" trick so hadn't investigated how to solve a problem they didn't know existed). One day, while I was at our engineering lab and was waiting for some other automated tasks to run, I had one of those "I wonder if this will work" moments that allowd me to finally figure out how to “massage” the RPM database so that the basesystem RPM can reflect a newer install date:

# rpm -q --qf '%{installtime:date}\n' basesystem
Tue 12 Jul 2011 11:24:06 AM EDT
# rpm -i --force --justdb basesystem-10.0-4.el6.noarch.rpm
# rpm -q --qf '%{installtime:date}\n' basesystem
Wed 11 Jun 2014 09:21:13 PM EDT

Thus, if you drop something similar to the above into your VM's system prep/cloudinit/etc. scripts, your resultant VM will have its instantiation-date captured and not just its template-build date.

Monday, October 21, 2013

Visualizing Deduplication

The enterprise I work for makes heavy use of deduplicating storage arrays for storing backup data. If you're frequently backing up the same data over and over again (e.g., multiple backup clients all running the same OS, weekly full backups of low change-rate application data-sets, etc.), deduplication can save you a ton of space.

In general, most people in the organization don't really bother to think how things work. It's just magic on the backend that allows their data to be backed up. They don't really grok that the hundreds of TiB of blocks they've pushed at the array actually only took up a few tens of TiB. Unfortunately, like any storage device, deduplicating storage arrays can run out of space. When they do, people's lack of understanding of what goes on under the covers really comes to the fore. You'll hear plaintive cries of "but I deleted 200TiB - what do you mean there's only 800GiB free, now??"

Given our data storage tendencies, if we realize a 10% return of the amount deleted at the backup server, it's a good day. When we get more than that, it's gravy. Trying to verbally-explain this to folks who don't understand what's going on at the arrays, is mostly a vain effort.

To try to make things a bit more understandable, I came up with three simplified illustrations of array utilization patterns with respect to backup images and the nature of the application's data change patterns. Each of the illustrations should be interpreted to represent data sets that are backed up. Each box represents a same-sized chunk of data (e.g., 200GiB). Each letter represents a unique segment of data within a larger data-set. Each row represents a backup job - with time being represented oldest to most recent ordered top to bottom. Each column represents a copy of a unique chunk of data.

All of what follows is massive over-simplification and ignores the effects of file-compression and other factors that can contribute to space-used on or reclaimed from an array.

100% Data Change Rate:
100% Change-rate (full backup to full backup)
The above diagram attempts to illustrate a data set that changes 100% - with no growth in the data set-size - between full backups. A given full backup is represented by five lettered-boxes on a single line. Incremental backups are represented by single lettered-boxes on a single line.

The application's full data-set is represented by the five boxes labeled "A" through "E". Changes to a given segment are signified by using the prime symbol. That is, "A'" indicates that data-segment "A" has experienced a change to every block of its contents - presumably sufficient change to prevent the storage array from being able to deduplicate that data either against the original data-segment "A" or any of the other lettered data-segments.

This diagram illustrates that, between the time the first and the last full backup has occurred, all of the data-segments have changed. Further, the changes happened such that one (and only one) full data-segment changed before a given incremental backup job was run.

Given these assumptions about the characteristics of the backup data:

  • if one were to erase only the original full backup from the the backup server, up to 100% of the blocks consumed at the array might be returned to the pool of available blocks.
  • If one were to erase the original full backup and any (up to all) of the incrementals, the only blocks that would be returned to the pool of available blocks would be the ones associated with the initial full backup. None of the blocks associated with the incremental backups would be returned. Thus, the greater the number of incrementals deleted, the lower the overall yield of blocks returned to the available blocks pool would be. Thus, if all of the backup images - except for the most recent full backup - were deleted, the maximum return-yield might be as high as 50% of the blocks erased at the backup host.

This would be a scenario where you might hear the "but I deleted xGiB of data and you're telling me only half that got freed up??"

100% Growth Rate:
The above diagram attempts to illustrate a data set that grows by 100% between full backups - growing by 20% of the original set-size between each backup window. A given full backup is represented by five or more lettered-boxes on a single line. Incremental backups are represented by single lettered-boxes on a single line.

The above is kind of ugly. Assuming no deduplicability between data segments, each backup job will consume additional space on the storage array. Assuming 200GiB segments, the first full backup would take up 1TiB of space on the array. Each incremental would consume a further 200GiB. By the time the last incremental has run, 2TiB of space will be consumed in the array.

That ugliness aside, when the second full backup is run, almost no additional disk space will be consumed: the full backups segments would be duplicates of all prior jobs.

However, because the second full backup duplicates all prior data, erasing the first full backup and/or any (or all) of the incrementals would result in a 0% yield of the deleted blocks being returned to the available blocks pool. In this scenario, the person deleting data on their backup server will be incredulous when you tell them "that 2TiB of image data you deleted freed up no additional space on the array". Unfortunately, they won't really care that the 4TiB of blocks that they'd sent to the array only actually consumed 2TiB of space on the array.

20% Data Change Rate:
The above diagram attempts to illustrate a data set that changes continuously changes 20% of the data set - with no growth in the data set-size - between full backups. Further, only one chunk of data is changing - 80% of the original data-set remains completely unchanged between backups. A given full backup is represented by five lettered-boxes on a single line. Incremental backups are represented by single lettered-boxes on a single line.

The application's full data-set is represented by the five boxes labeled "A" through "E". Changes to a given segment are signified by using the prime symbol. That is, "A'" indicates that data-segment "A" has experienced a change to every block of its contents - presumably sufficient change to prevent the storage array from being able to deduplicate that data either against the original data-segment "A" or any of the other lettered data-segments.

Assuming 200GiB data-segments, the first full backup plus the incrementals would have resulted in 2TiB having been sent to the array. The total space consumed on the array would be similar.

If the first full backup is erased, of the 1TiB erased, up to 200GiB would be returned to the free blocks pool. This is because the first full backup and the second full backup overlap by 800GiB worth of deduplicated (non-unique) data.

In the above image, each of the first four incrementals has unique data not found in the second full backup. The fifth incremental's data is incorporated into the last full backup. Thus, deleting each of the first four incrementals may return up to 200GiB of blocks to the free blocks pool. Deletion of the last incremental will result in no blocks being returned to the free blocks pool. By extension, if the first full backup plus all of the incrementals are deleted, though 2TiB of data would be deleted at the backup server, only 1TiB would be returned to the free block pool.


Each of the above illustrations aside, unless you're undertaking the very expensive practice of allocating one deduplication pool per application/data-set, the deletion-yields tend to become even more apalling. The compression and deduplication algorithms in your array may have found storage optimizations across data sets. End result will be that your deduplication ratios will go down, but your actual blocks consumed on the array won't go down nearly as quickly as the folks deleting data might hope.

Similarly poor deletion-yields will occur when your backup system is keeping more than just two full backups. This is because the amount of aggregate overlaps across backup sets will be greater as the number of retained full backups increases.

Wednesday, September 25, 2013

The Case of the Broken ARP (In Progress)

Recently, I was tasked with a project that, as part of the preparation-phase, required patching up a whole bunch of servers to the same patch-level. In total, I patched about 20 systems that were dual-homed and equipped with asymmetrical, 10/1 active/passive bonds. All but the last system went aces. The last system ...was weird.

After patching the final system, my secondary network-pair was no longer able to talk on the network. While diagnosing, attempts to ping out to any hosts on the local LAN segment resulted in "host unreachable" errors. Any hosts that I did try to ping, ended up in the afflicted host's ARP table with a missing ("<incomplete>") MAC address entry.

Our builds are normally fairly locked down. That means, many troubleshooting tools (such as tcpdump) are not loaded. At my last straw, I opted to temporarily load tcpdump to see what, if anything, the afflicted bond (and sub-interfaces) were seeing on the network. Interestingly, as soon as I started snooping either the parent bond or the active interface, networking activity became "normal" and the previously "<incomplete>" ARP table entries populated the other hosts' MAC addresses. As soon as I stopped my tcpdump runs, networking reverted to its broken state and the other hosts' MAC address entries in the ARP table returned to "<incomplete>".

Still don't have a fix - this is a "work in progress" article, at the moment.What I ended up doing as a workaround - since this server is a critical infrastructure component - is did a `ifconfig bond1 promisc` and updated its /etc/sysconfig/network-scripts file to preserve the state should the system reboot before I find a more suitable fix. So, for right now, in order to get this one bond (of two on the system) to work, I need to leave it in promiscuous mode.

Obviously, I have our networking guys looking at the switches to see if there's a difference between how the ports for bond0 and bond1 are configured. I figure, it has to be the network, since: A) one bond works but the other doesn't; and, B) no promiscuous-mode changes were required for any of the other hosts that were patched.

At any rate, if you happen to stumble on this article before I get it beyond a "work in progress" state, please feel free to comment if you know a likely fix.

Tuesday, September 3, 2013

Password Encryption Methods

As a systems administrator, there are times where you have to find programatic ways to update passwords for accounts. On some operating systems, your user account modification tools don't allow you to easily set passwords in a programmatic fashion. Solaris used to be kind of a pain, in this regard, when it came time to do an operations-wide password reset. Fortunately, Linux is a bit nicer about this.

The Linux `usermod` utility allows you to (relatively easily) specify password in a programmatic fashion. The one "gotcha" of the utility is the requirement to use hashed password-strings rather than cleartext. The question becomes, "how best to generate those hashes".

The answer will likely depend on your security requirements. If MD5 hashes are acceptable, then you can use OpenSSL or the `grub-md5-crypt` utilities to generate them. If, however, your security requirements require SHA256- or even SHA512-based hashes neither of those utilities will work for you.

Newer Linux distributions (e.g. RHEL 6+) essentially replace the `grub-md5-crypt` utility with the `grub-crypt` utility. This utility supports not only the older MD5 that its predecessor supported, but also SHA256 and SHA512.

However, what do you do when `grub-crypt` is missing (e.g., you're running RedHat 5.x) or you just want one method that will work across different Linux versions (e.g., your operations environment consists of a mix of RHEL 5 and RHEL 6 systems)? While you can use a tool like `openssl` to do the dirty work, if your security requirements dictate an SHA-based hashing algorithm, it's not currently up to the task. If you want the SHAs in a cross-distribution manner, you have to leverage more generic tools like Perl or Python.

The following examples will show you how to create a hashed password-string from the cleartext password "Sm4<kT*TheFace". Some tools (e.g., OpenSSL's "passwd" command) allowed you to choose to use a fixed-salt or a random-salt. From the standpoint of being able to tell "did I generate this script", using a fixed-salt can be useful; however, using a random-salt may be marginally more secure. The Perl and Python methods pretty much demand the specification of a salt. In the examples below, the salt I'm using is "Ay4p":
  • Perl (SHA512) Method: `perl -e 'print crypt("Sm4<kT*TheFace", "\$6\$Ay4p\$");'`
  • Python (SHA512) Method: `python -c 'import crypt; print crypt.crypt(""Sm4<kT*TheFace", "$6$Ay4p$")'
Note that you specify the encryption-type used by specifying an numerical representation of the standard encryption-types. The standard encryption types for Linux operating systems (from the crypt() manpage):
  • 1  = MD5
  • 2a = BlowFish (not present in all Linux distributions)
  • 5  = SHA256 (Linux with GlibC 2.7+)
  • 6  = SHA512 (Linux with GlibC 2.7+) 

    Tuesday, May 28, 2013

    Fixing CIFS ACLs On a DataDomain

    On my current project, our customer makes use of DataDomain storage to provide nearline backups of system data. Backups to these devices is primarily done through tools like Veeam (for ESX-hosted virtual machines and associated application data) and NetBackup.

    However, a small percentage of the tenants my customer hosts are "self backup" tenants. In this case, "self backup" means that, rather than leveraging the enterprise backup frameworks, the tenants are given an NFS or CIFS share directly off of the DataDomain that is: A) closest to their system(s); and, B) has the most free space to accommodate their data.

    "Self backup" tenants that use NFS shares are minimally problematic. Most of the backup problems come from the fact that DataDomains weren't really designed for multi-tenancy. Things like quota controls are fairly lacking. So, it's possible for a tennants of a shared DataDomain to screw each other over by either soaking up all of the device's bandwidth or soaking up all the space.

    Still, those problems aside, providing NFS service to tenants is fairly straight-forward. You create a directory, you go into the NFS share-export interface, create the share and access controls and call it a day. CIFS shares, on the other hand...

    While we'd assumed that providing CIFS service would be on a par to providing NFS service, it's proven to be otherwise. While the DataDomains provide an NTFS-style ACL capability in their filesystem, it hasn't proven to work quite as one might expect.

    The interface for creating shares allows you to set share-level access controls based on both calling-host as well as assigned users and/or groups. One would reasonably assume that this would mean that the correct way to set up a share is to export it with appropriate client-allow lists and user/group-allow lists and that the shares would be set with appropriate filesystem permissions automagically. This isn't exactly how it's turned out to work.

    What we've discovered is that you pretty much have to set the shares up as being universally accessible from all CIFS clients and that you grant global "full control" access to the top-level share-folder. Normally, this would be a nightmare, but, once created, you can lock the shares down. You just have to manage the NTFS attributes from a Windows-based host. Basically, you create the share, present it to a Windows-based administrative host, then use the Windows folder security tools to modify the permissions on the share (e.g., remove all the "Everyone" rights, then manually assign appropriate appropriate ownerships and posix gropus to the folder and set up the correct DACLs.

    From an engineering perspective, it means that you have to document the hell out of things and try your best to train the ops folks on how to do things The Right Way™. Then, with frequent turnovers in Operations and other "shit happens" kind of things, you have to go back and periodically audit configurations for correctness and repair the brokenness that has crept in.

    Unfortunately, one of the biggest sources of brokenness that creeps in is broken permissions structures. When doing the initial folder-setup, it's absolutely critical that the person setting up the folder remembers to click the "Replace all child object permissions with inheritable permissions from this object" checkbox (accessed by clicking on the "Change Permissions" button within the "Advanced Security Settings" section for the folder). Failure to do so makes it so that each folder, subfolder and file created (by tenants) in the share have their own, tenant-created permissions structures. What this results in is a share whose permissions are not easily maintainable by the array-operators. Ultimately, it results in trouble tickets opened by tenants whose applications and/or operational folks eventually break access for themselves

    Once those tickets come in, there's not much that can be easily done if the person who "owns" the share has left the organization. If you find yourself needing to fix such a situation, you need to either involve DataDomain's support staff to fix it (assuming your environment is reachable via an WebEx-type of support session) or get someone to slip you instructions on how to access the array's "Engineering Mode"

    There's actually two engineering modes: there's the regular SE shell and the BASH shell. The SE shell is basically a super-set of the regular system administration CLI. The BASH shell is basically a Linux BASH shell with DataDomain-specific management commands enabled. For the most part, the two modes are interchangable. However, if you need the ability to do mass modifications or script on your array, you'll need to access the DataDomain's BASH shell mode to do it. See my prior article on accessing the DataDomain's BASH shell mode.

    Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed. In addition to the standard Linux commands will be some DataDomain-specific commands. For the purposes of fixing your NTFS ACL mess, you'll be wanting to use the "dd_xcacls" command:

    • Use "dd_xcacls -O '[DomainObject]' [Object]" to set the Ownership of an object. For example, to set the ownership attribute to your AD domain account, issue the command "dd_xcacls -O 'MDOMAIN\MYUSER' /data/col1/backup/ShareName".
    • Use "dd_xcacls -G '[DomainObject]' [Object]" to set the POSIX group of an object.  For example, to set the POSIX group attribute to your AD domain group, issue the command "dd_xcacls -O 'MDOMAIN\MYUSER' /data/col1/backup/ShareName".
    • Use "dd_xcacls -D '[ActiveDirectorySID]:[Setting]/[ScopeMask]/[RightsList]' [OBJECT]" to set the POSIX group of an object. For example, to give "Full Control" rights to your domain account, issue the command "dd_xcacls -D 'MDOMAIN\MYUSER:ALLOW/4/FullControl' /data/col1/backup/ShareName".

    A couple of notes apply to the immediately preceding:

    1. While the "dd_xcacls" command can notionally set rights-inheritance, I've discovered that this isn't 100% reliable in the DDOS 5.1.x family. It will likely be necessary that once you've placed the desired DACLs on the filesystem objects, you'll need to use a Windows system to set/force inheritance onto objects lower in the filesystem hierarchy.
    2. When you set a DACL with "dd_xcacls -D", it replaces whatever DACLS are in place. Any permissions previously on the filesystem object will be removed. If you want more than one user/group DACL applied to the filesystem-object, you need to apply them all at once. Use the ";" token to separate DACLs within the quoted text-argument to the "-D" flag

    Because you'll need to fix all of your permissions, one at a time, from this mode, you'll want to use the Linux `find` command to power your use of the  "dd_xcacls" command. On a normal Linux system, when dealing with filesystems that have spaces in directory or file object-names, you'd do something like `find [DIRECTORY] -print0 | xargs -0 [ACTION]` to more efficiently handle this. However, that doesn't seem to work exactly like on a generic Linux system, at least not on the DDOS 5.x systems I've used. Instead, you'll need to use a `find [Directory] -exec [dd_xcacls command-string] {} \;`. This is very slow and resource intensive. On a directory structure with thousands of files, this can take hours to run. Further, because of how resource-intensive using this method is, you won't be able to run more than one such job at a time. Attempting to do so will result in SEGFAULTs - and the more you attempt to run concurrently, the more frequent the SEGFAULTs will be. These SEGFAULTs will cause individual "dd_xcacls" iterations to fail, potentially leaving random filesystem objects permissions unmodified.

    Friday, May 24, 2013

    DataDomain Bash Shell (or: So You Want to Wreck Your DataDomain)

    When dealing with the care and feeding of DataDomain arrays, there are occasions where it helps to know how to access the array's "Engineering Mode". In actuality, there are two levels of engineering mode for DataDomains:

    • SE Shell: the SE (system engineer) shell mode is a superset of the normal system administration shell. It includes all of the management commands of the normal administration shell plus some powerful utilities for doing lower-level maintenance tasks on your DataDomain. These include things like fixing ACLs on your CIFS shares, changing networking settings (e.g., timeouts related to OST sessions) and other knobs that are nice to be able to twizzle
    • BASH Shells: While the SE shell mode gives you more utilities for managing the array, they're still wrapped in the overall DDOS command-shell construct. The BASH shell mode is pretty much just like a normal root shell on a Linux system: you're able to script tasks in it, use tools like `find`, etc. Take all the damage you can do in the SE mode and add on the capability of doing those tasks on a massive, automated scale.
    While enabling SE mode can likened to enabling you to shoot your foot off with a .22, the BASH mode could be likened to enabling you to shoot your foot off with a howitzer. Where SE mode is merely dangerous, I can't really begin to characterize the level of risk you expose yourself to when you start taking full advantage of the DataDomain's BASH shell.

    Since accessing either of these modes isn't well-documented (though there's a decent number of Google searches that will turn up the basic "SE" mode) and I use this site as a personal-reminder on how to do things. I'm going to put the procedures here.

    Please note: use of engineering mode allows you to do major amounts of damage to your data with a frightening degree of ease and rapidity. Don't try to access engineering mode unless you're fully prepared to have to re-install your DataDomain - inclusive of destroying what's left of the data on it.

    Accessing SE Mode:
    1. SSH to the DataDomain.
    2. Login with an account that has system administrator privileges (this may be one of the default accounts your array was installed with, a local account you've set up for the purpose or an Active Directory managed account that has been placed into a Active Directory security-group that has been granted the system administrator role on the DataDomain
    3. Get the array's serial number. The easiest way to do this is type `system show serialno` at the default command prompt
    4. Access SE mode by typing `priv set se`. You will be prompted for a password - the password is the serial number from the prior step.
    At this point, your command prompt will change to "SE@<ARRAYNAME>" where "<ARRAYNAME>" will be the nodename of your DataDomain. While in this mode, an additional command-set will be enabled. These commands are accessed by typing "se". You can get a further listing of the "se" sub-commands in much the same way you can get help at the normal system administration shell (in this particular case: by typing "se ?").


    Accessing the SE BASH Shell:
    Once you're in SE mode, the following command-sequence will allow you to access the engineering mode's BASH shell:

    1. Type "fi st"
    2. Type "df"
    3. Type <CTRL>-C three times
    4. Type "shell-escape"
    At this point, a warning banner will come up to remind you of the jeopardy you've put your configuration in. The prompt will also change to include a warning. This is DataDomain's way of reminding you, at every step, the danger of the access-level you've entered.

    Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed.

    In addition to the standard Linux commands will be some DataDomain-specific commands. These are the commands that are accessible from the "se" command and its subcommands. The primary use-case for exercising these commands in BASH mode is that the BASH mode is pretty much as fully-scriptable as a root prompt on a normal Linux host. In other words, take all the danger and power of SE mode and wrap it in the sweaty-dynamite of an automated script (you can do a lot of modifications/damage by horsing the se sub-commands to a BASH `find` command or script).

    Friday, February 22, 2013

    Posterous Alternatives?

    So, Posterous is shutting down at the end of the month. It was a great meta-poster. Anyone know any suitable alternatives. Really liked having it for replicating my technical ramblings across multiple sites (really helpful when you work at multiple job-sites, each with their own web-filter rule-sets).

    Looking for any recommendations - maybe even paid-for services.

    Friday, February 8, 2013

    The "Which LUN Is That" Game

    One of the fun things about SAN storage has to do with its tendency to be multipathed. Different operating systems handle this differently - but most of the ones I've worked with tend to do what I refer to as "ghosting".

    "Ghosting" is, by no means, a technical term. It's mostly meant to convey that, when an OS sees a device down multiple paths, it sees the device multiple times. While you may only have one actual chunk of storage presented from your SAN to your host, you might see that chunk 2, 3, 4 or more times in your storage listings. I call these "extra" presences in the output "ghosts".

    At any rate, one of the joys associated with ghosting is determining, "am I seeing all of the LUNs I expect to see and am I seeing each of them an appropriate number of times. If you're only presenting one chunk of storage down four paths and you see four copies of the LUN show up, it's easy enough to say, "yep: things look right". It's even easy(ish) when you have multiple chunks of different-sized chunks of storage to determine if things are right: 1) you have PATHsxLUNs number of storage chunks visible; 2) each set of chunks is an identifiable size. However, if you're presenting multiple chunks that are the same size or multiple chunks with differing levels of pathing-redundancy, things get a bit trickier.

    One of the things I'd sorta liked about Solaris's STMS drivers were, once you'd activated the software for a given LUN, the ghosts disappeared into a single meta-device. For better or worse, not everyone used STMS - particularly not on older Solaris releases or configurations that used things like EMC's PowerPath or VERITAS's DMP software. PowerPath actually kinda made the problem worse, as, in addition to the normal ghosts, you added a PowerPath meta-device for each group of LUN-paths. This made the output from the `format` command even longer.

    All of that aside, how do you easily identify which disks are ghosts of each other and which ones are completely separate LUNS? The most reliable method I've found is looking for the LUNs' serial numbers. If you have eight storge chunks visible, four of which have one serial number and four of which have a different serial number, you know that you've got two LUNs presented to your host and that each is visible down four paths. But how do you get the serial numbers?

    Disks' and LUNs' serial numbers are generally found in their SCSI inquiry responses in what's referred to as "code page 83". How you get to that information is highly OS dependent.

    • On Solaris - at least prior to Solaris 10 - you didn't generally have good utilities for pulling serial numbers from LUNs. If you wanted to pull that info, you'd have to fire up the `format` utility in "expert" mode, issue the command-sequence "scsi → inquire". By default, this dumps out code page 83 as part of the response. It dumps this info in two parts: a big, multi-line block of hex-codes and a smaller multi-line block of ASCII text. Your disk/LUN serial number is found by ignorg the big, multiline block of hex values and looking at the third line from the smaller ASCII block.
    • On Linux, they provide you a nice little tool that allows you to directly dump out the target SCSI inquiry code-page. Its default behavior is pretty much to dump just the serial number (actually, the serial number is embedded in a longer string, but if you send that string over to your SAN guys, they'll generally recognize the relevant substring and match it up to what they've presented to your host). The way you dump out that string is to use the command `scsi_id -ugs /block/sdX` (where "sdX" is something like "sda", "sdh", etc.).

    At any rate, multi-pathing software and associated utilities aside, once you've determined which serial numbers correspond to which disk device-nodes, it because a trivial exercise to determine "am I seeing all of the LUNs I expect to see" and "are my LUNs presented down the expected number of SAN-fabric paths".

    Note: if you're running Solaris 10 or an earlier Solaris release with appropriate storage device management packages installed, you may have access to tools like `prtpicl`, `luxadm` and `fcinfo` with which to pull similarly-useful pathing information.