DataDomain Bash Shell (or: So You Want to Wreck Your DataDomain)

When dealing with the care and feeding of DataDomain arrays, there are occasions where it helps to know how to access the array's "Engineering Mode". In actuality, there are two levels of engineering mode for DataDomains:

• SE Shell: the SE (system engineer) shell mode is a superset of the normal system administration shell. It includes all of the management commands of the normal administration shell plus some powerful utilities for doing lower-level maintenance tasks on your DataDomain. These include things like fixing ACLs on your CIFS shares, changing networking settings (e.g., timeouts related to OST sessions) and other knobs that are nice to be able to twizzle
• BASH Shells: While the SE shell mode gives you more utilities for managing the array, they're still wrapped in the overall DDOS command-shell construct. The BASH shell mode is pretty much just like a normal root shell on a Linux system: you're able to script tasks in it, use tools like `find`, etc. Take all the damage you can do in the SE mode and add on the capability of doing those tasks on a massive, automated scale.

While enabling SE mode can likened to enabling you to shoot your foot off with a .22, the BASH mode could be likened to enabling you to shoot your foot off with a howitzer. Where SE mode is merely dangerous, I can't really begin to characterize the level of risk you expose yourself to when you start taking full advantage of the DataDomain's BASH shell.

Since accessing either of these modes isn't well-documented (though there's a decent number of Google searches that will turn up the basic "SE" mode) and I use this site as a personal-reminder on how to do things. I'm going to put the procedures here.

Please note: use of engineering mode allows you to do major amounts of damage to your data with a frightening degree of ease and rapidity. Don't try to access engineering mode unless you're fully prepared to have to re-install your DataDomain - inclusive of destroying what's left of the data on it.

Accessing SE Mode:

1. SSH to the DataDomain.
2. Login with an account that has system administrator privileges (this may be one of the default accounts your array was installed with, a local account you've set up for the purpose or an Active Directory managed account that has been placed into a Active Directory security-group that has been granted the system administrator role on the DataDomain
3. Get the array's serial number. The easiest way to do this is type `system show serialno` at the default command prompt
4. Access SE mode by typing `priv set se`. You will be prompted for a password - the password is the serial number from the prior step.

At this point, your command prompt will change to "SE@<ARRAYNAME>" where "<ARRAYNAME>" will be the nodename of your DataDomain. While in this mode, an additional command-set will be enabled. These commands are accessed by typing "se". You can get a further listing of the "se" sub-commands in much the same way you can get help at the normal system administration shell (in this particular case: by typing "se ?").


Accessing the SE BASH Shell:
Once you're in SE mode, the following command-sequence will allow you to access the engineering mode's BASH shell:

1. Type "fi st"
2. Type "df"
3. Type <CTRL>-C three times
4. Type "shell-escape"

At this point, a warning banner will come up to remind you of the jeopardy you've put your configuration in. The prompt will also change to include a warning. This is DataDomain's way of reminding you, at every step, the danger of the access-level you've entered.

Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed.

In addition to the standard Linux commands will be some DataDomain-specific commands. These are the commands that are accessible from the "se" command and its subcommands. The primary use-case for exercising these commands in BASH mode is that the BASH mode is pretty much as fully-scriptable as a root prompt on a normal Linux host. In other words, take all the danger and power of SE mode and wrap it in the sweaty-dynamite of an automated script (you can do a lot of modifications/damage by horsing the se sub-commands to a BASH `find` command or script).

CLARiiON Report Data Verification

Earlier this year, the organization I work for decided to put into production an enterprise-oriented storage resource management (SRM) system. The tool we bought is actually pretty cool. We install collectors into each of our major data centers and they pull storage utilization data off of all of our storage arrays, SAN switches and storage clients (you know: the Windows and UNIX boxes that use up all that array-based storage). Then, all those collectors pump out the collected data to a reporting server at our main data center. The reporting server is capable of producing all kinds of nifty/pretty reports: configuration snapshots, performance reports, trending reports, utilization profiles, etc.

As cool as all this is, you have the essential problem of "how do I know that the data in all those pretty reports is actually accurate?" Ten or fifteen years ago, when array-based storage was fairly new and storage was still the realm of systems administrators with coding skills, you'd ask you nearest scruffy misanthrope, "could you verify the numbers on this report," and get an answer back within a few hours (and then within minutes each subsequent time you asked). Unfortunately, in the modern, GUI-driven world, asking your storage guys to verify numbers can be like pulling teeth. Many modern storage guys aren't really coders and frequently don't know the quick and easy way to get you hard numbers out of the devices they manage. In some cases, you may watch them cut and paste from the individual array's management web UIs into something like MicroSoft Calculator. So, you'll have to wait and, often times, you'll have to continually prod them for the data because it's such a pain in the ass for them to produce.

With our SRM rollout, I found myself in just such a situation. Fortunately, I've been doing Unix system adminstration for the best part of 20 years and, therefore, am rather familiar with scripting. I frequently wish I was able to code in better reporting languages, but I just don't have the time to keep my "real" coding skills up to par. I'm also not terribly patient. So, after waiting a couple weeks for our storage guys to get me the numbers I'd asked for, I said to myself, "screw it: there's gotta be a quicker/better way."

In the case of our CLARiiONs, that better way was to use the NaviCLI (or, these days, the NaviSECCLI). This is a tool set that has been around a looooooong time, in one form or another, and has been available for pretty much any OS that you might attach to a CLARiiON as a storage client. These days, it's a TCP/IP-based commandline tool - prior to NaviCLI, you either had platform-specific tools (IRIX circa 1997 had a CLI-based tool that did queries through the SCSI bus to the array) or you logged directly into the array's RS232 port and used its onboard tools (hopefully, you had a terminal or terminal program that allowed you to capture output) ...but I digress.

If you own EMC equipment, you've hopefully got maintenance contracts that give you rights to download tools and utilities from the EMC support site. NaviCLI is one such tool. Once you install it, you have a nifty little command-line tool that you can wrap inside of scripts. You can create these scripts to both provisioning tasks and reporting tasks. My use, in this case, was reporting.'

The SRM we bought came with a number of canned-reports - including ones for CLARiiON devices. Unfortunately, the numbers we were getting from our SRM were indicating that we only had about 77TiB on one of our arrays when the EMC order sheets said we should have had about 102TiB. That's a bit of a discrepancy. I was able to wrap some NaviCLI commands into a couple scripts (one that reported on RAID-group capacity and one that reported physical and logical disk capacities [ed.: please note that these scripts are meant to be illustrative of what you can do, but aren't really something you'd want to have as the nexus of your enterprise-reporting strategy. They're slow to run, particularly on larger arrays]) and verify that the 77TiB was sort of right and that the 102TiB was also sorta right. The group capacity script basically just spits out two numbers - total raw capacity and total capacity allocatable to clients (without reporting on how much of either is already allocated to clients). The disk capacity script reports how the disks are organized (e.g., RAID1, RAID5, Spare, etc.) - printing total number of disks in each configuration category and how much raw capacity that represented. Basically, the SRM tool was reporting the maximum number of blocks that were configured into RAID groups, not the total raw physical blocks in the array that we'd thought it was supposed to report.

Having these number in hand allowed us to tear apart the SRM's database queries and tables so that we could see what information it was grabbing, how it was storing/organizing it and how to improve on the vendor-supplied standard reports. Mostly, it consisted of changing the titles of some existing fields and adding some fields to the final report.

Yeah, all of this begs the question "what was the value of buying an SRM when you had to reverse-engineer it to make the presented data meaningful?" To be honest, "I dunno." I guess, at the very least, we bought a framework through which we could put together pretty reports and ones that were more specifically meaningful to us (though, to be honest, I'm a little surprised that we're the only customers of the SRM vendor to have found the canned-reports to be "sadly lacking"). It also gave me an opportunity to give our storage guys a better idea of the powerful tools they had available to them if only they were willing to dabble at the command line (even on Windows).

Still the vendor did provide a technical resource to help us get things sorted out faster than we might have done without that assistance. So, I guess that's something?

Automating Yum Setup

Recently, as part of my employer's "employee development" programs, I took Red Hat's RH255 class so that I could prep for getting my RHCSA and RHCE (my opinions on the experience are grist for another post - and I don't know, yet, whether that post should be in this blog or one of my personal blogs). The RH255 class I took was based on RHEL 6. In the US, they moved to training based on RHEL 6.0 about six months ago. One of the interesting things (I thought) my instructor said was that RHEL 6.0 included a function that tried to automatically configure `yum` to take mounted CDROMs and ISOs and treat them as installation repositories.

I may have misheard or misinterpreted what he said. It may also be a case that, since my instructor is in the RHEL 6.1 beta program, he was referring to a feature in RHEL 6.1 and not RHEL 6.0. Whatever the case may be, popping an RHEL 6.0 DVD (or ISO) into and RHEL 6.0 machine does not (yet) cause that DVD to be automatically included in `yum`'s repository search. Fortunately, the RHEL 6.x media has been set up (prior RHELs may also have been, it's just been so long since I've built an RHEL 5 system from other than an automated-build system) so that it's easy enough to automatically set up the DVD or ISO for inclusion in `yum`'s seaches. All of the yum repository is already on the media. So, you needn't muck about with doing the createrepo stuff (and the time that running createrepo against a multi-gigabyte DVD can take). Below is the script I wrote to take this already-present data and make it available to `yum`:

#!/bin/sh

MNTDIR=${1:-UNDEF}

# Make sure we passed a location to check

if [ ${MNTDIR} == "UNDEF" ]

then

   echo "Usage: $0 <ISO Mount Location>"

   exit

fi

# Make sure it's actually a directory

if [ ! -d ${MNTDIR} ]

then

   echo "No such directory"

   exit

fi

#ID directories containing repository data

REPODIRS=$(find ${MNTDIR} -type d -name repodata)

if [ "${REPODIRS}" == "" ]

then

   echo "No repository data found"

   exit

fi

for DIR in ${REPODIRS}

to

   DIRNAME=$(echo ${DIR} | sed -e 's#'${MNTDIR}'/##' -e 's#/repodata##')

   BASEURL=$(dirname ${DIR})

   echo "[${DIRNAME}]"

   echo "baseurl=${BASEURL}"

   echo "enabled=1"

   echo

done

The above script takes a directory location (presumably where you mounted your DVD or ISO) and produces yum repository configuration output. I opted to have it output as a capturable stream rather than as a file. That way, I'd have to option of either capturing to a temp location or  directly into /etc/yum.repos.d. Doing so gave more flexibility in what to name a file (so that I didn't clobber any existing files). Granted, I could have parameterized the name and location of the output file, but chose not to. If you want to use the above and would rather have the output go directly to an output file in /etc/yum.repos.d (or elsewhere), it's a trivial modification to the above. Have at it.

At any rate, once you run the above and take the resultant output and move it into a file in /etc/yum.repos.d, subsequent invocations of the `yum` command should now include your DVD or ISO. If it fails to do so (or errors result) it's probably because the new .repo file contains section IDs that collide with exiting ones. Just edit your new file to eliminate any collisions.

While I've done some basic error-checking in the file (specifically: ensuring the user passed a location and that the location is a valid directory), I didn't write it to gracefully handle repository directories that have spaces in their path-names. It's also fairly trivial to fix the script to accommodate it. However, it's probably even easier to just make sure you don't mount your DVD or ISO with spaces in the path-names. I'm lazy, so, I'm choosing the latter option and avoiding the coding exercise. If you find you need to have spaces in your path-names and want to use my script, feel free to make the necessary modifications to accommodate your use of space-containing path names.