Friday, May 24, 2013

DataDomain Bash Shell (or: So You Want to Wreck Your DataDomain)

When dealing with the care and feeding of DataDomain arrays, there are occasions where it helps to know how to access the array's "Engineering Mode". In actuality, there are two levels of engineering mode for DataDomains:

  • SE Shell: the SE (system engineer) shell mode is a superset of the normal system administration shell. It includes all of the management commands of the normal administration shell plus some powerful utilities for doing lower-level maintenance tasks on your DataDomain. These include things like fixing ACLs on your CIFS shares, changing networking settings (e.g., timeouts related to OST sessions) and other knobs that are nice to be able to twizzle
  • BASH Shells: While the SE shell mode gives you more utilities for managing the array, they're still wrapped in the overall DDOS command-shell construct. The BASH shell mode is pretty much just like a normal root shell on a Linux system: you're able to script tasks in it, use tools like `find`, etc. Take all the damage you can do in the SE mode and add on the capability of doing those tasks on a massive, automated scale.
While enabling SE mode can likened to enabling you to shoot your foot off with a .22, the BASH mode could be likened to enabling you to shoot your foot off with a howitzer. Where SE mode is merely dangerous, I can't really begin to characterize the level of risk you expose yourself to when you start taking full advantage of the DataDomain's BASH shell.

Since accessing either of these modes isn't well-documented (though there's a decent number of Google searches that will turn up the basic "SE" mode) and I use this site as a personal-reminder on how to do things. I'm going to put the procedures here.

Please note: use of engineering mode allows you to do major amounts of damage to your data with a frightening degree of ease and rapidity. Don't try to access engineering mode unless you're fully prepared to have to re-install your DataDomain - inclusive of destroying what's left of the data on it.

Accessing SE Mode:
  1. SSH to the DataDomain.
  2. Login with an account that has system administrator privileges (this may be one of the default accounts your array was installed with, a local account you've set up for the purpose or an Active Directory managed account that has been placed into a Active Directory security-group that has been granted the system administrator role on the DataDomain
  3. Get the array's serial number. The easiest way to do this is type `system show serialno` at the default command prompt
  4. Access SE mode by typing `priv set se`. You will be prompted for a password - the password is the serial number from the prior step.
At this point, your command prompt will change to "SE@<ARRAYNAME>" where "<ARRAYNAME>" will be the nodename of your DataDomain. While in this mode, an additional command-set will be enabled. These commands are accessed by typing "se". You can get a further listing of the "se" sub-commands in much the same way you can get help at the normal system administration shell (in this particular case: by typing "se ?").


Accessing the SE BASH Shell:
Once you're in SE mode, the following command-sequence will allow you to access the engineering mode's BASH shell:

  1. Type "fi st"
  2. Type "df"
  3. Type <CTRL>-C three times
  4. Type "shell-escape"
At this point, a warning banner will come up to remind you of the jeopardy you've put your configuration in. The prompt will also change to include a warning. This is DataDomain's way of reminding you, at every step, the danger of the access-level you've entered.

Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed.

In addition to the standard Linux commands will be some DataDomain-specific commands. These are the commands that are accessible from the "se" command and its subcommands. The primary use-case for exercising these commands in BASH mode is that the BASH mode is pretty much as fully-scriptable as a root prompt on a normal Linux host. In other words, take all the danger and power of SE mode and wrap it in the sweaty-dynamite of an automated script (you can do a lot of modifications/damage by horsing the se sub-commands to a BASH `find` command or script).

18 comments:

  1. Step #1 for Accessing the SE BASH shell should be Type "uname".

    ReplyDelete
  2. This doesn't seem to work on a 5.1.1 system, I can get into SE mode but "shell-escape" gives me a command not found error.

    ReplyDelete
    Replies
    1. At the time or original publishing of this article, we were still mostly running 4.9 in our environment. It worked on 5.1 until we applied one of the updates: DataDomain patched the 5.x releases a couple months after this article was originally published See the `uname` comment posted on August 12 for the updated procedure.

      Delete
  3. Unfortunately the "uname" way not works under 5.4.1 system.
    The "shell-escape" gives me a command not found error.

    ReplyDelete
    Replies
    1. Yeah. Even though a DataDomain support rep might tell you "I can't tell you the procedures, but you can find them online with a quick search" (which usually leads to this page), DataDomain still makes periodic patches to their code to make accessing engineering mode require slightly-different key sequences. Unfortunately, I've moved on from my prior position and no longer have ready access to any DataDomains, let alone ones running 5.4.x

      Delete
  4. Can you please tell me how to replace step #1 with uname type.

    Is it: filesys ????

    Thanks much

    ReplyDelete
    Replies
    1. Simply just insert this step (Type "uname") before the sequence above and start with it...
      It works on 5.4.x too....

      Delete
  5. Here is how to do this:

    Log into system
    system show serialno
    paste this when it asks for password
    ctrl-c three times
    uname
    fi st
    df
    ctrl-c three times
    shell-escape

    ReplyDelete
  6. Hi,

    Do you know how I can exit from SE mode?

    Thanks.

    ReplyDelete
  7. Step 1: Log into Data Domain
    Step 2: type system show serial
    Step 3: type priv set se
    Step 4: enter the serial no as password
    Step 5: you will the SE mode now with ##
    Step 6: type uname
    Step 7: type fi st
    Step 8: type df
    Step 9: -C three times
    Step 10: Type "shell-escape"
    Now you will see the Wanining Message and type uname again it will show "linux"

    Here UR good to crash ur DD :P

    ReplyDelete
  8. all the above steps working well with 5.0.2 to 5.5.2

    ReplyDelete
  9. how do you find your directories now that you are root? Ex) if your NFS or CIFS shares are at /data/col1/[folder name] how do you actually browse there using linux? I've "cd" and "ls" and "ls -a" and I still can't find my data. I can cd to /data or even /backup but there is no /col1 under it. Anyone know?

    Thanks.

    ReplyDelete
    Replies
    1. Haven't reason to touch DataDomain in several years, now (work has me working primarily within cloud services like AWS). That said, if you know the name of a file you've written to the DataDomain, you can always brute-force things by doing a `find / -name -type f`. This will likely be slow, but it will tell you where your version of DDOS is storing files.

      Note: looking for the name of a file backed up through an application like NetBackup won't work as those files tend to be encapsulated within a larger file-object. In the case of NetBackup, you'd be looking for the name of the image file rather than a backed-up file within that image-file.

      Delete
    2. They've pretty much swallowed files into the database.

      If you want to list files on an MTree, you can use "filesys show compression MTREE recursive". There's no delete or move (you can filesys fastcopy at least).

      Delete
    3. Eef... Not a lot of databases that handle blobs of arbitrary size all that well. Plus, if you've truly moved from having the data objects hosted in a filesystem construct to be hosted in a database, you've added the pain of having to write an abstraction/presentation-layer from the database to the file-sharing protocols (i.e., DB-to-NFS and DB-to-CIFS gateways). Seems a potentially "sub-optimal" path to pursue.

      You sure they haven't just put object-tracking into the database but still leave the objects, themselves, in a legacy filesystem construct? Seems that would be marginally less fraught than a complete blob-ingest model.

      Delete