Supporting Dynamic root-disk in LVM-enabled Templates

One of the main customers I support has undertaken adoption of cloud-based services. This customer's IA team also requires that the OS drive be carved up to keep logging and audit activities separate from the rest of the OS disks. Previous to adoption of cloud-based services, this was a non-problem.

Since moving to the cloud — and using a build-method that generates launch-templates directly in the cloud (EL6 and EL7) — the use of LVM has proven problematic - particularly with EL6. Out-of-th-box, EL6 does not support dynamic resizing of the boot disk. This means that specifying a larger-than-default root-disk when launching a template is pointless if using a "stock" EL6 template. This can be overcome by creating a custom launch-template and that uses the dracut-modules-growroot from EPEL in that template.

Unfortunately, this EPEL RPM is only part of the picture. The downloaded dracut-modules-growroot RPM only supports growing "/" partition to the size of the larger-than-default disk if the template's root disk is either wholly unpartitioned or the disk is partitioned such that the "/" partition is the last partition on disk. It does not support a case where the "/" filesystem is hosted within an LVM2 volume-group. To get around this, it is necessary to patch the growroot.sh script that the dracut-modules-growroot RPM installs:

--- /usr/share/dracut/modules.d/50growroot/growroot.sh  2013-11-22 13:32:42.000000000 +0000
+++ growroot.sh 2016-08-02 15:56:54.308094011 +0000
@@ -18,8 +18,20 @@
 }

 _growroot() {
-       # Remove 'block:' prefix and find the root device
-       rootdev=$(readlink -f "${root#block:}")
+       # Compute root-device
+       if [ -z "${root##*mapper*}" ]
+       then
+               set -- "${root##*mapper/}"
+               VOLGRP=${1%-*}
+               ROOTVOL=${1#*-}
+               rootdev=$(readlink -f $(pvs --noheadings | awk '/'${VOLGRP}'/{print $1}'))
+               _info "'/' is hosted on an LVM2 volume: setting \$rootdev to ${rootdev}"
+       else
+               # Remove 'block:' prefix and find the root device
+               rootdev=$(readlink -f "${root#block:}")
+       fi
+
+       # root arg was nulled at some point...
        if [ -z "${rootdev}" ] ; then
                _warning "unable to find root device"
                return

Once the growroot.sh script has been patched, it will be necessary to generate and update the template's initramfs with the grow functionality enabled. If the template has multiple kernels installed, it will be desirable to ensure that each is functionally-enabled. A quick way to ensure that all of the initramfs files in the template are properly-enabled is to execute:

rpm -qa kernel | sed 's/^kernel-//' | \
   xargs -I {} dracut -f /boot/initramfs-{}.img

Note¹: The above will likely put data into the template's /var/log/dracut.log file. It is likely desirable to null-out this file (along with all other log files) prior to sealing the template.

Note²: Patching the growroot.sh script will cause RPM-verification to fail in VMs launched from the template. This can either be handled as a known/expected exception or can be averted by performing a `yum reinstall dracut-modules-growroot` in the template or in the VMs launched from the template.

Credit: The above is an extension to a solution that I found at Backslasher.Net (my Google-fu was strong the day that I wanted to solve this problem!)

Retrospective Automatic Image Replication in NetBackup

In version 7.x of NetBackup, VERITAS added the Automatic Image Replication functionality. This technology is more commonly referred to as "AIR". Its primary use case is to enable a NetBackup administrator to easily configure data replication between two different — typically geographically-disbursed — NetBackup domains.

Like many tools that are designed for a given use-case, AIR can be used for things it wasn't specifically designed for. Primary down-side to these not-designed-for use-cases is the documentation and tool-sets for such usage is generally pretty thin.

A customer I was assisting wanted to upgrade their appliance-based NetBackup system but didn't want to have to give up their old data. Because NetBackup appliances use Media Server Deduplication Pools (MSDP), it meant that I had a couple choices in how to handle their upgrade. I opted to try to use AIR to help me quickly and easily migrate data from their old appliance's MSDP to their new appliance's.

Sadly, as is typical of  not-designed-for use-case, documentation for doing it was kind of thin-on-the ground. Worse, because Symantec had recently spun VERITAS back off as its own entity, many of the forums that survived the transition had reference- and discussion-links that pointed to nowhere. Fortunately, I had access to a set of laboratory systems (AWS/Azure/Google Cloud/etc. is great for this - both from the standpoint of setup speed and "ready to go" OS templates). I was also able to borrow some of my customer's NetBackup 7.7 keys to use for the testing.

I typically prefer to work with UNIX/Linux-based systems to host NetBackup. However, my customer is a Windows-based shop. My customer's planned migration was also going to have the new NetBackup domain hosted on a different VLAN from their legacy NetBackup domain. This guided my lab design: I created a cloud-based "lab" configuration using two subnets and two Windows Server 2012 instance-templates. I set up each of my instances with enough storage to host the NetBackup software on one disk and the MSDPs on another disk ...and provisioned each of my test master servers with four CPUs and 16GiB or RAM. This is considerably smaller then both their old and new appliances, but I also wasn't trying to simulate an enterprise outpost's worth of backpup traffic. I also set up a mix of about twenty Windows and Linux instances to act as testing clients (customer is beginning to add Linux systems as virtualization and Linux-based "appliances" have started to creep into their enterprise-stacks).

I set up two very generic NetBackup domains. Into each, I built an MSDP. I also set up a couple of very generic backup policies on the one NetBackup Master Server to backup all of the testing clients to the MSDP. I configured the policies for daily fulls and hourly incrementals, and set up each of the clients to continuously regenerate random data-sets in their filesystems. I let this run for forty-eight hours so that I could get a nice amount of seed-data into the source NBU domain's MSDP.

Note: If you're not familiar with MSDP setup, the SETTLERSOMAN website has a good, generic walkthrough.

After populating the source site's MSDP, I converted from using the MSDP by way of a direct STorage Unit definition (STU) to using it by way of a two-stage Storage Lifecycle Policy (SLP). I configured the SLP to use the source-site MSDP as the stage-one destination in the lifecycle and added the second NBU domain's MSDP as the stage-two destination in the lifecycle. I then seeded the second NBU domain's MSDP with data by executing a full backup of all clients against the SLP.

Note: For a discussion on setting up an AIR-based replication SLP, again, the SETLLERSOMAN website has a good, generic walkthrough.

All of the above is fairly straight-forward and well documented (both within the NBU documentation and sites like SETTLERSOMAN). However, it only addresses the issue of how you get newly-generated data from one NBU domain's MSDP to another's. Getting older data from an existing MSDP to a new MSDP is a bit more involved ...and not for the command-line phobic (or, in my case, PowerShell-phobic.)

At a high level, what you do is:

1. Use the `bpimmedia` tool to enumerate all of the backup images stored on the source-site's MSDP
2. Grab only the media-IDs of the enumerated backup images
3. Feed that list of media-IDs to the `nbreplicate` tool so that it can copy that old data to the new MSDP

Note: The vendor documentation for the `bpimmedia` and  `nbreplicate` tools can be found at the VERITAS website.

When using the `bpimmedia` tool to automate image-ID enumeration, using the `-l` flag puts the output into a script-parsable format. The desired capture-item is the fourth field in all lines that begin 'IMAGE':

• In UNIX/Linux shell, use an invocation similar to: `bpimmedia -l | awk '/^IMAGE/{print $4}`
• In PowerShell, use an invocation similar to:`bpimmedia -l | select-string -pattern "IMAGE *" | ForEach-Object { $data = $_ -split " " ; "{0}" -f $data[3] }`

The above output can then be either captured to a file — so that one the `nbreplicate` job can be launched to handle all of the images — or each individual image-ID can be passed to an individual `nbreplicate` job (typically via a command-pipeline in a foreach script). I ended up doing the latter because, even though the documentation indicates that the tool supports specifying an image-file, when executed under PowerShell, `nbreplicate` did not seem to know what to do with said file.

The `nbreplicate` command has several key flags we're interested in for this exercise:

• -backupid: The backup-identifier captured via the `bpimmedia` tool
• -cn: The copy-number to replicate — in most circumstances, this should be "1"
• -rcn: The copy-number to assign to the replicated backup-image — in most circumstances, this should be "1"
• -slp: the name of the SLP hosted on the destination NetBackup domain
• -target_sts: the FQDN of the destination storage-server (use `nbemmcmd -listhosts` to verify names - or the replication jobs will fail with a status 191, sub-status 174)
• -target_user: the username of a user that has administrative rights to the destination storage-server
• -target_user: the password of the the -target_user username

 If you don't care about minimizing the number of replication operations, this can all be put together similar to the following:

• For Unix:

  for ID in $(bpimmedia -l | awk '/^IMAGE/{print $4}')
  do
     nbreplicate -backupid ${ID} -cn 1 -slp_name <REMOTE_SLP_NAME> \
       -target_sts <REMOTE_STORAGE_SERVER> -target_user <REMOTE_USER> \
       -target_pwd <REMOTE_USER_PASSWORD>
  done
  

• For Windows:

  @(bpimmedia -l | select-string -pattern "IMAGE *" | \
     ForEach-Object { $data = $_ -split " " ; "{0}" -f $data[3] }) | \
     ForEach-Object { nbreplicate -backupid $_ -cn 1 \
       -slp_name <REMOTE_SLP_NAME> -target_sts <REMOTE_STORAGE_SERVER> \
       -target_user <REMOTE_USER> -target_pwd <REMOTE_USER_PASSWORD> }
  

Seriously, CentOS?

One of my (many) duties in our shop is doing cross-platform maintenance of RPMs. Previously, when I was maintaining them for EL5 and EL6, things were fairly straight-forward. You got or made a SPEC file to packages your SOURCES into RPMS and you were pretty much good to go. Imagine my surprise when I went to start porting things to EL7 and all my freaking packages had el7.centos in their damned names. WTF?? These RPMs are for use on all of our EL7 systems, not just CentOS: why the hell are you dropping the implementation into my %{dist}-string?? So, now I have to make sure that my ${HOME}/.rpmmacros file has a line in it that looks like:

%dist .el7

If I don't want my %{dist}-string get crapped-up.

Screw you, CentOS. Stuff was just fine on CentOS 5 and CentOS 6. This change is not an improvement.

SSH Via GateOne

If you're like me, you occasionally work from locations that have restrictive firewall rules. Those rules may even be so restrictive that you're unable to SSH to remote hosts that you need to manage or do development work on.

The solution to this scenario is to use an SSH-over-HTML proxy service. In my case, I use GateOne. It's a light-weight service that's quick and easy to configure. The software maintainers even make Docker-ready bundles for it.

In my case, I create a GateOne-enabled SSH-proxy in AWS. Doing so is as simple as picking an EL6-compatible AMI (CentOS, RHEL, Scientific Linux, Amazon Linux, etc.) and giving the AMI launch-tool the following user-data:

#!/bin/sh

yum update -y
yum install -y git
pip install --upgrade pip
pip install --upgrade tornado
git clone https://github.com/liftoff/GateOne.git /tmp/GateOne
(cd /tmp/GateOne ; python setup.py install)
chkconfig gateone on
service gateone start
printf "Sleeping for 15s..."
sleep 15
echo "Done!"
pkill gateone
sed -i -e '/"https_redirect"/s/: .*$/: true,/' \
    -e '/"origins":/s/:.*$/: ["*"],/' \
    $(readlink -f /etc/gateone/conf.d/10server.conf)
sed -i '/"auth"/s/: .*$/: "pam",/' \
    $(readlink -f /etc/gateone/conf.d/20authentication.conf)
service gateone restart

When installed from its Git repository, the default configuration of GateOne only allows connection from an instances local interfaces. The `sed` operation against the "/etc/gateone/conf.d/10server.conf" configuration-file opens up this restriction. Setting the "origins" definition to '"*"' will allow browser connections from anywhere (if this is unacceptable, you can use AWS security-groups to lock things back down a bit).

Because I like to think of myself as lazy, I also enable a port 80 → 443 automatic redirect. Saves me from having to type "https://" into my browser (every keystroke saved counts!). This redirect is created by setting the "/etc/gateone/conf.d/10server.conf" configuration-file's "https_redirect" configuration-option to "true".

Because I prefer to do key-based logins and I don't want to make my SSH keys easily snarfable, I disable anonymous logins. This also has the side-effect of making it so it's not as easy for randos to discover your SSH proxy and use it for their own purposes. Disabling anonymous logins is done by setting the "/etc/gateone/conf.d/20authentication.conf" file's "auth" parameter. By setting it to "pam", as above, you instruct GateOne to allow local user-credentials to be used for logins.

When GateOne's authentication is set to "pam", it will be necessary to set up a valid user/password at the OS layer to be able to login via GateOne. You can do this via userdata or by logging into the instance and manually creating the GateOne user-credentials. I prefer the userdata method, as it means I can fully-automate the whole GateOne deployment process (e.g., via a CLI-script, CloudFormation template, etc.).

Note on the sleep statement: GateOne's configuration files are not present until the first time the service starts. Without the sleep, the `sed` operations will fail with a "missing file" error.

But I Don't Like That Username

One of the clients I do work for is in the process of adopting commercial cloud solutions. Early in the process, they had a private network that they were doing executing virtualization and private-cloud efforts. The maintainers of those two environments had created standard builds for use within those environments. For better or worse, my client has a number of developer teams they've contracted out to whose primary efforts are conducted either in-house (to the contractor) or within AWS, Azure or Google Compute Engine.

The group I work for has been tasked with standardizing and automating the deployment of enterprise components across all of the various environments and providing stewardship of the other contracted-out development efforts. When we were first given this task, the customer's build engineers would not provide any methods to replicate the production build - not even sufficient documentation that would allow us to accurately mimic it well enough to enable the other developers with a seamless dev -> test -> prod workflow.

Early in our involvement, I ended up creating my own build. Eventually, in the previously-described vacuum, others decided to adopt my build. Now, there's enough different groups using that build that it's pressuring the maintainers of the internal build to abandon theirs and adopt mine.

Fun part of DevOps is that when tends to be consensus-driven. If there's a slow or unresponsive link in the chain, the old "top down" approach frequently becomes a casualty when critical mass is achieved bottom up..

At any rate, we were recently in discussions with the enterprise build maintainers to show them how to adopt the consensus build. One of their pushbacks was "but that build doesn't include the 'break-glass' account that the legacy builds do." They wanted to know if we could modify the build to be compliant with that user-account.

This struck me odd, since, our group's (and other dev-groups') approach to such issues is "modify it in code". This isn't an approach familiar to the enterprise team. They're very "golden image" oriented. So, I provideded them a quick set of instructions on how to take the incoming standard build and make it compatible with their tools' expectations

#cloud-config
system_info:
  default_user:
    name: ent-adm

For enterprise components that will be migrated from private virtualization and cloud solutions to commercial cloud offerings, the above allows them to take not just my build, but any build that's enabled for automated provisioning and inject their account. Instead of launching a system with whatever the default-user is that's baked in, the above allows them to reset any system's initial username to be whatever they want (the above's 'ent-adm' is just an example - I dunno what their preferred account name is - I've simply used the above whenever I'm deploying instance-templates and don't want to remember "instance X uses userid A; instance Y uses userid B; and instance Z uses userid C").

Even more fun if they don't like any of the attributes associated with the default user's account, they can override it with any user-parameter available in cloud-init.

Per-Sender SMTP SASL Authentication

One of the customers I do work for runs a multi-tenant environment. One of the issues this customer has been having is "how do we notify tenants that their systems want patching". For their Linux systems, the immediate answer was to use the `yum-cron` facility to handle it.

Unfortunately, the system that would receive these types of notification emails is different than the ones that handle generic SMTP relaying. Instead of being able to set up each Linux system's Postfix service to use a single Smart-relay, we needed to be able to have just the account that's used to send the ready-to-patch notifications relay into through the notification gateway while all other emails get directed through a broader-scope relay.

It took several search iterations to finally uncover the "trick" for allowing this (credit to zmwangx for his GitHub Gist-posting). Overall, it's pretty straight-forward. Only thing that was not immediately obvious was which tokens mapped-through (what was the common key). In summary, to create the solution one needs to do three things:

1. Modify the Postfix configuration:
   1. Do the standard tasks around defining a "smart" relay
   2. Do the standard tasks around enabling Postfix to act as a SASL-authenticated client
   3. Enable per-sender authentication
   4. Define a per-sender relay-map
2. Create a file to map a local sender-address to a SASL-credential
3. Create a file to map a local sender-address to a specific relay-host

One that's done, it's simply a matter of verifying that things are working the way you think they should be.

Modify Postfix: Define Default Smart Relay
As with much of my experimentation the past year or two, I did my testing using Amazon Web Services resources. In this case, I used AWS's Simple Email Service (SES). Configuring Postfix to relay is trivial. Since my testing was designed to ensure that everything except my wanted sender-address to deny relaying, I configured my test system to point to its local SES relay without actually configuring an SES account to enable that relaying. For postfix, this was simply a matter of doing:

postconf -e "relayhost = [email-smtp.us-west-2.amazonaws.com]:587"

Appends creates line to your /etc/postfix/main.cf that looks like:

relayhost = [email-smtp.us-west-2.amazonaws.com]:587

Note: The AWS SES relays are currently only available within a few regions (as of this writing, NoVA/us-east-1, Oregon/us-west-2 and Ireland/eu-west-1). Each relay requires a SASL credential be created to allow relaying. So, no big deal publicizing the relay's name if spammers don't have such credentials.

At this point, any mail sent via Postfix will attempt to relay through the listed relay-host. Further, because SASL client-credentials are not yet set up, those relay attempts will fail.

Modify Postfix: Define Default SASL-Client Credentials
Add a block similar to the following to begin to turn Postfix into an SMTP SASL-client:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_mechanism_filter = plain
smtp_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt
smtp_use_tls = yes
smtp_tls_security_level = encrypt

Two critical items above are the "smtp_sasl_password_maps" and "smtp_tls_CAfile" parameters:

1. The former instructs Postfix where to look for SASL credentials.
2. The latter tells postfix where to look for root Certificate Authorities so it can verify that the relay host's SSL certificates are valid. The path listed in the above is the default for Red Hat-derived distributions: alter to fit your distribution's locations

The rest of the parameters instruct Postfix to use SASL-authentication over TLS-encrypted channels (required when connecting to an SMTP-relay via port 587) and to use the "plain" mechanism for sending credentials to the relay-host.

Modify Postfix: Enable Per-Sender Authentication

Use the command `postconf -e "smtp_sender_dependent_authentication = yes"` to enable Postfix's per-sender authentication modules. This will add a line to the /etc/postfix/main.cf that looks like:

smtp_sender_dependent_authentication = yes

Modify Postfix: Define SASL-Sender Map

Once per-sender authentication is enabled, Postfix needs to be instructed where to find mappings of senders to credentials. Use the command,  `postconf -e "sender_dependent_relayhost_maps = hash:/etc/postfix/sender_relay"` to enable Postfix's per-sender authentication modules. This will add a line to the /etc/postfix/main.cf that looks like:

sender_dependent_relayhost_maps = hash:/etc/postfix/sender_relay

Create a Sender-to-Credential Map

Edit/create the sender-to-credential mapping file /etc/postfix/sender_passwd. Its contents should be similar to the following:

# Sender Address                                 <userid>:<password>
patch-alert@ses-test.cloudlab.xanthia.com        AKIAICAOGQX5UA0ACDSVJ:pDHM1n4uYLGN4BQOnzGcTSeQXSRDjcKCy6VkmQk+CoBV

Postfix will use the "patch-alert@ses-test.cloudlab.xanthia.com" sender-address as a common key with the value in the relay-map (following).


Create a Sender-to-Relay Map

Edit/create the sender-to-credential mapping file /etc/postfix/sender_relay. Its contents should be similar to the following:

# Sender Address                            [relay-host]:port
patch-alert@ses-test.cloudlab.xanthia.com        [email-smtp.us-west-2.amazonaws.com]:587

Postfix will use "patch-alert@ses-test.cloudlab.xanthia.com" sender-address as a common key with the value in the credential-map (preceding).

Verification

A quick verification test is to send an email from a mapped user-address and a non-mapped user address:

# sendmail -f patch-alert@ses-test.cloudlab.xanthia.com -t <<EOF
To: fubar@cloudlab.xanthia.com
Subject: Per-User SASL Test
Content-type: text/html

If this arrived, things are probably set up correctly
EOF
# sendmail -f unmapped-user@ses-test.cloudlab.xanthia.com -t <<EOF
To: fubar@cloudlab.xanthia.com
Subject: Per-User SASL Test
Content-type: text/html

If this bounced, things are probably set up correctly
EOF

This should result in an SMTP log-snippet that resembles the following:

Feb 16 18:08:09 ses-test maintuser: MARK == MARK == MARK
Feb 16 18:08:22 ses-test postfix/pickup[5484]: 2B7D244AB: uid=0 from=<patch-alert@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:22 ses-test postfix/cleanup[5583]: 2B7D244AB: message-id=<20160216180822.2B7D244AB@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:22 ses-test postfix/qmgr[5485]: 2B7D244AB: from=<patch-alert@ses-test.cloudlab.xanthia.com>, size=403, nrcpt=1 (queue active)
Feb 16 18:08:22 ses-test postfix/smtp[5585]: 2B7D244AB: to=<thjones2@gmail.com>, relay=email-smtp.us-west-2.amazonaws.com[54.187.123.10]:587, delay=0.37, delays=0.02/0.03/0.19/0.13, dsn=2.0.0, status=sent (250 Ok 00000152eb44d396-408494a9-93f0-4f21-8985-460c057537bf-000000)
Feb 16 18:08:22 ses-test postfix/qmgr[5485]: 2B7D244AB: removed
Feb 16 18:08:32 ses-test postfix/pickup[5484]: A339E44AB: uid=0 from=<bad-sender@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:32 ses-test postfix/cleanup[5583]: A339E44AB: message-id=<20160216180832.A339E44AB@ses-test.cloudlab.xanthia.com>
Feb 16 18:08:32 ses-test postfix/qmgr[5485]: A339E44AB: from=<bad-sender@ses-test.cloudlab.xanthia.com>, size=408, nrcpt=1 (queue active)
Feb 16 18:08:32 ses-test postfix/smtp[5585]: A339E44AB: to=<thjones2@gmail.com>, relay=email-smtp.us-west-2.amazonaws.com[54.69.81.169]:587, delay=0.09, delays=0.01/0/0.08/0, dsn=5.0.0, status=bounced (host email-smtp.us-west-2.amazonaws.com[54.69.81.169] said: 530 Authentication required (in reply to MAIL FROM command))

As can be seen in the snippet, the first message (from the mapped sender) was relayed while the second message (from the unmapped sender) was rejected.

Solving Root VG Collisions in LVM-Enabled Virtualization Templates

The nature of template-based Linux OS deployments means that, if a template uses LVM for its root filesystems, any system built from that template will have non-unique volume group (VG) names. In most situations, non-unique VG names are not a problem. However, if you encounter a situation where you need to a broken instance by correcting a problem within the instance's root filesystems, non-unique VG names can make that task more difficult.

To avoid this eventuality, the template user can easily modify each launched template by executing steps similar to the following:

#!/bin/sh

DEFIF=$(ip route show | awk '/^default/{print $5}')
BASEIP=$(printf '%02X' \
         $(ip addr show ${DEFIF} | \
           awk '/inet /{print $2}' | \
           sed -e 's#/.*$##' -e 's/\./ /g' \
          ))

vgrename -v VolGroup00 VolGroup00_${BASEIP}
sed -i 's/VolGroup00/&_'${BASEIP}'/' /etc/fstab
sed -i 's/VolGroup00/&_'${BASEIP}'/g' /boot/grub/grub.conf

for KRNL in $(awk '/initrd/{print $2}' /boot/grub/grub.conf | \
              sed -e 's/^.*initramfs-//' -e 's/\.img$//')
do
   mkinitrd -f -v /boot/initramfs-${KRNL}.img ${KRNL}
done

init 6

Note that the above script assumes that the current root VG name is "VolGroup00". If your current root VG name is different, change the value in the script above as appropriate.

This script may be executed either at instance launch-time or anywhere in the life-cycle of an an instance. The above script takes the existing root VG name and tacks on a uniqueness component. In this case, the uniqueness is achieved by taking the IP address of the instance's primary interface and converting it to a hexadecimal string. So long as a group of systems does not contain any repeated primary IP addresses, this should provide a sufficient level of uniqueness for a group of deployed systems.

Note: renaming the root VG will _not_ solve the problems caused by PV UUID non-uniqueness. Currently, there is no known-good solution to this issue. The general recommendation is to avoid that problem by using a different template to build your recovery-host than used to build your broken host.

Simple Guacamole

If the enterprise you work for is like mine, access through the corporate firewall is tightly-controlled. You may find that only we-related protocols are left through (mostly) unfettered. When you're trying to work with a serivce like AWS, this can make management of Linux- and/or Windows-based resources problematic.

A decent solution to such a situation is the use of HTML-based remote connection gateway services. If all you're looking to do is SSH, the GateOne SSH-over-HTTP gateway is a quick and easy to setup solution. If you need to manage instances via graphical desktops - most typically Windows but some people like it for Linux as well - a better solution is Guacamole.

Guacamole is an extensible, HTTP-based solution. It runs as a Java servlet under a Unix hosted service like Tomcat. If you're like me, you may also prefer to encapsulate/broker the Tomcat service through a generic HTTP service like Apache or Nginx. My preference has been Apache - but mostly because I've been using Apache since not long after it was formally forked off of the NCSA project. I also tend to favor Apache because it's historically been part of the core repositories of my Linux of choice, Red Hat/CentOS.

Guacamole gives you HTTP-tunneling options for SSH, Telnet, RDP and VNC. this walk through is designed to get you quickly running Guacamole as an web-based SSH front end. Once you've got the SSH component running, adding other management protocols is easy. This procedure is also designed to be doable even if you don't yet actually have the ability to SSH to a AWS-hosted instance.

1. Start the AWS web console's "launch instance" wizard.
2. Select an appropriate EL6-based AMI.
3. Select an appropriate instance type (the free tier instances are suitable for a basic SSH proxy)
4. On the "Configure Instance" page, expand the "Advanced Details" section.
5. In the now-available text box, paste in the contents of this script. Note that this script is flexible enough that, if the version of Guacamole hosted via the EPEL project is updated, the script should continue to work. With a slight bit of massaging, the script could also be made to work with EL 7 and associated Tomcat and EPEL-hosted RPMs.
6. If the AMI you've picked does not provide the option of password-based logins for the default SSH user, add steps (in the "Advanced Details" text box) for creating an interactive SSH user with a password. Ensure that the user also has the ability to use `sudo` to get root privileges.
7. Finish up the rest of the process for deploying an instance.

Once the instance finishes deploying, you should be able to set your browser to the public hostname shown for the instance in the AWS console. Add "/guacamole/" after the hostname. Assuming all went well, you will be presented with a Guacamole login prompt. Enter the credentials:
Note that these credentials can be changing the:

printf "\t<authorize username=\"admin\" password=\"PASSWORD\">\n"

Line of the pasted-in script. Once you've authenticated to Guacamole, you'll be able to login to the hosting-instance via SSH using the instance's no-privileged user's credentials. Once logged in, you can escalate privileges and then configure additional authentication mechanisms and connection destinations and protocols.

Note: Guacamole doesn't currently support key-based login mechanisms. If key-based logins are a must make use of GateOne, instead.

EXTn and the Tyranny of AWS

One of the organizations I provide consulting services to opted to start migrating from an in-house, VMware-based virtualization solution to an Amazon-hosted cloud solution. The transition has been somewhat fraught, here and there - possibly moreso than the prior transition from physical (primarily Solaris-based) servers to virtualized (Linux) servers.

One of the huge "problems" is that the organization's various sub-units have habits formed of decade or longer lifecycles. Staff are particularly used to being able to get on console for various things (using GUI-based software-installers, graphical IDEs as well as tasks that actually require console access - like recovering a system stuck in its startup sequence).

For all that AWS offers, console access isn't one of them. Amazon's model for how customers should deploy and manage systems means that they don't figure that console access is strictly necessary.

In a wholly self-service model, this is probably an ok assumption. Unfortunately, the IT model that the organization in question is moving to doesn't offer instance-owners true self-service. They're essentially trying to transport an on-premises, managed multi-tenancy model into AWS. The model they're moving from didn't have self-service, so they're not interested in enabling self-service (at least not during phase one of the move). Their tenants not only don't have console access in the new environment, they don't have the ability to execute the AWS-style recovery-methods you'd use in the absence of console access. The tenants are impatient and the group supporting them is small, so it's a tough situation.

The migration has been ongoing for a sufficiently long period of time that the default `mkfs` behavior for EXT-based filesystems is starting to rear its head. Being well beyond the 180 day mark since the inception of the migration, tenants are finding that, when their Linux instances reboot, they're not coming back as quickly as they did towards the beginning of the migration ...because their builds still leave autofsck enabled.

If you're reading this, you may have run into similar issues.

The solution to this, while still maintaining the spirit of the "fsck every 180 days or so" best practices for EXTn-based filesystems is fairly straight forward:

1. Disable the autofsck settings on your instances' EXTn filesystems: use `tune2efs -i 0 -c -1 /dev/<DEVNODE>`
2. Schedule periodic fsck "simulations". This can be done either by running fsck in "dryrun" mode or by doing an fsck of a filesystem metadata image.

The "dryrun" method is fairly straight forward: just run fsck with the "-N" option. I'm not super much a fan of this as it doesn't feel like it gives me the info I'm looking for to feel good about the state of my filesystems.

The "fsck of a filesystem metadata image" is pretty straight forward, automatable and provides a bit more on the "warm fuzzies" side of thing. To do it:

1. Create a metadata image file using `e2image -fr /dev/<DEVNODE> /IMAGE/FILE/PATH` (e.g. `e2image -fr /dev/RootVG/auditVol /tmp/auditVol.img`)
2. Use `losetup` to create an fsck'able block-device from the image file (e.g., `losetup /dev/loop0 /tmp/auditVol.img`)
3. Execute an fsck against the loopback device (e.g., `fsck /dev/loop0`). Output will look similar to the following:

   # fsck /dev/loop0
   fsck from util-linux-ng 2.17.2
   e2fsck 1.41.12 (17-May-2010)
   /dev/loop0: recovering journal
   /dev/loop0: clean, 13/297184 files, 56066/1187840 blocks

4. If the output indicates anything other than good health, schedule an outage to do a proper repair of your live filesystem(s)

Granted, if you find you need to do the full check of the real filesystem(s), you're still potentially stuck with the "no console" issue. Even that is potentially surmountable:

1. Create a "/forcefsck" file
2. Create a "/fsckoptions" file with the contents "-sy"
3. Schedule your reboot

When the reboot happens, depending how long the system takes to boot, the EC2 launch monitors may time out: just be patient. If you can't be patient, just monitor the boot logs (either in the AWS console or using the AWS CLI's equivalent option).

Could You Make It Just A Little More Difficult?

Every so often, you need to share out an HTML file via a public web-server. Back in the pre-cloud days, this meant that you'd toss it up on your vanity web server and call it a day. In the cloud era, you have other options. Amazon's S3 was the first one that I used, but other cloud-storage services can be leveraged in a similar manner for static content.

One of those others is Google's "Drive" service. Unfortunately, Google doesn't exactly seem to want to make it a straight forward affair to share static web content straight from drive. It's easy if you want viewers to see the raw HTML, but not so great if you want them to see rendered HTML.

At any rate, as of the writing of this document (warning: judging by my Google results, the method seems to change over time and even Google's own help pages aren't really kept up to date), this was what I had to do:

1. Create a new folder in Google Drive (optional: if you're willing to make your top-level gDrive folder publicly-browsable or already have a folder that's set - or you're willing to set - as publicly-browsable, you can skip this step)
2. Edit the sharing on the folder, setting the permission to "Public View"
3. Navigate into the folder. Take note of its path. It will look something like:
   

   https://drive.google.com/drive/#folders/0F3SA-qkPpztNflU1bUtyekYYC091a2ttHZJpMElwTm9UcFNqN1pNMlf3iUlTUkJ0UU5PUVk

4. Take the left part of the URL, up to and including "/#folders/" and nuke it
5. Replace the deleted part of the original URL and replace it with:
   

   http://www.googledrive.com/host/

   The browsable URL to your publicly-viewable folder will now look like:
   

   http://www.googledrive.com/host/0F3SA-qkPpztNflU1bUtyekYYC091a2ttHZJpMElwTm9UcFNqN1pNMlf3iUlTUkJ0UU5PUVk

6. Clicking on  that link will take you to the folder holding your static web content. To get the sharable URL for your file(s), click on the link to the file.
7. When the file opens in your browser, copy the URL for the file, then send it out (for the sake of your recipients' sanity, you might want to pump it through a URL-shorter service, first - like maybe goo.gl)

In my case, I was trying to collaborate on a system-hardening toolset. I'd run my system through a security scanner that had flagged a number of false findings (actually, all the "fail" findings turned out to be bogus). I wanted to share the report with him and the rules files that the security tool had reported against. So, I sorted out the above so I could post links into our collaboration tool.

Maybe one day Google will make sharing static web content from Drive as (relatively) easy as Amazon has with S3. A "share as web page" button sure would be nice.