Showing posts with label cloud. Show all posts
Showing posts with label cloud. Show all posts

Tuesday, August 2, 2016

Supporting Dynamic root-disk in LVM-enabled Templates - EL7 Edition

In my previous article,  Supporting Dynamic root-disk in LVM-enabled Templates, I discussed the challenges around supporting LVM-enabled VM-templates in cloud-based deployments of Enterprise Linux 6 VMs. The kernel used for Enterprise Linux 7 distributions makes template-based deployment of LVM-enabled VMs a bit easier. Instead of having to add an RPM from EPEL and then hack that RPM to make it support LVM2-encapsulated root volumes/filesystems, one need only ensure that the cloud-utils-growpart RPM is installed and do same launch-time massaging via cloud-init. By way of example:
#cloud-config
runcmd:
  - /usr/bin/growpart /dev/xvda 2
  - pvresize /dev/xvda2
  - lvresize -r -L +2G VolGroup00/logVol
  - lvresize -r -L +2G VolGroup00/auditVol
Will cause the launched instance to:
  1. Grow the second partition on the boot disk to the end of the disk
  2. Instruct LVM to resize the PV to match the new partition-size
  3. Instruct LVM to grow the VolGroup00/logVol volume — and the filesystem on top of it — by 2GiB
  4. Instruct LVM to grow the VolGroup00/auditVol volume — and the filesystem on top of it — by 2GiB
Upon login, the above launch-time configuration-actions can be verified by using `vgdisplay -s` and `lvs --segments -o +devices`:
# vgdisplay -s
  "VolGroup00" 29.53 GiB [23.53 GiB used / 6.00 GiB free]
# lvs --segments -o +devices
  LV       VG         Attr       #Str Type   SSize Devices
  auditVol VolGroup00 -wi-ao----    1 linear 8.53g /dev/xvda2(2816)
  auditVol VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(5512)
  homeVol  VolGroup00 -wi-ao----    1 linear 1.00g /dev/xvda2(1536)
  logVol   VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(2304)
  logVol   VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(5000)
  rootVol  VolGroup00 -wi-ao----    1 linear 4.00g /dev/xvda2(0)
  swapVol  VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(1024)
  varVol   VolGroup00 -wi-ao----    1 linear 2.00g /dev/xvda2(1792)

Supporting Dynamic root-disk in LVM-enabled Templates

One of the main customers I support has undertaken adoption of cloud-based services. This customer's IA team also requires that the OS drive be carved up to keep logging and audit activities separate from the rest of the OS disks. Previous to adoption of cloud-based services, this was a non-problem.

Since moving to the cloud — and using a build-method that generates launch-templates directly in the cloud (EL6 and EL7) — the use of LVM has proven problematic - particularly with EL6. Out-of-th-box, EL6 does not support dynamic resizing of the boot disk. This means that specifying a larger-than-default root-disk when launching a template is pointless if using a "stock" EL6 template. This can be overcome by creating a custom launch-template and that uses the dracut-modules-growroot from EPEL in that template.

Unfortunately, this EPEL RPM is only part of the picture. The downloaded dracut-modules-growroot RPM only supports growing "/" partition to the size of the larger-than-default disk if the template's root disk is either wholly unpartitioned or the disk is partitioned such that the "/" partition is the last partition on disk. It does not support a case where the "/" filesystem is hosted within an LVM2 volume-group. To get around this, it is necessary to patch the growroot.sh script that the dracut-modules-growroot RPM installs:
--- /usr/share/dracut/modules.d/50growroot/growroot.sh  2013-11-22 13:32:42.000000000 +0000
+++ growroot.sh 2016-08-02 15:56:54.308094011 +0000
@@ -18,8 +18,20 @@
 }

 _growroot() {
-       # Remove 'block:' prefix and find the root device
-       rootdev=$(readlink -f "${root#block:}")
+       # Compute root-device
+       if [ -z "${root##*mapper*}" ]
+       then
+               set -- "${root##*mapper/}"
+               VOLGRP=${1%-*}
+               ROOTVOL=${1#*-}
+               rootdev=$(readlink -f $(pvs --noheadings | awk '/'${VOLGRP}'/{print $1}'))
+               _info "'/' is hosted on an LVM2 volume: setting \$rootdev to ${rootdev}"
+       else
+               # Remove 'block:' prefix and find the root device
+               rootdev=$(readlink -f "${root#block:}")
+       fi
+
+       # root arg was nulled at some point...
        if [ -z "${rootdev}" ] ; then
                _warning "unable to find root device"
                return
Once the growroot.sh script has been patched, it will be necessary to generate and update the template's initramfs with the grow functionality enabled. If the template has multiple kernels installed, it will be desirable to ensure that each is functionally-enabled. A quick way to ensure that all of the initramfs files in the template are properly-enabled is to execute:

rpm -qa kernel | sed 's/^kernel-//' | \
   xargs -I {} dracut -f /boot/initramfs-{}.img
Note1: The above will likely put data into the template's /var/log/dracut.log file. It is likely desirable to null-out this file (along with all other log files) prior to sealing the template.

Note2: Patching the growroot.sh script will cause RPM-verification to fail in VMs launched from the template. This can either be handled as a known/expected exception or can be averted by performing a `yum reinstall dracut-modules-growroot` in the template or in the VMs launched from the template.

Credit: The above is an extension to a solution that I found at Backslasher.Net (my Google-fu was strong the day that I wanted to solve this problem!)

Tuesday, April 19, 2016

But I Don't Like That Username

One of the clients I do work for is in the process of adopting commercial cloud solutions. Early in the process, they had a private network that they were doing executing virtualization and private-cloud efforts. The maintainers of those two environments had created standard builds for use within those environments. For better or worse, my client has a number of developer teams they've contracted out to whose primary efforts are conducted either in-house (to the contractor) or within AWS, Azure or Google Compute Engine.

The group I work for has been tasked with standardizing and automating the deployment of enterprise components across all of the various environments and providing stewardship of the other contracted-out development efforts. When we were first given this task, the customer's build engineers would not provide any methods to replicate the production build - not even sufficient documentation that would allow us to accurately mimic it well enough to enable the other developers with a seamless dev -> test -> prod workflow.

Early in our involvement, I ended up creating my own build. Eventually, in the previously-described vacuum, others decided to adopt my build. Now, there's enough different groups using that build that it's pressuring the maintainers of the internal build to abandon theirs and adopt mine.

Fun part of DevOps is that when tends to be consensus-driven. If there's a slow or unresponsive link in the chain, the old "top down" approach frequently becomes a casualty when critical mass is achieved bottom up..

At any rate, we were recently in discussions with the enterprise build maintainers to show them how to adopt the consensus build. One of their pushbacks was "but that build doesn't include the 'break-glass' account that the legacy builds do." They wanted to know if we could modify the build to be compliant with that user-account.

This struck me odd, since, our group's (and other dev-groups') approach to such issues is "modify it in code". This isn't an approach familiar to the enterprise team. They're very "golden image" oriented. So, I provideded them a quick set of instructions on how to take the incoming standard build and make it compatible with their tools' expectations
#cloud-config
system_info:
  default_user:
    name: ent-adm
For enterprise components that will be migrated from private virtualization and cloud solutions to commercial cloud offerings, the above allows them to take not just my build, but any build that's enabled for automated provisioning and inject their account. Instead of launching a system with whatever the default-user is that's baked in, the above allows them to reset any system's initial username to be whatever they want (the above's 'ent-adm' is just an example - I dunno what their preferred account name is - I've simply used the above whenever I'm deploying instance-templates and don't want to remember "instance X uses userid A; instance Y uses userid B; and instance Z uses userid C").

Even more fun if they don't like any of the attributes associated with the default user's account, they can override it with any user-parameter available in cloud-init.

Thursday, March 26, 2015

Could You Make It Just A Little More Difficult?

Every so often, you need to share out an HTML file via a public web-server. Back in the pre-cloud days, this meant that you'd toss it up on your vanity web server and call it a day. In the cloud era, you have other options. Amazon's S3 was the first one that I used, but other cloud-storage services can be leveraged in a similar manner for static content.

One of those others is Google's "Drive" service. Unfortunately, Google doesn't exactly seem to want to make it a straight forward affair to share static web content straight from drive. It's easy if you want viewers to see the raw HTML, but not so great if you want them to see rendered HTML.

At any rate, as of the writing of this document (warning: judging by my Google results, the method seems to change over time and even Google's own help pages aren't really kept up to date), this was what I had to do:

  1. Create a new folder in Google Drive (optional: if you're willing to make your top-level gDrive folder publicly-browsable or already have a folder that's set - or you're willing to set - as publicly-browsable, you can skip this step)
  2. Edit the sharing on the folder, setting the permission to "Public View"
  3. Navigate into the folder. Take note of its path. It will look something like:
    https://drive.google.com/drive/#folders/0F3SA-qkPpztNflU1bUtyekYYC091a2ttHZJpMElwTm9UcFNqN1pNMlf3iUlTUkJ0UU5PUVk
  4. Take the left part of the URL, up to and including "/#folders/" and nuke it
  5. Replace the deleted part of the original URL and replace it with:
    http://www.googledrive.com/host/
    The browsable URL to your publicly-viewable folder will now look like:
    http://www.googledrive.com/host/0F3SA-qkPpztNflU1bUtyekYYC091a2ttHZJpMElwTm9UcFNqN1pNMlf3iUlTUkJ0UU5PUVk
  6. Clicking on  that link will take you to the folder holding your static web content. To get the sharable URL for your file(s), click on the link to the file.
  7. When the file opens in your browser, copy the URL for the file, then send it out (for the sake of your recipients' sanity, you might want to pump it through a URL-shorter service, first - like maybe goo.gl)
In my case, I was trying to collaborate on a system-hardening toolset. I'd run my system through a security scanner that had flagged a number of false findings (actually, all the "fail" findings turned out to be bogus). I wanted to share the report with him and the rules files that the security tool had reported against. So, I sorted out the above so I could post links into our collaboration tool.

Maybe one day Google will make sharing static web content from Drive as (relatively) easy as Amazon has with S3. A "share as web page" button sure would be nice.