I Hate PowerShell (Installment 3783)

So, I know I've skipped a few numbers since my I Hate PowerShell (Installment 3769) post. That post still applies and, yes, there've been several more annoyances encountered since authoring that post.

For the project I'm currently working on, I can only access my remote Windows (Server 2022) hosts using AWS's  FleetManager service. The VPCs are otherwise configured to block access via direct RDP.

At any rate, the first time I login, I fire up PowerShell. I go to type into the resulting "terminal" window and… nothing. The window doesn't accept my keyboard input. Perplexed, I Slack the teammate who asked me to write the automation for the Windows provisioning-tasks that he'd been doing manually. I note my problem to him. He tells me, "install PowerShell 7".

I update the userData I pass as part of launching my EC2 so as to implement that advice. I wait for the userData's execution to complete and then login. Once the remote desktop session becomes fully responsive, I fire up powershell …and find that I am still unable to type. Perplexed, I do some digging around and find that "upgrade to PowerShell 7" doesn't _actually_ upgrade PowerShell, it just installs a second, newer version of PowerShell alongside the system default. If I want to access the newer version, I need to call `pwsh`, instead. So, I do so. The new window opens and, as advertised, I'm able to type into that window.

Now, I'm a person possessed of significant curiousity. As such, my curious brain thinks, "what happens if I call `powershell` from the PS7 window I'm able to type into". To sate my curiosity, I do so. Suddenly, the window I was previously able to type into no longer accepts keyboard input.

So, yeah, if I want to be able to type into windows opened from the default PowerShell version, I need to add some further plumbing to my userData. Specifically, I need to include some logic to update the default PowerShell installation's `PSReadLine` module.

Notionally, I could skip the installation of PS7. However, its installation seems to be the default work-around for this project (still don't know if I'd call that an "anti-pattern" or not). As such, I fear that not having it installed will result in the eventual users of my automation screaming about it not being installed.

At any rate, if I want to be able to type into the default PowerShell version and I want PowerShell 7 available for others' use, my userData payload needs to look like:

# Ensure the default PowerShell (v5.x) windows accept typed-in content
Install-PackageProvider -name NuGet -Force
Install-Module `
  -Name PSReadLine `
  -Repository PSGallery `
  -Force

# Install PowerShell 7
iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"

So. Yeah. But, at least my deployment target works. 

Interestingly, I don't seem to run into the "can't type into the window" problem if I use the AWS SSM CLI to login. Or maybe I didn't start trying that route until I was already including the PSReadLine update-juju into my userData payload?

Also interestingly, because the PS7 installation is a parallel installation, if I want to use the  AWS SSM CLI to login directly to a PS7 session, I have to update my CLI-access invocation from:

aws ssm start-session --target <INSTANCE_ID>

To:

aws ssm start-session --target <INSTANCE_ID> \
  --document-name AWS-StartInteractiveCommand \
  --parameters command="pwsh"

I guess the "don't try to change the system-wide PS version" guidance that search-results are showing me are a lot like "don't try to replace the system-default Python version" cautions for Red Hat distros?

I Hate PowerShell (Installment 3769)

One of my customers has a group of developers. They're also tasked with SRE-style responsibilities as things move from development to production. Since the production environment is significantly locked down, these developers cannot access it from their laptops. As such, they don't have the kind of service-management tooling available to them in case shit goes sideways. To work around this issue, they requested the creation of a "jump box" in the production-environment that would have a reliable set of tools installed. Even though everything in production is either Linux or Kubernetes based, they wanted the "jump box" to be Windows-based. So, one of my peers hand-created a suitable "jump box" for them — installing all of the desired tools and creating the desired, RDP-enabled users.

In general, we don't like to have such hand-made boxes. They tend to be poorly maintained (such that their security auditors become sadder and sadder with the passing of each patching cycle) and, if they're the victim of system-breakage. Further, automating builds makes it much easier to do cost-control since you can better implement an "intantiate as you need it" deployment (or repair/replace) model. As such, they needed someone to automate what had previously been hand-jammed.

Since I'd just come off of a project where I'd delivered a bunch of "hardening" automation, I was the stuckee. I, uh, don't generally work with Windows. I especially don't really do automation for Windows-based systems. So, I accepted the task know that, "fun times be ahead," for me.

Part of the task involved hardening the "jump box" using a framework we'd been pushing our various customers to use for the past decade or so. Since there was already a "bootstrap" (PowerShell) script written for this purpose, I opted to use that script as my starting-point — why (wholly) reinvent the wheel, eh?

That script worked by downloading and executing the hardening framework. Easiest path forward, for me, was to update that script to similarly download and execute a my script.

At any rate, the first-pass at authoring my script was just to install a set of developer-oriented (really, more like "SRE oriented") tools. I wrote it in such a way that the automation user could specify specific versions of the desired tooling (by giving the URLs of the tools they wanted installed). It worked well enough. However, it didn't include the creation of RDP-enabled users. As such, once the "jump box" was built, the developers/SREs couldn't login without someone hand-jamming the creation of local users.

Today, I extended my script to allow the use of an external, JSON-formatted, user-specification file.

I wanted the various users' attributes to be flexible (e.g., the "is an admin user" attribute should allow a value that was any of `true`, `false` or undefined). The version of Powershell on the automation-targets apparently has some kind of "strict" mode set as the default behavior. So, when three of my JSON-file's testing user-definitions didn't have the "is an admin user" attribute defined, Powershell noped-out on that "problem". 

So, I did some digging around (Microsoft documentation, StackExchange, etc.). Figured out how to write the block in a manner that operates properly when the "strict" mode is set.

It was, to me, quite ugly compared to automation I've written for Linux-based hosts. To implement in a way that left the "strict" mode interpreter happy, I implemented my function like:

function Parse-JsonFile {
  # Where to write downloaded user-creation spec-file to
  $UserCreationFile = "${SaveDir}\$(${UserCreationUrl}.split("/")[-1])"

  # Download user-creation spec-file
  Download-File -Url ${UserCreationUrl} -SavePath ${UserCreationFile}

  # Abort if given file-path is not valid
  if ( -not ( Test-Path $UserCreationFile ) ) {
      Write-Error "File not found: $UserCreationFile"
      return
  }

  # Load JSON-payload from file and convert to PS object
  $JsonStream = Get-Content -Raw -Path "${UserCreationFile}" | ConvertFrom-Json

  # The structure has a 'Users' array containing a single object with dynamic keys
  foreach ($userContainer in $JsonStream.Users) {
    # Iterate through each dynamic key (the usernames)
    foreach ($username in $userContainer.psobject.Properties.Name) {
      # Get the array associated with that username
      $userDetails = $userContainer.$username

      foreach ($detail in $userDetails) {
        # Create "full name" attribute to user-creation function
        $FullName = ${detail}.givenName + " " + ${detail}.surname

        # Safely set WantsAdmin in a Strict-mode safe way
        if ( $detail.psobject.Properties.Name -contains "localAdmin" ) {
          $WantsAdmin = ${detail}.localAdmin
        } else {
          $WantsAdmin = $null
        }

        if ( ${WantsAdmin} -eq "true" ) {
          Create-User -UserUidName "$username" `
            -UserFullName ${FullName} `
            -UserPasswd ${detail}.initialPassword `
            -UserIsAdmin
        } else {
          Create-User -UserUidName "$username" `
            -UserFullName ${FullName} `
            -UserPasswd ${detail}.initialPassword
        }
      }
    }
  }
}

I understand that this is far from optimal way to do things. I hate how non-terse it feels like PowerShell makes me write things. However, I am not a PowerShell guy. It works. This is the first pass at it. I'll probably try to improve it in future iterations. 

I still hold out hope that we can convince their to use a Linux-based "jump box" (since, as mentioned early, their production environment is wholly Linux and Kubernetes-based) …At which point, I'll probably be tasked with figuring out how to RDP-enable a box with similar tooling and user-access. 

Getting the Most Out of EC2 Transfers With Private S3 Endpoints

Recently, I was given a project to help a customer migrate an on-premises GitLab installation into AWS. The current GitLab was pretty large: a full export of the configuration was nearly 500GiB in size.

It turned out a good chunk of that 500GiB was due to disk-hosted artifacts and LFS objects. Since I was putting it all into AWS, I opted to make use of GitLab's ability to store BLOBs in S3. Ultimately, that turned out to be nearly 8,000 LFS objects and nearly 150,000 artifacts (plus several hundred "uploads").

The first challenge was getting the on-premises data into my EC2. Customer didn't want to give me access to their on-premises network, so I needed to have them generate the export TAR-file and upload it to S3. Once in S3, I needed to get it into an EC2.

Wanting to make sure that the S3→EC2 task was as quick as possible, I selected an instance-type rated to 12.5Gbps of network bandwidth and 10Gbps of EBS bandwidth. However, my first attempt at downloading the TAR-file from S3 took nearly an hour to run: it was barely creeping along at 120MiB/s. Abysmal.

I broke out `iostat` and found that my target EBS was reporting 100% utilization and a bit less than 125MiB/s of average throughput. That seemed "off" to me, so I looked at the EBS. Was then that I noticed that the default volume-throughput was only 125MiB/s. So, I upped the setting to its maximum: 1000MiB/s. I re-ran the transfer only to find that, while the transfer-speed had improved, it had only improved to a shade under 150MiB/s. Still abysmal.

So, I started rifling through the AWS documentation to see what CLI settings I could change to improve things. First mods were:

max_concurrent_requests = 40
multipart_chunksize = 10MB
multipart_threshold = 10MB

This didn't really make much difference. `iostat` was showing really variable utilization-numbers, but mostly that my target-disk was all but idle. Similarly, `netstat` was showing only a handful of simultaneous-streams between my EC2 and S3.

Contacted AWS support. They let me know that S3 multi-part upload and download was limited to 10,0000 chunks. So, I did the math (<FILE_SIZE> / <MAX_CHUNKS>) and changed the above to:

max_concurrent_requests = 40
multipart_chunksize = 55MB
multipart_threshold = 64MB

This time, the transfers were running about 220-250MiB/s. While that was a 46% throughput increase, it was still abysmal. While `netstat` was finally showing the expected number of simultaneous connections, my `iostat` was still saying that my EBS was mostly idle.

Reached back out to AWS support. They had the further suggestion of adding:

preferred_transfer_client = crt
target_bandwidth = 10GB/s

To my S3 configuration. Re-ran my test and was getting ≈990MiB/s of continuous throughput for the transfer! This knocked the transfer speed down from fifty-five minutes to a shade over eight minutes. In other words, I was going to be able to knock nearly an hour off the upcoming migration-task.

In digging back through the documentation, it seems that, when one doesn't specify a preferred_transfer_client value, the CLI will select the `classic` (`python`) client. And, depending on your Python version, the performance ranges from merely-horrible to ungodly-bad: using RHEL 9 for my EC2, it was pretty freaking bad, but had been less-bad when using AWS for my EC2's OS. Presumably a difference in the two distro's respective Python versions?

Specifying a preferred_transfer_client value of `crt` (C run-time client) unleashed the full might and fury of my EC2's and GP3's capabilities.

Interestingly, this "use 'classic'" behavior isn't a universal auto-selection. If you've selected an EC2 with any of the instance-types:

• p4d.24xlarge
• p4de.24xlarge
• p5.48xlarge
• trn1n.32xlarge
• trn1.32xlarge

The auto-selection gets you `crt`. Not sure why `crt` isn't the auto-selected value for Nitro-based instance-types. But, "it's what it's". 

Side note: just selecting `crt` probably wouldn't have completely roided-out the transfer. I assume the further setting of `target_bandwidth` to `10GB/s` probably fully-unleashed things. There definitely wasn't much bandwidth leftover for me to actually monitor the transfer. I assume that the `target_bandwidth` parameter has a default value that's less than "all the bandwidth". However, I didn't actually bother to verify that.

Update: 

After asking support "why isn't `crt` the default for more instance-types", I got back the reply:

Thank you for your response. I see that these particular P5, P4d and Trn1 instances are purpose built for high-performance ML training¹. Hence I assume the throughput needed for this ML applications needs to high and CRT is auto enabled for these instance types.

Currently, the CRT transfer client does not support all of the functionality available in the classic transfer client.

These are few limitations for CRT configurations²:

• Region redirects - Transfers fail for requests sent to a region that does not match the region of the targeted S3 bucket.
• max_concurrent_requests, max_queue_size, multipart_threshold, and max_bandwidth configuration values - Ignores these configuration values.
• S3 to S3 copies - Falls back to using the classic transfer client


All of which is to say that, once I set `preferred_transfer_client = crt` all of my other, prior settings got ignored.

Keeping It Clean: EKS and `kubectl` Configuration

Previously, I was worried about, "how do I make it so that kubectl can talk to my EKS clusters".  However, after several days of standing up and tearing down EKS clusters across a several accounts, I discovered that my ~/.kube/config file had absolutely exploded in size and its manageability reduced to all but zero. And, while aws eks update-kubeconfig --name <CLUSTER_NAME> is great, its lack of a `--delete` suboption is kind of horrible when you want or need to clean out long-since-deleted clusters from your environment. So, onto "next best thing", I guess…

Ultimately, that "next best thing" was setting a KUBECONFIG environment-variable as part of my configuration/setup tasks (e.g., something like `export KUBECONFIG=${HOME}/.kube/config.d/MyAccount.conf`). While not as good as I'd like to think a `aws eks update-kubeconfig --name <CLUSTER_NAME> --delete would be, it at least means that:

1. Each AWS account's EKS's configuration-stanzas are kept wholly separate from each other
2. Reduces cleanup to simply overwriting – or straight up nuking – per-account ${HOME}/.kube/config.d/MyAccount.conf files

…I tend to like to keep my stuff "tidy". This kind of configuration-separation facilitates scratching that (OCDish) itch. 

The above is derived, in part, from the Organizing Cluster Access Using kubeconfig Files document

Crib Notes: Accessing EKS Cluster with `kubectl`

While AWS does provide a CLI tool – eksctl –for talking to EKS resources, it's not suitable for all Kubernetes actions one might wish to engage in. Instead, one must use the more-generic access provided through the more-broadly used tool, kubectl. Both tools will generally be needed, however.

If, like me, your AWS resources are only reachable through IAM roles – rather than IAM user credentials – it will be necessary to use the AWS CLI tool's eks update-kubeconfig subcommand. The general setup workflow will look like:

1. Set up your profile definition(s)
2. Use the AWS CLI's sso login to authenticate your CLI into AWS (e.g., `aws sso login --no-browser`)
3. Verify that you've successfully logged in to your target IAM role (e.g., `aws sts get-caller-identity` …or any AWS command, really)
4. Use the AWS CLI to update your ~/.kube/config file with the `eks update-kubeconfig` subcommand (e.g., `aws eks update-kubeconfig --name thjones2-test-01`)
5. Validate that you're able to execute kubectl commands and get back the kind of data that you expect to get (e.g., `kubectl get pods --all-namespaces` to get a list of all running pods in all namespaces within the target EKS cluster)
   

So You Work in Private VPCs and Want CLI Access to Your Linux EC2s?

 Most of the AWS projects I work on, both currently and historically, have deployed most, if not all, of their EC2s into private VPC subnets. This means that, if one wants to be able directly login to their Linux EC2s' interactive shells, they're out of luck. Historically, to get something akin to direct access one had to set up bastion-hosts in a public VPC subnet, and then jump through to the EC2s one actually wanted to login to. How well one secured those bastion-hosts could make-or-break how well-isolated their private VPC subnets – and associated resources – were.

If you were the sole administrator or part of a small team, or were part of an arbitrary-sized administration-group that all worked from a common network (i.e., from behind a corporate firewall or through a corporate VPN), keeping a bastion-host secure was fairly easy. All you had to do was set up a security-group that allowed only SSH connections and only allowed them from one or a few source IP addresses (e.g. your corporate firewall's outbound NAT IP address). For a bit of extra security, one could eve prohibit password-based logins on the Linux bastions (instead, using SSH key-based login, SmartCards, etc. for authenticating logins). However, if you were a member of a team of non-trivial size and your team members were geographically-distributed, maintaining whitelists to protect bastion-hosts could become painful. That painfulness would be magnified if that distributed team's members were either frequently changing-up their work locations or were coming from locations where their outbound IP address would change with any degree of frequency (e.g., work-from-home staff whose ISPs would frequently change their routers' outbound IPs).

A few years ago, AWS introduced SSM and the ability to tunnel SSH connections through SSM (see the re:Post article for more). With appropriate account-level security-controls, the need for dedicated bastion-hosts and maintenance of whitelists effectively vanished. Instead, all one had to do was:

• Register an SSH key to the target EC2s' account
• Set up their local SSH client to allow SSH-over-SSM
• Then SSH "directly" to their target EC2s

SSM would, effectively, "take care of the rest" …including logging of connections. If one were feeling really enterprising, one could enable key-logging for those SSM-tunneled SSH connections (a good search-engine query should turn up configuration guides; one such guide is toptal's). This would, undoubtedly make your organization's IA team really happy (and may even be required depending on security-requirements your organization is legally-required to adhere to) – especially if they don't yet have an enterprise session-logging tool purchased.

But what if your EC2s are hosting applications that require GUI-based access to set up and/or administer? Generally, you have two choices:

• X11 display-redirection
• SSH port-forwarding

Unfortunately, SSM is a fairly low-throughput solution. So, while doing X11 display-redirection from an EC2 in a public VPC subnet may be more than adequately performant, the same cannot be said when done through an SSH-over-SSM tunnel. Doing X11 display-redirection of a remote browser session – or, worse, an entire graphical desktop session (e.g., KDE or Gnome desktops) – is paaaaaainfully slow. For my own tastes, it's uselessly slow. 

Alternately, one can use SSH port-forwarding as part of that SSH-over-SSM session. Then, instead of trying to send rendered graphics over the tunnel, one only sends the pre-rendered data. It's a much lighter traffic load with the result being a much quicker/livelier response. It's also pretty easy to set up. Something like:

ssh -L localhost:8080:$(
  aws ec2 describe-instances \
    --query 'Reservations[].Instances[].PrivateIpAddress' \
    --output text \
    --instance-ids <EC2_INSTANCE_ID>
):80 <USERID>@<EC2_INSTANCE_ID>

Is all you need. In the above, the argument to the -L flag is saying, "set up a tcp/8080 listener on my local machine and have it forward connections to the remote machine's tcp/80". The local and remote ports can be varied for your specific needs. You can even set up dynamic-forwarding by creating a SOCKS proxy (but this document is meant to be a starting point, not dive into the weeds).

Note that, while the above is using a subshell (via the $( … ) shell-syntax) to snarf the remote EC2's private IP address, one should be able to simply substitute "localhost". I simply prefer to try to speak to the remote's ethernet, rather than loopback, interface, since doing so can help identify firewall-type issues that might interfere with others' use of the target service.

TIL: I Am Probably Going To Come To Hate `fapolicyd`

One of the things I do in my role is write security automation. Part of that requires testing systems' hardening-compliance each time one of the security-benchmarks my customers use is updated.

In a recent update, the benchmarks for Red Hat Enterprise Linux 8 (and derivatives) added the requirement to enable and run the application-whitelisting service, fapolicyd. I didn't immediately notice this change …until I went to offload the security-scans from my test EC2 to an S3 bucket. The AWS CLI was suddently broken.

Worse, it was broken in an absolutely inscrutable way: if one executed and  AWS CLI command, even something as simple and basic as `aws help`, it would immediately return having neither performed the requested action nor emitting an error. As an initial debug attempt, I did:

echo $( aws help )$?

Which got me the oh-so-useful `255` for my troubles. But, it did allow me to start about to Googling. My first hit was this guy. It had the oh-so-helpful guidance:

• 255 -- Command failed. There were errors from either the CLI or the service the request was made to.

Like I said: "oh-so-helpful guidance".

So, I opened a support request with Amazon. Initially, they were at least as in the dark as I was. 

Fortunately, another member of the team I work on noticed the support case when it came into his Inbox via our development account's auto-Cc configuration. Unlike me, he, apparently, hasn't deadmailed everything sent to that distro (which, given how much is sent to that distro, deadmailing anything that arrived through the distro was the only way to preserve my sanity). He'd indicated that he had previously had similar issues and that he got around them by disabling the fapolicyd service. I quickly tested stopping the service and the AWS CLI happily resumed functioning as it had before the hardening tasks had executed.

I knew that wholly disabling the service was not going to be acceptable to our cyber-defense team. But, knowing where the problem originated meant I had a useful investigatory path.

The first (seeming) solution I found was to execute something like:

fapolicyd-cli --file add /usr/local/bin/aws
fapolicyd -u

This allowed both the AWS CLI to function and for the fapolicyd service to continue to be run.

For better or worse, though, I'm a curious type. I wanted to see what the rule looked like that the fapolicyd-cli utility had created. So, I dug around the documentation to find where I might be able to eyeball the exception.

# AUTOGENERATED FILE VERSION 2
# This file contains a list of trusted files
#
#  FULL PATH        SIZE                             SHA256
# /home/user/my-ls 157984 61a9960bf7d255a85811f4afcac51067b8f2e4c75e21cf4f2af95319d4ed1b87
/usr/local/aws-cli/v2/2.11.6/dist/aws 6658376 c48f667b861182c2785b5988c5041086e323cf2e29225da22bcd0f18e411e922

Which immediately rung alarm bells in my skull (strong "danger Will Robinson"  vibes). By making the exception not only conditional on the binary's (real) path, but also to its size and, particularly, its SHA256 signature, I knew that if anyone ever updated the installed binary, their exception-description was no longer going to match. This in turn would mean that the utility would stop working. Not wanting to deal with tickets that I could easily prevent, I continued my investigation.

Knowing that what I actually wanted was to give a blanket exemption to everything under the "/usr/local/aws-cli/v2" directory, I started investigating how to do that. Ultimately, I came up with a exeption-set that looked like:

allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1

I saved the contents as `/etc/fapolicyd/rules.d/30-aws.rules`and reloaded the `fapolicyd` configuration. However, I was sadly disappointed to discover that the AWS CLI was still broken. On the plus side, it was broken differently. Now, instead of immediately and silently exiting (with a 255 error-code), it was giving me:

[2030] Error loading Python lib '/usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0':
dlopen: /usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0: cannot open shared object
file: Operation not permitted

Much more meat to chew on. Further searching later, I had two additional commands to help me in my digging:

ausearch --start today -m fanotify --raw | aureport --file --summary

Which gave me:

File Summary Report
===========================
total  file
===========================
16  /usr/local/aws-cli/v2/2.11.6/dist/libz.so.1
16  /usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0
2  /usr/local/aws-cli/v2/2.11.6/dist/aws

And:

fapolicyd-cli --list

Which gave me:

-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,
application/java-archive,text/x-java,application/x-java-applet,application/javascript,
text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,
text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,
text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
 1. allow perm=any uid=0 : dir=/var/tmp/
 2. allow perm=any uid=0 trust=1 : all
 3. allow perm=open exe=/usr/bin/rpm : all
 4. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
 5. deny_audit perm=any pattern=ld_so : all
 6. deny_audit perm=any all : ftype=application/x-bad-elf
 7. allow perm=open all : ftype=application/x-sharedlib trust=1
 8. deny_audit perm=open all : ftype=application/x-sharedlib
 9. allow perm=execute all : trust=1
10. allow perm=open all : ftype=%languages trust=1
11. deny_audit perm=any all : ftype=%languages
12. allow perm=any all : ftype=text/x-shellscript
13. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
14. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
15. deny_audit perm=execute all : all
16. allow perm=open all : all

Which told me why the binary was at least trying to work, but was unable to load its shared-library. Since I'd named the file `/etc/fapolicyd/rules.d/80-aws.rules`, it was loading later than a rule that was preventing access to shared libraries not in standard trust-paths. In the above, it was the file that created rule #8.

I grepped through the `/etc/fapolicyd/rules.d/` directory looking for the file that created rule #8. Having found it, I moved my rule-file upwards with a quick `mv /etc/fapolicyd/rules.d/{8,3}0-aws.rules` and reloaded my rules. This time, my rules-list came up like:

-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,
application/java-archive,text/x-java,application/x-java-applet,application/javascript,
text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,
text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,
text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
 1. allow perm=any uid=0 : dir=/var/tmp/
 2. allow perm=any uid=0 trust=1 : all
 3. allow perm=open exe=/usr/bin/rpm : all
 4. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
 5. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
 6. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
 7. deny_audit perm=any pattern=ld_so : all
 8. deny_audit perm=any all : ftype=application/x-bad-elf
 9. allow perm=open all : ftype=application/x-sharedlib trust=1
10. deny_audit perm=open all : ftype=application/x-sharedlib
11. allow perm=execute all : trust=1
12. allow perm=open all : ftype=%languages trust=1
13. deny_audit perm=any all : ftype=%languages
14. allow perm=any all : ftype=text/x-shellscript
15. deny_audit perm=execute all : all
16. allow perm=open all : all

With my sharedlib allow-rule now ahead of the default-deny sharedlib rule, I tested out the AWS CLI command again. Success!

Unfortuantely, while I solved the problem I set out to solve, my `ausearch`  output was telling me that a few other standard tools were also likely having similar whitelisting issues. Ironically, those "other standard tools" are all security-related.

Fun fact: a number of security-vendors write their products for Windows, first and foremost. Their Linux tooling is almost an afterthought. As such, it's often not well-delivered: if they deliver their software in RPMs at all, the RPMs are often poorly-constructed. I almost never see signed RPMS from security vendors. When I do actually get signed RPMs, they're very rarely signed in a way that's compatible with a Red Hat system that's configured to run in FIPS-mode. So, I guess I shouldn't be super surprised that these same security-tools aren't aware of the need or how to work with fapolicyd. Oh well, that's someone else's problem (realistically, probably "future me's").

Crib Notes: Assuming a Role

Several of my current customers leverage AWS IAM's role-assumption capability. In particular, one of my customers leverages it for automating the execution of the Terragrunt-based IaC. For the automated-execution, they run the Terragrunt code from an EC2 that has an attached IAM role that allows code executed on the hosting-EC2 to assume roles in other accounts.

Sometimes, when writing updates to their Terragrunt code, it's helpful to be able to audit the target account's state before and after the execution, but outside the context of Terragrunt, itself. In these cases, knowing how to use the AWS CLI to switch roles can be quite handy. A quick one-liner template for doing so looks like:

$ eval "$(
  aws sts assume-role \
    --role-arn "arn:<AWS_PARTITION>:iam::<TARGET_ACCOUNT_NUMBEr>:role/<TARGET_ROLE_NAME>" \
    --role-session-name <userid> --query 'Credentials' | \
  awk '/(Key|Token)/{ print $0 }' | \
  sed -e 's/",$/"/' \
      -e 's/^\s*"/export /' \
      -e 's/": "/="/' \
      -e 's/AccessKeyId/AWS_ACCESS_KEY_ID/' \
      -e 's/SecretAccessKey/AWS_SECRET_ACCESS_KEY/' \
      -e 's/SessionToken/AWS_SESSION_TOKEN/'
)"

What the above does is:

1. Opens a subshell to execute a series of commands in
2. Executes `aws sts assume-role` to fetch credentials, in JSON format, for accessing the target AWS account as the target IAM role
3. Uses `awk` to select which parts of the prior command's JSON output to keep (`grep` or others are likely more computationally-efficient, but you get the idea)
4. Uses `sed` to convert the JSON parameter/value pair-strings into BASH-compatible environment-variable delcarations
5. Uses `eval` to take the output of the subshell and read it into the current shell's environment

Once this is executed, your SHELL will grant you privileges to execute commands in the target account – be that using the AWS CLI or any other tool that understands the "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY" and "AWS_SESSION_TOKEN" environment variables.

Using `aws sts get-caller-identity` will allow you to see your new IAM role.

You CAN Make the Logs' Format Worse?!

Most of my customers have security compliance mandates that makes it necessary to offload their Linux auditd event-logs to an external/centralized logging-destination. One of my customers leverages a third-party tool to offload their EC2 logs directly to S3. However, because they use a common compliance-framework to guide their EC2s' hardening configurations, they had been configuring their systems to use the "normal" auditd dispatch plugin service, audisp. Unfortunately, prior my arrival, no one had actually bothered to validate their auditing configuration. Turns out, audisp was trying to off-host the event logs to a non-existent, centralized event-collector that didn't actually exist.

My "solution" to their problem was to simply eliminate the CM-automation that sets up the errant event-offloading. However, before I could suggest summarily nuking this automation-content, I had to verify that their S3-based logging solution was actually working. So, decided to cobble together a quick-n-dirty AWS CLI command – because their log-stream names are the same as the associated EC2s' instance-IDs, doing the quick test was easy:

$ aws logs filter-log-events\
  --log-group-name  \
  --log-stream-names $(
    curl -s http://169.254.169.254/latest/meta-data/instance-id
  )   --start-time $(
    date -d '-15 minutes' '+%s%N' | \
    cut -b 1-13
  )

To explain the above - particularly the nested subshells…

Executing:

curl -s http://169.254.169.254/latest/meta-data/instance-id

Makes use of the EC2 metadata service to return the EC2s own AWS instance-ID. Because it's being executed in a subshell – using the $( COMMAND ) notation – The instance's ID is returned as the value fed to the `--log-stream-names` command-option.

Similarly, executing:

date -d '-15 minutes' '+%s%N' | cut -b 1-13

Makes use of the `date` command to return an `aws logs` compatible time specification for use by the --start-time command-option. Specifically, it tells the date command, "take the time 15 minutes ago and convert it to milliseconds since epoch-time. Because the time-shifted, date-string is longer than what the  `aws logs` command-option expects, we use the `cut` command to truncate it to a compatible length. Ultimately, this will return output similar to:

{
  "logStreamName": "i-0a8b14c42b651476f",
  "timestamp": 1664815386793,
  "message": "{\"a0\":\"b75480\",\"a1\":\"0\",\"a2\":\"0\",\"a3\":\"7ffeb692d220\",\"arch\":\"c000003e\",\"auid\":\"239203505\",\"az\":\"us-gov-west-1a\",\"comm\":\"vim\",\"ec2_instance_id\":\"i-0a8b14c42b651476f\",\"egid\":\"239203505\",\"euid\":\"239203505\",\"exe\":\"/usr/bin/vim\",\"exit\":\"0\",\"fsgid\":\"239203505\",\"fsuid\":\"239203505\",\"gid\":\"239203505\",\"items\":\"2\",\"key\":\"delete\",\"log_file\":\"/var/log/audit/audit.log\",\"node\":\"ip-10-244-0-100.dev.ac2sp.army.mil\",\"pid\":\"10503\",\"ppid\":\"3961\",\"ses\":\"3\",\"sgid\":\"239203505\",\"subj\":\"staff_u:staff_r:staff_t:s0-s0:c0.c1023\",\"success\":\"yes\",\"suid\":\"239203505\",\"syscall\":\"84\",\"tty\":\"pts1\",\"type\":\"SYSCALL\",\"uid\":\"239203505\"}",
  "ingestionTime": 1664815388879,
  "eventId": "37126623743463896942643741908611915331411397433463734292"
}

Kind of horrible. But, you can extract the actual message payload with a tool like `jq` which will give you output like:

{
    "a0": "b75480",
    "a1": "0",
    "a2": "0",
    "a3": "7ffeb692d220",
    "arch": "c000003e",
    "auid": "239203505",
    "az": "us-gov-west-1a",
    "comm": "vim",
    "ec2_instance_id": "i-0a8b14c42b651476f",
    "egid": "239203505",
    "euid": "239203505",
    "exe": "/usr/bin/vim",
    "exit": "0",
    "fsgid": "239203505",
    "fsuid": "239203505",
    "gid": "239203505",
    "items": "2",
    "key": "delete",
    "log_file": "/var/log/audit/audit.log",
    "node": "ip-10-244-0-100.dev.ac2sp.army.mil",
    "pid": "10503",
    "ppid": "3961",
    "ses": "3",
    "sgid": "239203505",
    "subj": "staff_u:staff_r:staff_t:s0-s0:c0.c1023",
    "success": "yes",
    "suid": "239203505",
    "syscall": "84",
    "tty": "pts1",
    "type": "SYSCALL",
    "uid": "239203505"
}

Yeah… If you're familiar with audit.log output, that ain't normally what it looks like. Here, the OS's event-data has been intermingled with the CSP's event log-tracking data. Not great. Not something you can feed "as-is" to generic tools that expect to directly interact with auditd data (e.g., the tools in the policyutils RPMs). But, you can write filters to get it back into a tool-compatible format. But still, it manages to make an ugly log format markedly-uglier to deal with. So, uh, congratulations?

Crib Notes: Quick Audit of EC2 Instance-Types

Was recently working on a project for a customer who was having performance issues. Noticed the customer was using t2.* for the problematic system. Also knew that I'd seen them using pre-Nitro instance-types on some other systems they'd previously complained about performance problems with. Wanted to put a quick list of "you might want to consider updating these guys" EC2s. Ended up executing:

$ aws ec2 describe-instances \
   --query 'Reservations[].Instances[].{Name:Tags[?Key == `Name`].Value,InstanceType:InstanceType}' \
   --output text | \
sed -e 'N;s/\nNAME//;P;D'

Because the describe-instances's command-output is multi-line – even with the applied --query filter – adding the sed filter was necessary to provide a nice, table-like output:

t3.medium       ingress.dev-lab.local
t2.medium       etcd1.dev-lab.local
m5.xlarge       k8snode.dev-lab.local
m6i.large       runner.dev-lab.local
t2.small        dns1.dev-lab.local
t3.medium       k8smaster.dev-lab.local
t2.medium       bastion.dev-lab.local
t3.medium       ingress.dev-lab.local
t2.medium       etcd0.dev-lab.local
m5.xlarge       k8snode.dev-lab.local
m6i.large       runner.dev-lab.local
m5.xlarge       k8snode.dev-lab.local
t2.xlarge       workstation.dev-lab.local
t2.medium       proxy.dev-lab.local
t2.small        dns0.dev-lab.local
t3.medium       ingress.dev-lab.local
t2.medium       etcd2.dev-lab.local
m5.xlarge       k8snode.dev-lab.local
t2.medium       mail.dev-lab.local
m6i.large       runner.dev-lab.local
t2.small        dns2.dev-lab.local
t3.medium       k8smaster.dev-lab.local
t2.medium       bastion.dev-lab.local
t2.medium       proxy.dev-lab.local

Dense Coding and Code Explainers

My role with my employer is frequently best described as "smoke jumper". That is, a given project they're either priming or, more frequently, subbing on is struggling and the end customer has requested further assistance getting things more on the track they were originally expecting to be on. How I'm usually first brought onto such projects is automation-support "surge".

In this context, "surge" means resources brought in to either fill gaps in the existing project-team created by turnover or augmenting that team with additional expertise. Most frequently, that's helping them either update and improve existing automation or write new automation. Either way, the code I tend to deliver tends to be both fairly compact and dense as well as flexible compared to what they've typically delivered to date.

One of my first-principles in delivering new functionality is to attempt to do so in a way that is easily deactivated or backed out. This project team, like others I've helped, uses Terraform, but in a not especially modular or function-isolating way. All of the deployments consist of main.tf, vars.tf and outputs.tf files and occational "template" files (usually simple HERE documents with simple variable-substitution actions). While they do (fortunately) make use of some data providers, they're not real disciplined about where they implement them. They embed them in either or both of a given service's the main.tf and vars.tf files. Me? I generally like all of my data-providers in data.tf type of files as it aids consistency and keeps the type of contents in the various, individual files "clean" from an offered-functionality standpoint.

Similarly, if I'm using templated content, I prefer to deliver it in ways that sufficiently externalizes the content to allow appropriate linters to be run on it. This kind of externalization not only allows such files to be more easily linted but, because it tends to remove encapsulation effects, it tends to make either debugging or extending the externalized content easier to do.

On a recent project, I was tasked with helping them automate the deployment of VPC endpoints into their AWS accounts. The customer was trying, to the greatest extent possible, prevent as much of their project-traffic from leaving their VPCs as possible.

When I started the coding-task, the customer wasn't able to tell me which specific services they wanted or needed so-enabled. Knowing that each such service-endpoint comes with recurring costs and not wanting them to accidentally break the bank, I opted to write my code in a way that, absent operator input, would deploy all AWS endpoint services into their VPCs but also easily allow them to easily dial things back when the first (shocking) bills came due.

The code I delivered worked well. However, as familiar with the framework as the incumbent team was, they were left a bit perplexed by the code I delivered. They asked me to do a walkthrough of the code for them. Knowing the history of the project – both from a paucity-of-documentation and staff-churn perspective – I opted to write an explainer document. What follows is that explanation.

Firstly, I delivered my contents as four, additional files rather than injecting my code into their existing  main.tf, vars.tf and outputs.tf file-set. Doing so allowed them to wholly disable functionality simply by nuking the files I delivered rather than having to do file-surgery on their normal file-set. As my customer is operating in multiple AWS partitions, this makes dealing with partitions' API differences easier to roll back changes if the deployment-partition's APIs are older then their development-partition's are. The file-set I delivered was an endpoints_main.tf, endpoints_data.tf,  endpoints_vars.tf and an endpoints_services.tpl.hcl file. Respectively these files encapsulate: primary functionality; data-provider definitions; definition of variables used in the "main" and "data" files; and an HCL-formatted default-endpoints template-file.

The most basic/easily-explained file is the default-endpoints template file, endpoints_services.tpl.hcl. The file consists of map-objects encapsulated in a larger list structure. The map-objects consist of name and type attribute-pairs. The name values were derived by executing:

aws ec2 describe-vpc-endpoint-services \
  --query 'ServiceDetails[].{Name:ServiceName,Type:ServiceType[].ServiceType}' | \
sed -e '/\[$/{N;s/\[\n *"/"/;}' -e '/^[        ][      ]*]$/d' | \
tr '[:upper:]' '[:lower:]'

And then changing the literal region-names to "${endpoint-region}". This change allows Terraform's templatefile() function to sub in the desired-value when the template file is read – making the automation portable across both regions and partitions. The template-file's contents are also encapsulate with Terraform's jsondecode() function. This encapsulation is necesary to allow the templatefile() function to properly read the file in (so that the variable-substitution can occur).

Because I wanted the use of this template-file to be the fallback (default) behavior, I needed to declare its use as a fallback. This was done in the endpoint_data.tf file's locals {} section:

locals {
  vpc_endpoint_services = length(var.vpc_endpoint_services) == 0 ? jsondecode(
    templatefile(
      "./endpoint_services.tpl.hcl",
      {
        endpoint_region = var.region
      }
    )
  ) 
}  

In the above, we're using a ternary evaluation to set the value of the locally-scoped vpc_endpoint_services variable. If the size of the globally-scoped vpc_endpoint_services variable is "0", then the template file is used; otherwise, the content of the globally-scoped vpc_endpoint_services variable is used. The template-file's use is effected by using the templatefile() function to read the file in while substituting all occurrences of "${endpoint-region}" in the file with the value of the globally-scoped "region" variable.

Note: The surrounding jsondecode() function is used to convert the file-stream from the format previously set using the jsonencode() function at the beginning of the file. I'm not a fan of having to resort to this kind of kludgery, but, without it, the templatefile() function would error out when trying to populate the vpc_endpoint_services variable. If any reader has a better idea of how to attain the functionality desired in a less-kludgey way, please comment.

Where my customer needed the most explanation was the logic in the section:

data "aws_vpc_endpoint_service" "this" {
  for_each = {
    for service in local.vpc_endpoint_services :
    "${service.name}:${service.type}" => service
  }

  service_name = length(
    regexall(
      var.region,
      each.value.name
    )
  ) == 1 ? each.value.name : "com.amazonaws.${var.region}.${each.value.name}"
  service_type = title(each.value.type)
}

This section leverages Terraform's aws_vpc_endpoint_service data-source. My code gives it the reference id "this". Not a terribly original or otherwise noteworthy label, but, absent the need for multiple such references, it will do.

The for_each function iterates over the values stored in the locally-scoped vpc_endpoint_services object-variable. As it loops, assigns each dictionary-object – the name and type attribute-pairs – to the service loop-variable. In turn, the loop iteratively-exports an each.value.name and each.value.type variable.

I could have set the service_name variable to more-simply equal the each.value.name variable's value, however, I wanted to make life a bit less onerous for the automation-user. Instead of needing to specify the full service-name path-string, the short-name could be specified. Using the regexall() function to see if the value of the region globally-scoped variable was present in the each.value.name variable's value allows the length() function to be used as part of a ternary definition for the service_name variable. If returned length is "0", the operator-passed service-name is prepended with the fully-qualified service-path typically valid for the partition's region; if the returned length is "1", then the value already stored in the each.value.name variable is used. 

Similarly, I didn't want the operator to need to care about the case of the service-type they were specifying. As such, I let Terraform's title(function) take care of setting the proper case of the each.value.type variable's value.

The service_type and service_name values are then returned when the data-provider is called from the endpoint_main.tf file's locals {} block is processed:

locals {
  […elided…]
  # Split Endpoints by their type
  gateway_endpoints = toset(
    [
      for e in data.aws_vpc_endpoint_service.this :
      e.service_name if e.service_type == "Gateway"
    ]
  )
  interface_endpoints = toset(
    [
      for e in data.aws_vpc_endpoint_service.this :
      e.service_name if e.service_type == "Interface"
    ]
  )
  […elided…]
}

The gateway_endpoints and interface_endpoints locally-scoped variables are each list-variables. Each is populated by taking the service-name returned from the data.aws_vpc_endpoint_service.this data-provider if the selected-for service_type value matches. This list-vars are then iteratively-processed in the relevant resource "aws_vpc_endpoint" "interface_services" and resource "aws_vpc_endpoint" "gateway_services" stanzas.

What Permissions Do I Need

 In recent months, I've been converting some automation I originally wrote under CloudFormation to instead work under Terraform. Ultimately, the automation I wrote is going to be used in a different account than I (re)developed it in. As part of the customer's "least-privileges" deployment model, I needed to be able to specify to them all of the specific AWS IAM permissions that my TerraForm-based automation would need. Since the development account I've been working in doesn't provide me CloudTrail or other similarly-useful access, I had to find another way. Turns out, that "another way" is effectively built into Terraform, itself!

When one uses the TF_LOG=trace environment-variable, the activity-logging becomes very verbose. Burried amongst the storm of output is all of the IAM permissions that Terraform needs in order to perform its deployment, configuration and removal actions. Extracting it all was a matter of:

1. Execute a `terraform apply` using:
     TF_LOG=trace terraform apply -autoapprove > apply.log
2. Execute a `terraform apply` using:
     TF_LOG=trace terraform apply --autoapprove \
       -refresh-only > refresh.log`
3. Execute a `terraform apply` using:
     TF_LOG=trace terraform destroy -autoapprove > destroy.log

Once each of the above completes successfully, one has three looooong output files. To extract the information (and put it in a format IAM administrators are more used to), a simple set of filters can be applied:

cat *.log | \
grep 'DEBUG: Request ' | \
sed -e 's/.*: Request//' \
    -e 's/ Details:.*$//' \
    -e 's#/#:#' | \
sort -u

This filter-set gives you a list that looks something like:

ec2:AuthorizeSecurityGroupEgress
 ec2:AuthorizeSecurityGroupIngress
 ec2:CreateSecurityGroup
 ec2:DescribeImages
 ec2:DescribeInstanceAttribute
 ec2:DescribeInstanceCreditSpecifications
 ec2:DescribeInstances
 ec2:DescribeSecurityGroups
 ec2:DescribeTags
 ec2:DescribeVolumes
 ec2:DescribeVpcs
 ec2:RevokeSecurityGroupEgress
 ec2:RunInstances
 elasticloadbalancing:AddTags
 elasticloadbalancing:CreateListener
 elasticloadbalancing:CreateLoadBalancer
 elasticloadbalancing:CreateTargetGroup
 elasticloadbalancing:DescribeListeners
 elasticloadbalancing:DescribeLoadBalancerAttributes
 elasticloadbalancing:DescribeLoadBalancers
 elasticloadbalancing:DescribeTags
 elasticloadbalancing:DescribeTargetGroupAttributes
 elasticloadbalancing:DescribeTargetGroups
 elasticloadbalancing:ModifyLoadBalancerAttributes
 elasticloadbalancing:ModifyTargetGroup
 elasticloadbalancing:ModifyTargetGroupAttributes
 elasticloadbalancing:SetSecurityGroups
 s3:GetObject
 s3:ListObjects

Which you can then pass on to the parties that set up your IAM roles.

Working Around Errors Caused By Poorly-Built AMIs (Networking Edition)

Over the past several years, the team I work on created a set of provisioning-automation tools that we've used with/for a NUMBER of customers. The automation is pretty well designed to run "anywhere".

Cue current customer/project. They're an AWS-using customer. They maintain their own AMIs. Unfortunately, our automation would break during the hardening phase of the deployment automation. After a waste of more than a man-day, discovered the root cause of the problem: when they build their EL7 AMIs, they don't do an adequate cleanup job.

Discovered that there were spurious ifcfg-* files in the resultant EC2s' /etc/sysconfig/network-scripts directory. Customer's AMI-users had never really noticed this oversight. All they really knew was that "networking appears to work", so had never noticed that the network.service systemd unit was actually in a fault state. Whipped out journalctl to find that the systemd unit was attempting to online interfaces that didn't exist on their EC2s ...because, while there were ifcfg-* files present, corresponding interface-directories in /sys/class/net didn't actually exist.

Because our hardening tools, as part of ensuring that network-related hardenings all get applied, does (the equivalent of) systemctl restart network.service. Unfortunately, due to the aforementioned problem, this action resulted in a non-zero exit. Consequently, our tools were aborting.

So, how to pre-clean the system so that the standard provisioning automation would work? Fortunately, AWS lets you inject boot-time logic via cloud-init scripts. I whipped up a quick script to eliminate the superfluous ifcfg-* files:  

for IFFILE in $( echo /etc/sysconfig/network-scripts/ifcfg-* )
do
   [[ -e /sys/class/net/${IFFILE//*ifcfg-/} ]] || (
      printf "Device %s not found. Nuking... " "${IFFILE//*ifcfg-/}" &&
      rm "${IFFILE}" || ( echo FAILED ; exit 1 )
      echo "Success!"
   )
done

Launched a new EC2 with the userData addition. When the "real" provisioning automation ran, no more errors. Dandy.

Ugh... Hate having to kludge to work around error-conditions that simply should not occur.

Taming the CUDA

Recently, I was placed on a new contract supporting a data science project. I'm not doing any real data-science work, simply improving the architecture and automation of the processes used to manage and deploy their data-science tooling.

Like most of my customers, the current customer is an Enterprise Linux shop and an AWS shop. Amazon makes available several GPU-enabled instance-types that are well-disposed to running data science types of tasks. And, while RHEL is generically suitable to running on GPU-enabled instance types, to get the best performance out of them, you need to run the GPU drivers published by the GPU-vendor rather than the ones bundled with RHEL.

Unfortunately, as third-party drivers, there's some gotchas with using them. The one they'd been most-plagued by was updating drivers as the GPU-vendor made further updates available. While doing a simple `yum upgrade` works for most packagings, it can be problematic when using third-party drivers. When you try to do `yum upgrade` (after having ensured the new driver-RPMs are available via `yum`), you'll ultimately get a bunch of dependency errors due to the driver DSOs being in use.

Ultimately, what I had to move to was a workflow that looked like:

1. Uninstall the current GPU-driver RPMs
2. Install the new GPU-driver RPMs
3. Reboot

Unfortunately, "uninstall the current GPU-driver RPMs" actually means "uninstall just the 60+ RPMs that were previously installed ...and nothing beyond that. And, while I could have done something like `yum uninstall <DRIVER_STUB-NAME>`, doing so would result in more packages being removed than I intended.

Fortunately, RHEL (7+) include a nice option with the `yum` package-management utility: `yum history undo <INSTALL_ID>`.  

Due to the data science users individual EC2s being of varying vintage (and launched from different AMIs), the value of <INSTALL_ID> is not stable across their entire environment.

The automation gods giveth; the automation gods taketh away.

That said, there's a quick method to make the <INSTALL_ID> instability pretty much a non-problem:

yum history undo $( yum history info <rpm_name>| \
   sed -n '/Transaction ID/p' | \
   cut -d: -f 2 )

Which is to say "Undo the yum transaction-ID returned when querying the yum history for <rpm_name>". Works like a champ and made the overall update process go very smoothly.

Now to wrap it up within the automation framework they're leveraging (Ansible). I don't think it natively understands the above logic, so, I'll probably have to shell-escape to get step #1 done.

TIL: You Gotta Be Explicit

Started working on a new contract, recently. This particular customer makes use of S3FS. To be honest, in the past half-decade, I've had a number of customers express interest in S3FS, but they're pretty much universally turned their noses up at it (due to any number of reasons that I can't disagree with — trying to use S3 like a shared filesystem is kind of horrible).

At any rate, this customer also makes use of Ansible for their provisioning automation. One of their "plays" is designed to mount the S3 buckets via s3fs. However, the manner in which they implemented it seemed kind of jacked to me: basically, they set up a lineinfile-based play to add to add s3fs commands to the /etc/rc.d/rc.local file, and then do a reboot to get the filesystems to mount up.

It wasn't a great method, to begin with, but, recently, their their security people made a change to the IAM objects they use to enable access to the S3 buckets. It, uh, broke things. Worse, because of how they implemented the s3fs-related play, there was no error trapping in their work-flow. Jobs that relied on /etc/rc.d/rc.local having worked started failing with no real indication as to why (when you pull a file directly from S3 rather than an s3fs mount, things are pretty immediately obvious what's going wrong).

At any rate, I decided to try to see if there might be a better way to manage the s3fs mounts. So, I went to the documentation. I wanted to see if there was a way to make them more "managed" by the OS such that, if there was a failure in mounting, the OS would put a screaming-halt to the automation. Overall, if I think a long-running task is likely to fail, I'd rather it fail early in the process than after I've been waiting for several minutes (or longer). So I set about simulating how they were mounting S3 buckets with s3fs.

As far as I can tell, the normal use-case for mounting S3 buckets via s3fs is to do something like:

s3fs <bucket> <mount> -o <OPTIONS>

However, they have their buckets cut up into "folders" and sub-folders and wanted to mount them individually. The s3fs documentation indicated that you could both mount individual folders and that you could do it via /etc/fstab. You simply needed an /etc/fstab that looks sorta like:

s3fs-build-bukkit:/RPMs    /provisioning/repo       fuse.s3fs    _netdev,allow_other,umask=0000,nonempty 0 0
s3fs-build-bukkit:/EXEs    /provisioning/installer  fuse.s3fs    _netdev,allow_other,umask=0000,nonempty 0 0
s3fs-users-bukkit:/build   /Data/personal           fuse.s3fs    _netdev,allow_other,umask=0000,nonempty 0 0

However, I was finding that, even though the mount-requests weren't erroring, they also weren't mounting. So, hit up the almighty Googs and found an issue-report in the S3FS project that matched my symptoms. The issue ultimately linked to a (poorly-worded) FAQ-entry. In short, I was used to implicit "folders" (ones that exist by way of an S3 object containing a slash-delimited key), but s3fs relies on explicitly-created "folders" (e.g., null objects with key-names that end in `/` — as would be created by doing `aws s3api put-object --bucket s3fs-build-bukkit --key test-folder/`). Once I explicitly created these trailing-slash null-objects, my /etc/fstab entries started working the way the documentation indicated they ought to have been doing all along.

Crib-Notes: EC2 UserData Audit

Sometimes, I find that I'll return to a customer/project and forget what's "normal" for them in how they deploy their EC2s. If I know a given customer/project tends to deploy EC2s that include UserData, but they don't keep good records of what they tend to do for said UserData, I find the following BASH scriptlet to be useful for getting myself back into the swing of things:

for INSTANCE in $( aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' | \
                   sed -e '/^\[/'d -e '/^]/d' -e 's/^ *"//' -e 's/".*//' )
do
   printf "###############\n# %s\n###############\n" "${INSTANCE}"
   aws ec2 describe-instance-attribute --instance-id "${INSTANCE}" --attribute userData | \
   jq -r .UserData.Value | base64 -d
   echo
done | tee /tmp/DiceLab-EC2-UserData.log

To explain, what the above does is:

1. Initiates a for-loop using ${INSTANCE} as the iterated-value
2. With each iteration, the value injected into ${INSTANCE} is derived from a line of output from the aws ec2 describe-instances command. Normally, this command outputs a JSON document containing a bunch of information about each instance in the account-region. Using the --query option, the output is constrained to only output each EC2 instance's InstanceId value. This is then piped through sed so that the extraneous characters are removed, resulting in a clean list of EC2 instance-IDs.
3. The initial printf line creates a bit of an output-header. This will make it easier to pore through the output and keep each iterated instance's individual UserData content separate
4. Instance UserData is considered to be an attribute of a given EC2 instance. The aws ec2 describe-instance-attribute command is what is used to actually pull this content from the target EC2. I could have used a --query filter to constrain my output. However, I instead chose to use jq as it allows me to both constrain my output as well as do output-cleanup, eliminating the need for the kind of complex sed statement I used in the loop initialization (cygwin's jq was crashing this morning when I was attempting to use it in the loop-initialization phase - in case you were wondering about the inconsistent constraint/cleanup methods). Because the UserData output is stored as a BASE64-encoded string, I have to pipe the cleaned-up output through the base64 utility to get my plain-text data back.
5. I inject a closing blank line into my output stream (via the echo command) to make the captured output slightly easier to scan.
6. I like to watch my scriptlet's progress, but still like to capture that output into a file for subsequent perusal, thus I pipe the entire loop's output through tee so I can capture as I view.

I could have set it up so that each instance's data was dumped to an individual output-file. This would have saved the need for the printf and echo lines. However, I like having one, big file to peruse (rather than having to hunt through scads of individual files) ...and a single file-open/close action is marginally faster than scads of open/closes.

In an account-region that had hundreds of EC2s, I'd probably have been more selective with which instance-IDs I initiated my loop. I would have used a --filter statement in my aws ec2 describe-instances command - likely filtering by VPC-ID and one or two other selectors.

Crib-Notes: Offline Delta-Syncs of S3 Buckets

In the normal world, synchronizing two buckets is as simple as doing `aws s3 sync <SYNC_OPTIONS> <SOURCE_BUCKET> <DESTINATION_BUCKET>`. However, due to the information security needs of some of my customers, it's occasionally necessary to perform data-synchronizations between two S3 buckets, but using methods that amount to "offline" transfers.

To illustrate what is meant by "offline":

1. Create a transfer-archive from a data source
2. Copy the transfer-archive across a security boundary
3. Unpack the transfer-archive to its final destination

Note that things are a bit more involved than the summary of the process – but this gives you the gist of the major effort-points.

The first time you do an offline bucket sync, transferring the entirety of a bucket is typically the goal. However, for a refresh-sync – particularly for a bucket of greater than a trivial content-size, this can be sub-ideal. For example, it might be necessary to do monthly syncs of a bucket that grows by a few Gigabytes per month. After a year, a full sync can mean having to move tens to hundreds of gigabytes. A better way is to only sync the deltas – copying only what's changed between the current and immediately-prior sync-tasks (a few GiB rather than tens to hundreds).

The AWS CLI tools don't really have a "sync only the files that have been added/modified since <DATE>". That said, it's not super difficult to work around that gap. A simple shell script like the following works a trick:

for FILE in $( aws s3 ls --recursive s3://<SOURCE_BUCKET>/  | \
   awk '$1 > "2019-03-01 00:00:00" {print $4}' )
do
   echo "Downloading ${FILE}"
   install -bDm 000644 <( aws s3 cp "s3://<SOURCE_BUCKET>/${FILE}" - ) \
     "<STAGING_DIR>/${FILE}"
done

To explain the above:

1. Create a list of files to iterate:
   1. Invoke a subprocess using the $() notation. Within that subprocess...
   2. Invoke the AWS CLI's S3 module to recursively list the source-bucket's contents (`aws s3 ls --recursive`)
   3. Pipe the output to `awk` – looking for any date-string that's newer than the value in s3 ls's first output-column (the file-modification date column) and print out only the fourth column (the S3 object-path)
   The output from the subprocess is captured that output as an iterable list-structure
2. Use a for loop-method to iterate the previously-assembled list, assigning each S3 object-path to the ${FILE} variable
3. Since I hate sending programs off to do things in silence (I don't trust them to not hang), my first looped-command is to say what's happening via the echo "Downloading ${FILE}" directive.
4. The install line makes use of some niftiness within both BASH and the AWS CLI's S3 command:
   1. By specifying "-" as the "destination" for the file-copy operation, you tell the S3 command to write the fetched object-contents to STDOUT.
   2. BASH allows you take a stream of output and assign a file-handle to it by surrounding the output-producing command with <( ).
   3. Invoking the install command with the -D flag tells the command to "create all necessary path-elements to place the source 'file' in the desired location within the filesystem, even if none of the intervening directory structure exists, yet."
      
   Putting it all together, the install operation takes the streamed s3 cp output, and installs it as a file (with mode 000644) at the location derived from the STAGING_DIR plus the S3 object-path ...thus preserving the SOURCE_BUCKET's content-structure within the STAGING_DIR

Obviously, this method really only works for additive/substitutive deltas. If you need to account for deletions and/or moves, this approach will be insufficient.

Crib-Notes: End-to-End SSL Within AWS

In general, when using an Elastic Load-Balancer (ELB) to do SSL-encrypted proxying for an AWS-hosted, Internet-facing application, it's typically sufficient to simply do SSL-termination at the ELB and call it a day. That said:

• If you're paranoid, you can ensure that only the proxied EC2(s) and the ELB are able to communicate with each other via security groups.
• If you're under compliance-requirements (e.g., PCI/DSS), you can enable end-to-end SSL such that:
  1. Connections between Internet-based client and the ELB are encrypted
  2. Connections between the ELB and the application-hosting EC2(s) is encrypted

Either/both amount to a "belt and suspenders"  approach. Other than "meeting policy", security isn't meaningfully improved: given AWS's overarching security-design, even if someone else has access to your application-EC2(s) and ELB's VPC, they won't be able to sniff packets/data – encrypted or not.

Technical need aside... Implementing end-to-end SSL is trivial:

• ACM allows easy provisioning of SSL certificates for the ELB (with the security-bonus of automatically rotating said certificates). 
• You can very generic, self-signed certificates on your application-hosting EC2s:
  
  • The certificate's Subject doesn't matter
  • The certificate's validity window doesn't matter (no need to worry about rotating certificates that have expired)

Thus setup comes down to:

1. Create an EC2-hosted application/service:
   1. Launch EC2
   2. Install HTTPS-capable application
   3. Generate a self-signed certificate (setting the -days to as little as 1 day). Example (using the OpenSSL utility):
      
      

      openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
         -keyout peer.key -out peer.crt

      
      When prompted for input, just hit the <RETURN> key (this will create a cert with defaulted values ...which, as noted previously, don't really have bearing on the ELB's trust of the certificate). Similary, one can wholly omit the -days 1 flag and value – the default certificate will be valid for 30 days (but, ELB doesn't care about the validity time-window).
      
   4. Configure the HTTPS-capable application to load the certificate
   5. Configure the EC2's host-based firewall to allow connections to whatever port the application listens on for SSL-protected connections
   6. Configure the EC2's security group to allow connections to whatever port the application listens on for SSL-protected connections
2. Create an ELB:
   1. Set the ELB to listen for SSL-based connection-requests (using a certificate from ACM or IAM)
   2. Set the ELB to forward connections using the HTTPS protocol to connect to the target EC2(s) over whatever port the application listens on for SSL-protected connections
   3. Ensure the ELB's healthcheck is requesting a suitable URL to establish the health of the application 

Once the ELB's healthcheck goes green, it should be possible to connect to the EC2-hosted application via SSL.If one wants to verify the encryption-state of the connetction between the ELB and EC2(s), one would need to login to the EC2(s) and sniff the inbound packets (e.g., by using a tool like WireShark).

Crib-Notes: Validating Consistent ENA and SRIOV Support in AMIs

One of the contracts I work on, we're responsible for producing the AMIs used for the entire enterprise. At this point, the process is heavily automated. Basically, we use a pipeline that leverages some CI tools, Packer and a suite of BASH scripts to do all the grunt-work and produce not only an AMI but artifacts like AMI configuration- and package-manifests.

When we first adopted Packer, it had some limitations on how it registered AMIs (or, maybe, we just didn't find the extra flags back when we first selected Packer – who knows, it's lost to the mists of time at this point). If you wanted the resultant AMIs to have ENA and/or SRIOV support baked in (we do), your upstream AMI needed to have it baked in as well. This necessitated creating our own "bootstrap" AMIs as you couldn't count on these features being baked in – not even within the upstream vendor's (in our case, Red Hat's and CentOS's) AMIs.

At any rate, because the overall process has been turned over from the people that originated the automation to people that basically babysit automated tasks, the people running the tools don't necessarily have a firm grasp of everything that the automation's doing. Further, the people that are tasked with babysitting the automation differe from run-to-run. While automation should see to it that this doesn't matter, sometimes it pays to be paranoid. So a quick way to assuage that paranoia is to run quick reports from the AWS CLI. The following snippet makes for an adequate, "fifty-thousand foot" consistency-check:

aws ec2 describe-images --owner <AWS_ACCOUNT_ID> --query \
      'Images[].[ImageId,Name,EnaSupport,SriovNetSupport]' \
      --filters 'Name=name,Values=<SEARCH_STRING_PATTERN>' \
      --out text | \
   aws 'BEGIN {
         printf("%-18s%-60s%-14s-10s\n","AMI ID","AMI Name","ENA Support","SRIOV Support")
      } {
         printf("%-18s%-60s%-14s-10s\n",$1,$2,$3,$4)
      }'

• There's a lot of organizations and individuals publishing AMIs. Thus, we use the --owner flag to search only for AMIs we've published.
• We produce a couple of different families of AMIs. Thus, we use the --filter statement to only show the subset of our AMIs we're interested in.
• I really only care about four attributes of the AMIs being reported on: ImageId, Name, EnaSupport and SriovNetSupport. Thus, the use of the JMSE --query statement to suppress all output except for that in which I'm interested.
• Since I want the output to be pretty, I used the compound awk statement to create a formatted header and apply the same formatting to the output from the AWS CLI (using but a tiny bit of the printf routine's many capabilities).

This will produce output similar to:

   AMI ID                 AMI Name                                        ENA Support  SRIOV Support
   ami-187af850f113c24e1  spel-minimal-centos-7-hvm-2019.03.1.x86_64-gp2  True         simple
   ami-91b38c446d188643e  spel-minimal-centos-7-hvm-2019.02.1.x86_64-gp2  True         simple
   ami-22867cf08bb264ac4  spel-minimal-centos-7-hvm-2019.01.1.x86_64-gp2  True         simple
   [...elided...]
   ami-71c3822ed119c3401  spel-minimal-centos-7-hvm-2018.03.1.x86_64-gp2  None         simple
   [...elided...]
   ami-8057c2bf443dc01f5  spel-minimal-centos-7-hvm-2016.06.1.x86_64-gp2  None         None

As you can see, not all of the above AMIs are externally alike. While this could indicate a process or personnel problem, what my output actually shows is evolution in our AMIs. Originally, we weren't doing anything to support SRIOV or ENA. Then we added SRIOV support (because our AMI users were finally asking for it). Finally, we added ENA support (mostly so we could use the full range and capabilities of the fifth-generation EC2 instance-types).

At any rate, running a report like the above, we can identfy if there's unexpected differences and, if a sub-standard AMI slips out, we can alert our AMI users "don't use <AMI> if you have need of ENA and/or SRIOV".

Crib-Notes: Tracing Permissions for an Instance Role That Uses Inline Policies

The rationale for this crib-note is essentially identical to that for the previous topic on tracing IAM permissions in instance-roles that use managed-policies. So I'm just going to crib the rest of the intro...

When working with AWS EC2 instances – particularly when automating the deployments and lifecycles thereof – it's common to make use of the AWS IAM system's Instance Role Policy feature. Occasionally, you might get asked, "what permissions have been given to that instance." To answer, you might use AWS's IAM web console or the IAM CLI. In the former case, to get the information to the requestor, you're left with the options of either taking screen-shots or trying to copy and paste the text from the UI to your email/slack/etc. reply. In the latter case, you can just dump out the JSON-formatted policy-document that enumerates the permissions.

Generally, I prefer the CLI option since I don't have to worry "what does the recipient need to do with the results". Just dumping a text file, I needn't worry about being yelled at that the contents of a screen-shot can't be used to easily create another, similar policy. It's also easy to simply redirect the output to a file and attach it to whatever media I'm responding to ...or, if mail is enabled within the CLI environment, simply pipe the output directly to a CLI-based email tool and save a step in the process (laziness for the win!).

But, how to go from "there's an instance in this account: what privileges does it have" to "here's what it can do?". Basically, it's a three-step process:

1. Get the name of the Instance Role attached to the EC2 instance. This can be done with a method similar to:
   
   

   $ aws ec2 describe-instances --instance-id <INSTANCE_ID> \
         --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' --out text | \
     sed 's/^.*arn:.*profile\///'

   
   
2. Get th list of inline policies attached to role. This can be done with a method similar to:
   
   

   $ aws iam list-role-policies --role-name <OUPUT_FROM_PREVIOUS>

   
   
3. Get the list of permissions associated with the instance role's inline policy/policies. This can be done with a method similar to:
   
   

   $ aws iam get-role-policy --role-name INSTANCE --policy-name <OUPUT_FROM_PREVIOUS>

   
   

The above steps will dump out the full IAM policy/permissions of the queried inline policy:

• If the inline policy was the only policy attached to the role, the output will show all of the permissions the instance role grants any EC2s it is attached to.
• If the inline policy was the not the only inline policy attached to the role, it will be necessary to iterate over the remaining policies attached to the role to get the aggregated permission-set.

To facilitate the iteration, one can use a script similar to the following to encapsulate the second and third steps from prior process description:

#!/bin/bash
#
# Script to dump out all permissions granted through an IAM role with multiple
# inline IAM policies attached.
###########################################################################

if [[ $# -eq 0 ]]
then
   echo "Usage: ${0} <instance-id>" >&2
   exit 1
fi

PROFILE_NAME=$( aws ec2 describe-instances --instance-id "${1}" \
   --query 'Reservations[].Instances[].IamInstanceProfile[].Arn' --out text | \
  tr '\t' '\n' | sort -u | sed 's/^.*arn:.*profile\///' )

POLICY_LIST_RAW=$( aws iam list-role-policies --role-name ${PROFILE_NAME} )
POLICY_LIST_CLN=($( echo ${POLICY_LIST_RAW} | jq .PolicyNames[] | sed 's/"//g' ))

for ITER in $( seq 0 $(( ${#POLICY_LIST_CLN[@]} - 1 )) )
do
  aws iam get-role-policy --role-name "${PROFILE_NAME}" \
    --policy-name "${POLICY_LIST_CLN[${ITER}]}"
done