Mixed Data-Types and Keeping Things Clean

This year, one of the projects I've been assigned to has me assisting a customer in implementing a cloud-monitoring solution for their multi-cloud deployment. The tool uses the various CSPs APIs to monitor the creation/modification/deletion of resources and how those resources are configured.

The tool, itself, is primarily oriented for use and configuration via web UI. However, one can configure it via Terraform. This makes it easier to functionally-clone the monitoring tool's configuration as well a reconstitute it if someone blows it up.

That said, the tool uses Nunjucks and GraphQL to implement some of its rule-elements. Further, most of the data it handles comes in the form of JSON streams. The Nunjucks content, in particular, can be used to parse those JSON streams and static JSON content can be stored within the  monitoring-application. Because Terraform is used for CLI-based configuration, the Terraform resources can consist of pure Terraform code as well as a mix of encapsulated Nunjucks, GraphQL and JSON.

Most of the vendor's demonstration configuration-code has the Nunjucks, GraphQL and JSON contents wholly encapsulated in Terraform resource-definitions. If one wants to lint their configuration-code prior to pushing it into the application, the vendor-offered method for formatting the code can work counter to that. That said, with careful coding, one can separate the content-types from each other and use reference-directives to allow Terraform to do the work of merging it all together. While this may seem more complex, separating the content-types means that each chunk of content is more-easily validated and checked for errors. Rather than blindly hitting "terraform apply" and hoping for the best, you can lint your JSON, Nunjucks and GraphQL separately. This means that, once you've authored all of your initial code and wish to turn it over to someone else to do lifecycle tasks, you can horse it all to a CI workflow that ensures that humans that have edited any given file hasn't introduced content-type violations that can lead to ugly surprises.

Honestly, I have more confidence that the people I turn things over to will know how to massage single content-type files than mixed content-type files. This means I feel like I'm less likely to get confused help requests after I'm moved to another assignment.

Crib Notes: Has My EC2's AMI-Publisher Updated Their AMIs

Discovered, late yesterday afternoon, that the automation I'd used to deploy my development-EC2, had had some updates and that these updates (rotating some baked-in query-credentials) were breaking my  development-EC2's ability to use some resources (since they could no longer access those resources). So, today, I'll need to re-deploy my EC2. I figured that, since it'd been since September 14th since I'd launched my EC2, perhaps a new AMI was available (our team publishes new AMIs each month). So, what's a quick way to tell if a new AMI is available? Some nested AWS CLI commands:

aws \
  --output text ec2 describe-images \
  --owner $(
  aws ec2 describe-images \
    --output text \
    --image-ids $(
    aws ec2 describe-instances \
      --output text \
      --query 'Reservations[].Instances[].ImageId' \
      --instance-ids <CURRENT_EC2_ID>  ) \
  --query 'Images[].OwnerId'
) \
  --query 'Images[].{ImageId:ImageId,CreationDate:CreationDate,Name:Name}' | \
sort -nk 1

Sadly, the answer, today, is, "no". I apparently, this month's release-date is going to be later today or tomorrow. So, I'll just be re-deploying from September's AMI.

ACTUALLY Deleting Emails in gSuite/gMail

Each month, I archive all the contents of my main gSuite account to a third-party repository. I do this via an IMAP-based transfer.

Unfortunately, when you use an IMAP-based transfer to move files, Google doesn't actually delete the emails from your gMail/gSuite account. No, it simply removes all labels from them. This means that instead of getting space back – space that Google charges for in gSuite – the messages simply become not-easily-visible within gMail. None of your space is freed up and, thus, space-charges for those unlabeled emails continue to accrue.

Discovered this annoyance a couple years ago when my mail-client was telling me I was getting near the end of my quota. When I first got the quota-warning, I was like, "how??? I've offloaded all my old emails. There's only a month's worth of email in my Inbox, Sent folder and my per-project folders!" That prompted me to dig around to discover the de-labled/not-deleted fuckery. So, I dug around further to find a method for viewing those de-labeled/not-deleted files. Turns out, putting:

-has:userlabels -in:sent -in:chat -in:draft -in:inbox

In your webmail search-bar will show you them …and allow you to delete them.

My gSuite account was a couple years old when I discovered all this. So, when I selected all the unlabeled emails for deletion, it took a while for Google to actually delete them. However, once the deletion completed, I recovered nearly 2GiB worth of space in my gSuite account.

Tailoring `oscap` Profiles for Dummies

Several of the projects I am or have been matrixed to leverage the oscap utility to perform hardening based on common security-benchmarks. However, some of the profile-defaults are either too strict or too lax for a given application-deployment. While one can wholly ignore the common security-benchmarks selected hardenings and create one's own custom hardening-profile(s), that's a bit too much like reinventing the wheel.

Checking Which Security-Profiles Are Available

The oscap utility can be used to quickly show what profile-names are available for use. This is done by executing:

$ oscap info /PATH/TO/OS/<XCCDF>.xml

On Red Hat systems (and derivatives) with the scap-security-guide RPM installed, the XCCDF files will be installed in the /usr/share/xml/scap/ssg/content directory. To see which profiles are available for Red Hat 8 distros, one would execute:

$ oscap info /usr/share/xml/scap/ssg/content/ssg-rhel8-xccdf.xml

Which would give an output like:

Document type: XCCDF Checklist
Checklist version: 1.2
Imported: 2023-02-13T11:49:00
Status: draft
Generated: 2023-02-13
Resolved: true
Profiles:
        Title: ANSSI-BP-028 (enhanced)
                Id: xccdf_org.ssgproject.content_profile_anssi_bp28_enhanced
        Title: ANSSI-BP-028 (high)
                Id: xccdf_org.ssgproject.content_profile_anssi_bp28_high
        Title: ANSSI-BP-028 (intermediary)
                Id: xccdf_org.ssgproject.content_profile_anssi_bp28_intermediary
        Title: ANSSI-BP-028 (minimal)
                Id: xccdf_org.ssgproject.content_profile_anssi_bp28_minimal
        Title: CIS Red Hat Enterprise Linux 8 Benchmark for Level 2 - Server
                Id: xccdf_org.ssgproject.content_profile_cis
        Title: CIS Red Hat Enterprise Linux 8 Benchmark for Level 1 - Server
                Id: xccdf_org.ssgproject.content_profile_cis_server_l1
        Title: CIS Red Hat Enterprise Linux 8 Benchmark for Level 1 - Workstation
                Id: xccdf_org.ssgproject.content_profile_cis_workstation_l1
        Title: CIS Red Hat Enterprise Linux 8 Benchmark for Level 2 - Workstation
                Id: xccdf_org.ssgproject.content_profile_cis_works
        Title: Unclassified Information in Non-federal Information Systems and Organizations (NIST 800-171)
                Id: xccdf_org.ssgproject.content_profile_cui
        Title: Australian Cyber Security Centre (ACSC) Essential Eight
                Id: xccdf_org.ssgproject.content_profile_e8
        Title: Health Insurance Portability and Accountability Act (HIPAA)
                Id: xccdf_org.ssgproject.content_profile_hipaa
        Title: Australian Cyber Security Centre (ACSC) ISM Official
                Id: xccdf_org.ssgproject.content_profile_ism_o
        Title: Protection Profile for General Purpose Operating Systems
                Id: xccdf_org.ssgproject.content_profile_ospp
        Title: PCI-DSS v3.2.1 Control Baseline for Red Hat Enterprise Linux 8
                Id: xccdf_org.ssgproject.content_profile_pci-dss
        Title: DISA STIG for Red Hat Enterprise Linux 8
                Id: xccdf_org.ssgproject.content_profile_stig
        Title: DISA STIG with GUI for Red Hat Enterprise Linux 8
                Id: xccdf_org.ssgproject.content_profile_stig_gui
Referenced check files:
        ssg-rhel8-oval.xml
                system: http://oval.mitre.org/XMLSchema/oval-definitions-5
        ssg-rhel8-ocil.xml
                system: http://scap.nist.gov/schema/ocil/2
        https://access.redhat.com/security/data/oval/com.redhat.rhsa-RHEL8.xml.bz2
                system: http://oval.mitre.org/XMLSchema/oval-definitions-5

The critical items, here, are the lines that begin with "Title" and the lines that begin with "Id".

• The lines that begine with "Title" are what will appear in graphical tools like the SCAP WorkBench GUI.
• The lines that begin with "Id" are used with the `oscap` utility. These identifiers one given as arguments to the utility's --profile flag (when using the `oscap` utility to scan and/or remediate a system).
  Note: When using the values from the "Id" lines, either the fully-qualified ID-string may be given or just the parts after the "…profile_" substring. As such, one could specify either "xccdf_org.ssgproject.content_profile_stig" or just "stig".

Creating Tailored Security-Profiles:

The easiest method for tailoring security-profiles is to use the SCAP Workbench to generate the appropriately-formatted XML. However, if one already has an appropriately-formatted tailoring XML file, a plain text-editor (such as vim) is a very quick way to add or remove content.

It's worth noting that the SCAP Workbench is a GUI. As such, it will be necessary to either have access to the graphical console of Linux or OSX host or the ability to display a remote Linux host's GUI-programs locally. Remote display to local system can be the entire remote desktop (via things like Xrdp, VNC, XNest or other) or just the SCAP Workbench client, itself (personally, I leverage X11-over-SSH tunnels).

On a Red Hat (or derivative) system, you'll want to install the scap-workbench and the scap-security-guide RPMs. The former provides the SCAP Workbench GUI while the latter provides the content you'll typically want to use. Alternate to the scap-security-guide RPM, you can install SCAP content from the Compliance As Code project (the upstream source for the scap-security-guide RPM's contents).

To generate a "null" tailoring-profile – one that doesn't change the behavior of a given profile's execution – use the following generic procedure:

1. Establish GUI access to the system that will run the scap-workbench binary
2. Execute `scap-workbench`. This will bring up a banner that looks like:
   
   
   
   
   The above is shown with the "content to load" list expanded. This demonstrates the content that's loaded to a Red Hat 8 system by way of the scap-security-guide RPM.
3. Select the appropriate content from the dropdown: if using vendor-content, one of the RHELn menu items; if opening content from another source (e.g. the Compliance as Code project), select the "Other SCAP Content" option
4. Click the "Load Content" button. if the "Other SCAP Content" option was selected, this will open up a dialog for navigating to the desired content. Otherwise, the vendor-content for the selected RHEL version will be opened.
5. Once the selected content has been read, the GUI will display a page with the Title of the opened-content, a "Customization" dropdown-menu and a "Profile" drop-down menu.
6. Select the appropriate hardening-profile from the "Profile" drop-down menu (e.g., "DISA STIG for Red Hat Enterprise Linux 8")
7. Click on the "Customize" button next to the selected "Profile":
   
   
8. This will bring up a window like:
   
   
   Accept the default value for the "New Profile ID" field and click on the "Ok" button
   
9. This will open a new window:
   • To save a "null" tailoring-file, immediately hit the "Ok" button
   • Alternately, pick through the list of available settings, selecting or unselecting boxes as seems appropriate for the target use-case, then hit the "Ok" button 
10. This will close the customization-window and change the main window's "Customization" drop-down to include the string "(unsaved changes)"
11. Click on the  "File" dropdown-menu at the top of the window and select the  "Save Customization Only" menu-item. Select a file-name that makes sense (I typically choose something like "tailoring-<OS_VERSION>-<PROFILE_NAME>.xml" (e.g., "tailoring-el8-stig.xml",  "tailoring-el8-cis-server-l1.xml", etc.)
12. Exit the SCAP workbench.

The resultant file will contain a line similar to:

<xccdf:Profile id="xccdf_org.ssgproject.content_profile_stig_customized" extends="xccdf_org.ssgproject.content_profile_stig">

The actual contents of the line will vary, but the critical components are the "id" and "extends" tokens:

• id: the name of the profile to use when invoking the oscap utility
• extends: the name of the profile that will get modified by the tailoring-file's contents

The contents of the tailoring-file are generally pretty basic – something like:

<xccdf:tailoring id="xccdf_scap-workbench_tailoring_default" xmlns:xccdf="http://checklists.nist.gov/xccdf/1.2">
  <xccdf:benchmark href="/tmp/scap-workbench-WOghwr/ssg-rhel8-xccdf.xml">
  <xccdf:version time="2023-09-11T17:10:35">1</xccdf:version>
  <xccdf:profile extends="xccdf_org.ssgproject.content_profile_stig" id="xccdf_org.ssgproject.content_profile_stig_customized">
    <xccdf:title override="true" xml:lang="en-US" xmlns:xhtml="http://www.w3.org/1999/xhtml">DISA STIG for Red Hat Enterprise Linux 8 [CUSTOMIZED]</xccdf:title>
    <xccdf:description override="true" xml:lang="en-US" xmlns:xhtml="http://www.w3.org/1999/xhtml">This profile contains configuration checks that align to the
DISA STIG for Red Hat Enterprise Linux 8 V1R9.</xccdf:description>
    <xccdf:select idref="xccdf_org.ssgproject.content_rule_rpm_verify_hashes" selected="false">
  </xccdf:select></xccdf:profile>
</xccdf:benchmark></xccdf:"tailoring>

Rules that have been added to the execution list will look something like (note the "true" condition/key):

<xccdf:select idref="xccdf_org.ssgproject.content_rule_rpm_verify_hashes" selected="true"/>

While rules that have been deslected for execution will look something like (note the "false" condition/key):

<xccdf:select idref="xccdf_org.ssgproject.content_rule_rpm_verify_hashes" selected="false"/>

Whether adding extra rules or deselecting rules from the execution-profile, the rules will be placed after the "</xccdf:description>" token and before the "</xccdf:Profile>" token.

Note that the action/rule is effectively null if the condition/key for a rule in the tailoring-file has the same value as the action/rule value in the profile referenced by the "extends" token.

Using Tailored Security-Profiles:

Once generated, the tailoring-file is used by calling the oscap utility in the normal way but for:

• Adding a "--tailoring" flag (with the path of the tailoring-file as its argument)
• Ensuring the value of the "--profile" matches the profile "id" token's value in the tailoring-file (and that the "extends" token's value in the tailoring-file matches the "id" token's value in the referenced XCCDF file)

For example, if executing a remediation using a tailored-execution of the STIG profile, one would execute something like:

oscap xccdf eval \
  --profile stig_custom \
  --tailoring-file /root/tailoring-el8-stig.xml \
  /usr/share/scap-content/openscap/ssg-rhel8-xccdf.xml

The above tells the oscap utility to use the tailoring-file located at "/root/tailoring-watchmaker-el8.xml" to modify the behavior of the "stig" profile as defined in the "/usr/share/scap-content/openscap/ssg-rhel8-xccdf.xml" file.

TIL: I Am Probably Going To Come To Hate `fapolicyd`

One of the things I do in my role is write security automation. Part of that requires testing systems' hardening-compliance each time one of the security-benchmarks my customers use is updated.

In a recent update, the benchmarks for Red Hat Enterprise Linux 8 (and derivatives) added the requirement to enable and run the application-whitelisting service, fapolicyd. I didn't immediately notice this change …until I went to offload the security-scans from my test EC2 to an S3 bucket. The AWS CLI was suddently broken.

Worse, it was broken in an absolutely inscrutable way: if one executed and  AWS CLI command, even something as simple and basic as `aws help`, it would immediately return having neither performed the requested action nor emitting an error. As an initial debug attempt, I did:

echo $( aws help )$?

Which got me the oh-so-useful `255` for my troubles. But, it did allow me to start about to Googling. My first hit was this guy. It had the oh-so-helpful guidance:

• 255 -- Command failed. There were errors from either the CLI or the service the request was made to.

Like I said: "oh-so-helpful guidance".

So, I opened a support request with Amazon. Initially, they were at least as in the dark as I was. 

Fortunately, another member of the team I work on noticed the support case when it came into his Inbox via our development account's auto-Cc configuration. Unlike me, he, apparently, hasn't deadmailed everything sent to that distro (which, given how much is sent to that distro, deadmailing anything that arrived through the distro was the only way to preserve my sanity). He'd indicated that he had previously had similar issues and that he got around them by disabling the fapolicyd service. I quickly tested stopping the service and the AWS CLI happily resumed functioning as it had before the hardening tasks had executed.

I knew that wholly disabling the service was not going to be acceptable to our cyber-defense team. But, knowing where the problem originated meant I had a useful investigatory path.

The first (seeming) solution I found was to execute something like:

fapolicyd-cli --file add /usr/local/bin/aws
fapolicyd -u

This allowed both the AWS CLI to function and for the fapolicyd service to continue to be run.

For better or worse, though, I'm a curious type. I wanted to see what the rule looked like that the fapolicyd-cli utility had created. So, I dug around the documentation to find where I might be able to eyeball the exception.

# AUTOGENERATED FILE VERSION 2
# This file contains a list of trusted files
#
#  FULL PATH        SIZE                             SHA256
# /home/user/my-ls 157984 61a9960bf7d255a85811f4afcac51067b8f2e4c75e21cf4f2af95319d4ed1b87
/usr/local/aws-cli/v2/2.11.6/dist/aws 6658376 c48f667b861182c2785b5988c5041086e323cf2e29225da22bcd0f18e411e922

Which immediately rung alarm bells in my skull (strong "danger Will Robinson"  vibes). By making the exception not only conditional on the binary's (real) path, but also to its size and, particularly, its SHA256 signature, I knew that if anyone ever updated the installed binary, their exception-description was no longer going to match. This in turn would mean that the utility would stop working. Not wanting to deal with tickets that I could easily prevent, I continued my investigation.

Knowing that what I actually wanted was to give a blanket exemption to everything under the "/usr/local/aws-cli/v2" directory, I started investigating how to do that. Ultimately, I came up with a exeption-set that looked like:

allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1

I saved the contents as `/etc/fapolicyd/rules.d/30-aws.rules`and reloaded the `fapolicyd` configuration. However, I was sadly disappointed to discover that the AWS CLI was still broken. On the plus side, it was broken differently. Now, instead of immediately and silently exiting (with a 255 error-code), it was giving me:

[2030] Error loading Python lib '/usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0':
dlopen: /usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0: cannot open shared object
file: Operation not permitted

Much more meat to chew on. Further searching later, I had two additional commands to help me in my digging:

ausearch --start today -m fanotify --raw | aureport --file --summary

Which gave me:

File Summary Report
===========================
total  file
===========================
16  /usr/local/aws-cli/v2/2.11.6/dist/libz.so.1
16  /usr/local/aws-cli/v2/2.11.6/dist/libpython3.11.so.1.0
2  /usr/local/aws-cli/v2/2.11.6/dist/aws

And:

fapolicyd-cli --list

Which gave me:

-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,
application/java-archive,text/x-java,application/x-java-applet,application/javascript,
text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,
text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,
text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
 1. allow perm=any uid=0 : dir=/var/tmp/
 2. allow perm=any uid=0 trust=1 : all
 3. allow perm=open exe=/usr/bin/rpm : all
 4. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
 5. deny_audit perm=any pattern=ld_so : all
 6. deny_audit perm=any all : ftype=application/x-bad-elf
 7. allow perm=open all : ftype=application/x-sharedlib trust=1
 8. deny_audit perm=open all : ftype=application/x-sharedlib
 9. allow perm=execute all : trust=1
10. allow perm=open all : ftype=%languages trust=1
11. deny_audit perm=any all : ftype=%languages
12. allow perm=any all : ftype=text/x-shellscript
13. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
14. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
15. deny_audit perm=execute all : all
16. allow perm=open all : all

Which told me why the binary was at least trying to work, but was unable to load its shared-library. Since I'd named the file `/etc/fapolicyd/rules.d/80-aws.rules`, it was loading later than a rule that was preventing access to shared libraries not in standard trust-paths. In the above, it was the file that created rule #8.

I grepped through the `/etc/fapolicyd/rules.d/` directory looking for the file that created rule #8. Having found it, I moved my rule-file upwards with a quick `mv /etc/fapolicyd/rules.d/{8,3}0-aws.rules` and reloaded my rules. This time, my rules-list came up like:

-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,
application/java-archive,text/x-java,application/x-java-applet,application/javascript,
text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,
text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,
text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
 1. allow perm=any uid=0 : dir=/var/tmp/
 2. allow perm=any uid=0 trust=1 : all
 3. allow perm=open exe=/usr/bin/rpm : all
 4. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
 5. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-sharedlib trust 1
 6. allow perm=any all : dir=/usr/local/aws-cli/v2/ type=application/x-executable trust 1
 7. deny_audit perm=any pattern=ld_so : all
 8. deny_audit perm=any all : ftype=application/x-bad-elf
 9. allow perm=open all : ftype=application/x-sharedlib trust=1
10. deny_audit perm=open all : ftype=application/x-sharedlib
11. allow perm=execute all : trust=1
12. allow perm=open all : ftype=%languages trust=1
13. deny_audit perm=any all : ftype=%languages
14. allow perm=any all : ftype=text/x-shellscript
15. deny_audit perm=execute all : all
16. allow perm=open all : all

With my sharedlib allow-rule now ahead of the default-deny sharedlib rule, I tested out the AWS CLI command again. Success!

Unfortuantely, while I solved the problem I set out to solve, my `ausearch`  output was telling me that a few other standard tools were also likely having similar whitelisting issues. Ironically, those "other standard tools" are all security-related.

Fun fact: a number of security-vendors write their products for Windows, first and foremost. Their Linux tooling is almost an afterthought. As such, it's often not well-delivered: if they deliver their software in RPMs at all, the RPMs are often poorly-constructed. I almost never see signed RPMS from security vendors. When I do actually get signed RPMs, they're very rarely signed in a way that's compatible with a Red Hat system that's configured to run in FIPS-mode. So, I guess I shouldn't be super surprised that these same security-tools aren't aware of the need or how to work with fapolicyd. Oh well, that's someone else's problem (realistically, probably "future me's").

Crib Notes: Assuming a Role

Several of my current customers leverage AWS IAM's role-assumption capability. In particular, one of my customers leverages it for automating the execution of the Terragrunt-based IaC. For the automated-execution, they run the Terragrunt code from an EC2 that has an attached IAM role that allows code executed on the hosting-EC2 to assume roles in other accounts.

Sometimes, when writing updates to their Terragrunt code, it's helpful to be able to audit the target account's state before and after the execution, but outside the context of Terragrunt, itself. In these cases, knowing how to use the AWS CLI to switch roles can be quite handy. A quick one-liner template for doing so looks like:

$ eval "$(
  aws sts assume-role \
    --role-arn "arn:<AWS_PARTITION>:iam::<TARGET_ACCOUNT_NUMBEr>:role/<TARGET_ROLE_NAME>" \
    --role-session-name <userid> --query 'Credentials' | \
  awk '/(Key|Token)/{ print $0 }' | \
  sed -e 's/",$/"/' \
      -e 's/^\s*"/export /' \
      -e 's/": "/="/' \
      -e 's/AccessKeyId/AWS_ACCESS_KEY_ID/' \
      -e 's/SecretAccessKey/AWS_SECRET_ACCESS_KEY/' \
      -e 's/SessionToken/AWS_SESSION_TOKEN/'
)"

What the above does is:

1. Opens a subshell to execute a series of commands in
2. Executes `aws sts assume-role` to fetch credentials, in JSON format, for accessing the target AWS account as the target IAM role
3. Uses `awk` to select which parts of the prior command's JSON output to keep (`grep` or others are likely more computationally-efficient, but you get the idea)
4. Uses `sed` to convert the JSON parameter/value pair-strings into BASH-compatible environment-variable delcarations
5. Uses `eval` to take the output of the subshell and read it into the current shell's environment

Once this is executed, your SHELL will grant you privileges to execute commands in the target account – be that using the AWS CLI or any other tool that understands the "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY" and "AWS_SESSION_TOKEN" environment variables.

Using `aws sts get-caller-identity` will allow you to see your new IAM role.

Why Prompt Uncomfortable Questions

I do automation work for a number of enterprise customers. Each of these customers deploys Red Hat and Red Hat derivatives in their hosting environments (be those environments physical, on-premises virtualized or "in the cloud"). Basically, they use "for real" Red Hat for their production systems, then use one of the free clones for the development efforts …until the CentOS 8 Core → CentOS 8 Stream debacle, they used Centos for the development efforts. My focus with such customers is almost always constrained to their cloud-hosted Enterprise Linux hosts.

However, due to that CentOS 8 Stream debacle, one of my customers fell for Oracle's "use OL8: it's free" come-on. Now, if you've been in IT for even five minutes, you realize that anything "free" offered by Oracle is only so offered because Oracle sees it as a foot in the door. That said, with OL8, specifically, they try to add features to it to make it "a more compelling offering" (or something). Unfortunately, to date, the deltas between OL8 and RHEL8 are significantly greater than between RHEL8 and Rocky 8 or Alma 8 ...and not for the better. Automation that my team has written that "just works" for creating RHEL 8, Rocky 8 and Alma 8 AMIs and VM-templates doesn't "just work" for creating OL8 AMIs.

For example, while Rocky 8, Alma 8 and CentOS 8 Stream (and CentOS 8 Core, before it) all know how to do weak dependency resolution, OL8 doesn't (or, at least, didn't until maybe recently: a couple months after reporting the issue to Oracle via their bugzilla, they finally sent back a notification saying the problem's fixed, I just haven't had the opportunity to verify).

At any rate, each time I've run into various brokenness within OL8, the customer's Oracle sales team keeps trying to sell support contracts.

Similarly, when trying to tout the value of OL8, they tried to flog OL8's ksplice feature. When I asked "why would I use that when RHEL 8 and all of its non-Oracle derivatives have kpatch", the Oracle rep responded back with a list of things that ksplice is able to do but kpatch isn't. That said, the wording of his response elicited a "but you also seem to be saying that feature isn't in the free edition" response from me. Eventually, the representative replied back saying that my assessment was accurate – the flogged feature wasn't in the free edition.

He also tried to salvage the "Oracle support" thread by pointing out that Oracle would also provide support for my customer's Red Hat systems under an uber-contract. Now, my customer uses pay as you go (PAYG) EC2s in AWS but wanted free EL8 alternatives – thus the consideration of OL8 – for their non-production workloads. As such, if they're doing PAYG RHEL instances and wanting free OL8 for their non-production workloads, why would suggesting my customer buy Oracle's support for all of them make any sense? I mean, if my customer were not doing PAYG RHEL instances, they've presumably bought instance-licenses (and support along with it) from Red Hat, so, again, "why am would they want to buy Oracle's support for them"

…similarly, if they're already doing static licensing for RHEL instances, then they're probably also managing their instance-licenses through Satellite (etc.). As such, they'd then be able to take advantage of the "free for developers" licenses for their non-production EC2s …at which point the question would be "why would they even bother with OL8" let alone have to ask the "why would we buy Oracle's support for them" question?

Yeah, I get that Oracle sees OL8 more as a foot-in-the-door than an actual product, but still: send me information that doesn't beg questions that are going to be uncomfortable for you to answer.

Rate-limit Testing

Recently, was working on a project where I needed to enable customers to join their Linux EC2s to their on-premises Windows AD domain. I noticed that I was occasionally getting errors like:

adcli: couldn't connect to dev.lab domain: Couldn't authenticate as: svc_dev_joiner@DEV.LAB: Client's credentials have been revoked

Initially, I'd thought I was triggering the lockup by trying to rejoin the same host to the domain in too quick of succession. But then I suspected that I might actually be running into a broader-scope rate-limiting problem with the joiner-account. So, I set up a userData file that contained a block like:

hostnamectl set-hostname "ip-$(
  cat /dev/urandom | \
  tr -dc '[:alpha:]' | \
  tr '[:upper:]' '[:lower:]' | \
  fold -w ${1:-11} | \
  head -n 1
).dev.lab"

Then updated my `aws ec2 run-instances …` command to include a `--count 12` option. The above code-snippet ensures that I get an randomized FQDN where the node-name consists of the string "ip-" followed by 11 (relatively) random characters. This creates a 15-character node-name …necessitated by the domain-controller's refusal to allow domain-joins by clients that want node-names longer than the NETBIOS character-limit (because the DC is running in 2003 compatibility-mode). I had previously tried using:

hostnamectl set-hostname "$( 
    printf '%02X' $(
      hostname -I | sed 's/\./ /g'
    )
  ).dev.lab"

However, with my testing subnet being small, I realized that I might be generating hostnames that were already in the AD domain, which might cause its own problems. Thus, the desire for greater uniqueness in my node-names.

In either case, the domain-owners are going to be pissed that I'm dicking up their domain database with a crapton of "nonconformant" hostnames.

Stop Tramping On My Toes

 A week or so ago, I was asked by my customer to write some (AWS) account-import scripts to help their Ops team bring account-resources under Terragrunt contol. No big deal.

I inform the customer, "I need an account that I'm free to spin up and tear down so I can test the scripts as I write them." Customer responds back, supplying me with an account in which to do my work.

Now, there's a lot of stuff that needs to be accounted for in the import automation. Things were going decently well, until late last week. Then things got squirrely. Upon investigating, I see the telltales signs that I haven't been the only person using the target account and that the other person's activities have undermined all of the assumptions that were in my scripts.

I ask for help razing the account and point out that it looks like someone else had been using the account while it was supposed to be for my sole use. Another "engineer" on the team – one who has been regularly tramping on everyone else's toes by just accessing accounts and working in them – chimes in on the thread saying, "yeah: I ran <CODE> in that account, last week?"

Color me utterly non-shocked. Also, color wanting to choke the shit out of the asshole for breaking shit, again by conducting activities with no coordination.

So, now I have to wait until someone with more permissions than I have on the account can go in and flatten the account, so I can build it back to the state I was expecting to be working from.

I mean, I supposed I should be used to it, by now. Further, I suppose that I shouldn't be surprised that an "engineer" who barely knows how to use git would operate like a freaking wrecking ball. But it's no less frustrating each time he blows me or someone else up with his carelessness.

The Hatefulness of SELinux Compounded By IPA

A couple months ago, one of my customers' cyber-security teams handed down an edict that all IPA-managed users needed to have SELinux confinements applied. For admistrative accounts, this meant that administrators would SSH system and have an SELinux assignment of:

$ id -Z
staff_u:staff_r:staff_t:s0-s0:c0.c1023

And that when such administrative-users executed `sudo`, they would end up with an SELinux assignment of:

# id -Z
staff_u:sysadm_r:sysadm_t:s0-s0:c0.c1023

Not long after this, that same cyber-security team opened up a bug-ticket complaining that their scanners were no longer able to conduct all the tests they were able to conduct prior to the implementation of the confinement scheme (shocking, I know).

Since their scan-user operated with the same privilege set as human administrators did, their scan-user was getting similarly constrained. As an example of the practical impacts of this confinement, one need only try to look at the contents of the /etc/shadow file:

$ sudo cat /etc/shadow
/bin/cat: /etc/shadow: Permission denied

Since their security tooling was using that confined user-account to do – among a whole host of other check-tasks – functionally similar tests, naturally, some of their tests started failing and red lights started showing up on their dashboards. Worse (for them), their boss started asking why his summary-reports suddenly started to look like the elevator scene in The Shining after months of being mostly green.

Their problem ended up in my task-pile. I'm not really an SELinux guy …even if I can usually figure out how to get most third-party applications working that don't come with SELinux policy without having to resort to disabling SELinux. And, as much of "not really an SELinux guy" as I am, I'm really not an IPA guy. So, "fun times ahead".

Not wanting to waste time on the problem, and knowing that my customer had a Red Hat Enterprise Support entitlement, I opted to try to avail myself of that support. With my customer being part of a large, very-siloed organization, just getting a support case opened – and then actually able to directly read and comment on it rather than having to play "grape vine" – proved to be its own, multi-week ordeal.

After several days of back-and-forth, I was finally able to get the case escalated to people with SELinux and IPA expertise …but, seemingly, no one with both sets of expertise (yay). So, I was getting SELinux answers that included no "how do we actually do this wholly within IPA" and I was getting generic IPA guidance. The twain never quite met.

Ultimately, I was given guidance to do (after creating a scan-user POSIX-group to put the scan-user into):

  ipa selinuxusermap-add --hostcat='all' --selinuxuser=unconfined_u:s0-s0:c0.c1023 <SCAN_GROUP_NAME>
  ipa selinuxusermap-add-user <SCAN_GROUP_NAME> --group=<SCAN_GROUP_NAME>
  ipa sudorule-add <SCAN_USER_SUDO_RULE_NAME> --hostcat='all' --cmdcat='all'
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption '!authenticate'
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption role=unconfined_r
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption type=unconfined_t
  ipa sudorule-add-user <SCAN_GROUP_RULE_NAME> --group=<SCAN_GROUP_NAME>
  sudo bash -c "service sssd stop ; rm -f /var/lib/sss/db/* ; service sssd start"

Unfortunately, this didn't solve my problem. My scan-user continued to be given the same SELinux profile upon logging in and executing `sudo` that my "normal" administrative users were. Support came back and told me to re-execute the above, but to give the sudorule a precedence-setting:

  ipa selinuxusermap-add --hostcat='all' --selinuxuser=unconfined_u:s0-s0:c0.c1023 <SCAN_GROUP_NAME>
  ipa selinuxusermap-add-user <SCAN_GROUP_NAME> --group=<SCAN_GROUP_NAME>
  ipa sudorule-add <SCAN_USER_SUDO_RULE_NAME> --hostcat='all' --cmdcat='all' --order=99
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption '!authenticate'
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption role=unconfined_r
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption type=unconfined_t
  ipa sudorule-add-user <SCAN_GROUP_RULE_NAME> --group=<SCAN_GROUP_NAME>
  sudo bash -c "service sssd stop ; rm -f /var/lib/sss/db/* ; service sssd start"

Still, even having set a precedence on the `sudo` rule, my scan-user wasn't getting the right confinement rules applied. The support rep had me execute `ipa user-show <SCAN_USER> --all`. Upon doing that, we noticed that the <SCAN_USER> account was affiliated with two `sudo` rules. Then using `ipa sudorule-show <SUDO_RULE_NAME> --all` for each rule, was able to find that one of the two rules was applying the `!authenticate`, `role=sysadm_r` and `type=sysadm_t` sudo-options.

I made the assumption that, even though we'd explicitly set a precedence-value on the one rule, the previously-existing rule likely didn't have a precedence-value set and that, as a result, implicit-precedence rules were still being applied ...and that the previously-existing rule "won" in that scenario. To test, I did:

ipa sudorule-mod <EXISTING_RULE> --order=10

And then retested with my scan-user. At this point, the scan-user was functioning as desired. This meant that:

1. My supposition about implicit and explicit rule-ordering was correct
2. That the rules are processed on a "higher-number == higher-priority" precedence-processing

Unfortunately, due to limitations in RHEL7 and the version of IPA in use by my customer (vendor-rep checked internal documentation for this determination), I had to leave the scan-user wholly unconstrained. Customer's security-team is likely to gripe about that, but I figure that, since they own the offending user, they can deal with the consequences (paperwork) associated with adequately-relaxing security for that user so that their scans would return to a functional state.