Smashing Walls of Text

On a few projects I work on, we make use of Ansible to automate system-configuration tasks. For better or worse, the automation relies on some upstream content that is outside the control of the customer. Translation: every few months, automation that has worked for months will suddenly no longer work.

By itself, this is more an inconvenience than a real problem. However, because the automation is handed off to other, more-junior people to execute, when errors are encountered, those others are frequently at a loss as to what to do.

Frequently, they don't bother to include logs of any sort ...or if they do, they include log snippets (or, worse: screen-caps of text-based log snippets!), the snippets frequently don't contain the information critical to solve the problem. So, first response in the support request is a "send me all the logs" type of reply.

Now, depending on the size of the Ansible job, the associate log-file might be HUGE. Generally, I'm only interested in where Ansible has failed. Parsing through an entire Ansible log-file can be like trying to find a specific brick in the Great Wall of China. So, to help preserve my sanity, I cooked up a quick BASH script to both help "cut to the chase" and provide more-easily readable log-output. That script looks like:

#!/bin/bash
#
# Script to filter the output of Ansible log files to something more-readable
#
###########################################################################
OUTFILE="${1:-/tmp/Ansible.log}"

mapfile -t ERROUT < <(
   grep ^failed: "${OUTFILE}"
)

# Make sure we got a mount-string
if [[ ${#ERROUT[@]} -lt 1 ]]
then
   echo "No failure-messages found in ${OUTFILE}"
fi

# Iterate over error-string
ITER=0
while [[ ${ITER} -lt ${#ERROUT[@]} ]]
do
   printf '##########\n## %s\n##########\n' "$(
      echo "${ERROUT[${ITER}]}" | sed 's/^failed:.* => //' | 
      python3 -c "import sys, json; print(json.load(sys.stdin)['item'])"
   )"
   
   echo "${ERROUT[${ITER}]}" | sed 's/^failed:.* => //' | \
     python3 -c "import sys, json; print(json.load(sys.stdin)['stderr'])" | sed 's/^/    /'
     
   printf '####################\n\n\n'
   ITER=$(( ITER + 1 ))
done

Crib-Notes: Manifest Deltas

Each month, the group I work for publishes new CentOS and Red Hat AMIs (and Azure templates and Vagrant boxes). When we complete the publication-event, we post a news announcement to our user-portal so that subscribers can receive an alert of the new publication. Included in that news announcement is a "what's changed" section.

In prior months, figuring out "what changed" was left as a manual step for the team-member charged with running the automation for a given month's publication event. This month, no one generated that news article and there were several updated and new RPMs included in the new image. So, I set about figuring out "how to extract this information programmatically so as to more-easily suss-out what to include in the announcement posting." The following does so (though, presumably, in a not-particularly-optimized) fashion:

git diff $(
      git log --pretty='%H' --follow -- <PATH_TO_MANIFEST_FILE> | \
      head -2 | \
      tac | \
      sed 'N;s/\n/../'
   ) -- <PATH_TO_MANIFEST_FILE> | \
grep -E '(amazon|aws|ec2)-' | \
sed 's/^./& /' | \
sort -k 2

To explain:

1. Use `git log` to output the commit-hashes for all the commits for the target file (in this case, the project's manifest-file)
2. Use `head -2` to grab only the two most-recent commit hashes from the output-stream
3. Use the `tac` command to invert the order of the two lines returned from the `head` command
4. Use the `sed` command to join the two lines, replacing the first line's line-ending newline character with ".."
5. Use `git diff` against the output created in steps 1-4, and constrain the diff-activity to just the manifest-file.
6. Pipe that output through `grep` to suppress all information other than the bits containing the `amazon-`, `aws-` and `ec2-` substrings.
7. Pipe that through `sed` so that the +/- that `git diff` uses to show new and removed files, respectively, becomes an easily-tokenized substring.
8. Sort the remaining output-stream (with `sort`) so that the lines are groups by manifest-element (the second key/token in the sorted output)

Taking that output and converting to a news article is still manual, but it at least makes it a lot easier to do than either hand-diffing two files or having to "just know" what's changed.

Notes

Because Red Hat has placed EL6 is in its final stage of de-support, we've stopped publishing CentOS6 and RHEL6. We did this to discourage our subscribers from doing new deployments on EL6 (since the underlying platform will go into final de-support come November of this year).

Similarly, due to current lack of CentOS offering for EL8, lack of security-related build- or hardening-guidance for EL8 and associated lack of subscriber-demand for an EL8 build, we don't yet include builds for CentOS8 or RHEL8 in our process. Thus, for the time being, we only need to provide a "whats changed" for EL7 builds. Given this, we currently only need to do change-queries against the "manifests/spel-minimal-centos-7-hvm.manifest.txt" file.