Crib-Notes: End-to-End SSL Within AWS

In general, when using an Elastic Load-Balancer (ELB) to do SSL-encrypted proxying for an AWS-hosted, Internet-facing application, it's typically sufficient to simply do SSL-termination at the ELB and call it a day. That said:

• If you're paranoid, you can ensure that only the proxied EC2(s) and the ELB are able to communicate with each other via security groups.
• If you're under compliance-requirements (e.g., PCI/DSS), you can enable end-to-end SSL such that:
  1. Connections between Internet-based client and the ELB are encrypted
  2. Connections between the ELB and the application-hosting EC2(s) is encrypted

Either/both amount to a "belt and suspenders"  approach. Other than "meeting policy", security isn't meaningfully improved: given AWS's overarching security-design, even if someone else has access to your application-EC2(s) and ELB's VPC, they won't be able to sniff packets/data – encrypted or not.

Technical need aside... Implementing end-to-end SSL is trivial:

• ACM allows easy provisioning of SSL certificates for the ELB (with the security-bonus of automatically rotating said certificates). 
• You can very generic, self-signed certificates on your application-hosting EC2s:
  
  • The certificate's Subject doesn't matter
  • The certificate's validity window doesn't matter (no need to worry about rotating certificates that have expired)

Thus setup comes down to:

1. Create an EC2-hosted application/service:
   1. Launch EC2
   2. Install HTTPS-capable application
   3. Generate a self-signed certificate (setting the -days to as little as 1 day). Example (using the OpenSSL utility):
      
      

      openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
         -keyout peer.key -out peer.crt

      
      When prompted for input, just hit the <RETURN> key (this will create a cert with defaulted values ...which, as noted previously, don't really have bearing on the ELB's trust of the certificate). Similary, one can wholly omit the -days 1 flag and value – the default certificate will be valid for 30 days (but, ELB doesn't care about the validity time-window).
      
   4. Configure the HTTPS-capable application to load the certificate
   5. Configure the EC2's host-based firewall to allow connections to whatever port the application listens on for SSL-protected connections
   6. Configure the EC2's security group to allow connections to whatever port the application listens on for SSL-protected connections
2. Create an ELB:
   1. Set the ELB to listen for SSL-based connection-requests (using a certificate from ACM or IAM)
   2. Set the ELB to forward connections using the HTTPS protocol to connect to the target EC2(s) over whatever port the application listens on for SSL-protected connections
   3. Ensure the ELB's healthcheck is requesting a suitable URL to establish the health of the application 

Once the ELB's healthcheck goes green, it should be possible to connect to the EC2-hosted application via SSL.If one wants to verify the encryption-state of the connetction between the ELB and EC2(s), one would need to login to the EC2(s) and sniff the inbound packets (e.g., by using a tool like WireShark).

Crib-Notes: Validating Consistent ENA and SRIOV Support in AMIs

One of the contracts I work on, we're responsible for producing the AMIs used for the entire enterprise. At this point, the process is heavily automated. Basically, we use a pipeline that leverages some CI tools, Packer and a suite of BASH scripts to do all the grunt-work and produce not only an AMI but artifacts like AMI configuration- and package-manifests.

When we first adopted Packer, it had some limitations on how it registered AMIs (or, maybe, we just didn't find the extra flags back when we first selected Packer – who knows, it's lost to the mists of time at this point). If you wanted the resultant AMIs to have ENA and/or SRIOV support baked in (we do), your upstream AMI needed to have it baked in as well. This necessitated creating our own "bootstrap" AMIs as you couldn't count on these features being baked in – not even within the upstream vendor's (in our case, Red Hat's and CentOS's) AMIs.

At any rate, because the overall process has been turned over from the people that originated the automation to people that basically babysit automated tasks, the people running the tools don't necessarily have a firm grasp of everything that the automation's doing. Further, the people that are tasked with babysitting the automation differe from run-to-run. While automation should see to it that this doesn't matter, sometimes it pays to be paranoid. So a quick way to assuage that paranoia is to run quick reports from the AWS CLI. The following snippet makes for an adequate, "fifty-thousand foot" consistency-check:

aws ec2 describe-images --owner <AWS_ACCOUNT_ID> --query \
      'Images[].[ImageId,Name,EnaSupport,SriovNetSupport]' \
      --filters 'Name=name,Values=<SEARCH_STRING_PATTERN>' \
      --out text | \
   aws 'BEGIN {
         printf("%-18s%-60s%-14s-10s\n","AMI ID","AMI Name","ENA Support","SRIOV Support")
      } {
         printf("%-18s%-60s%-14s-10s\n",$1,$2,$3,$4)
      }'

• There's a lot of organizations and individuals publishing AMIs. Thus, we use the --owner flag to search only for AMIs we've published.
• We produce a couple of different families of AMIs. Thus, we use the --filter statement to only show the subset of our AMIs we're interested in.
• I really only care about four attributes of the AMIs being reported on: ImageId, Name, EnaSupport and SriovNetSupport. Thus, the use of the JMSE --query statement to suppress all output except for that in which I'm interested.
• Since I want the output to be pretty, I used the compound awk statement to create a formatted header and apply the same formatting to the output from the AWS CLI (using but a tiny bit of the printf routine's many capabilities).

This will produce output similar to:

   AMI ID                 AMI Name                                        ENA Support  SRIOV Support
   ami-187af850f113c24e1  spel-minimal-centos-7-hvm-2019.03.1.x86_64-gp2  True         simple
   ami-91b38c446d188643e  spel-minimal-centos-7-hvm-2019.02.1.x86_64-gp2  True         simple
   ami-22867cf08bb264ac4  spel-minimal-centos-7-hvm-2019.01.1.x86_64-gp2  True         simple
   [...elided...]
   ami-71c3822ed119c3401  spel-minimal-centos-7-hvm-2018.03.1.x86_64-gp2  None         simple
   [...elided...]
   ami-8057c2bf443dc01f5  spel-minimal-centos-7-hvm-2016.06.1.x86_64-gp2  None         None

As you can see, not all of the above AMIs are externally alike. While this could indicate a process or personnel problem, what my output actually shows is evolution in our AMIs. Originally, we weren't doing anything to support SRIOV or ENA. Then we added SRIOV support (because our AMI users were finally asking for it). Finally, we added ENA support (mostly so we could use the full range and capabilities of the fifth-generation EC2 instance-types).

At any rate, running a report like the above, we can identfy if there's unexpected differences and, if a sub-standard AMI slips out, we can alert our AMI users "don't use <AMI> if you have need of ENA and/or SRIOV".

Crib-Notes: Tracing Permissions for an Instance Role That Uses Inline Policies

The rationale for this crib-note is essentially identical to that for the previous topic on tracing IAM permissions in instance-roles that use managed-policies. So I'm just going to crib the rest of the intro...

When working with AWS EC2 instances – particularly when automating the deployments and lifecycles thereof – it's common to make use of the AWS IAM system's Instance Role Policy feature. Occasionally, you might get asked, "what permissions have been given to that instance." To answer, you might use AWS's IAM web console or the IAM CLI. In the former case, to get the information to the requestor, you're left with the options of either taking screen-shots or trying to copy and paste the text from the UI to your email/slack/etc. reply. In the latter case, you can just dump out the JSON-formatted policy-document that enumerates the permissions.

Generally, I prefer the CLI option since I don't have to worry "what does the recipient need to do with the results". Just dumping a text file, I needn't worry about being yelled at that the contents of a screen-shot can't be used to easily create another, similar policy. It's also easy to simply redirect the output to a file and attach it to whatever media I'm responding to ...or, if mail is enabled within the CLI environment, simply pipe the output directly to a CLI-based email tool and save a step in the process (laziness for the win!).

But, how to go from "there's an instance in this account: what privileges does it have" to "here's what it can do?". Basically, it's a three-step process:

1. Get the name of the Instance Role attached to the EC2 instance. This can be done with a method similar to:
   
   

   $ aws ec2 describe-instances --instance-id <INSTANCE_ID> \
         --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' --out text | \
     sed 's/^.*arn:.*profile\///'

   
   
2. Get th list of inline policies attached to role. This can be done with a method similar to:
   
   

   $ aws iam list-role-policies --role-name <OUPUT_FROM_PREVIOUS>

   
   
3. Get the list of permissions associated with the instance role's inline policy/policies. This can be done with a method similar to:
   
   

   $ aws iam get-role-policy --role-name INSTANCE --policy-name <OUPUT_FROM_PREVIOUS>

   
   

The above steps will dump out the full IAM policy/permissions of the queried inline policy:

• If the inline policy was the only policy attached to the role, the output will show all of the permissions the instance role grants any EC2s it is attached to.
• If the inline policy was the not the only inline policy attached to the role, it will be necessary to iterate over the remaining policies attached to the role to get the aggregated permission-set.

To facilitate the iteration, one can use a script similar to the following to encapsulate the second and third steps from prior process description:

#!/bin/bash
#
# Script to dump out all permissions granted through an IAM role with multiple
# inline IAM policies attached.
###########################################################################

if [[ $# -eq 0 ]]
then
   echo "Usage: ${0} <instance-id>" >&2
   exit 1
fi

PROFILE_NAME=$( aws ec2 describe-instances --instance-id "${1}" \
   --query 'Reservations[].Instances[].IamInstanceProfile[].Arn' --out text | \
  tr '\t' '\n' | sort -u | sed 's/^.*arn:.*profile\///' )

POLICY_LIST_RAW=$( aws iam list-role-policies --role-name ${PROFILE_NAME} )
POLICY_LIST_CLN=($( echo ${POLICY_LIST_RAW} | jq .PolicyNames[] | sed 's/"//g' ))

for ITER in $( seq 0 $(( ${#POLICY_LIST_CLN[@]} - 1 )) )
do
  aws iam get-role-policy --role-name "${PROFILE_NAME}" \
    --policy-name "${POLICY_LIST_CLN[${ITER}]}"
done

Crib-Notes: Tracing Permissions for an Instance Role That Uses Managed Policies

When working with AWS EC2 instances – particularly when automating the deployments and lifecycles thereof – it's common to make use of the AWS IAM system's Instance Role Policy feature. Occasionally, you might get asked, "what permissions have been given to that instance." To answer, you might use AWS's IAM web console or the IAM CLI. In the former case, to get the information to the requestor, you're left with the options of either taking screen-shots or trying to copy and paste the text from the UI to your email/slack/etc. reply. In the latter case, you can just dump out the JSON-formatted policy-document that enumerates the permissions.

Generally, I prefer the CLI option since I don't have to worry "what does the recipient need to do with the results". Just dumping a text file, I needn't worry about being yelled at that the contents of a screen-shot can't be used to easily create another, similar policy. It's also easy to simply redirect the output to a file and attach it to whatever media I'm responding to ...or, if mail is enabled within the CLI environment, simply pipe the output directly to a CLI-based email tool and save a step in the process (laziness for the win!).

But, how to go from "there's an instance in this account: what privileges does it have" to "here's what it can do?" Basically, it's a four-step process:

1. Get the name of the Instance Role attached to the EC2 instance. This can be done with a method similar to:
   

   $ aws ec2 describe-instances --instance-id <INSTANCE_ID> \
       --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' --out text | \
     sed 's/^.*arn:.*profile\///'
   $ aws iam get-instance-profile --instance-profile-name <OUPUT_FROM_PREVIOUS> \
       --query 'InstanceProfile.Roles[].RoleName' --out text
    

2. Get list of policies attached to role. This can be done with a method similar to:
   

   aws iam list-attached-role-policies --role-name <OUPUT_FROM_PREVIOUS> \
       --query 'AttachedPolicies[].PolicyArn[]' --out text
    

3. Find current version of attached policy/policies. Thi can be done with a method similar to:
   

   aws iam get-policy --policy-arn <OUPUT_FROM_PREVIOUS> \
       --query 'Policy.DefaultVersionId' --out text
   

4. Get contents of attached policy/policies active version. This can be done – using the outputs from steps #2 and #3 – with a method similar to:
   

   aws iam get-policy-version --policy-arn <OUPUT_FROM_STEP_2> \
        --version-id <OUPUT_FROM_STEP_3>
    

The above steps will dump out the full IAM policy/permissions of the queried managed-policy:

• If the inline policy was the only policy attached to the role, the output will show all of the permissions the instance role grants any EC2s it is attached to.
  
• If the inline policy was the not the only inline policy attached to the role, it will be necessary to iterate over the remaining policies attached to the role to get the aggregated permission-set.

To facilitate the iteration, one can use a script similar to the following to encapsulate the steps from prior process description:

#!/bin/bash
#
# Script to dump out all permissions granted through an IAM role with multiple
# managed IAM policies attached.
###########################################################################

if [[ $# -eq 0 ]]
then
   echo "Usage: ${0} <instance-id>" >&2
   exit 1
fi

PROFILE_NAME="$( aws ec2 describe-instances --instance-id "${1}" \
  --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' \
  --out text | sed 's/^.*arn:.*profile\///' )"
ROLE_NAME="$( aws iam get-instance-profile \
  --instance-profile-name "${PROFILE_NAME}" \
  --query 'InstanceProfile.Roles[].RoleName' \
  --out text | sed 's/^.*arn:.*profile\///' )"
ATTACHED_POLICIES=($( aws iam list-attached-role-policies \
  --role-name "${ROLE_NAME}" --query 'AttachedPolicies[].PolicyArn[]' | \
  jq .[] | sed 's/"//g' ))

for ITER in $( seq 0 $(( ${#ATTACHED_POLICIES[@]} - 1 )) )
do
   POLICY_VERSION=$( aws iam get-policy --policy-arn \
     ${ATTACHED_POLICIES[${ITER}]} --query \
     'Policy.DefaultVersionId' --out text )
   aws iam get-policy-version --policy-arn ${ATTACHED_POLICIES[${ITER}]} \
     --version-id "${POLICY_VERSION}"
done