Crib-Notes: Tracing Permissions for an Instance Role That Uses Inline Policies

The rationale for this crib-note is essentially identical to that for the previous topic on tracing IAM permissions in instance-roles that use managed-policies. So I'm just going to crib the rest of the intro...

When working with AWS EC2 instances – particularly when automating the deployments and lifecycles thereof – it's common to make use of the AWS IAM system's Instance Role Policy feature. Occasionally, you might get asked, "what permissions have been given to that instance." To answer, you might use AWS's IAM web console or the IAM CLI. In the former case, to get the information to the requestor, you're left with the options of either taking screen-shots or trying to copy and paste the text from the UI to your email/slack/etc. reply. In the latter case, you can just dump out the JSON-formatted policy-document that enumerates the permissions.

Generally, I prefer the CLI option since I don't have to worry "what does the recipient need to do with the results". Just dumping a text file, I needn't worry about being yelled at that the contents of a screen-shot can't be used to easily create another, similar policy. It's also easy to simply redirect the output to a file and attach it to whatever media I'm responding to ...or, if mail is enabled within the CLI environment, simply pipe the output directly to a CLI-based email tool and save a step in the process (laziness for the win!).

But, how to go from "there's an instance in this account: what privileges does it have" to "here's what it can do?". Basically, it's a three-step process:

1. Get the name of the Instance Role attached to the EC2 instance. This can be done with a method similar to:
   
   

   $ aws ec2 describe-instances --instance-id <INSTANCE_ID> \
         --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' --out text | \
     sed 's/^.*arn:.*profile\///'

   
   
2. Get th list of inline policies attached to role. This can be done with a method similar to:
   
   

   $ aws iam list-role-policies --role-name <OUPUT_FROM_PREVIOUS>

   
   
3. Get the list of permissions associated with the instance role's inline policy/policies. This can be done with a method similar to:
   
   

   $ aws iam get-role-policy --role-name INSTANCE --policy-name <OUPUT_FROM_PREVIOUS>

   
   

The above steps will dump out the full IAM policy/permissions of the queried inline policy:

• If the inline policy was the only policy attached to the role, the output will show all of the permissions the instance role grants any EC2s it is attached to.
• If the inline policy was the not the only inline policy attached to the role, it will be necessary to iterate over the remaining policies attached to the role to get the aggregated permission-set.

To facilitate the iteration, one can use a script similar to the following to encapsulate the second and third steps from prior process description:

#!/bin/bash
#
# Script to dump out all permissions granted through an IAM role with multiple
# inline IAM policies attached.
###########################################################################

if [[ $# -eq 0 ]]
then
   echo "Usage: ${0} <instance-id>" >&2
   exit 1
fi

PROFILE_NAME=$( aws ec2 describe-instances --instance-id "${1}" \
   --query 'Reservations[].Instances[].IamInstanceProfile[].Arn' --out text | \
  tr '\t' '\n' | sort -u | sed 's/^.*arn:.*profile\///' )

POLICY_LIST_RAW=$( aws iam list-role-policies --role-name ${PROFILE_NAME} )
POLICY_LIST_CLN=($( echo ${POLICY_LIST_RAW} | jq .PolicyNames[] | sed 's/"//g' ))

for ITER in $( seq 0 $(( ${#POLICY_LIST_CLN[@]} - 1 )) )
do
  aws iam get-role-policy --role-name "${PROFILE_NAME}" \
    --policy-name "${POLICY_LIST_CLN[${ITER}]}"
done

Crib-Notes: Tracing Permissions for an Instance Role That Uses Managed Policies

When working with AWS EC2 instances – particularly when automating the deployments and lifecycles thereof – it's common to make use of the AWS IAM system's Instance Role Policy feature. Occasionally, you might get asked, "what permissions have been given to that instance." To answer, you might use AWS's IAM web console or the IAM CLI. In the former case, to get the information to the requestor, you're left with the options of either taking screen-shots or trying to copy and paste the text from the UI to your email/slack/etc. reply. In the latter case, you can just dump out the JSON-formatted policy-document that enumerates the permissions.

Generally, I prefer the CLI option since I don't have to worry "what does the recipient need to do with the results". Just dumping a text file, I needn't worry about being yelled at that the contents of a screen-shot can't be used to easily create another, similar policy. It's also easy to simply redirect the output to a file and attach it to whatever media I'm responding to ...or, if mail is enabled within the CLI environment, simply pipe the output directly to a CLI-based email tool and save a step in the process (laziness for the win!).

But, how to go from "there's an instance in this account: what privileges does it have" to "here's what it can do?" Basically, it's a four-step process:

1. Get the name of the Instance Role attached to the EC2 instance. This can be done with a method similar to:
   

   $ aws ec2 describe-instances --instance-id <INSTANCE_ID> \
       --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' --out text | \
     sed 's/^.*arn:.*profile\///'
   $ aws iam get-instance-profile --instance-profile-name <OUPUT_FROM_PREVIOUS> \
       --query 'InstanceProfile.Roles[].RoleName' --out text
    

2. Get list of policies attached to role. This can be done with a method similar to:
   

   aws iam list-attached-role-policies --role-name <OUPUT_FROM_PREVIOUS> \
       --query 'AttachedPolicies[].PolicyArn[]' --out text
    

3. Find current version of attached policy/policies. Thi can be done with a method similar to:
   

   aws iam get-policy --policy-arn <OUPUT_FROM_PREVIOUS> \
       --query 'Policy.DefaultVersionId' --out text
   

4. Get contents of attached policy/policies active version. This can be done – using the outputs from steps #2 and #3 – with a method similar to:
   

   aws iam get-policy-version --policy-arn <OUPUT_FROM_STEP_2> \
        --version-id <OUPUT_FROM_STEP_3>
    

The above steps will dump out the full IAM policy/permissions of the queried managed-policy:

• If the inline policy was the only policy attached to the role, the output will show all of the permissions the instance role grants any EC2s it is attached to.
  
• If the inline policy was the not the only inline policy attached to the role, it will be necessary to iterate over the remaining policies attached to the role to get the aggregated permission-set.

To facilitate the iteration, one can use a script similar to the following to encapsulate the steps from prior process description:

#!/bin/bash
#
# Script to dump out all permissions granted through an IAM role with multiple
# managed IAM policies attached.
###########################################################################

if [[ $# -eq 0 ]]
then
   echo "Usage: ${0} <instance-id>" >&2
   exit 1
fi

PROFILE_NAME="$( aws ec2 describe-instances --instance-id "${1}" \
  --query 'Reservations[].Instances[].IamInstanceProfile[].Arn[]' \
  --out text | sed 's/^.*arn:.*profile\///' )"
ROLE_NAME="$( aws iam get-instance-profile \
  --instance-profile-name "${PROFILE_NAME}" \
  --query 'InstanceProfile.Roles[].RoleName' \
  --out text | sed 's/^.*arn:.*profile\///' )"
ATTACHED_POLICIES=($( aws iam list-attached-role-policies \
  --role-name "${ROLE_NAME}" --query 'AttachedPolicies[].PolicyArn[]' | \
  jq .[] | sed 's/"//g' ))

for ITER in $( seq 0 $(( ${#ATTACHED_POLICIES[@]} - 1 )) )
do
   POLICY_VERSION=$( aws iam get-policy --policy-arn \
     ${ATTACHED_POLICIES[${ITER}]} --query \
     'Policy.DefaultVersionId' --out text )
   aws iam get-policy-version --policy-arn ${ATTACHED_POLICIES[${ITER}]} \
     --version-id "${POLICY_VERSION}"
done

Fixing CIFS ACLs On a DataDomain

On my current project, our customer makes use of DataDomain storage to provide nearline backups of system data. Backups to these devices is primarily done through tools like Veeam (for ESX-hosted virtual machines and associated application data) and NetBackup.

However, a small percentage of the tenants my customer hosts are "self backup" tenants. In this case, "self backup" means that, rather than leveraging the enterprise backup frameworks, the tenants are given an NFS or CIFS share directly off of the DataDomain that is: A) closest to their system(s); and, B) has the most free space to accommodate their data.

"Self backup" tenants that use NFS shares are minimally problematic. Most of the backup problems come from the fact that DataDomains weren't really designed for multi-tenancy. Things like quota controls are fairly lacking. So, it's possible for a tennants of a shared DataDomain to screw each other over by either soaking up all of the device's bandwidth or soaking up all the space.

Still, those problems aside, providing NFS service to tenants is fairly straight-forward. You create a directory, you go into the NFS share-export interface, create the share and access controls and call it a day. CIFS shares, on the other hand...

While we'd assumed that providing CIFS service would be on a par to providing NFS service, it's proven to be otherwise. While the DataDomains provide an NTFS-style ACL capability in their filesystem, it hasn't proven to work quite as one might expect.

The interface for creating shares allows you to set share-level access controls based on both calling-host as well as assigned users and/or groups. One would reasonably assume that this would mean that the correct way to set up a share is to export it with appropriate client-allow lists and user/group-allow lists and that the shares would be set with appropriate filesystem permissions automagically. This isn't exactly how it's turned out to work.

What we've discovered is that you pretty much have to set the shares up as being universally accessible from all CIFS clients and that you grant global "full control" access to the top-level share-folder. Normally, this would be a nightmare, but, once created, you can lock the shares down. You just have to manage the NTFS attributes from a Windows-based host. Basically, you create the share, present it to a Windows-based administrative host, then use the Windows folder security tools to modify the permissions on the share (e.g., remove all the "Everyone" rights, then manually assign appropriate appropriate ownerships and posix gropus to the folder and set up the correct DACLs.

From an engineering perspective, it means that you have to document the hell out of things and try your best to train the ops folks on how to do things The Right Way™. Then, with frequent turnovers in Operations and other "shit happens" kind of things, you have to go back and periodically audit configurations for correctness and repair the brokenness that has crept in.

Unfortunately, one of the biggest sources of brokenness that creeps in is broken permissions structures. When doing the initial folder-setup, it's absolutely critical that the person setting up the folder remembers to click the "Replace all child object permissions with inheritable permissions from this object" checkbox (accessed by clicking on the "Change Permissions" button within the "Advanced Security Settings" section for the folder). Failure to do so makes it so that each folder, subfolder and file created (by tenants) in the share have their own, tenant-created permissions structures. What this results in is a share whose permissions are not easily maintainable by the array-operators. Ultimately, it results in trouble tickets opened by tenants whose applications and/or operational folks eventually break access for themselves

Once those tickets come in, there's not much that can be easily done if the person who "owns" the share has left the organization. If you find yourself needing to fix such a situation, you need to either involve DataDomain's support staff to fix it (assuming your environment is reachable via an WebEx-type of support session) or get someone to slip you instructions on how to access the array's "Engineering Mode"

There's actually two engineering modes: there's the regular SE shell and the BASH shell. The SE shell is basically a super-set of the regular system administration CLI. The BASH shell is basically a Linux BASH shell with DataDomain-specific management commands enabled. For the most part, the two modes are interchangable. However, if you need the ability to do mass modifications or script on your array, you'll need to access the DataDomain's BASH shell mode to do it. See my prior article on accessing the DataDomain's BASH shell mode.

Once you've gotten the engineering BASH shell, you have pretty much unfettered access to the guts of the DataDomain. The BASH shell is pretty much the same as you'd encounter on a stock Linux system. Most of the GNU utilities you're used to using will be there and will work the same way they do on Linux. You won't have man pages, so, if you forget flags to a given shell command, look them up on a Linux host that has the man pages installed. In addition to the standard Linux commands will be some DataDomain-specific commands. For the purposes of fixing your NTFS ACL mess, you'll be wanting to use the "dd_xcacls" command:

• Use "dd_xcacls -O '[DomainObject]' [Object]" to set the Ownership of an object. For example, to set the ownership attribute to your AD domain account, issue the command "dd_xcacls -O 'MDOMAIN\MYUSER' /data/col1/backup/ShareName".
• Use "dd_xcacls -G '[DomainObject]' [Object]" to set the POSIX group of an object.  For example, to set the POSIX group attribute to your AD domain group, issue the command "dd_xcacls -O 'MDOMAIN\MYUSER' /data/col1/backup/ShareName".
• Use "dd_xcacls -D '[ActiveDirectorySID]:[Setting]/[ScopeMask]/[RightsList]' [OBJECT]" to set the POSIX group of an object. For example, to give "Full Control" rights to your domain account, issue the command "dd_xcacls -D 'MDOMAIN\MYUSER:ALLOW/4/FullControl' /data/col1/backup/ShareName".

A couple of notes apply to the immediately preceding:

1. While the "dd_xcacls" command can notionally set rights-inheritance, I've discovered that this isn't 100% reliable in the DDOS 5.1.x family. It will likely be necessary that once you've placed the desired DACLs on the filesystem objects, you'll need to use a Windows system to set/force inheritance onto objects lower in the filesystem hierarchy.
2. When you set a DACL with "dd_xcacls -D", it replaces whatever DACLS are in place. Any permissions previously on the filesystem object will be removed. If you want more than one user/group DACL applied to the filesystem-object, you need to apply them all at once. Use the ";" token to separate DACLs within the quoted text-argument to the "-D" flag

Because you'll need to fix all of your permissions, one at a time, from this mode, you'll want to use the Linux `find` command to power your use of the  "dd_xcacls" command. On a normal Linux system, when dealing with filesystems that have spaces in directory or file object-names, you'd do something like `find [DIRECTORY] -print0 | xargs -0 [ACTION]` to more efficiently handle this. However, that doesn't seem to work exactly like on a generic Linux system, at least not on the DDOS 5.x systems I've used. Instead, you'll need to use a `find [Directory] -exec [dd_xcacls command-string] {} \;`. This is very slow and resource intensive. On a directory structure with thousands of files, this can take hours to run. Further, because of how resource-intensive using this method is, you won't be able to run more than one such job at a time. Attempting to do so will result in SEGFAULTs - and the more you attempt to run concurrently, the more frequent the SEGFAULTs will be. These SEGFAULTs will cause individual "dd_xcacls" iterations to fail, potentially leaving random filesystem objects permissions unmodified.

ACL Madness

In our environment, security is a concern. So, many programs and directories that you might take being available to you on a "standard" Linux system will give you a "permission denied" on one of our systems. Traditionally, you might work around this by changing the group-ownership and permissions on the object to allow a subset of the system users the expected level of access to those files or directories. And, this will work great if all the people that need access share a common group. If they don't you have to explore other avenues. One of these avenues is the extened permissions attributes found in Linux's ACL handling (gonna ignore SELinux, here, mostly because: A) I'm lazy; B) I freaking hate SELinux - probably because of "A"; and, C) ACLs are a feature that is available across different implementations of UNIX and Linux, and is thus more portable than SELinux or other vendor-specific extended security attribute mechanisms). Even better, as a POSIX filesystem extension, you can use it on things like NFS shares and have your ACLs work across systems and *N*X platforms (assuming everone implements the POSIX ACLs the same way).

And, yes, I know things like `sudo` can be used to delegate access, but that's fraught with its own concerns, as well, least of all its lack of utility for users that aren't logging in with interactive shells.

Say you have a file <tt>/usr/local/sbin/killmenow</tt> that is currently set to mode 700 and you want to give access to it to members of the ca_opers and md_admins groups. You can do something like:

# setfacl -m g:ca_opers:r-x /usr/local/sbin/killmenow
# setfacl -m g:md_admins:r-x /usr/local/sbin/killmenow

Now, members of both the ca_opers and md_admins groups can run this program.

All is well and good until someone asks, "what files have had their ACLs modified to allow this" and you've (or others on your team) gone crazy with the setfacl command. With standard file permissions, you can just use the `find` command to locate files that have a specific permissions-setting. With ACLs, `find` is mostly going to let you down. So, what to do? Fortunately,  setfacl 's partner-command, `getfacl ` can come to your rescue. Doing something like

# getfacl --skip-base -R 2> /dev/null | sed -n 's/^# file://p`

Will walk the directory-structure, from downward, giving you a list of files with ACLs added to them. Once you've identified such-modified files, you can then run `getfacl` against them, individually, to show what the current ACLs are.