The Hatefulness of SELinux Compounded By IPA

A couple months ago, one of my customers' cyber-security teams handed down an edict that all IPA-managed users needed to have SELinux confinements applied. For admistrative accounts, this meant that administrators would SSH system and have an SELinux assignment of:

$ id -Z
staff_u:staff_r:staff_t:s0-s0:c0.c1023

And that when such administrative-users executed `sudo`, they would end up with an SELinux assignment of:

# id -Z
staff_u:sysadm_r:sysadm_t:s0-s0:c0.c1023

Not long after this, that same cyber-security team opened up a bug-ticket complaining that their scanners were no longer able to conduct all the tests they were able to conduct prior to the implementation of the confinement scheme (shocking, I know).

Since their scan-user operated with the same privilege set as human administrators did, their scan-user was getting similarly constrained. As an example of the practical impacts of this confinement, one need only try to look at the contents of the /etc/shadow file:

$ sudo cat /etc/shadow
/bin/cat: /etc/shadow: Permission denied

Since their security tooling was using that confined user-account to do – among a whole host of other check-tasks – functionally similar tests, naturally, some of their tests started failing and red lights started showing up on their dashboards. Worse (for them), their boss started asking why his summary-reports suddenly started to look like the elevator scene in The Shining after months of being mostly green.

Their problem ended up in my task-pile. I'm not really an SELinux guy …even if I can usually figure out how to get most third-party applications working that don't come with SELinux policy without having to resort to disabling SELinux. And, as much of "not really an SELinux guy" as I am, I'm really not an IPA guy. So, "fun times ahead".

Not wanting to waste time on the problem, and knowing that my customer had a Red Hat Enterprise Support entitlement, I opted to try to avail myself of that support. With my customer being part of a large, very-siloed organization, just getting a support case opened – and then actually able to directly read and comment on it rather than having to play "grape vine" – proved to be its own, multi-week ordeal.

After several days of back-and-forth, I was finally able to get the case escalated to people with SELinux and IPA expertise …but, seemingly, no one with both sets of expertise (yay). So, I was getting SELinux answers that included no "how do we actually do this wholly within IPA" and I was getting generic IPA guidance. The twain never quite met.

Ultimately, I was given guidance to do (after creating a scan-user POSIX-group to put the scan-user into):

  ipa selinuxusermap-add --hostcat='all' --selinuxuser=unconfined_u:s0-s0:c0.c1023 <SCAN_GROUP_NAME>
  ipa selinuxusermap-add-user <SCAN_GROUP_NAME> --group=<SCAN_GROUP_NAME>
  ipa sudorule-add <SCAN_USER_SUDO_RULE_NAME> --hostcat='all' --cmdcat='all'
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption '!authenticate'
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption role=unconfined_r
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption type=unconfined_t
  ipa sudorule-add-user <SCAN_GROUP_RULE_NAME> --group=<SCAN_GROUP_NAME>
  sudo bash -c "service sssd stop ; rm -f /var/lib/sss/db/* ; service sssd start"

Unfortunately, this didn't solve my problem. My scan-user continued to be given the same SELinux profile upon logging in and executing `sudo` that my "normal" administrative users were. Support came back and told me to re-execute the above, but to give the sudorule a precedence-setting:

  ipa selinuxusermap-add --hostcat='all' --selinuxuser=unconfined_u:s0-s0:c0.c1023 <SCAN_GROUP_NAME>
  ipa selinuxusermap-add-user <SCAN_GROUP_NAME> --group=<SCAN_GROUP_NAME>
  ipa sudorule-add <SCAN_USER_SUDO_RULE_NAME> --hostcat='all' --cmdcat='all' --order=99
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption '!authenticate'
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption role=unconfined_r
  ipa sudorule-add-option <SCAN_GROUP_RULE_NAME> --sudooption type=unconfined_t
  ipa sudorule-add-user <SCAN_GROUP_RULE_NAME> --group=<SCAN_GROUP_NAME>
  sudo bash -c "service sssd stop ; rm -f /var/lib/sss/db/* ; service sssd start"

Still, even having set a precedence on the `sudo` rule, my scan-user wasn't getting the right confinement rules applied. The support rep had me execute `ipa user-show <SCAN_USER> --all`. Upon doing that, we noticed that the <SCAN_USER> account was affiliated with two `sudo` rules. Then using `ipa sudorule-show <SUDO_RULE_NAME> --all` for each rule, was able to find that one of the two rules was applying the `!authenticate`, `role=sysadm_r` and `type=sysadm_t` sudo-options.

I made the assumption that, even though we'd explicitly set a precedence-value on the one rule, the previously-existing rule likely didn't have a precedence-value set and that, as a result, implicit-precedence rules were still being applied ...and that the previously-existing rule "won" in that scenario. To test, I did:

ipa sudorule-mod <EXISTING_RULE> --order=10

And then retested with my scan-user. At this point, the scan-user was functioning as desired. This meant that:

1. My supposition about implicit and explicit rule-ordering was correct
2. That the rules are processed on a "higher-number == higher-priority" precedence-processing

Unfortunately, due to limitations in RHEL7 and the version of IPA in use by my customer (vendor-rep checked internal documentation for this determination), I had to leave the scan-user wholly unconstrained. Customer's security-team is likely to gripe about that, but I figure that, since they own the offending user, they can deal with the consequences (paperwork) associated with adequately-relaxing security for that user so that their scans would return to a functional state.

You CAN Make the Logs' Format Worse?!

Most of my customers have security compliance mandates that makes it necessary to offload their Linux auditd event-logs to an external/centralized logging-destination. One of my customers leverages a third-party tool to offload their EC2 logs directly to S3. However, because they use a common compliance-framework to guide their EC2s' hardening configurations, they had been configuring their systems to use the "normal" auditd dispatch plugin service, audisp. Unfortunately, prior my arrival, no one had actually bothered to validate their auditing configuration. Turns out, audisp was trying to off-host the event logs to a non-existent, centralized event-collector that didn't actually exist.

My "solution" to their problem was to simply eliminate the CM-automation that sets up the errant event-offloading. However, before I could suggest summarily nuking this automation-content, I had to verify that their S3-based logging solution was actually working. So, decided to cobble together a quick-n-dirty AWS CLI command – because their log-stream names are the same as the associated EC2s' instance-IDs, doing the quick test was easy:

$ aws logs filter-log-events\
  --log-group-name  \
  --log-stream-names $(
    curl -s http://169.254.169.254/latest/meta-data/instance-id
  )   --start-time $(
    date -d '-15 minutes' '+%s%N' | \
    cut -b 1-13
  )

To explain the above - particularly the nested subshells…

Executing:

curl -s http://169.254.169.254/latest/meta-data/instance-id

Makes use of the EC2 metadata service to return the EC2s own AWS instance-ID. Because it's being executed in a subshell – using the $( COMMAND ) notation – The instance's ID is returned as the value fed to the `--log-stream-names` command-option.

Similarly, executing:

date -d '-15 minutes' '+%s%N' | cut -b 1-13

Makes use of the `date` command to return an `aws logs` compatible time specification for use by the --start-time command-option. Specifically, it tells the date command, "take the time 15 minutes ago and convert it to milliseconds since epoch-time. Because the time-shifted, date-string is longer than what the  `aws logs` command-option expects, we use the `cut` command to truncate it to a compatible length. Ultimately, this will return output similar to:

{
  "logStreamName": "i-0a8b14c42b651476f",
  "timestamp": 1664815386793,
  "message": "{\"a0\":\"b75480\",\"a1\":\"0\",\"a2\":\"0\",\"a3\":\"7ffeb692d220\",\"arch\":\"c000003e\",\"auid\":\"239203505\",\"az\":\"us-gov-west-1a\",\"comm\":\"vim\",\"ec2_instance_id\":\"i-0a8b14c42b651476f\",\"egid\":\"239203505\",\"euid\":\"239203505\",\"exe\":\"/usr/bin/vim\",\"exit\":\"0\",\"fsgid\":\"239203505\",\"fsuid\":\"239203505\",\"gid\":\"239203505\",\"items\":\"2\",\"key\":\"delete\",\"log_file\":\"/var/log/audit/audit.log\",\"node\":\"ip-10-244-0-100.dev.ac2sp.army.mil\",\"pid\":\"10503\",\"ppid\":\"3961\",\"ses\":\"3\",\"sgid\":\"239203505\",\"subj\":\"staff_u:staff_r:staff_t:s0-s0:c0.c1023\",\"success\":\"yes\",\"suid\":\"239203505\",\"syscall\":\"84\",\"tty\":\"pts1\",\"type\":\"SYSCALL\",\"uid\":\"239203505\"}",
  "ingestionTime": 1664815388879,
  "eventId": "37126623743463896942643741908611915331411397433463734292"
}

Kind of horrible. But, you can extract the actual message payload with a tool like `jq` which will give you output like:

{
    "a0": "b75480",
    "a1": "0",
    "a2": "0",
    "a3": "7ffeb692d220",
    "arch": "c000003e",
    "auid": "239203505",
    "az": "us-gov-west-1a",
    "comm": "vim",
    "ec2_instance_id": "i-0a8b14c42b651476f",
    "egid": "239203505",
    "euid": "239203505",
    "exe": "/usr/bin/vim",
    "exit": "0",
    "fsgid": "239203505",
    "fsuid": "239203505",
    "gid": "239203505",
    "items": "2",
    "key": "delete",
    "log_file": "/var/log/audit/audit.log",
    "node": "ip-10-244-0-100.dev.ac2sp.army.mil",
    "pid": "10503",
    "ppid": "3961",
    "ses": "3",
    "sgid": "239203505",
    "subj": "staff_u:staff_r:staff_t:s0-s0:c0.c1023",
    "success": "yes",
    "suid": "239203505",
    "syscall": "84",
    "tty": "pts1",
    "type": "SYSCALL",
    "uid": "239203505"
}

Yeah… If you're familiar with audit.log output, that ain't normally what it looks like. Here, the OS's event-data has been intermingled with the CSP's event log-tracking data. Not great. Not something you can feed "as-is" to generic tools that expect to directly interact with auditd data (e.g., the tools in the policyutils RPMs). But, you can write filters to get it back into a tool-compatible format. But still, it manages to make an ugly log format markedly-uglier to deal with. So, uh, congratulations?