Showing posts with label security. Show all posts
Showing posts with label security. Show all posts

Wednesday, March 18, 2026

Why, Palo Alto, WHY??

Yesterday, I was doing up some SaltStack based automation to help a customer automate the installation of the Cortex XDR Agent on RHEL-based Linux hosts. The vendor delivers the agent in the form of a ZIP-archived RPM. Yeah, I was a bit unimpressed by them deciding an RPM needed to be encapsulated in a ZIP-archive.

When you read the installation documentation, there's a link in the page they tell you to download. Yesterday, the embedded link was:

https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip

This URL was actually set up as an HTTP 302 (redirect) to:

https://docs-cortex.paloaltonetworks.com/api/khub/documents/Im1wc74y4HN15mXxBu3nYQ/content?Ft-Calling-App=ft%2Fturnkey-portal&Ft-Calling-App-Version=5.2.49&download=true&locationValue=viewer

SaltStack didn't care for trying to use file.managed to try to download from a redirect. I had to whip up some logic to chase the redirect and stuff it into a (Jinja) variable. Worked well once I got it in place.

Today, I was attempting to continue with the refactoring that I'd started, yesterday. This means launching an EC2 with the new automtion. I was surprised to find that the automation — unaltered since yesterday's day-ending push — was failing. When I checked the logs, SaltStack was complaining that I was trying to pass a null value to file.managed's source parameter. Perplexed, I started troubleshooting.

Ultimately, what I found was that the URL I was doing redirection-chasing on was no longer redirecting. Using curl like:

curl -Ls -o /dev/null 
  -w %{url_effective} "https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip"

Was returning null. So, I opted to try to make it provide more-definitive output like:

curl -Ls -o /dev/null \
  -w "Status: %{http_code}\nEffective URL: %{url_effective}\n" \
  "https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip"

This returned:

Status: 200
Effective URL: https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip

Which is to say, the reference URL is no longer returning an HTTP 302. Thus receiving a null-value when looking for a "url_effective" value. 

So, I revisited the documentation. I scrolled down to the where the document called out the signature-file's link. When I hovered over the link, it's value had changed since yesterday. I guess they pushed out a documentation-portal change and, along with that, nuked the previous URL's redirect action. This meant that the URL I was expecting was now returning a data-stream. Looking at the data-stream's first and last five lines:

curl -Ls https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip | \
sed -n '1,5p; :a;$p;N;11,$D;ba'

It was obvious from the output that the HTTP data-stream was now sending me JavaScript, presumably substituting the prior HTTP redirect with JS-based navigation-aids. Unfortunately, those kinds of navigation-aids work well enough for graphical browsers but not at all well for curl-type methods. 

None of this would have been necessary if they just maintained a file-repository of the RPMs and their signing-keys. Or, since they were ZIPing up the RPMs, include the damned signing-key in the archive-file. But, no, that would be too fucking easy and way too sensible.

I wish I could say this was surprising. However, I've had to integrate enough tooling — particularly security tooling — that I've gotten sort of used to (especially security) vendors  doing things in absolutely baffling ways. At least this vendor wasn't doing things in ways that required reducing system-security to allow installation of their tool: that is a hallmark of the brain-damage I frequently witness with security vendors' tooling.

Friday, January 3, 2025

SSH Problems When Using `sysadm_u` SELinux Confinement

Decided to be proactive on the security-setup for my one project. Opted to confine my default-user to sysadm_u. However, as soon as I did that, I stopped being able to ssh into the resulting EC2 as the default-user. Turns out there's a bit more requirede in order to use that confinement with a user that also needs to be able to SSH into the host.

For those reading and who don't have a Red Hat login, if I want to confine a user to the to sysadm_u, I also need to ensure that my system-configuration automation includes:

setsebool ssh_sysadm_login on
setsebool -P ssh_sysadm_login on
Without the above, doing an ssh -v to the target user will show a spurious:
Authenticated to 0.0.0.0 ([127.0.0.1]:22) using "publickey".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem full
client_loop: send disconnect: Broken pipe

The `pledge: filesystem full` kinda threw me, at first, since I knew that neither my local nor my remote filesystem was full. So, I assumed that it was just a misleading error message (as seems to so often be the case when SELinux is involved). So, I searched for ssh login problems associated with the selected SELinux-confinement, which led me to the previously-linked Red Hat article.

I guess that's why the hardening guidelines show `staff_u` as the recommended confinement for administrator users?

 Ultimately, I opted to use  `staff_u`, instead. Having a cloud-config block like:

user: {
  gecos: "GitLab Provisioning-Account (LOCAL)",
  name: "${PROVISIONING_USER}",
  selinux_user: 'staff_u',
  sudo: ['ALL=(root) TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL']
}
Ensuring to have ROLE and TYPE SELinux transition-mappings defined for my default-user eliminates the confusion that can result when confining a user to staff_u and not supplying a mapping. Without the mapping, if an confined admin-user executes `sudo -i`, they get all sorts of unexpected `permission denied` errors.

Thursday, May 16, 2024

So You Work in Private VPCs and Want CLI Access to Your Linux EC2s?

 Most of the AWS projects I work on, both currently and historically, have deployed most, if not all, of their EC2s into private VPC subnets. This means that, if one wants to be able directly login to their Linux EC2s' interactive shells, they're out of luck. Historically, to get something akin to direct access one had to set up bastion-hosts in a public VPC subnet, and then jump through to the EC2s one actually wanted to login to. How well one secured those bastion-hosts could make-or-break how well-isolated their private VPC subnets – and associated resources – were.

If you were the sole administrator or part of a small team, or were part of an arbitrary-sized administration-group that all worked from a common network (i.e., from behind a corporate firewall or through a corporate VPN), keeping a bastion-host secure was fairly easy. All you had to do was set up a security-group that allowed only SSH connections and only allowed them from one or a few source IP addresses (e.g. your corporate firewall's outbound NAT IP address). For a bit of extra security, one could eve prohibit password-based logins on the Linux bastions (instead, using SSH key-based login, SmartCards, etc. for authenticating logins). However, if you were a member of a team of non-trivial size and your team members were geographically-distributed, maintaining whitelists to protect bastion-hosts could become painful. That painfulness would be magnified if that distributed team's members were either frequently changing-up their work locations or were coming from locations where their outbound IP address would change with any degree of frequency (e.g., work-from-home staff whose ISPs would frequently change their routers' outbound IPs).

A few years ago, AWS introduced SSM and the ability to tunnel SSH connections through SSM (see the re:Post article for more). With appropriate account-level security-controls, the need for dedicated bastion-hosts and maintenance of whitelists effectively vanished. Instead, all one had to do was:

  • Register an SSH key to the target EC2s' account
  • Set up their local SSH client to allow SSH-over-SSM
  • Then SSH "directly" to their target EC2s

SSM would, effectively, "take care of the rest" …including logging of connections. If one were feeling really enterprising, one could enable key-logging for those SSM-tunneled SSH connections (a good search-engine query should turn up configuration guides; one such guide is toptal's). This would, undoubtedly make your organization's IA team really happy (and may even be required depending on security-requirements your organization is legally-required to adhere to) – especially if they don't yet have an enterprise session-logging tool purchased.

But what if your EC2s are hosting applications that require GUI-based access to set up and/or administer? Generally, you have two choices:

  • X11 display-redirection
  • SSH port-forwarding

Unfortunately, SSM is a fairly low-throughput solution. So, while doing X11 display-redirection from an EC2 in a public VPC subnet may be more than adequately performant, the same cannot be said when done through an SSH-over-SSM tunnel. Doing X11 display-redirection of a remote browser session – or, worse, an entire graphical desktop session (e.g., KDE or Gnome desktops) – is paaaaaainfully slow. For my own tastes, it's uselessly slow. 

Alternately, one can use SSH port-forwarding as part of that SSH-over-SSM session. Then, instead of trying to send rendered graphics over the tunnel, one only sends the pre-rendered data. It's a much lighter traffic load with the result being a much quicker/livelier response. It's also pretty easy to set up. Something like:

ssh -L localhost:8080:$( aws ec2 describe-instances \ --query 'Reservations[].Instances[].PrivateIpAddress' \ --output text \ --instance-ids <EC2_INSTANCE_ID> ):80 <USERID>@<EC2_INSTANCE_ID>

Is all you need. In the above, the argument to the -L flag is saying, "set up a tcp/8080 listener on my local machine and have it forward connections to the remote machine's tcp/80". The local and remote ports can be varied for your specific needs. You can even set up dynamic-forwarding by creating a SOCKS proxy (but this document is meant to be a starting point, not dive into the weeds).

Note that, while the above is using a subshell (via the $( … ) shell-syntax) to snarf the remote EC2's private IP address, one should be able to simply substitute "localhost". I simply prefer to try to speak to the remote's ethernet, rather than loopback, interface, since doing so can help identify firewall-type issues that might interfere with others' use of the target service.

Friday, May 22, 2020

But We Want a Pre-Authentication Consent Banner!

My primary client-base is security- and compliance-focussed. As part of this, they follow security guidelines that include things like "system must display consent-to-monitor banners".

Up until recently, I only ever worked on the server end of things — almost exclusively UNIX- or Linux-based (though, the past decade has been almost exclusively Linux). This meant that all I really had to worry about was ensuring a suitable "/etc/issue file was in place and that any interactive login services (mostly SSH; occasionally FTP) were configured to reference that file."

Any GUI-dependent developers that my customers might have had did all of their development work on Windows boxes or specially-prepared Linux desktops. In either case, they were using on-premises desktops that I didn't have to worry about.

With COVID19 and the mass adoption of telework, those carefully-prepared development desktops are sitting disused at my customers offices. Developers have been forced to work remote. As such, we've been asked, as part of our "enablement" mandate, to help these newly-remote developers set up cloud-hosted development-systems. A significant percentage of these developers rely on graphical tools. While most — if not all — of these graphical tools would be usable as individually-launched applications with their X11 tunneled through SSH back to the workstations they're using, that hasn't been enough for some of them:
  • Many don't have X11 servers installed on their laptops
    • I usually like to point BYOD Windows users to Cygwin/X, MobaXterm, Xming, VcXsrv and the corporate-issued Windows users to contact their employers and ask that they add any one of the commercial Xserver offerings to their laptops
    • Though, given the number of Macintosh users in their ranks, they have Xservers, but it seems like they don't quite understand that fact:
  • In either case, most don't seem to understand that the "easy button" solution is to add:
    Host *
      ForwardAgent true
    To their ${HOME}/.ssh/config files. And that doing so allows them to SSH to their development-host and, by executing <GRAPHICAL_UTILITY> on the remote server, it will cause the utility to magically appear on their laptop.
  • Even once you show them the magic of X11 forwarding, many will whine "I don't want to have to have a terminal open just to executed programs, I want the entire remote desktop so I can just use the launchers."
Result, I've been having to help Linux neophytes install the full Gnome desktop on top of our standard AMIs and figuring out how to access them.

Our standard AMIs are RHEL- and CentOS-based have little more on them than the @Core RPM-group. The AMIs also have boot-EBSes that are as small as we could make them and meet the server security-requirements that require the AMIs to be carved up into partitions. These initially led to several common questions:

  1. "Can you create new AMIs that have enough room for us to install the Gnome desktop onto"; and
  2. "Can you create an AMI with the Gnome-desktop preinstalled". 
The answer to both has always been, "no, but if you follow the FAQ on how to expand the disk at launch-time, you'll have plenty of space for installing the Gnome desktop yourself".

Unfortunately, with the sudden proliferation of people building themselves graphical, our various customers' security assessors have taken notice. That means that our (headless) server-oriented automated-hardening tools are not accounting for the now Gnome-enabled desktops. By itself, not a problem, but different assessors have different demands around banners.
For some assessors, simply painting the contents of /etc/issue on the Gnome login screen's root window is sufficient. Fortunately, the Gnome project provides instructions that are also somewhere in the neighborhood of "dead-easy to do". Once done, you have a login screen that looks something like:



Unfortunately, some assessors insist that your consent banner must be a pop-up that is displayed prior to the would-be-user being allowed to enter their username or password. Meeting this requirement is a skosh more-involved. Instead of doing the above Gnome mods, one needs to do:
  1. Move the existing /etc/gdm/Init/Default file contents aside (e.g., `mv /etc/gdm/Init/Default{,-DIST}`)
  2. Install a new /etc/gdm/Init/Default file with contents similar to:
    /usr/bin/zenity --text-info --width=700 --height=300 \
    --title="Security Message" --filename=/etc/issue
    Which uses Zenity to create a create a dialogue-box from the /etc/issue file.
  3. Restart the GDM service (e.g., `systemctl restart gdm`)
Doing this results in production of a pre-authentication pop-up similar to the following:



If you want something marginally fancier, you can install `yad` (from EPEL). If using `yad`, changing your /etc/gdm/Init/Default contents to something like:
/usr/bin/yad \
  --center \
  --buttons-layout=center \
  --button=OK:0 \
  --no-escape \
  --fontname="Monospace Regular 10" \
  --fore tomato4 \
  --text-info \
  --width=700 \
  --height=300 \
  --wrap \
  --title="Security Message" \
  --filename=/etc/issue

Will produce a screen like:



Primary differences being ability to set the text-color and get rid of the extraneous "Cancel" button (and center the "Ok" button).

Wednesday, April 24, 2019

Crib-Notes: End-to-End SSL Within AWS

In general, when using an Elastic Load-Balancer (ELB) to do SSL-encrypted proxying for an AWS-hosted, Internet-facing application, it's typically sufficient to simply do SSL-termination at the ELB and call it a day. That said:
  • If you're paranoid, you can ensure that only the proxied EC2(s) and the ELB are able to communicate with each other via security groups.
  • If you're under compliance-requirements (e.g., PCI/DSS), you can enable end-to-end SSL such that:
    1. Connections between Internet-based client and the ELB are encrypted
    2. Connections between the ELB and the application-hosting EC2(s) is encrypted
Either/both amount to a "belt and suspenders"  approach. Other than "meeting policy", security isn't meaningfully improved: given AWS's overarching security-design, even if someone else has access to your application-EC2(s) and ELB's VPC, they won't be able to sniff packets/data – encrypted or not.

Technical need aside... Implementing end-to-end SSL is trivial:
  • ACM allows easy provisioning of SSL certificates for the ELB (with the security-bonus of automatically rotating said certificates). 
  • You can very generic, self-signed certificates on your application-hosting EC2s:
    • The certificate's Subject doesn't matter
    • The certificate's validity window doesn't matter (no need to worry about rotating certificates that have expired)
Thus setup comes down to:
  1. Create an EC2-hosted application/service:
    1. Launch EC2
    2. Install HTTPS-capable application
    3. Generate a self-signed certificate (setting the -days to as little as 1 day). Example (using the OpenSSL utility):

      openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
         -keyout peer.key -out peer.crt

      When prompted for input, just hit the <RETURN> key (this will create a cert with defaulted values ...which, as noted previously, don't really have bearing on the ELB's trust of the certificate). Similary, one can wholly omit the -days 1 flag and value – the default certificate will be valid for 30 days (but, ELB doesn't care about the validity time-window).
    4. Configure the HTTPS-capable application to load the certificate
    5. Configure the EC2's host-based firewall to allow connections to whatever port the application listens on for SSL-protected connections
    6. Configure the EC2's security group to allow connections to whatever port the application listens on for SSL-protected connections
  2. Create an ELB:
    1. Set the ELB to listen for SSL-based connection-requests (using a certificate from ACM or IAM)
    2. Set the ELB to forward connections using the HTTPS protocol to connect to the target EC2(s) over whatever port the application listens on for SSL-protected connections
    3. Ensure the ELB's healthcheck is requesting a suitable URL to establish the health of the application 

Once the ELB's healthcheck goes green, it should be possible to connect to the EC2-hosted application via SSL.If one wants to verify the encryption-state of the connetction between the ELB and EC2(s), one would need to login to the EC2(s) and sniff the inbound packets (e.g., by using a tool like WireShark).

Thursday, January 3, 2019

Isolated Network, You Say

The vast majority of my clients, for the past decade and a half, have been very security conscious. The frequency with which other companies end up in the news for data-leaks — either due to hackers or simply leaving an S3 bucket inadequately protected — has made many of them extremely cautious as they move to the cloud.

One of my customers has been particularly wary. As a result, their move to the cloud has included significant use of very locked-down and, in some cases, isolated VPCs. It has made implementing things both challenging and frustrating.

Most recently, I had to implement self-hosted GitLab solution within a locked down VPC. And, when I say "locked down VPC", I mean that even the standard AWS service-endpoints have been (effectively) replaced with custom, heavily-controlled endpoints. It's, uh, fun.

As I was deploying a new GitLab instance, I noticed that its backup jobs were failing. Yeah, I'd done what I thought was sufficient configuration via the gitlab.rb file's gitlab_rails['backup_upload_connection'] configuration-block. I'd even dug into the documentation to find the juju necessary for specifying the requisite custom-endpoint. While I'd ended up following a false lead to the documentation for fog (the Ruby module GitLab uses to interact with cloud-based storage options), I ultimately found the requisite setting is in the Digital Ocean section of the backup and restore document (simply enough, it requires setting an appropriate value for the "endpoint" parameter).

However, that turned out to not be enough. When I looked through git's error logs, I saw that it was getting SSL errors from the Excon Ruby module. Yes, everything in the VPC was using certificates from a private certificate authority (CA), but I'd installed the root CA into the OS's trust-chain. All the OS level tools were fine with using certificates from the private CA. All of the AWS CLIs and SDKs were similarly fine (since I'd included logic to ensure they were all pointing at the OS trust-store) - doing `aws s3 ls` (etc.) worked as one would expect. So, ended up digging around some more. Found the in-depth configuration-guidance for SSL and the note at the beginning of the Details on how GitLab and SSL work section:

GitLab-Omnibus includes its own library of OpenSSL and links all compiled programs (e.g. Ruby, PostgreSQL, etc.) against this library. This library is compiled to look for certificates in /opt/gitlab/embedded/ssl/certs.

This told me I was on the right path. Indeed, reading down just a page-scroll further, I found:

Note that the OpenSSL library supports the definition of SSL_CERT_FILE and SSL_CERT_DIR environment variables. The former defines the default certificate bundle to load, while the latter defines a directory in which to search for more certificates. These variables should not be necessary if you have added certificates to the trusted-certs directory. However, if for some reason you need to set them, they can be defined as envirnoment variables.

So, I added a:

gitlab_rails['env'] = {
        "SSL_CERT_FILE" => "/etc/pki/tls/certs/ca-bundle.crt"
}

To my gitlab.rb and did a quick `gitlab-ctl reconfigure` to make the new settings active in the running service. Afterwards, my GitLab backups to S3 worked without further issue.

Notes:

  • We currently use the Omnibus installation of GitLab. Methods for altering source-built installations will be different. See the GitLab documentation.
  • The above path for the "SSL_CERT_FILE" parameter is appropriate for RedHat/CentOS 7. If using a different distro, consult your distro's manuals for the appropriate location.

Thursday, July 5, 2018

2FA and the Heartbreak of SMS

So, a good long while ago, I started enabling all of my Internet-accessible accounts that supported it with two-factor authentication (a.k.a., "2FA"). It has been a somewhat 'fraught' road.

Typically, sites offer you the capability of using token-based — hardware or, thankfully, "virtual"/software — devices or SMS. I chose SMS because several of my customers make use of hardware-based tokens (like the YubiKeys I wanted to use) not practicable. Then, a couple years in, someone discovered "SS7 has a vulnerability that has the side-effect of rendering SMS-based 2FA no longer secure" and, not long after, a documented exploitation was discovered. Warnings were sent out saying "don't use SMS for 2FA".

However, when your work circumstances are like mine sometimes are, SMS-based 2FA was the only meaningful option, so you opted to accept the risk and keep going as is. Afterall, some form of 2FA - even if potentially exploitable - is better than no 2FA at all ...if only in that it made attacking you harder than someone that didn't have even that meager second level of protection. I mean, you lock the doors to your car and to your house even though you know that an adequately determined and/or skillful thief can still break-in, right?

At any rate, those warnings started coming out even before "in the wild" exploits were discovered. However, prior to those discoveries, the warnings were mostly advisory. Then, last year, AWS decided "we're sunsetting SMS-based 2FA: it was a beta capability, any way." That left me kind of "stuck". When I'd first set up my AWS accounts with 2FA, the only non-SMS options were either hardware keys or cellphone-based soft-keys. As mentioned previously, the hardware keys were not an option. Similarly, so were cellphone-based soft-keys. While there were other software-based key options, few of them had desktop-based clients. The few of those that had desktop-based options, most seemed to assume that you only ever used one desktop (such that, if you, say, set up your bank account for 2FA using such an app on your home computer, you no longer had the ability to use your bank's website at work without using one of the "backup" access codes). I thought I was going to be stuck going back to 2FA-less for my AWS accounts.

Fortunately, it seems like there's more options for 2FA, some of which have included the design-assumption, "many people don't have the ability to rely on just one token (virtual) device". Or, maybe they simply realized that no one was going to carry around an inch-thick deck of filing-cards, each with a given 2FA-enabled site's backup codes printed on it. Don't really care on the why, simply care that now I can "2FA all the things" (that support 2FA at all).

This morning, it was a dead day at work (a lot of people in the US take off the day following the Fourth of July). So, I decided to see whether/how to change my AWS account to use a 2FA protection that wasn't reliant on SMS. I looked at AWS's list of supported soft-keys — Google Authenticator (and presumably ones that operate compatibly – like LastPass's claims to) and Authy — it made my choice easy: the Google Authenticator only works on cellphones, thus, I could eliminate it. So, I opted to download and install Authy to see if I could make it work (surprisingly, they don't yet seem to have options for Linux desktop users). Also wanted to see how hard the setup was so I could make an informed "we should require all our AWS users to 2FA-enable their accounts" (or not) recommendation to our security-policy team. So, after downloading and installing Authy:

  1. Not wanting to lock myself out of my AWS account, I created a testing console-user with console login rights.
  2. I logged out of the account I have that's enabled for IAM-user creation
  3. Logged in to my test account using standard user/password credentials
  4. Went to the IAM console for my test-user
  5. Clicked on the "Security Credentials" tab
  6. Clicked on the (edit) pencil-icon next to the "Assigned MFA device" row
  7. Selected "A virtual MFA device" from the first dialog-popup
  8. Clicked "Next" till I got to the "Manage MFA Device" screen
  9. Clicked on the "Show secret key for manual configuration" button to expose the manual-configuration token-string (since, my computer had no ability to scan a QR code)
  10. Clicked on the "Tokens" icon in Authy, and then the "+" icon to add a new account-binding
  11. Copied the secret key from the AWS page to my cut-buffer
  12. Pasted the AWS secret key into the "Enter Code given by the website" text-box in Authy and hit the "Add Account" button
  13. Chose a meaningful account name and icon and hit the "Save" button
  14. Then copied the first and second six-charater tokens into the "Authentication Code 1" and "Authentication Code 2" fields, then clicked the "Activate Virtual MFA" button
  15. After AWS popped up its "The MFA device was successfully associated with your account.
    " message, I clicked the "Finish" button to exit the MFA setup tool
  16. I then logged into the account using a different browser. After entering the test user's normal user/password credentials, it prompted me for my Authy-generated string. I entered that and I was into the AWS console just like normal.
  17. I quickly logged out from my test account and then nuked it via my IAM-administrator enabled account.
I repeated the above with my other AWS accounts (I have several - each with different priv-sets). Each was a breeze to set up and use.

Now to see how many of my other 2FA-via-SMS accounts I can convert to Authy. I'd really rather have "one ring", but some sites want you to use particular 2FA tools.

Thursday, February 5, 2015

Attack of the Clones

One of the clients I do work for has a fairly significant Linux footprint. However, in these times of greater fiscal responsibility/austerity, my client is looking at even cheaper alternatives. This means that, for systems whose applications don't require  Red Hat for warranty-support, CentOS is being subbed into their environment. This is particularly true for their testing environments. There've even been arguments for doing it in production, applications' vendor-support be damned, because "CentOS is the same as Red Hat"

I've previously argued, "they're very, very similar, but they're not truly identical". In particular, Red Hat handles CVEs and errata somewhat differently than CentOS does (Red Hat backports many fixes to prior EL releases, CentOS's stance is generally "upgrade it").

Today, I got bit by one place where CentOS hews far too closely to "the same as Red Hat Enterprise Linux". Specifically, I was using the `oscap` security tool to do a security audit of a test system. I should say, "I was struggling to use the `oscap` security tool...". With later versions of EL6, Red Hat, and as a derivative, CentOS, implement the CPE system for Linux.

This is all fine and good, except where the tools you use rely on the correctness of CPE-related definitions. By the standard of CPE, Red Hat and CentOS are very much not "the same". Because the security-auditing tool I was using (`oscap`) leverages CPEs and because the CentOS maintainers simply repackage the Red Hat furnished security profiles without updating the CPE call-outs, first, the security tool fails horribly. Every test comes back as "notapplicable".

To fix this situation, a bit of `sed`-fu is required:
mv /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml-DIST && \
cp /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml-DIST \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml && \
sed -i '{
   s#Red Hat Enterprise Linux 6#CentOS 6##g
   s#cpe:/o:redhat:enterprise_linux:6#cpe:/o:centos:centos:6##g
}' /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-oval.xml


mv /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml-DIST && \
cp /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml-DIST \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml && \
sed -i \
   's#cpe:/o:redhat:enterprise_linux#cpe:/o:centos:centos##g' \
/usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml

Once the above is done, running `oscap` actually produces useful results.

NOTE: Ironically, doing the above edits will cause the various SCAP profiles to flag an error when running the tests that verify that RPMs have been unaltered. I've submitted a bug to the CentOS group so these fixes are included in future versions of the CentOS OpenSCAP RPMs, but, until then, you just need to be aware that the `oscap` tool  will flag the above two files.

...And if you found this page because you're trying to figure out how to run `oscap` to get results, here's a sample invocation that should act as a starting-point:

oscap xccdf eval --profile common --report \
   /var/tmp/oscap-report_`date "+%Y%m%d%H%M"`.html \
   --results /var/tmp/oscap-results_`date "+%Y%m%d%H%M"`.xml\
   --cpe /usr/share/xml/scap/ssg/content/ssg-rhel6-cpe-dictionary.xml \
   /usr/share/xml/scap/ssg/content/ssg-rhel6-xccdf.xml

Tuesday, August 7, 2012

Why Google's Two-Factor Authentication Is Junk

To be fair, I understand the goals that Google was trying to achieve. And, they're starting down a good path. However, there are some serious flaws (as I see it) with how they've decided to treat services that don't support two-factor authentication.

  • Google advertises that you can set per-service passwords for each application. That is to say, if you use third-party mail clients such as Thunderbird, third-party calendaring clients such as Lightning, and third-party chat clients such as Trillian, you can set up a password for each service. Conceivably, one could set one password for IMAP, one password for SMTP, one password for iCAL and yet another password for GoogleTalk. However, Google doesn't actually sandbox the passwords. By "sandbox" I mean restrict a given password to a specific protocol.  If I generate four passwords with the intention of using each password once for each service - as Google's per-application passwords would logically be inferred to work - one actually weakens the security of the associated services. Instead of each service having its own password, each of the four, generated passwords can be used with any of the four targeted services. Instead of having one guessable password, there are now four guessable passwords.
  • Google's "per-application" passwords do not allow you to set your password strings. You have to use their password generator. While I can give Google credit for generating 16-character passwords, the strength of the generated passwords is abysmally low. Google's generated passwords are comprised solely of lower case characters. When you go to a "how strong is my password site", Google's generated passwords are ridiculously easy. The Google password is rated at "very weak" - a massive cracking array would take 14 years to break it. By contrast, the password I used on my work systems, last December, is estimated take the best part of 16,000 centuries. For the record, my work password from last year is two characters shorter than the ones Google generates.

So, what you end up with is X number of services that are each authenticatable against with X number of incredibly weak passwords.

All in all, I'd have to rate Google's efforts, at this point, pretty damned close to #FAIL: you have all the inconvenience of two-factor authentication and you actually broaden your attack surfaces if you use anything that's not HTTP/HTTPS based.

Resources:

  1. GRC Password Cracker Estimator
  2. PasswordMeter Password Strength Checker

Thursday, March 22, 2012

ACL Madness

In our environment, security is a concern. So, many programs and directories that you might take being available to you on a "standard" Linux system will give you a "permission denied" on one of our systems. Traditionally, you might work around this by changing the group-ownership and permissions on the object to allow a subset of the system users the expected level of access to those files or directories. And, this will work great if all the people that need access share a common group. If they don't you have to explore other avenues. One of these avenues is the extened permissions attributes found in Linux's ACL handling (gonna ignore SELinux, here, mostly because: A) I'm lazy; B) I freaking hate SELinux - probably because of "A"; and, C) ACLs are a feature that is available across different implementations of UNIX and Linux, and is thus more portable than SELinux or other vendor-specific extended security attribute mechanisms). Even better, as a POSIX filesystem extension, you can use it on things like NFS shares and have your ACLs work across systems and *N*X platforms (assuming everone implements the POSIX ACLs the same way).

And, yes, I know things like `sudo` can be used to delegate access, but that's fraught with its own concerns, as well, least of all its lack of utility for users that aren't logging in with interactive shells.

Say you have a file <tt>/usr/local/sbin/killmenow</tt> that is currently set to mode 700 and you want to give access to it to members of the ca_opers and md_admins groups. You can do something like:

# setfacl -m g:ca_opers:r-x /usr/local/sbin/killmenow
# setfacl -m g:md_admins:r-x /usr/local/sbin/killmenow

Now, members of both the ca_opers and md_admins groups can run this program.

All is well and good until someone asks, "what files have had their ACLs modified to allow this" and you've (or others on your team) gone crazy with the setfacl command. With standard file permissions, you can just use the `find` command to locate files that have a specific permissions-setting. With ACLs, `find` is mostly going to let you down. So, what to do? Fortunately,  setfacl 's partner-command, `getfacl ` can come to your rescue. Doing something like

# getfacl --skip-base -R 2> /dev/null | sed -n 's/^# file://p`

Will walk the directory-structure, from downward, giving you a list of files with ACLs added to them. Once you've identified such-modified files, you can then run `getfacl` against them, individually, to show what the current ACLs are.

Friday, March 9, 2012

FaceBook Security Fail

Ok, this is a bit of a departure from my usual, work-related posting. That said, security fits into the rubric of "more serious postings", even if that security is related to social networks. Who knows: maybe I'll decide to move it, later. At any rate...

 

I'm a big fan of social networking. In addition to my various blogs (on Posterous, BlogSpot, Tumblr, etc.), I make heavy use of services like FaceBook, Plus and Twitter (and have "personal" and "work" personnae set up on each). On some services (FaceBook and LiveJournal)  I run my accounts fairly locked-down; on others (most everything else), I run things wide-open. In either case, I'm not exactly the most censored of individuals. The way I look at it, if someone's going to use my postings against me, I may as well make it easy for them to do it up front than to put myself in a position where I'm invested in an individual or an organization only to have my posting history negatively subsequently sabotage that. It's a pick your poison kind of thing.

That said, not everyone is quite as laissez-faire about their online sharing activities. So, in general, I prefer to keep my stuff at least as locked down as the members of my sharing community do. That way, there's a lower likelihood that my activities will accidentally compromise someone else.

Today, a FaceBook friend of mine was trying to sort through they myriad security settings available to her so that she could create a "close friends only" kind of profile. She'd thought that she'd gotten things pretty locked down, until an unexpected personal message revealed to her that she had information leakage, somewhere, in her FaceBook usage. I was trying to help her ID it.

While my friend had fairly restrictively locked down her profile, she wasn't aware that certain actions could compromise those settings. Specifically, she wasn't aware that if she posted a comment to a public (or at least more open) thread, that others would be able to "see her". She'd assumed that if she set all of her security buttons-n-dials to "friends only" that anything she did would be kept friends only. With FaceBook, that's mostly the case. However, if you comment on a thread started by another friend, then everyone who is able to see that thread can "see" (aspects of) you, as well. Thus, if a friend starts a thread and has the permissions set to public and you comment on it, the entire Internet can see that you have some kind of FaceBook presence, even if they don't have permission to view your profile/timeline.

In attempting to illustrate this, I took a screen capture of a post that had been set to public. I'd done so using the post of someone I thought was a shared friend (when I'd clicked on the poster's profile, both myself and my security-conscious friend appeared to show up in the poster's "mutual friends" list). It turns out I was mistaken in that thought.

When I'd posted the screen shot to my security-conscious friend's wall, I tagged the original poster in that wall post. My security-conscious friend had set her wall to "friends only". When she informed me that the public-poster was not a mutual "friend" but a "friend of a friend", I'd made the suppostion that the tagging of the public-posting friend would be moot. After all: what kind of security model would allow me to overrided my security-conscious friend's wall security settings with something so simple as a tag-event? Turns out, FaceBook's security model would. To me, that would fall into the general heading of a "broken security model".

Oh well, now to figure out how to rattle some cages in FaceBook's site usability group to get them to fix that.

Monday, December 12, 2011

Who Cut Off My `finger`

Overall, I'm not trying to make this, my "more serious blog" a dumping-ground for rants. So, please forgive me this rant and please feel free to skip this post....

I've been using UNIX and similar systems for a long time, now. So, I'm kind of set in my ways in the things I do on systems and the tools I expect to be there. When someone capriciously removes a useful tool, I get a touch upset.

`finger` is one of those useful tools. Sadly, because people have, in the mists of time` misconfigured finger, security folks now like to either simply disable it or remove it altogether. Fine. Whatever: I appreciate that there might be security concerns. However, if you're going to remove a given command, at least make sure you're accomplishing something useful for the penalty you make system users pay in frustration and lost time. If you decide to remove the finger command, then you should probably also make it so I can't get the same, damned information via:

• `id`
• `who`
• `whoami`
• `last`
• `getent passwd <USERID>`
• (etc.)

If I can run all of those commands, I've still got all the ability to get the data you're trying to hide by removing `finger`. So, what have you accomplished other than to piss me off and make it so I have to get data via other avenues? Seriously: "WTF"?

Why the hell is it that, when someone reads a "security best practice", they go ahead and blindly implement something without bothering to ask the next, logical questions: "does doing this, by itself, achieve my security goal," "is the potential negative impact on system users more than balanced-out by increased system security", "is there a better way to do this and achieve my security goals" and "what goal am I actually acheiving by taking this measure." If you don't ask these questions (and have good, strong answers to each), you probably shouldn't be following these "best practices."

Monday, May 2, 2011

Vanity Linux Servers and SSH Brute-Forcers

Let me start by saying, that, for years (think basically since OpenSSH became available) I have run my personal, public-facing SSH services relatively locked-down. No matter what the default security posture for the application was - whether compiled from source or using the host operating systems defaults - the first things I did was to ensure that PermitRootLogin was set to "no". I used to allow tunneled clear-text passwords (way back in the day), but even that I've habitually disabled for (probably) a decade, now. In other words, if you want to SSH into one of my systems, you had to do so as a regular user and you had to do it using key-based logins. Even if you did manage to break in as one of those non-privileged users, I used access controls to limit which users could elevate privileges to root.

Now, I never went as far as changing the ports my SSH servers listened on. This always seemed kind of pointless. I'm sure there's plenty of script kiddies whose cracking-scripts don't look for services running on alternate ports, but I've never found much value relying on "security by obscurity".

At any rate, I figured this was enough to keep me basically safe. And, to date, it seems to have. That said, I do periodically get annoyed at seeing my system logs filled with the "Too many authentication failures for root" and "POSSIBLE BREAK-IN ATTEMPT" messages. However, most of the solutions to such problems seemed to be log-scrapers that then blacklisted the attack sources. As I've indicated in prior posts, I'm lazy. Going through the effort of setting up log-scrapers and tying them to blacklisting scripts was more effort than I felt necessary to address something that seemed, primarily to be only a nuisance. So, I never bothered.

I've also been a longtime user of tools like PortSentry (and its equivalents). So, I usually picked up attacks before they got terribly far. Unfortunately, as Linux has become more popular, there seems to be a lot more service-specific attacks and less broad-spectrum attacks (attacks preceded by probing of all possible entry points). Net result: I'm seeing more of the nuisance alerts in my logs.

Still, there's that laziness thing. Fortunately, I'd recently sat through a RedHat training class. And, while I was absolutely floored when the instructor told me that even RHEL 6 still ships with PermitRootLogin set to "yes", he let me know that recent RHEL patch levels included iptables modules that made things like fail2ban somewhat redundant. Unfortunately, he didn't go into any further detail. So, I had to go and dig around for how to do it.

Note: previously, I'd never really bothered with using iptables. I mean, for services that don't require Internet-at-large access, I'd always used things like TCPWrappers or configuring to only listen on loopback or domain sockets to prevent exposing the services. Thus, with my systems, the only Internet-reachable ports were the ones that had to be. There never really seemed to be a point in enabling a local firewall when the system wasn't acting as a gateway to other systems. However, the possibility of leveraging iptables in a useful way kind of changed all that.

Point of honesty, here: the other reason I'd never bothered with iptables was that its syntax was a tad arcane. While I'd once bothered to learn the syntax for ipfilter - a firewall solution with similarly arcane syntax - so that I could use a Solaris-based system as a firewall for my house, converting my ipfilter knowledge to iptables didn't seem worth the effort.

So, I decided to dig into it. I read through manual pages. I looked at websites. I dug through my Linux boxes netfilter directories to see if I could find the relevant iptables modules and see if they were internally documented. Initially, I thought the iptables module my instructor had been referring to was the ipt_limit module. Reading up on it, the ipt_limit module looked kind of nifty. So, I started playing around with it. As I played with it (and dug around online), I found there was an even better iptables module, ipt_recent. I now assume the better module was the one he was referring to. At any rate, dinking with both, I eventually set about getting things to a state I liked.

First thing I did, when setting up iptables was decided to be a nazi about my default security stance. That was accommodated with one simple rule: `iptables -P INPUT DROP`. If you start up iptables with no rules, you get the equivalent default INPUT filter rule of `iptables -P INPUT ACCEPT`. I'd seen some documentation where people like to set there's to `iptables -P INPUT REJECT`. I like "DROP" better than "REJECT" - probably because it suits the more dickish side of me. I mean, if someone's going to chew up my systems resources by probing me or attempting to break in, why should I do them the favor of telling their TCP stack to end the connection immediately? Screw that: let their TCP stack send out SYNs and be ignored. Depending on whether they've cranked down their TCP stack, those unanswered SYNs will mean that they will end up with a bunch of connection attempts stuck in a wait sequence. Polite TCP/IP behavior says that, when you send out a SYN, you wait for an ACK for some predetermined period before you consider the attempt to be failed and execute your TCP/IP abort and cleanup sequence. That can be several tens of seconds to a few hours. During that interval, the attack source has resources tied up. If I sent a REJECT, they could go into immediate cleanup, meaning they can more quickly move onto their next attack with all their system resources freed up.

The down side of setting your default policy to either REJECT or DROP is that it applies to all your interfaces. So, not only will your public-facing network connectivity cease, so will your loopback traffic. Depending on how tightly you want to secure your system, you could bother to iterate all of the loopback exceptions. Most people will probably find it sufficient to simply set up the rule `iptables -A INPUT -i lo0 -j ACCEPT`. Just bear in mind that more wiley attackers can spoof things to make it appear to come through loopback and take advantage of that blanket exception to your DROP or REJECT rules (though, this can be mitigated by setting up rules to block loopback traffic that appears on your "real" intefaces - something like `-A INPUT -i eth0 -d 127.0.0.0/8 -j DROP` will do it).

The next thing you'll want to bear in mind with the defualt REJECT or DROP is that, without further fine-tuning, it will apply to each and every packet hitting that filterset. Some TCP/IP connections start on one port, but then get moved off to or involve other ports. If that happens, your connection's not gonna quite work right. One way to work around that, is to use a state table to manage established connections or related connections. Use a rule like `iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT` to accommodate that.

At this point you're ready to start punching the service-specific holes in your default-deny firewall. On a hobbyist or vanity type system, you might be running things like DNS, HTTP(S), SMTP, and IMAP. That will look like:

-A INPUT -p udp -m udp --dport 53 -j ACCEPT       # DNS via UDP (typically used for individual DNS lookups)
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT       # DNS via TCP (typically used for large zone transfers)
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT       # HTTP
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT       # HTTP over SSL
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT       # SMTP
-A INPUT -p tcp -m tcp --dport 587 -j ACCEPT       # SMTP submission via STARTTLS
-A INPUT -p tcp -m tcp --dport 993 -j ACCEPT       # IMAPv4 + SSL

What the ipt_limits module gets you is the ability to rate-limit connection attempts to a service. This can be a simple as ensuring that only "so many connections" per second are allowed access to the service, limiting the number of connections per time interval per source or outright blacklisting a source that too frequently connects.

Doing the first can be done within the SSH and/or TCP Wrappers (or, for services run through xinetd, through your xinetd config). Downside of this is, since it's not distinguishing sources, if you're being attacked, you won't be able to get in since the overall number of connections will have been exceeded. Generally, potentially allowing others to lock you out of your own system is considered to be "not a Good Thing™ to do". But, if you want to risk it, add a rule that looks something like  `-A INPUT -m limit --limit 3/minute -m tcp -p tcp --dport 22 -j ACCEPT` to your iptables configuration and be on about your way (using the ipt_limit module).

If you want to be a bit more targeted in your approach, the ipt_recent module can be leveraged. I used a complex of rules like the following:
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m recent --set --name sshtrack --rsource
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --name sshtrack --rsource -j LOG --log-prefix "ssh rejection: "
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update  --seconds 60 --hitcount 3 --name sshtrack --rsource -j DROP
   -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
What the above four rules do is:
  • For each new connection attempt to port 22, add the remote source address to the "sshtrack" tracking table
  • If this is the third such new connection within 60 seconds, update the remote source address entry in the tracking table and log rejection action
  • If this is the third such new connection within 60 seconds, update the remote source address entry in the tracking table and DROP the connection
  • Otherwise, accept the new connection.
I could have chosen to simply "rcheck" rather than "update" the "sshtrack" table. However, by using "update", it essentially resets the time to live from the last connect attempt packet to whatever might be the next attempt. This way, you get the full sixty second window rather than (60 - ConnectInterval). If it becomes apparent that attackers start to use slow attacks to get past the rule, one can up the "seconds" from 60 to some other value. I chose 60 as a start. It might be reasonable to up it to 300 or even 900 since it's unlikely that I'm going to want to start more than three SSH sessions to the box within a 15 minute interval.

As a bit of reference: on RHEL-based systems, you can check what iptables modules are available by listing out '/usr/include/linux/netfilter_ipv4/ipt_*'. You can then (for most) use `iptables -m [MODULE] --help` to show you the options for a given module. For example:
# iptables -m recent --help | sed -n '/^recent v.*options:/,$p'
     recent v1.3.5 options:
     [!] --set                       Add source address to list, always matches.
     [!] --rcheck                    Match if source address in list.
     [!] --update                    Match if source address in list, also update last-seen time.
     [!] --remove                    Match if source address in list, also removes that address from list.
         --seconds seconds           For check and update commands above.
                                     Specifies that the match will only occur if source address last seen within
                                     the last 'seconds' seconds.
         --hitcount hits             For check and update commands above.
                                     Specifies that the match will only occur if source address seen hits times.
                                     May be used in conjunction with the seconds option.
         --rttl                      For check and update commands above.
                                     Specifies that the match will only occur if the source address and the TTL
                                     match between this packet and the one which was set.
                                     Useful if you have problems with people spoofing their source address in order
                                     to DoS you via this module.
         --name name                 Name of the recent list to be used.  DEFAULT used if none given.
         --rsource                   Match/Save the source address of each packet in the recent list table (default).
         --rdest                     Match/Save the destination address of each packet in the recent list table.
     ipt_recent v0.3.1: Stephen Frost .  http://snowman.net/projects/ipt_recent/
Gives you the options for the "recent" iptables module and a URL for further infomation lookup.