`iptables` for the Obsessive Compulsive

While I probably don't meet the clinical definition for being obsessive compulsive, I do tend to like to keep things highly organized. This is reflected heavily in the way I like to manage computer systems.
My employer, as part of the default security posture for production Linux systems, requires the use of iptables. If you've ever looked at an iptables file, they tend to be a spaghetti of arcana. Most tables start out fairly basic and might look something like:

-A INPUT -i lo -j ACCEPT
   -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
   -A INPUT -p udp -m udp --dport 53 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 587 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 993 -j ACCEPT
   -A INPUT -j REJECT --reject-with icmp-host-prohibited

This would probably be typical of a LAMP server that's also providing DNS and mail services.As it stands, it's fairly manageble and easy to follow if you've got even a slight familiarity with iptables or even firewalls in general.
Where iptables starts to become unfun is when you start to get fancy with it. I started going down this "unfun" path when I put in place a defense against SSHD brute-forcers. I had to add a group of rules just to handle what, above, was done with a single line. Initially, this started to "spaghettify" my iptables configuration. It ended up making the above look like:

-A INPUT -i lo -j ACCEPT
   -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
   -A INPUT -p udp -m udp --dport 53 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name ssh_safe --rsource
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 3 --name ssh_safe --rsource -j LOG --log-prefix "SSH CONN. REJECT: "
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 3 --name ssh_safe --rsource -j DROP
   -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 587 -j ACCEPT
   -A INPUT -p tcp -m tcp --dport 993 -j ACCEPT
   -A INPUT -j REJECT --reject-with icmp-host-prohibited

Not quite as straight-forward any more. Not "tidy", as I tend to refer to these kinds of things. It gave me that "tick" I get whenever I see something messy. So, how to fix it? Well, my first step was to use iptables' comments module. That allowed me to make the configuration a bit more self-documenting (if you ever look at my shell scripts or my "real" programming, they're littered with comments - makes it easier to go back and remember what the hell you did and why). However, it still "wasn't quite right". So, I decided, "I'll dump all of those SSH-related rules into a single rule group" and then reference that group from the main iptables policy:

-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m comment --comment "Forward to SSH attack-handler" -m tcp --dport 22 -j ssh-defense
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 587 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 993 -j ACCEPT
-A ssh-defense -p tcp -m comment --comment "SSH: track" -m tcp --dport 22 -m state --state NEW -m recent --set --name ssh_safe --rsource
-A ssh-defense -p tcp -m comment --comment "SSH: attack-log" -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 3 --name ssh_safe --rsource -j LOG --log-prefix "SSH CONN. REJECT: "
-A ssh-defense -p tcp -m comment --comment "SSH: attack-block" -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 300 --hitcount 3 --name ssh_safe --rsource -j DROP
-A ssh-defense -p tcp -m comment --comment "SSH: accept" -m tcp --dport 22 -j ACCEPT

Ok, so the above doesn't really look any less spaghetti-like. That's ok. This isn't exactly where we-despaghettify things. The above is mostly meant to be machine read. If you want to see the difference in things, use the `iptables -L` command. Or, to really see the difference, issue `iptables -L INPUT ; iptables -L ssh-defense`:

Chain INPUT (policy ACCEPT)
   target     prot opt source               destination
   ACCEPT     all  --  anywhere             anywhere
   DROP       all  --  anywhere             loopback/8
   ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
   ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
   ssh-defense  tcp  --  anywhere             anywhere            /* Forward to SSH attack-handler */ tcp dpt:ssh
   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http
   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https
   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:smtp
   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:submission
   ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:imaps

   Chain ssh-defense (1 references)
   target     prot opt source               destination
              tcp  --  anywhere             anywhere            /* SSH: track */ tcp dpt:ssh state NEW recent: SET name: ssh_safe side: source
   LOG        tcp  --  anywhere             anywhere            /* SSH: attack-log */ tcp dpt:ssh state NEW recent: UPDATE seconds: 300 hit_count: 3 name: ssh_safe side: source LOG level warning prefix `SSH CONN. REJECT: '
   DROP       tcp  --  anywhere             anywhere            /* SSH: attack-block */ tcp dpt:ssh state NEW recent: UPDATE seconds: 300 hit_count: 3 name: ssh_safe side: source
   ACCEPT     tcp  --  anywhere             anywhere            /* SSH: accept */ tcp dpt:ssh

Even if you don't find the above any more self-documenting or easier to handle (in a wide xterm, it looks much better), it does have one other value: it makes it harder for people to muck up whatever flow or readability that your iptables configuration has. Because you've externalized a group of directives, someone's going to have to go out of their way to intersperse random rules into your iptables configuration. If it's just your own server, this probably has little value (unless you've got MPD). However, if you have shared administration duties, it can be a sanity-saver.

Linux Active Directory Integration and PAM

Previously, I've written about using LikeWise to provide Active Directory integration to Linux and Solaris hosts. One of the down sides of LikeWise (and several other similar integration tools) is that it tends to make it such that, if a user has an account in Active Directory, they can log into the UNIX or Linux boxes you've bound to your domain. In fact, while walking someone through setting up LikeWise with the automated configuration scripts I'd written, that person asked, "you mean anyone with an AD account can log in?"

Now, this had occurred to me when I was testing the package for the engineer who was productizing LikeWise for our enterprise build. But, it hadn't really been a priority, at the time. Unfortunately, when someone who isn't necessarily a "security first" kind of person hits you with that question/observation, you know that the folks for whom security is more of a "Job #1" are eventually going to come for you (even if you weren't the one who was responsible for engineering the solution). Besides, I had other priorities to take care of.

This week was a semi-slack week at work. There was some kind of organizational get-together going on that had most of the IT folks out of town discussing global information technology strategies. Fortunately, I'd not had to take part in that event. So, I've spent the week revisiting some stuff I'd rolled out (or been part of the rollout of) but wasn't completely happy with. The "AD integration giving everyone access" thing was one of them. So, I began by consulting the almighty Google. When I'd found stuff I that seemed promising, I fired up a test VM and started trying it out.

Now, SSH (and several other services) weren't really a problem. Many applications allow you to internally regulate who can use the service. For example, with OpenSSH, you can modify the sshd_config file to explicitly define which users and groups can and cannot access your box through that service (for those of you who hit this page looking for tips, do a `man sshd_config` and grep for AllowUsers and AllowGroups for more in-depth information). Unfortunately, it's predictable enough to figure that people that are gonna whine about AD integration giving away the farm are gonna bitch if you tell them they have to modify the configuration of each and ever service they want to protect. No, most people want to be able to go to one place and take care of things with on action or one set of consistent actions. I can't blame them: I feel the same way. Everyone wants things done easily. Part of "easily" generally implies "consistently" and/or "in one place".

Fortunately, any good UNIX or Linux implementation leverages the Pluggable Authentication Management system (aka. PAM). There's about a bazillion different PAM modules out there that allow you to configure any given service's authentication to do or test a similar variety of attributes. My assumption for solving this particular issue was that, while there might be dozens or hundreds of groups (and thousands of users) in an Active Directory forrest, one would only want to grant a very few groups access to an AD-bound UNIX/Linux host. So, I wasn't looking for something that made it easy to grants lots of groups access in one swell-foop. In fact, I was kind of looking for things that made that not an easy thing to do (after all, why lock stuff down if you're just going to blast it back open, again?). I was also looking for something that I could fairly reliably find on generic PAM implementations. The pam_succeed_if is just about tailor made for those requirements.

LikeWise (and the other AD integration methods) add entries into your PAM system to allow users allowed by those authentication subsystems to login, pretty much, unconditionally. Unfortunately, those PAM modules don't often include methods for controlling which users are able to login once their AD authentication has succeeded. Since the PAM system uses a stackable authentication module, you can insert access controls earlier into the stack to cause a user access to fail out earlier than the AD module would otherwise grant the access. If you wanted to be able to allow users in AD_Group1 and AD_Group2 to log in, but not other groups, you'd modify your pam stack to insert the control ahead of the AD allow module.

     account    [default=ignore success=1] pam_succeed_if.so user ingroup AD_Group1 quiet_success
     account    [default=ignore success=1] pam_succeed_if.so user ingroup AD_Group2 quiet_success
     account    [default=bad success=ignore] pam_succeed_if.so user ingroup wheel quiet_success
     account    sufficient    /lib/security/pam_lsass.so

The above is processed such that if a user is a member of the AD-managed group "AD_Group1" or "AD_Group2", it sets the test's success flag to true. If the user isn't a member of those two groups, testing falls through to the next group check - is the user a member of the group wheel (if yes, fall through to the next test; if no, then there's a failure and the user's access is denied). Downside of using this particular PAM module is that it's only availble to you on *N*X systems with a plethora of PAM modules. This is true for many Linux releases - and I know it to be part of RedHat-related releases - but probably won't be available on less PAM-rich *N*X systems (yet one more reason to cast Solaris on the dung-heap, frankly). If your particular *N*X system doesn't have it, you can probably find the sourcecode for it and create yourself the requisite model for your OS.

Vanity Linux Servers and SSH Brute-Forcers

Let me start by saying, that, for years (think basically since OpenSSH became available) I have run my personal, public-facing SSH services relatively locked-down. No matter what the default security posture for the application was - whether compiled from source or using the host operating systems defaults - the first things I did was to ensure that PermitRootLogin was set to "no". I used to allow tunneled clear-text passwords (way back in the day), but even that I've habitually disabled for (probably) a decade, now. In other words, if you want to SSH into one of my systems, you had to do so as a regular user and you had to do it using key-based logins. Even if you did manage to break in as one of those non-privileged users, I used access controls to limit which users could elevate privileges to root.

Now, I never went as far as changing the ports my SSH servers listened on. This always seemed kind of pointless. I'm sure there's plenty of script kiddies whose cracking-scripts don't look for services running on alternate ports, but I've never found much value relying on "security by obscurity".

At any rate, I figured this was enough to keep me basically safe. And, to date, it seems to have. That said, I do periodically get annoyed at seeing my system logs filled with the "Too many authentication failures for root" and "POSSIBLE BREAK-IN ATTEMPT" messages. However, most of the solutions to such problems seemed to be log-scrapers that then blacklisted the attack sources. As I've indicated in prior posts, I'm lazy. Going through the effort of setting up log-scrapers and tying them to blacklisting scripts was more effort than I felt necessary to address something that seemed, primarily to be only a nuisance. So, I never bothered.

I've also been a longtime user of tools like PortSentry (and its equivalents). So, I usually picked up attacks before they got terribly far. Unfortunately, as Linux has become more popular, there seems to be a lot more service-specific attacks and less broad-spectrum attacks (attacks preceded by probing of all possible entry points). Net result: I'm seeing more of the nuisance alerts in my logs.

Still, there's that laziness thing. Fortunately, I'd recently sat through a RedHat training class. And, while I was absolutely floored when the instructor told me that even RHEL 6 still ships with PermitRootLogin set to "yes", he let me know that recent RHEL patch levels included iptables modules that made things like fail2ban somewhat redundant. Unfortunately, he didn't go into any further detail. So, I had to go and dig around for how to do it.

Note: previously, I'd never really bothered with using iptables. I mean, for services that don't require Internet-at-large access, I'd always used things like TCPWrappers or configuring to only listen on loopback or domain sockets to prevent exposing the services. Thus, with my systems, the only Internet-reachable ports were the ones that had to be. There never really seemed to be a point in enabling a local firewall when the system wasn't acting as a gateway to other systems. However, the possibility of leveraging iptables in a useful way kind of changed all that.

Point of honesty, here: the other reason I'd never bothered with iptables was that its syntax was a tad arcane. While I'd once bothered to learn the syntax for ipfilter - a firewall solution with similarly arcane syntax - so that I could use a Solaris-based system as a firewall for my house, converting my ipfilter knowledge to iptables didn't seem worth the effort.

So, I decided to dig into it. I read through manual pages. I looked at websites. I dug through my Linux boxes netfilter directories to see if I could find the relevant iptables modules and see if they were internally documented. Initially, I thought the iptables module my instructor had been referring to was the ipt_limit module. Reading up on it, the ipt_limit module looked kind of nifty. So, I started playing around with it. As I played with it (and dug around online), I found there was an even better iptables module, ipt_recent. I now assume the better module was the one he was referring to. At any rate, dinking with both, I eventually set about getting things to a state I liked.

First thing I did, when setting up iptables was decided to be a nazi about my default security stance. That was accommodated with one simple rule: `iptables -P INPUT DROP`. If you start up iptables with no rules, you get the equivalent default INPUT filter rule of `iptables -P INPUT ACCEPT`. I'd seen some documentation where people like to set there's to `iptables -P INPUT REJECT`. I like "DROP" better than "REJECT" - probably because it suits the more dickish side of me. I mean, if someone's going to chew up my systems resources by probing me or attempting to break in, why should I do them the favor of telling their TCP stack to end the connection immediately? Screw that: let their TCP stack send out SYNs and be ignored. Depending on whether they've cranked down their TCP stack, those unanswered SYNs will mean that they will end up with a bunch of connection attempts stuck in a wait sequence. Polite TCP/IP behavior says that, when you send out a SYN, you wait for an ACK for some predetermined period before you consider the attempt to be failed and execute your TCP/IP abort and cleanup sequence. That can be several tens of seconds to a few hours. During that interval, the attack source has resources tied up. If I sent a REJECT, they could go into immediate cleanup, meaning they can more quickly move onto their next attack with all their system resources freed up.

The down side of setting your default policy to either REJECT or DROP is that it applies to all your interfaces. So, not only will your public-facing network connectivity cease, so will your loopback traffic. Depending on how tightly you want to secure your system, you could bother to iterate all of the loopback exceptions. Most people will probably find it sufficient to simply set up the rule `iptables -A INPUT -i lo0 -j ACCEPT`. Just bear in mind that more wiley attackers can spoof things to make it appear to come through loopback and take advantage of that blanket exception to your DROP or REJECT rules (though, this can be mitigated by setting up rules to block loopback traffic that appears on your "real" intefaces - something like `-A INPUT -i eth0 -d 127.0.0.0/8 -j DROP` will do it).

The next thing you'll want to bear in mind with the defualt REJECT or DROP is that, without further fine-tuning, it will apply to each and every packet hitting that filterset. Some TCP/IP connections start on one port, but then get moved off to or involve other ports. If that happens, your connection's not gonna quite work right. One way to work around that, is to use a state table to manage established connections or related connections. Use a rule like `iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT` to accommodate that.

At this point you're ready to start punching the service-specific holes in your default-deny firewall. On a hobbyist or vanity type system, you might be running things like DNS, HTTP(S), SMTP, and IMAP. That will look like:

-A INPUT -p udp -m udp --dport 53 -j ACCEPT		# DNS via UDP (typically used for individual DNS lookups)
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT		# DNS via TCP (typically used for large zone transfers)
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT		# HTTP
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT		# HTTP over SSL
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT		# SMTP
-A INPUT -p tcp -m tcp --dport 587 -j ACCEPT		# SMTP submission via STARTTLS
-A INPUT -p tcp -m tcp --dport 993 -j ACCEPT		# IMAPv4 + SSL

What the ipt_limits module gets you is the ability to rate-limit connection attempts to a service. This can be a simple as ensuring that only "so many connections" per second are allowed access to the service, limiting the number of connections per time interval per source or outright blacklisting a source that too frequently connects.

Doing the first can be done within the SSH and/or TCP Wrappers (or, for services run through xinetd, through your xinetd config). Downside of this is, since it's not distinguishing sources, if you're being attacked, you won't be able to get in since the overall number of connections will have been exceeded. Generally, potentially allowing others to lock you out of your own system is considered to be "not a Good Thing™ to do". But, if you want to risk it, add a rule that looks something like  `-A INPUT -m limit --limit 3/minute -m tcp -p tcp --dport 22 -j ACCEPT` to your iptables configuration and be on about your way (using the ipt_limit module).

If you want to be a bit more targeted in your approach, the ipt_recent module can be leveraged. I used a complex of rules like the following:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -m recent --set --name sshtrack --rsource
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --name sshtrack --rsource -j LOG --log-prefix "ssh rejection: "
   -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update  --seconds 60 --hitcount 3 --name sshtrack --rsource -j DROP
   -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

What the above four rules do is:

• For each new connection attempt to port 22, add the remote source address to the "sshtrack" tracking table
• If this is the third such new connection within 60 seconds, update the remote source address entry in the tracking table and log rejection action
• If this is the third such new connection within 60 seconds, update the remote source address entry in the tracking table and DROP the connection
• Otherwise, accept the new connection.

I could have chosen to simply "rcheck" rather than "update" the "sshtrack" table. However, by using "update", it essentially resets the time to live from the last connect attempt packet to whatever might be the next attempt. This way, you get the full sixty second window rather than (60 - ConnectInterval). If it becomes apparent that attackers start to use slow attacks to get past the rule, one can up the "seconds" from 60 to some other value. I chose 60 as a start. It might be reasonable to up it to 300 or even 900 since it's unlikely that I'm going to want to start more than three SSH sessions to the box within a 15 minute interval.

As a bit of reference: on RHEL-based systems, you can check what iptables modules are available by listing out '/usr/include/linux/netfilter_ipv4/ipt_*'. You can then (for most) use `iptables -m [MODULE] --help` to show you the options for a given module. For example:

# iptables -m recent --help | sed -n '/^recent v.*options:/,$p'
     recent v1.3.5 options:
     [!] --set                       Add source address to list, always matches.
     [!] --rcheck                    Match if source address in list.
     [!] --update                    Match if source address in list, also update last-seen time.
     [!] --remove                    Match if source address in list, also removes that address from list.
         --seconds seconds           For check and update commands above.
                                     Specifies that the match will only occur if source address last seen within
                                     the last 'seconds' seconds.
         --hitcount hits             For check and update commands above.
                                     Specifies that the match will only occur if source address seen hits times.
                                     May be used in conjunction with the seconds option.
         --rttl                      For check and update commands above.
                                     Specifies that the match will only occur if the source address and the TTL
                                     match between this packet and the one which was set.
                                     Useful if you have problems with people spoofing their source address in order
                                     to DoS you via this module.
         --name name                 Name of the recent list to be used.  DEFAULT used if none given.
         --rsource                   Match/Save the source address of each packet in the recent list table (default).
         --rdest                     Match/Save the destination address of each packet in the recent list table.
     ipt_recent v0.3.1: Stephen Frost .  http://snowman.net/projects/ipt_recent/

Gives you the options for the "recent" iptables module and a URL for further infomation lookup.