AD Integration Woes

For a UNIX guy, I'm a big fan of leveraging Active Directory form centralized system and user management purposes. For me, one of the big qualifications for any application/device/etc. to be able to refer to itself as "enterprise" is that it has to be able to pull user authentication information via Active Directory. I don't care whether it does it the way that Winbind-y things do, or if they just pull data via Kerberos or LDAP. All I care is that I can offload my user-management to a centralized source.

In general, this is a good thing. However, like many things that are "in general, a good thing" it's not always a good thing. If you're in an enterprise that's evolving it's Active Directory infrastructure, tying things to AD means that you're impacted when AD changes or breaks. Someone decides, "hey, we need to reorganize the directory tree" and stuff can break. Someone decides, "hey, we need to upgrade our AD servers from 2003 to 2008" and stuff can break.

Recently, I started getting complaints from the users of our storage resource management (SRM) system that they couldn't login any more. It'd been nearly a year since I'd set it up, so sorting it out was an exercise in trying to remember what the hell I did ...and then Googling.

The application, itself, runs on a Windows platform. The login module that it uses for centralized authentication advertises itself as "Active Directory". In truth, the login module is a hybrid LDAP/Kerberos software module. Even though it's a "Windows" application, they actually use the TomCat Java server for the web UI (and associated login management components). TomCat is what uses Kerberos for authentication data.

Sometime in recent months, someone had upgraded Active Directory. Users that had been using the software before the AD-upgrade were able to authenticate, just fine. Users that had tried to start using the software after the AD-upgrade couldn't get in. Turns out that, when Active Directory had gotten upgraded, the encryption algorithms had gotten changed. Ironically, I didn't find the answer to my problem in any of the forums related to the application: I found it in the forums for another application that used TomCat's Kerberos components.

To start the troubleshooting process, one needs to first modify TomCat's bsclogin.conf file. Normally, this file is only used to tell TomCat where to find the Kerberos configuration file. However, if you modify your bsclogin.conf file and add the directive "debug=true" to it like so:

     com.sun.security.auth.module.Krb5LoginModule required debug=true;

Enhanced user login debugging messages are enabled. Once this is added and tomcat is restarted, login-related messages will start showing up in your ${TOMCATHOME}/logs/stdout.log file. What started showing up in mine were messages like:

     [Krb5LoginModule] user entered username: thjones2@SRMdomain.NET
Acquire TGT using AS Exchange

     [Krb5LoginModule] authentication failed KDC has no support for encryption type (14)

With this error message in hand, I was able to find out that TomCat's Kerberos modules were using the wrong encryption routines to get data out of Active Directory. The fix was to update my Kerberos initialization file and add the following two lines to my [libdefaults] stanza (I just added it right after the dns_lookup_realm line and before the next stanza of directives):

     default_tkt_enctypes = rc4-hmac
     default_tgs_enctypes = rc4-hmac

Making this change (and restarting TomCat) resulted in the failing users suddenly being able to get in.

I'd normally rag on my Windows guys for this, but, the Windows guys aren't exactly used to providing AD-related service notifications to anyone but Windows users. This application, while (currently) running on a Windows platform, isn't something that's traditionally thought of as AD-reliant. Factor in that true, native AD modules pretty much just auto-adjust to such changes, and it didn't occur to them to notify us of the changes.

Oh well. AD-integration is a learning experience for everyone involved, I suppose.

Issues With LikeWise Open for UNIX/Linux Active Directory Integration

Recently it came up on management's radar that, "we're starting to get more UNIX systems out in our farms and we need some way to manage logins for those hosts in much the same way we do for our Windows hosts." Previous central login managment was done through the traditional tools like NIS or Kerberos (but mostly, just relied on ever-expiring local password tables). In a security conscious environment, NIS really isn't a responsible choice, any longer. Standing up a Kerberos infrastructure just to support UNIX systems is kind of an inefficient use of resources - especially when you already have a serviceable Active Directory infrastructure out there.

Fortunately, in the UNIX world, particularly in the Open Systems (Linux, BSD, etc.) realm, there are a number of choices available for joining a system to Active Directory. Many modern UNIX operating systems include this kind of functionality through Winbind. However, the various UNIX vendors or packagers don't necessarily keep fully up to date with the Windbind version they ship in their OS (looking in your direction, here, Sun Oracle). So, if you're running or looking to run Active Directory 2008 the Winbind included with your UNIX may not be a workable solution. Even if your UNIX does include an up to date version of Winbind (but not the bleeding-edge ones that are part of the next-generation Samba project), it may not be up to the task if you have a particularly large or complex Active Directory namespace. In this case, you'll probably be stuck using a commercial offering (or, a commercial AD-integration product's "free" version).
We were kind of stuck in the latter boat. Our AD namespaces tend to be rather large and rather complex. So, we started experimenting with the free version of the LikeWise product, LikeWise Open. In the test labs, with the smaller and simpler Active Directory deployment that supports it, we ran into no issues in testing. However, when we tried to take it into production, we ran into some issues in environments that had "messy" Active Directory deployments.

Specifically, the error we were getting (and could never really find a workaround for in the LikeWise Open support forums) was the "LW_ERROR_ENUM_DOMAIN_TRUSTS_FAILED" error. The domain that was giving us fits was a large domain (tens of thousands of users and thousands more groups, server objects and other, miscellaneous AD objects) that consisted of many AD servers scattered around the globe. Further complicating matters was the fact that these ADs were members of multiple domains and therefore had cross-realm/domain-trust components, as well. Further complicating it is the fact that not all of these AD servers, particularly the ones that had trust relationships with other realms, fully agreed on what time it was.
On the down side, the problem and the lack of easily-Googleable solutions cast doubt as to whether we'd be able to use this product in our enterprise. On the plus side, it gave me a chance to do some troubleshooting. I'm one of those masochists that likes a good challenge and digging into the guts of things. So, I did a bunch of online research as well as poring through the LikeWise administration guides.

Older versions of Likewise used to allow configuration-tweaking through an lsassd.conf file (indeed, other vendors, such as VMware still have this). The latest iteration of LikeWise, unfortunately, does not. Instead, the makers of LikeWise have decided to implement a Windows-esque "registry" for their product. Dunno why plain text files don't work (or even XML files, for that matter) - probably just wanted to make things more familiar for Windows admins that might get saddled with brining those evil Unix boxes into their domains. Whatever. It's painful but not insurmountable. LikeWise provides tools for hacking this file: lwregshell and lw-edit-reg. For me, lw-edit-reg was a more comfortable tool to use. All I had to do was make sure my EDITOR environmental variable was set to VI and I could hack the file to my heart's content with `vi`.

At any rate, I fired up lw-edit-reg and began to dig around for likely tweakables. Given that my error mentioned "TRUSTS", I did a global search for any parameters metniong "TRUSTS". I found the parameter, "DomainManagerIgnoreAllTrusts", and saw that it was set to false (well, the registry equivalent which was "dword:00000000"). So, I tried changing that to "true" (modifying the value to "dword:00000001"). I then bounced all of the LikeWise processes and re-attempted to join my box to the messy domain. Voilà, it worked and all was happy in UNIX-as-AD-Client land!

Of course, it wasn't until after I'd found my fix through the lw-edit-reg manipulations that I thought to check to see if lwconfig could be used. It seems like there's two variants of the lwconfig command in the LikeWise 6 release family. One supports the "--dump" option (which shows all tunable parameters along with their currently-set values) and one does not (thus, seeing available tunables and their current values are a bit less straight-forward). At any rate, upon further investigation, I found that I could use lwconfig to make my settings changes. Thus, it makes the software installation and configuration a lot more scriptable. I can modify my automated installation policies to do an `lwconfig DomainManagerIgnoreAllTrusts true` operation if it has issues with the normal `domainjoin-cli ...` operation.