The Pain of LDAP

Recently, we've had a push to enable 2FA on all of the internet-facing services we offer. A manager asked, "how many users do we have to worry about?" We have three basic types of users:

• Those who only have access to the Remote Desktops
• Those who only have access to the various applications
• Those who have access to both the Remote Desktops and to the various appliations

All such access is unprivileged. All accesses are managed via Active Directory group memberships. 

• Our RDSH users are in a "remote desktops" group.
• Our application users are in a mix of groups. Fortunately, we long ago implemented standard naming for our groups. All group-names for one application-class start with "T_"; all group-names for the other application-class start with "E_"

Fortunately, the above means that I could get an answer for the manager by way of LDAP queries. A user's membership in an Active Directory is conveyed via the user-object's meberOf attribute. This means that the basic query-objects I needed to consider were:

• RDSH Users: In LDAP-speak, this works out to CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>
• "T" Application-group: In LDAP-speak, this works out to CN=T_*
• "E" Application group: In LDAP-speak, this works out to CN=E_*

LDAP queries generally operate on object-sets. Sets are denoted by parentheses. Using the previously-mentioned groups, a single-element set for each of my group-memberships would be:

• (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• (memberof=CN=T_*)
• (memberof=CN=E_*)

Where things get really freaking ugly is when you need to combine these into appropriate queries. Combining query-objects is a matter of using the booleans:

• "&": the logical AND operator
• "|": the logical OR operator
• "!": the logical NOT operator
• If you're familiar with logical operators, you'll notice that there isn't a built in eXclusive OR type of operator. 

Why I say things are ugly is that you apply a given operator within a set and the syntax is a little non-intutive:

• ANDing sets: Uses the syntax (&(SET1)(SET2)(...)(SET_N)). Basically, this says, "AND all of the following sets within this AND-set. An AND-set can consist of 2+ sets for evaluation
• ORing sets: Similarly, this uses the syntax (|(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this OR-set. An OR-set can consist of 2+ sets for evaluation
• NOTin sets: Similarly, this uses the syntax (!(SET1)(SET2)(...)(SET_N)). Basically, this says, "OR all of the following sets within this NOT-set. An NOT-set can consist of 2+ sets for evaluation

Where it gets really ugly is when a logical operation acually requires two or more sub operations. For example:

• When you want to tell if a user is a member of one group but not another, you need to do something like: (&(GROUP1)(!(GROUP2))
• Similarly, when you want to tell if a user is a member of one group but not a member of both of two other groups, you would do something like (&(GROUP1)(!(&(GROUP2)(GROUP3))
• Similarly, when you want to tell if a user is a member of one group but not a member of either of two other groups, you would do something like (&(GROUP1)(!(|(GROUP2)(GROUP3))

As you can probably tell, as the number of selectors goes up, so does the ugliness.

At any rate, back to my original queries:

• Getting a list of all people with RDSH access, my query looks like: (memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)
• Getting a list of people with membership in either of the application groups, my query looks like: (|(memberof=CN=T_*)(memberof=CN=T_*))
• Getting a list of people with RDSH access and with membership in either of the application groups, my query looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(|(memberof=CN=T_*)(memberof=CN=T_*)))
• Getting a list of people with RDSH access but not access to either of the other groups , my query now looks like: (&(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>)(!(|(memberof=CN=T_*)(memberof=CN=T_*))))
• Conversely, getting a list of people without RDSH access but with access to either of the other groups, my query now looks like: (&(!(memberof=CN=<RDSH_GROUP>,cn=users,dc=<F>,dc=<Q>,dc=<D>,dc=<N>))(|(memberof=CN=T_*)(memberof=CN=T_*)))

All that ugliness aside, I was able to get the manager the answer to his question and was able to do it quickly. This post is mostly a result of him asking "what the hell does all that gobbledygook mean" when I sent him the queries I ran to get him the answer. I'd sent that as part of the thread for the same reason I'm writing this: so I have the queries the next time the information is asked for again (and don't have to re-work it all out in my head again).

Finding Patterns

I know that most of my posts are of the nature "if you're trying to accomplish 'X' here's a way that you can do it". Unfortunately, this time, my post is more of a, "I've got yet to be solved problems going on". So, there's no magic bullet fix currently available for those with similar problems who found this article. That said, if you are suffering similar problmes, know that you're not alone. Hopefully that's small consolation and what follows may even help you in investigating your own problem.

Oh: if you have suggestions, I'm all ears. Given the nature of my configuration, there's not much in the way of useful information I've yet found via Google. Any help would be much appreciated...

---

The shop I work for uses Symantec's Veritas NetBackup product to perform backups of physical servers. As part of our effort to make more of the infrastructure tools we use more enterprise-friendly, I opted to leverage NetBackup 7.1's NetBackup Access Control (NBAC) subsystem. On its own, it provides fine-grained rights-delegation and role-based access control. Horse it to Active Directory and you're able to roll out a global backup system with centralized authentication and rights-management. That is, you have all that when things work.

For the past couple months, we've been having issues with one of the dozen NetBackup domains we've deployed into our global enterprise. When I first began trougleshooting the NBAC issues, the authentication and/or authorization failures had always been associated with a corruption of LikeWise's sqlite cachedb files. At the time the issues first cropped up, these corruptions always seemed to coincide with DRS moving the NBU master server from one ESX host to another. It seemed like, when under sufficiently heavy load - the kind of load that would trigger a DRS event - LikeWise didn't respond well to having the system paused and moved. Probably something to do with the sudden apparent time-jump that happens when a VM is paused for the last parts of the DRS action. My "solution" to the problem was to disable automated-relocation for my VM.

This seemed to stabilize things. LikeWise was no longer getting corrupted and seems like I'd been able to stabilize NBAC's authentication and authorization issues. Well, they stabilized for a few weeks.

Unfortunately, the issues have begun to manifest themselves, again, in recent weeks. We've now had enough errors that some patterns are starting to emerge. Basically, it looks like something is horribly bogging the system down around the time that the nbazd crashes are happening. I'd located all the instances of nbazd crashing from its log files ("ACI" events are loged to the /usr/openv/netbackup/logs/nbazd logfiles), and then began to try correlating them with system load shown by the host's sadc collections. I found two things: 1) I probably need to increase my sample frequency - it's currently at the default 10-minute interval - if I want to more-thoroughly pin-down and/or profile the events; 2) when the crashes have happened within a minute or two of an sadc poll, I've found that the corresponding poll was either delayed by a few seconds to a couple minutes or was completely missing. So, something is causing the server to grind to a standstill and nbazd is a casualty of it.

For the sake of thoroughness (and what's likely to have matched on a Google-search and brought you here), what I've found in our logs are messages similar to the following:

/usr/openv/netbackup/logs/nbazd/vxazd.log

07/28/2012 05:11:48 AM VxSS-vxazd ERROR V-18-3004 Error encountered during ACI repository operation.
07/28/2012 05:11:48 AM VxSS-vxazd ERROR V-18-3078 Fatal error encountered. (txn.c:964)
07/28/2012 05:11:48 AM VxSS-vxazd LOG V-18-4204 Server is stopped.
07/30/2012 01:13:31 PM VxSS-vxazd LOG V-18-4201 Server is starting.

/usr/openv/netbackup/logs/nbazd/debug/vxazd_debug.log

07/28/2012 05:11:48 AM Unable to set transaction mode. error = (-1)
07/28/2012 05:11:48 AM SQL error S1000 -- [Sybase][ODBC Driver][SQL Anywhere] Connection was terminated
07/28/2012 05:11:48 AM Database fatal error in transaction, error (-1)
07/30/2012 01:13:31 PM _authd_config.c(205) Conf file path: /usr/openv/netbackup/sec/az/bin/VRTSaz.conf
07/30/2012 01:22:40 PM _authd_config.c(205) Conf file path: /usr/openv/netbackup/sec/az/bin/VRTSaz.conf

Our NBU master servers are hosted on virtual machines. It's a supported configuration and adds a lot of flexibility and resiliency to the overall enterprise-design. It also means that I have some additional metrics available to me to check. Unfortunately, when I checked those metrics, while I saw utilization spikes on the VM, those spikes corresponded to healthy operations of the VM. There weren't any major spikes (or troughs) during the grind-downs. So, to ESX, the VM appeared to be healthy.

At any rate, I've requested our ESX folks see if there might be anything going on on the physical systems hosting my VM that aren't showing up in my VM's individual statistics. I'd previously had to disable automated DRS actions to keep LikeWise from eating itself - those DRS actions wouldn't have been happening had the hosting ESX system not been experiencing loading issues - perhaps whatever was causing those DRS actions is still afflicting this VM.

I've also tagged one of our senior NBU operators to start picking through NBU's logs. I've asked him to look to see if there are any jobs (or combinations of jobs) that are always running during the bog-downs. If it's a scheduling issue (i.e., we're to blame for our problems), we can always reschedule jobs to exert less loading or we can scale up the VM's memory and/or CPU reservations to accommodate such problem jobs.

For now, it's a waiting-game. At least there's an investigation path, now. It's all in finding the patterns.

AD Integration Woes

For a UNIX guy, I'm a big fan of leveraging Active Directory form centralized system and user management purposes. For me, one of the big qualifications for any application/device/etc. to be able to refer to itself as "enterprise" is that it has to be able to pull user authentication information via Active Directory. I don't care whether it does it the way that Winbind-y things do, or if they just pull data via Kerberos or LDAP. All I care is that I can offload my user-management to a centralized source.

In general, this is a good thing. However, like many things that are "in general, a good thing" it's not always a good thing. If you're in an enterprise that's evolving it's Active Directory infrastructure, tying things to AD means that you're impacted when AD changes or breaks. Someone decides, "hey, we need to reorganize the directory tree" and stuff can break. Someone decides, "hey, we need to upgrade our AD servers from 2003 to 2008" and stuff can break.

Recently, I started getting complaints from the users of our storage resource management (SRM) system that they couldn't login any more. It'd been nearly a year since I'd set it up, so sorting it out was an exercise in trying to remember what the hell I did ...and then Googling.

The application, itself, runs on a Windows platform. The login module that it uses for centralized authentication advertises itself as "Active Directory". In truth, the login module is a hybrid LDAP/Kerberos software module. Even though it's a "Windows" application, they actually use the TomCat Java server for the web UI (and associated login management components). TomCat is what uses Kerberos for authentication data.

Sometime in recent months, someone had upgraded Active Directory. Users that had been using the software before the AD-upgrade were able to authenticate, just fine. Users that had tried to start using the software after the AD-upgrade couldn't get in. Turns out that, when Active Directory had gotten upgraded, the encryption algorithms had gotten changed. Ironically, I didn't find the answer to my problem in any of the forums related to the application: I found it in the forums for another application that used TomCat's Kerberos components.

To start the troubleshooting process, one needs to first modify TomCat's bsclogin.conf file. Normally, this file is only used to tell TomCat where to find the Kerberos configuration file. However, if you modify your bsclogin.conf file and add the directive "debug=true" to it like so:

     com.sun.security.auth.module.Krb5LoginModule required debug=true;

Enhanced user login debugging messages are enabled. Once this is added and tomcat is restarted, login-related messages will start showing up in your ${TOMCATHOME}/logs/stdout.log file. What started showing up in mine were messages like:

     [Krb5LoginModule] user entered username: thjones2@SRMdomain.NET
Acquire TGT using AS Exchange

     [Krb5LoginModule] authentication failed KDC has no support for encryption type (14)

With this error message in hand, I was able to find out that TomCat's Kerberos modules were using the wrong encryption routines to get data out of Active Directory. The fix was to update my Kerberos initialization file and add the following two lines to my [libdefaults] stanza (I just added it right after the dns_lookup_realm line and before the next stanza of directives):

     default_tkt_enctypes = rc4-hmac
     default_tgs_enctypes = rc4-hmac

Making this change (and restarting TomCat) resulted in the failing users suddenly being able to get in.

I'd normally rag on my Windows guys for this, but, the Windows guys aren't exactly used to providing AD-related service notifications to anyone but Windows users. This application, while (currently) running on a Windows platform, isn't something that's traditionally thought of as AD-reliant. Factor in that true, native AD modules pretty much just auto-adjust to such changes, and it didn't occur to them to notify us of the changes.

Oh well. AD-integration is a learning experience for everyone involved, I suppose.

NetBackup with Active Directory Authentication on UNIX Systems

While the specific hosts that I used for this exercise were all RedHat-based, it should work for any UNIX platform that both NetBackup 6.0/6.5/7.0/7.1 and Likewise Open are installed onto.

I'm a big fan of leveraging centralized-authentication services wherever possible. It makes life in a multi-host environment - particularly where hosts can number from the dozens to the thousands - a lot easier when you only have to remember one or two passwords. It's even more valuable in modern security environments where policies require frequent password changes (or, if you've even been through the whole "we've had a security incident, all the passwords on all of the systems and applications need to be changed, immediately" exercise). Over the years, I've used things like NIS, NIS+, LDAP, Kerberos and Active Directory to do my centralized authentication. If your primary platforms are UNIX-based, NIS, NIS+, LDAP and Kerberos have traditionally been relatively straight-forward to set up and use.

I use the caveat of "relatively" because, particularly in the early implementations of each service, things weren't always dead-simple. Right now, we seem to be mid-way through the "easiness" life-cycle of using Active Directory as a centralized authentication source for UNIX operating systems and UNIX-hosted applications. Linux and OSX seem to be leading the charge in the OS space for ease of integration via native tools. There's also a number of third-party vendors out there who provide commercial and free solutions to do it for you, as well. In our enterprise, we chose LikeWise, because, at the time, it was the only free option that also worked reasonably-well with our large and complex Active Directory implementation. Unfortunately, not all of the makers of software that runs on the UNIX hosts seem to have been keeping up on the whole "AD-integration within UNIX operating environment" front.

My latest pain in the ass, in this arena, is Veritas NetBackup. While Symantec likes to tout the value of NetBackup Access Control (NBAC) in a multi-administrator - particularly one where different administrators may have radically different NetBackup skill sets or other differentiating factors - using it in a mixed-platform environment is kind of sucktackular to set up. While modern UNIX systems have the PAM framework to make writing an application's authentication framework relatively trivial, Symantec seems to still be stuck in the pre-PAM era. NBAC's group lookup components appear to still rely on direct consultation of a server's locally-maintained group files rather than just doing a call to the host OS's authentication frameworks.

When I discovered this problem, I opened a support case with Symantec. Unfortunately, their response was "set up a Windows-based authentication broker". My NetBackup environment is almost entirely RedHat-based (actually, unless/until we implement BareMetal Restore (BMR) or other backup modules that require specific OSes be added into the mix, it is entirely RedHat-based). The idea of having to build a Windows server just to act as an authentication broker struck me as a rather stupid way to go about things. It adds yet another server to my environment and, unless I cluster that server, it introduces a single point of failure into and otherwise fairly resilient NetBackup design. I'd designed my NetBackup environment with a virtualized master server (with DRS and SRM supporting it) and multiple media servers for both throughput and redundancy

We already use LikeWise Open to provide AD-base user and group management service for our Linux and Solaris hosts. When I first was running NetBackup through my engineering process, using the old Java auth.conf method for login management worked like a champ. The Java auth.conf-based systems just assumes that any users trying to access the Java UI are users that are managed through /etc/passwd. All you have to do is add the requisite user/rights entries into the auth.conf file and Java treats AD-provided users the same as it treats locally-managed users. Because of this, I suspected that I could work around Symantec's authorization coding lameness.

After a bit of playing around with NBAC, I discovered that, so long as the UNIX group I wanted to map rights to existed in /etc/group, NBAC would see it as a valid, mappable "UNIX PWD" group. I tested by seeing if it would at least let me map the UNIX "wheel" group to one of the NBAC privilege groups. Whereas, even if I could look up the group via getent, if it didn't exist in /etc/group, NBAC would tell me it was an invalid group. Having already verified that a group's presence in /etc/group allowed NBAC to use a group, I proceded to use getent to copy my NetBackup-related groups out of Active Directory and into my /etc/group file (all you have to do is a quick `getent [GROUPNAME] >> /etc/group` and you've populated your /etc/group file).

Unfortunately, I didn't quite have the full groups picture. When I logged in using my AD credentials, I didn't have any of the expected mapped-privileges. I remembered that I'd explicitly emptied the userids from the group entries I'd added to my /etc/group file (I'd actually sed'ed the getents to do it ...can't remembery why, at this point - probably just anticipating the issue of not including userids in the /etc/group file entries). So, I logged out of the Java UI and reran my getent's - this time leaving the userids in place. I logged back into the Java UI and this time I had my mapped privileges. Eureka. 

Still I wasn't quite done. I knew that, if I was going to roll this solution into production, I'd have to cron-out a job to keep the getent file up-to date with the changing AD group memberships. I noticed, while nuking my group entry out of getent, that only my userid was on the group line and not every member of the group. Ultimately, tracked it down to LikeWise not doing full group enumeration by default. So, I was going to have to force LikeWise to enumerate the group's membership before running my getent's.

I proceded to dig around in /opt/likewise/bin for likely candidates for forcing the enumeration. After trying several lw*group* commands, I found that doing a `lw-find-group-by-name [ADGROUP]` did the trick. Once that was run, my getent's produced fully-populated entries in my /etc/group file. I was then able to map rights to various AD groups and wrote a cron script to take care keeping my /etc/group file in sync with Active Directory.

In other words, I was able to get NBAC to work with Active Directory in an all RedHat environment and no need to set up a Windows server just to be an authentication broker. Overall, I was able to create a much lighter-weight, portable solution.

Linux Active Directory Integration and PAM

Previously, I've written about using LikeWise to provide Active Directory integration to Linux and Solaris hosts. One of the down sides of LikeWise (and several other similar integration tools) is that it tends to make it such that, if a user has an account in Active Directory, they can log into the UNIX or Linux boxes you've bound to your domain. In fact, while walking someone through setting up LikeWise with the automated configuration scripts I'd written, that person asked, "you mean anyone with an AD account can log in?"

Now, this had occurred to me when I was testing the package for the engineer who was productizing LikeWise for our enterprise build. But, it hadn't really been a priority, at the time. Unfortunately, when someone who isn't necessarily a "security first" kind of person hits you with that question/observation, you know that the folks for whom security is more of a "Job #1" are eventually going to come for you (even if you weren't the one who was responsible for engineering the solution). Besides, I had other priorities to take care of.

This week was a semi-slack week at work. There was some kind of organizational get-together going on that had most of the IT folks out of town discussing global information technology strategies. Fortunately, I'd not had to take part in that event. So, I've spent the week revisiting some stuff I'd rolled out (or been part of the rollout of) but wasn't completely happy with. The "AD integration giving everyone access" thing was one of them. So, I began by consulting the almighty Google. When I'd found stuff I that seemed promising, I fired up a test VM and started trying it out.

Now, SSH (and several other services) weren't really a problem. Many applications allow you to internally regulate who can use the service. For example, with OpenSSH, you can modify the sshd_config file to explicitly define which users and groups can and cannot access your box through that service (for those of you who hit this page looking for tips, do a `man sshd_config` and grep for AllowUsers and AllowGroups for more in-depth information). Unfortunately, it's predictable enough to figure that people that are gonna whine about AD integration giving away the farm are gonna bitch if you tell them they have to modify the configuration of each and ever service they want to protect. No, most people want to be able to go to one place and take care of things with on action or one set of consistent actions. I can't blame them: I feel the same way. Everyone wants things done easily. Part of "easily" generally implies "consistently" and/or "in one place".

Fortunately, any good UNIX or Linux implementation leverages the Pluggable Authentication Management system (aka. PAM). There's about a bazillion different PAM modules out there that allow you to configure any given service's authentication to do or test a similar variety of attributes. My assumption for solving this particular issue was that, while there might be dozens or hundreds of groups (and thousands of users) in an Active Directory forrest, one would only want to grant a very few groups access to an AD-bound UNIX/Linux host. So, I wasn't looking for something that made it easy to grants lots of groups access in one swell-foop. In fact, I was kind of looking for things that made that not an easy thing to do (after all, why lock stuff down if you're just going to blast it back open, again?). I was also looking for something that I could fairly reliably find on generic PAM implementations. The pam_succeed_if is just about tailor made for those requirements.

LikeWise (and the other AD integration methods) add entries into your PAM system to allow users allowed by those authentication subsystems to login, pretty much, unconditionally. Unfortunately, those PAM modules don't often include methods for controlling which users are able to login once their AD authentication has succeeded. Since the PAM system uses a stackable authentication module, you can insert access controls earlier into the stack to cause a user access to fail out earlier than the AD module would otherwise grant the access. If you wanted to be able to allow users in AD_Group1 and AD_Group2 to log in, but not other groups, you'd modify your pam stack to insert the control ahead of the AD allow module.

     account    [default=ignore success=1] pam_succeed_if.so user ingroup AD_Group1 quiet_success
     account    [default=ignore success=1] pam_succeed_if.so user ingroup AD_Group2 quiet_success
     account    [default=bad success=ignore] pam_succeed_if.so user ingroup wheel quiet_success
     account    sufficient    /lib/security/pam_lsass.so

The above is processed such that if a user is a member of the AD-managed group "AD_Group1" or "AD_Group2", it sets the test's success flag to true. If the user isn't a member of those two groups, testing falls through to the next group check - is the user a member of the group wheel (if yes, fall through to the next test; if no, then there's a failure and the user's access is denied). Downside of using this particular PAM module is that it's only availble to you on *N*X systems with a plethora of PAM modules. This is true for many Linux releases - and I know it to be part of RedHat-related releases - but probably won't be available on less PAM-rich *N*X systems (yet one more reason to cast Solaris on the dung-heap, frankly). If your particular *N*X system doesn't have it, you can probably find the sourcecode for it and create yourself the requisite model for your OS.

Issues With LikeWise Open for UNIX/Linux Active Directory Integration

Recently it came up on management's radar that, "we're starting to get more UNIX systems out in our farms and we need some way to manage logins for those hosts in much the same way we do for our Windows hosts." Previous central login managment was done through the traditional tools like NIS or Kerberos (but mostly, just relied on ever-expiring local password tables). In a security conscious environment, NIS really isn't a responsible choice, any longer. Standing up a Kerberos infrastructure just to support UNIX systems is kind of an inefficient use of resources - especially when you already have a serviceable Active Directory infrastructure out there.

Fortunately, in the UNIX world, particularly in the Open Systems (Linux, BSD, etc.) realm, there are a number of choices available for joining a system to Active Directory. Many modern UNIX operating systems include this kind of functionality through Winbind. However, the various UNIX vendors or packagers don't necessarily keep fully up to date with the Windbind version they ship in their OS (looking in your direction, here, Sun Oracle). So, if you're running or looking to run Active Directory 2008 the Winbind included with your UNIX may not be a workable solution. Even if your UNIX does include an up to date version of Winbind (but not the bleeding-edge ones that are part of the next-generation Samba project), it may not be up to the task if you have a particularly large or complex Active Directory namespace. In this case, you'll probably be stuck using a commercial offering (or, a commercial AD-integration product's "free" version).
We were kind of stuck in the latter boat. Our AD namespaces tend to be rather large and rather complex. So, we started experimenting with the free version of the LikeWise product, LikeWise Open. In the test labs, with the smaller and simpler Active Directory deployment that supports it, we ran into no issues in testing. However, when we tried to take it into production, we ran into some issues in environments that had "messy" Active Directory deployments.

Specifically, the error we were getting (and could never really find a workaround for in the LikeWise Open support forums) was the "LW_ERROR_ENUM_DOMAIN_TRUSTS_FAILED" error. The domain that was giving us fits was a large domain (tens of thousands of users and thousands more groups, server objects and other, miscellaneous AD objects) that consisted of many AD servers scattered around the globe. Further complicating matters was the fact that these ADs were members of multiple domains and therefore had cross-realm/domain-trust components, as well. Further complicating it is the fact that not all of these AD servers, particularly the ones that had trust relationships with other realms, fully agreed on what time it was.
On the down side, the problem and the lack of easily-Googleable solutions cast doubt as to whether we'd be able to use this product in our enterprise. On the plus side, it gave me a chance to do some troubleshooting. I'm one of those masochists that likes a good challenge and digging into the guts of things. So, I did a bunch of online research as well as poring through the LikeWise administration guides.

Older versions of Likewise used to allow configuration-tweaking through an lsassd.conf file (indeed, other vendors, such as VMware still have this). The latest iteration of LikeWise, unfortunately, does not. Instead, the makers of LikeWise have decided to implement a Windows-esque "registry" for their product. Dunno why plain text files don't work (or even XML files, for that matter) - probably just wanted to make things more familiar for Windows admins that might get saddled with brining those evil Unix boxes into their domains. Whatever. It's painful but not insurmountable. LikeWise provides tools for hacking this file: lwregshell and lw-edit-reg. For me, lw-edit-reg was a more comfortable tool to use. All I had to do was make sure my EDITOR environmental variable was set to VI and I could hack the file to my heart's content with `vi`.

At any rate, I fired up lw-edit-reg and began to dig around for likely tweakables. Given that my error mentioned "TRUSTS", I did a global search for any parameters metniong "TRUSTS". I found the parameter, "DomainManagerIgnoreAllTrusts", and saw that it was set to false (well, the registry equivalent which was "dword:00000000"). So, I tried changing that to "true" (modifying the value to "dword:00000001"). I then bounced all of the LikeWise processes and re-attempted to join my box to the messy domain. Voilà, it worked and all was happy in UNIX-as-AD-Client land!

Of course, it wasn't until after I'd found my fix through the lw-edit-reg manipulations that I thought to check to see if lwconfig could be used. It seems like there's two variants of the lwconfig command in the LikeWise 6 release family. One supports the "--dump" option (which shows all tunable parameters along with their currently-set values) and one does not (thus, seeing available tunables and their current values are a bit less straight-forward). At any rate, upon further investigation, I found that I could use lwconfig to make my settings changes. Thus, it makes the software installation and configuration a lot more scriptable. I can modify my automated installation policies to do an `lwconfig DomainManagerIgnoreAllTrusts true` operation if it has issues with the normal `domainjoin-cli ...` operation.