AI Chatbots Are Like Goldfish

My employer is a Google Partner. They also use gSuite for most of the stuff that other organizations use O365 for. As a result, they've been "encouraging" us to use Gemini.

Today, I was running into weirdness when trying to port a SaltStack formula I'd written for RHEL to alsow work on Windows servers. I'm not a Windows guy. Worse, I was running into problems with Saltstack's functionality that "just worked" on Linux. So, I opted to avail myself of Gemini to try to diagnose what turned out to be a fairly pernicious problem with a common templating mechanism I was using.

While I shouldn't be surprised, it turns out that SaltStack's modules for Linux are a lot more robust than for Windows. I say, "I shouldn't be surprised," because I've run into similar problems with putatively cross-platform tools like Terraform …or even just different CSPs' CLIs and APIs. Which is to say, much like automation-frameworks for AWS are considerably more mature than for Azure, frameworks that work well for Linux-oriented configuration-management can be rage-inducing when you try to use them on Windows.

Ultimately, working with Gemini helped me dig far enough into the weeds to get to my solution. However, Gemini's "working memory" (compared to its "long-term" memory) is almost ridiculously small. At many points, it felt like I was working with a goldfish. It kept seeming to "forget" information I'd previously shared with it. So, it would ask for the same tests/results and make the same fix suggestions over and over again. In frustration, I asked it what the hell was going on:

As bad as the "forgetfulness" was, it also seemed to fasten onto provably dead diagnostic and fix paths. Which is to say, it would tell me to do stuff "we" had already tried, sometimes multiple times, and had shown to be either merely not helpful or created new problems.

Net result? I kept having to say, "goldfish: we did that already" or "goldfish: I told you to let that go" or "goldfish: how many times do I need to tell you that hard-coding shit isn't acceptable to me".

For all of that "memory" problem, when I asked for an end-of-day summary, it was able to summarize (though, not especially well) the things we'd done. I had to tell it "hey, you seem to have forgotten <THING>" ...which would prompt it to find <THING> and include it in its summary. All in all, not a particularly reassuring experience and one that continues to leave me feeling like "AI chat tools for coding-support isn't for people that aren't experienced enough to know better". Worse, they're probably also not for people whose own memory isn't above-average.

Dunno Who To Blame This On

I'm currently working on extending some SaltStack-based automation. I'd initially written it to work on (RHEL) Linux-based hosts. Worked like a champ. I was recently asked to extend the automation to also work on Windows. My most recent "Why I hate Powershell" posts are related to other tasks around this recent request.

At any rate, the approach I've been taking to extending my SaltStack logic is to refactor the Linux logic so as to insert an execution-branch based on the detected "kernel" (what's returned when one does a `salt-call --local grains.get kernel`). Basically, I took my prior "install.sls" file and moved it to "lin_install.sls" and then made a new "install.sls" that looked like:

include:
{%- if grains.kernel == "Linux" %}
  - .lin_install
{%- elif grains.kernel == "Windows" %}
  - .win_install
{%- endif %}

Worked like a champ on Linux; on Windows, "not so much". So, I started debugging the logic.

I primarily write my code on Linux-based development-VMs. I'd chosen the "kernel" SaltStack-grain, as my branching-basis, because I was hoping to head off having to do compound if blocks (in case anyone ever asked, "can you make this support Ubuntu instead of just Enterprise Linux and Windows"). Otherwise, I'd have chosen the "os_family" SaltStack-grain. Prior to running the extended logic on a Windows-based host (Server 2022 for this exercise), I'd simply assumed that the code would work. Imagine my surprise when I executed it on a Server 2022-based EC2 and I was getting error messages that I should only have seen on a missconfigured Linux-based host.

First thing I did was test how saltstack was rendering my  new "install.sls" file. I executed:

& 'C:\Program Files\Salt Project\Salt\salt-call.exe' -c C:\<config_path> \
  slsutil.renderer C:<state_file_path>\install.sls

Interestingly, it returned

local:
    ----------
    include:
        - .win_install

This meant that the Jinja was returning the expected list-element for the include statement. So, why the hell was I getting errors as though the ".lin_install" logic were what was being executed??

Did some digging around. One of the search results I got back indicated that the SaltStack minion for Windows can be flaky when using relative pathing to invoke other SaltStack files. So, I updated my "install.sls" to look like:

include:
{%- if grains.kernel == "Linux" %}
  - <formula_root>.package.lin_install
{%- elif grains.kernel == "Windows" %}
  - <formula_root>.package..win_install
{%- endif %}

This time, when I executed `<formula_root>.package.install`, directly (on my Server 2022-based EC2), it worked like it always should have. The reason I don't know who to blame is that while the problem is in SaltStack, I can't help but think that the problem is resultant of Windows weirdness.

Oh well, at least I have a path forward. 

Why, Palo Alto, WHY??

Yesterday, I was doing up some SaltStack based automation to help a customer automate the installation of the Cortex XDR Agent on RHEL-based Linux hosts. The vendor delivers the agent in the form of a ZIP-archived RPM. Yeah, I was a bit unimpressed by them deciding an RPM needed to be encapsulated in a ZIP-archive.

When you read the installation documentation, there's a link in the page they tell you to download. Yesterday, the embedded link was:

https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip

This URL was actually set up as an HTTP 302 (redirect) to:

https://docs-cortex.paloaltonetworks.com/api/khub/documents/Im1wc74y4HN15mXxBu3nYQ/content?Ft-Calling-App=ft%2Fturnkey-portal&Ft-Calling-App-Version=5.2.49&download=true&locationValue=viewer

SaltStack didn't care for trying to use file.managed to try to download from a redirect. I had to whip up some logic to chase the redirect and stuff it into a (Jinja) variable. Worked well once I got it in place.

Today, I was attempting to continue with the refactoring that I'd started, yesterday. This means launching an EC2 with the new automtion. I was surprised to find that the automation — unaltered since yesterday's day-ending push — was failing. When I checked the logs, SaltStack was complaining that I was trying to pass a null value to file.managed's source parameter. Perplexed, I started troubleshooting.

Ultimately, what I found was that the URL I was doing redirection-chasing on was no longer redirecting. Using curl like:

curl -Ls -o /dev/null 
  -w %{url_effective} "https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip"

Was returning null. So, I opted to try to make it provide more-definitive output like:

curl -Ls -o /dev/null \
  -w "Status: %{http_code}\nEffective URL: %{url_effective}\n" \
  "https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip"

This returned:

Status: 200
Effective URL: https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip

Which is to say, the reference URL is no longer returning an HTTP 302. Thus receiving a null-value when looking for a "url_effective" value. 

So, I revisited the documentation. I scrolled down to the where the document called out the signature-file's link. When I hovered over the link, it's value had changed since yesterday. I guess they pushed out a documentation-portal change and, along with that, nuked the previous URL's redirect action. This meant that the URL I was expecting was now returning a data-stream. Looking at the data-stream's first and last five lines:

curl -Ls https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip | \
sed -n '1,5p; :a;$p;N;11,$D;ba'

It was obvious from the output that the HTTP data-stream was now sending me JavaScript, presumably substituting the prior HTTP redirect with JS-based navigation-aids. Unfortunately, those kinds of navigation-aids work well enough for graphical browsers but not at all well for curl-type methods. 

None of this would have been necessary if they just maintained a file-repository of the RPMs and their signing-keys. Or, since they were ZIPing up the RPMs, include the damned signing-key in the archive-file. But, no, that would be too fucking easy and way too sensible.

I wish I could say this was surprising. However, I've had to integrate enough tooling — particularly security tooling — that I've gotten sort of used to (especially security) vendors  doing things in absolutely baffling ways. At least this vendor wasn't doing things in ways that required reducing system-security to allow installation of their tool: that is a hallmark of the brain-damage I frequently witness with security vendors' tooling.