Dunno Who To Blame This On

I'm currently working on extending some SaltStack-based automation. I'd initially written it to work on (RHEL) Linux-based hosts. Worked like a champ. I was recently asked to extend the automation to also work on Windows. My most recent "Why I hate Powershell" posts are related to other tasks around this recent request.

At any rate, the approach I've been taking to extending my SaltStack logic is to refactor the Linux logic so as to insert an execution-branch based on the detected "kernel" (what's returned when one does a `salt-call --local grains.get kernel`). Basically, I took my prior "install.sls" file and moved it to "lin_install.sls" and then made a new "install.sls" that looked like:

include:
{%- if grains.kernel == "Linux" %}
  - .lin_install
{%- elif grains.kernel == "Windows" %}
  - .win_install
{%- endif %}

Worked like a champ on Linux; on Windows, "not so much". So, I started debugging the logic.

I primarily write my code on Linux-based development-VMs. I'd chosen the "kernel" SaltStack-grain, as my branching-basis, because I was hoping to head off having to do compound if blocks (in case anyone ever asked, "can you make this support Ubuntu instead of just Enterprise Linux and Windows"). Otherwise, I'd have chosen the "os_family" SaltStack-grain. Prior to running the extended logic on a Windows-based host (Server 2022 for this exercise), I'd simply assumed that the code would work. Imagine my surprise when I executed it on a Server 2022-based EC2 and I was getting error messages that I should only have seen on a missconfigured Linux-based host.

First thing I did was test how saltstack was rendering my  new "install.sls" file. I executed:

& 'C:\Program Files\Salt Project\Salt\salt-call.exe' -c C:\<config_path> \
  slsutil.renderer C:<state_file_path>\install.sls

Interestingly, it returned

local:
    ----------
    include:
        - .win_install

This meant that the Jinja was returning the expected list-element for the include statement. So, why the hell was I getting errors as though the ".lin_install" logic were what was being executed??

Did some digging around. One of the search results I got back indicated that the SaltStack minion for Windows can be flaky when using relative pathing to invoke other SaltStack files. So, I updated my "install.sls" to look like:

include:
{%- if grains.kernel == "Linux" %}
  - <formula_root>.package.lin_install
{%- elif grains.kernel == "Windows" %}
  - <formula_root>.package..win_install
{%- endif %}

This time, when I executed `<formula_root>.package.install`, directly (on my Server 2022-based EC2), it worked like it always should have. The reason I don't know who to blame is that while the problem is in SaltStack, I can't help but think that the problem is resultant of Windows weirdness.

Oh well, at least I have a path forward. 

Why, Palo Alto, WHY??

Yesterday, I was doing up some SaltStack based automation to help a customer automate the installation of the Cortex XDR Agent on RHEL-based Linux hosts. The vendor delivers the agent in the form of a ZIP-archived RPM. Yeah, I was a bit unimpressed by them deciding an RPM needed to be encapsulated in a ZIP-archive.

When you read the installation documentation, there's a link in the page they tell you to download. Yesterday, the embedded link was:

https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip

This URL was actually set up as an HTTP 302 (redirect) to:

https://docs-cortex.paloaltonetworks.com/api/khub/documents/Im1wc74y4HN15mXxBu3nYQ/content?Ft-Calling-App=ft%2Fturnkey-portal&Ft-Calling-App-Version=5.2.49&download=true&locationValue=viewer

SaltStack didn't care for trying to use file.managed to try to download from a redirect. I had to whip up some logic to chase the redirect and stuff it into a (Jinja) variable. Worked well once I got it in place.

Today, I was attempting to continue with the refactoring that I'd started, yesterday. This means launching an EC2 with the new automtion. I was surprised to find that the automation — unaltered since yesterday's day-ending push — was failing. When I checked the logs, SaltStack was complaining that I was trying to pass a null value to file.managed's source parameter. Perplexed, I started troubleshooting.

Ultimately, what I found was that the URL I was doing redirection-chasing on was no longer redirecting. Using curl like:

curl -Ls -o /dev/null 
  -w %{url_effective} "https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip"

Was returning null. So, I opted to try to make it provide more-definitive output like:

curl -Ls -o /dev/null \
  -w "Status: %{http_code}\nEffective URL: %{url_effective}\n" \
  "https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip"

This returned:

Status: 200
Effective URL: https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip

Which is to say, the reference URL is no longer returning an HTTP 302. Thus receiving a null-value when looking for a "url_effective" value. 

So, I revisited the documentation. I scrolled down to the where the document called out the signature-file's link. When I hovered over the link, it's value had changed since yesterday. I guess they pushed out a documentation-portal change and, along with that, nuked the previous URL's redirect action. This meant that the URL I was expecting was now returning a data-stream. Looking at the data-stream's first and last five lines:

curl -Ls https://docs-cortex.paloaltonetworks.com/v/u/cortex-xdr-agent.zip | \
sed -n '1,5p; :a;$p;N;11,$D;ba'

It was obvious from the output that the HTTP data-stream was now sending me JavaScript, presumably substituting the prior HTTP redirect with JS-based navigation-aids. Unfortunately, those kinds of navigation-aids work well enough for graphical browsers but not at all well for curl-type methods. 

None of this would have been necessary if they just maintained a file-repository of the RPMs and their signing-keys. Or, since they were ZIPing up the RPMs, include the damned signing-key in the archive-file. But, no, that would be too fucking easy and way too sensible.

I wish I could say this was surprising. However, I've had to integrate enough tooling — particularly security tooling — that I've gotten sort of used to (especially security) vendors  doing things in absolutely baffling ways. At least this vendor wasn't doing things in ways that required reducing system-security to allow installation of their tool: that is a hallmark of the brain-damage I frequently witness with security vendors' tooling.

I Hate PowerShell (Installment 3783)

So, I know I've skipped a few numbers since my I Hate PowerShell (Installment 3769) post. That post still applies and, yes, there've been several more annoyances encountered since authoring that post.

For the project I'm currently working on, I can only access my remote Windows (Server 2022) hosts using AWS's  FleetManager service. The VPCs are otherwise configured to block access via direct RDP.

At any rate, the first time I login, I fire up PowerShell. I go to type into the resulting "terminal" window and… nothing. The window doesn't accept my keyboard input. Perplexed, I Slack the teammate who asked me to write the automation for the Windows provisioning-tasks that he'd been doing manually. I note my problem to him. He tells me, "install PowerShell 7".

I update the userData I pass as part of launching my EC2 so as to implement that advice. I wait for the userData's execution to complete and then login. Once the remote desktop session becomes fully responsive, I fire up powershell …and find that I am still unable to type. Perplexed, I do some digging around and find that "upgrade to PowerShell 7" doesn't _actually_ upgrade PowerShell, it just installs a second, newer version of PowerShell alongside the system default. If I want to access the newer version, I need to call `pwsh`, instead. So, I do so. The new window opens and, as advertised, I'm able to type into that window.

Now, I'm a person possessed of significant curiousity. As such, my curious brain thinks, "what happens if I call `powershell` from the PS7 window I'm able to type into". To sate my curiosity, I do so. Suddenly, the window I was previously able to type into no longer accepts keyboard input.

So, yeah, if I want to be able to type into windows opened from the default PowerShell version, I need to add some further plumbing to my userData. Specifically, I need to include some logic to update the default PowerShell installation's `PSReadLine` module.

Notionally, I could skip the installation of PS7. However, its installation seems to be the default work-around for this project (still don't know if I'd call that an "anti-pattern" or not). As such, I fear that not having it installed will result in the eventual users of my automation screaming about it not being installed.

At any rate, if I want to be able to type into the default PowerShell version and I want PowerShell 7 available for others' use, my userData payload needs to look like:

# Ensure the default PowerShell (v5.x) windows accept typed-in content
Install-PackageProvider -name NuGet -Force
Install-Module `
  -Name PSReadLine `
  -Repository PSGallery `
  -Force

# Install PowerShell 7
iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"

So. Yeah. But, at least my deployment target works. 

Interestingly, I don't seem to run into the "can't type into the window" problem if I use the AWS SSM CLI to login. Or maybe I didn't start trying that route until I was already including the PSReadLine update-juju into my userData payload?

Also interestingly, because the PS7 installation is a parallel installation, if I want to use the  AWS SSM CLI to login directly to a PS7 session, I have to update my CLI-access invocation from:

aws ssm start-session --target <INSTANCE_ID>

To:

aws ssm start-session --target <INSTANCE_ID> \
  --document-name AWS-StartInteractiveCommand \
  --parameters command="pwsh"

I guess the "don't try to change the system-wide PS version" guidance that search-results are showing me are a lot like "don't try to replace the system-default Python version" cautions for Red Hat distros?