I Hate PowerShell (Installment 3783)

So, I know I've skipped a few numbers since my I Hate PowerShell (Installment 3769) post. That post still applies and, yes, there've been several more annoyances encountered since authoring that post.

For the project I'm currently working on, I can only access my remote Windows (Server 2022) hosts using AWS's  FleetManager service. The VPCs are otherwise configured to block access via direct RDP.

At any rate, the first time I login, I fire up PowerShell. I go to type into the resulting "terminal" window and… nothing. The window doesn't accept my keyboard input. Perplexed, I Slack the teammate who asked me to write the automation for the Windows provisioning-tasks that he'd been doing manually. I note my problem to him. He tells me, "install PowerShell 7".

I update the userData I pass as part of launching my EC2 so as to implement that advice. I wait for the userData's execution to complete and then login. Once the remote desktop session becomes fully responsive, I fire up powershell …and find that I am still unable to type. Perplexed, I do some digging around and find that "upgrade to PowerShell 7" doesn't _actually_ upgrade PowerShell, it just installs a second, newer version of PowerShell alongside the system default. If I want to access the newer version, I need to call `pwsh`, instead. So, I do so. The new window opens and, as advertised, I'm able to type into that window.

Now, I'm a person possessed of significant curiousity. As such, my curious brain thinks, "what happens if I call `powershell` from the PS7 window I'm able to type into". To sate my curiosity, I do so. Suddenly, the window I was previously able to type into no longer accepts keyboard input.

So, yeah, if I want to be able to type into windows opened from the default PowerShell version, I need to add some further plumbing to my userData. Specifically, I need to include some logic to update the default PowerShell installation's `PSReadLine` module.

Notionally, I could skip the installation of PS7. However, its installation seems to be the default work-around for this project (still don't know if I'd call that an "anti-pattern" or not). As such, I fear that not having it installed will result in the eventual users of my automation screaming about it not being installed.

At any rate, if I want to be able to type into the default PowerShell version and I want PowerShell 7 available for others' use, my userData payload needs to look like:

# Ensure the default PowerShell (v5.x) windows accept typed-in content
Install-PackageProvider -name NuGet -Force
Install-Module `
  -Name PSReadLine `
  -Repository PSGallery `
  -Force

# Install PowerShell 7
iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"

So. Yeah. But, at least my deployment target works. 

Interestingly, I don't seem to run into the "can't type into the window" problem if I use the AWS SSM CLI to login. Or maybe I didn't start trying that route until I was already including the PSReadLine update-juju into my userData payload?

Also interestingly, because the PS7 installation is a parallel installation, if I want to use the  AWS SSM CLI to login directly to a PS7 session, I have to update my CLI-access invocation from:

aws ssm start-session --target <INSTANCE_ID>

To:

aws ssm start-session --target <INSTANCE_ID> \
  --document-name AWS-StartInteractiveCommand \
  --parameters command="pwsh"

I guess the "don't try to change the system-wide PS version" guidance that search-results are showing me are a lot like "don't try to replace the system-default Python version" cautions for Red Hat distros?

I Hate PowerShell (Installment 3769)

One of my customers has a group of developers. They're also tasked with SRE-style responsibilities as things move from development to production. Since the production environment is significantly locked down, these developers cannot access it from their laptops. As such, they don't have the kind of service-management tooling available to them in case shit goes sideways. To work around this issue, they requested the creation of a "jump box" in the production-environment that would have a reliable set of tools installed. Even though everything in production is either Linux or Kubernetes based, they wanted the "jump box" to be Windows-based. So, one of my peers hand-created a suitable "jump box" for them — installing all of the desired tools and creating the desired, RDP-enabled users.

In general, we don't like to have such hand-made boxes. They tend to be poorly maintained (such that their security auditors become sadder and sadder with the passing of each patching cycle) and, if they're the victim of system-breakage. Further, automating builds makes it much easier to do cost-control since you can better implement an "intantiate as you need it" deployment (or repair/replace) model. As such, they needed someone to automate what had previously been hand-jammed.

Since I'd just come off of a project where I'd delivered a bunch of "hardening" automation, I was the stuckee. I, uh, don't generally work with Windows. I especially don't really do automation for Windows-based systems. So, I accepted the task know that, "fun times be ahead," for me.

Part of the task involved hardening the "jump box" using a framework we'd been pushing our various customers to use for the past decade or so. Since there was already a "bootstrap" (PowerShell) script written for this purpose, I opted to use that script as my starting-point — why (wholly) reinvent the wheel, eh?

That script worked by downloading and executing the hardening framework. Easiest path forward, for me, was to update that script to similarly download and execute a my script.

At any rate, the first-pass at authoring my script was just to install a set of developer-oriented (really, more like "SRE oriented") tools. I wrote it in such a way that the automation user could specify specific versions of the desired tooling (by giving the URLs of the tools they wanted installed). It worked well enough. However, it didn't include the creation of RDP-enabled users. As such, once the "jump box" was built, the developers/SREs couldn't login without someone hand-jamming the creation of local users.

Today, I extended my script to allow the use of an external, JSON-formatted, user-specification file.

I wanted the various users' attributes to be flexible (e.g., the "is an admin user" attribute should allow a value that was any of `true`, `false` or undefined). The version of Powershell on the automation-targets apparently has some kind of "strict" mode set as the default behavior. So, when three of my JSON-file's testing user-definitions didn't have the "is an admin user" attribute defined, Powershell noped-out on that "problem". 

So, I did some digging around (Microsoft documentation, StackExchange, etc.). Figured out how to write the block in a manner that operates properly when the "strict" mode is set.

It was, to me, quite ugly compared to automation I've written for Linux-based hosts. To implement in a way that left the "strict" mode interpreter happy, I implemented my function like:

function Parse-JsonFile {
  # Where to write downloaded user-creation spec-file to
  $UserCreationFile = "${SaveDir}\$(${UserCreationUrl}.split("/")[-1])"

  # Download user-creation spec-file
  Download-File -Url ${UserCreationUrl} -SavePath ${UserCreationFile}

  # Abort if given file-path is not valid
  if ( -not ( Test-Path $UserCreationFile ) ) {
      Write-Error "File not found: $UserCreationFile"
      return
  }

  # Load JSON-payload from file and convert to PS object
  $JsonStream = Get-Content -Raw -Path "${UserCreationFile}" | ConvertFrom-Json

  # The structure has a 'Users' array containing a single object with dynamic keys
  foreach ($userContainer in $JsonStream.Users) {
    # Iterate through each dynamic key (the usernames)
    foreach ($username in $userContainer.psobject.Properties.Name) {
      # Get the array associated with that username
      $userDetails = $userContainer.$username

      foreach ($detail in $userDetails) {
        # Create "full name" attribute to user-creation function
        $FullName = ${detail}.givenName + " " + ${detail}.surname

        # Safely set WantsAdmin in a Strict-mode safe way
        if ( $detail.psobject.Properties.Name -contains "localAdmin" ) {
          $WantsAdmin = ${detail}.localAdmin
        } else {
          $WantsAdmin = $null
        }

        if ( ${WantsAdmin} -eq "true" ) {
          Create-User -UserUidName "$username" `
            -UserFullName ${FullName} `
            -UserPasswd ${detail}.initialPassword `
            -UserIsAdmin
        } else {
          Create-User -UserUidName "$username" `
            -UserFullName ${FullName} `
            -UserPasswd ${detail}.initialPassword
        }
      }
    }
  }
}

I understand that this is far from optimal way to do things. I hate how non-terse it feels like PowerShell makes me write things. However, I am not a PowerShell guy. It works. This is the first pass at it. I'll probably try to improve it in future iterations. 

I still hold out hope that we can convince their to use a Linux-based "jump box" (since, as mentioned early, their production environment is wholly Linux and Kubernetes-based) …At which point, I'll probably be tasked with figuring out how to RDP-enable a box with similar tooling and user-access. 

WSL Space Recovery

Recently, the backup client I use for my laptop started popping up, "operation aborted: out of space" messages. I'd been confused because, the last time I'd looked — just a few days prior — I had over 100GiB of free space in my boot-drive. Yet, when I looked at my disk-utilization, I only had a few tens of MiB free. WHAT??

So, I began the process of tracking down what was suddenly chewing up all my disk-space. Ultimately, I found a nearly 70GiB "ext4.vhdx" buried deep within my Windows home-directory's "AppData" hierarchy.

Did a quick web search and found that what I was seeing was the virtual hard disk for my WSL2 instance. This confused me because I'd been pretty scrupulous in keeping my WSL2 instance's storage in check. In fact, when I checked, my WSL instances visible storage was only 23GiB, nowhere near the 70GiB+ of the "ext4.vhdx" file that was backing that 23GiB of visible storage-usage.

Further Googling turned up that WSL doesn't really reclaim freed storage. So, that differential between visible storage an the size of ext4.vhdx" file was effectively wastage. Presumably my virtual hard drive was significantly sparse.

Next thing I looked up was "how to reclaim wasted space in a WSL drive." Ultimately discovered that I needed to:

1. Stop my WSL instance
2. Back up my WSL instance
3. "Optimize" my instance's hard drive
4. Restart my WSL instance

The first step was dead easy: just fire up a command.exe (or PowerShell session), then issue a `wsl --shutdown`.

Second step was also fairly easy, as it was something I was doing every few months, any way: while my backup client should be backing up the virtual hard disk, I don't trust that those backups are anything beyond "crash consistent". At any rate, I took the opportunity to do a:

wsl --export <WSL_DISTRO> G:\WSL_Backups\<WSL_DISTRO>\backup-$( date '+%Y%m%d' ).tar

Once I found articles on the subject, the VHD compression was also pretty straight-forward (and, thus far, no need for the backups):

1. Open a PowerShell window (the optimization-command is a PowerShell commandlet) 
2. Navigate to the directory hosting the VHD file 
3. Execute Optimize-VHD -Path .\ext4.vhdx -Mode full.

The optimization crushed the VHD file back down to a hair larger than the (internal to the instance) disk-size. 

Restarting my WSL instance is just going into my Search box and typing in the name of my instance, then clicking on the menu-item that appears.

Situation fixed, my next problem was, "why the hell did I end up with such a huge disk-image in the first place". The answer to that didn't really come until a few days later when my VHD blew up again.

I primarily use my WSL instance for work-related stuff. Recently, I'd been using podman to do some — a bunch, actually — container work. Worse, some of that Podman-based container-work was resulting in Buildah images getting generated. Whenever I would run `podman system prune --all --volumes && podman system prune --external`, the tool would tell me I'd recovered 5-20GiB of space (particularly the --external runs). 

Those space-recovery numbers made it occurr to me, "are my podman activities blowing my disk up?" So, after a fresh `… Optimize-VHD …` run, I decided to see if I could intentionally provoke a "VHD is multiples of my visible-use" situation. And, yes, I could.

Moral of the story: while you can use WSL instances with things like Podman, doing so will likely make it so you'll need to habituate to doing more system cleanup activities. 

Retrospective Automatic Image Replication in NetBackup

In version 7.x of NetBackup, VERITAS added the Automatic Image Replication functionality. This technology is more commonly referred to as "AIR". Its primary use case is to enable a NetBackup administrator to easily configure data replication between two different — typically geographically-disbursed — NetBackup domains.

Like many tools that are designed for a given use-case, AIR can be used for things it wasn't specifically designed for. Primary down-side to these not-designed-for use-cases is the documentation and tool-sets for such usage is generally pretty thin.

A customer I was assisting wanted to upgrade their appliance-based NetBackup system but didn't want to have to give up their old data. Because NetBackup appliances use Media Server Deduplication Pools (MSDP), it meant that I had a couple choices in how to handle their upgrade. I opted to try to use AIR to help me quickly and easily migrate data from their old appliance's MSDP to their new appliance's.

Sadly, as is typical of  not-designed-for use-case, documentation for doing it was kind of thin-on-the ground. Worse, because Symantec had recently spun VERITAS back off as its own entity, many of the forums that survived the transition had reference- and discussion-links that pointed to nowhere. Fortunately, I had access to a set of laboratory systems (AWS/Azure/Google Cloud/etc. is great for this - both from the standpoint of setup speed and "ready to go" OS templates). I was also able to borrow some of my customer's NetBackup 7.7 keys to use for the testing.

I typically prefer to work with UNIX/Linux-based systems to host NetBackup. However, my customer is a Windows-based shop. My customer's planned migration was also going to have the new NetBackup domain hosted on a different VLAN from their legacy NetBackup domain. This guided my lab design: I created a cloud-based "lab" configuration using two subnets and two Windows Server 2012 instance-templates. I set up each of my instances with enough storage to host the NetBackup software on one disk and the MSDPs on another disk ...and provisioned each of my test master servers with four CPUs and 16GiB or RAM. This is considerably smaller then both their old and new appliances, but I also wasn't trying to simulate an enterprise outpost's worth of backpup traffic. I also set up a mix of about twenty Windows and Linux instances to act as testing clients (customer is beginning to add Linux systems as virtualization and Linux-based "appliances" have started to creep into their enterprise-stacks).

I set up two very generic NetBackup domains. Into each, I built an MSDP. I also set up a couple of very generic backup policies on the one NetBackup Master Server to backup all of the testing clients to the MSDP. I configured the policies for daily fulls and hourly incrementals, and set up each of the clients to continuously regenerate random data-sets in their filesystems. I let this run for forty-eight hours so that I could get a nice amount of seed-data into the source NBU domain's MSDP.

Note: If you're not familiar with MSDP setup, the SETTLERSOMAN website has a good, generic walkthrough.

After populating the source site's MSDP, I converted from using the MSDP by way of a direct STorage Unit definition (STU) to using it by way of a two-stage Storage Lifecycle Policy (SLP). I configured the SLP to use the source-site MSDP as the stage-one destination in the lifecycle and added the second NBU domain's MSDP as the stage-two destination in the lifecycle. I then seeded the second NBU domain's MSDP with data by executing a full backup of all clients against the SLP.

Note: For a discussion on setting up an AIR-based replication SLP, again, the SETLLERSOMAN website has a good, generic walkthrough.

All of the above is fairly straight-forward and well documented (both within the NBU documentation and sites like SETTLERSOMAN). However, it only addresses the issue of how you get newly-generated data from one NBU domain's MSDP to another's. Getting older data from an existing MSDP to a new MSDP is a bit more involved ...and not for the command-line phobic (or, in my case, PowerShell-phobic.)

At a high level, what you do is:

1. Use the `bpimmedia` tool to enumerate all of the backup images stored on the source-site's MSDP
2. Grab only the media-IDs of the enumerated backup images
3. Feed that list of media-IDs to the `nbreplicate` tool so that it can copy that old data to the new MSDP

Note: The vendor documentation for the `bpimmedia` and  `nbreplicate` tools can be found at the VERITAS website.

When using the `bpimmedia` tool to automate image-ID enumeration, using the `-l` flag puts the output into a script-parsable format. The desired capture-item is the fourth field in all lines that begin 'IMAGE':

• In UNIX/Linux shell, use an invocation similar to: `bpimmedia -l | awk '/^IMAGE/{print $4}`
• In PowerShell, use an invocation similar to:`bpimmedia -l | select-string -pattern "IMAGE *" | ForEach-Object { $data = $_ -split " " ; "{0}" -f $data[3] }`

The above output can then be either captured to a file — so that one the `nbreplicate` job can be launched to handle all of the images — or each individual image-ID can be passed to an individual `nbreplicate` job (typically via a command-pipeline in a foreach script). I ended up doing the latter because, even though the documentation indicates that the tool supports specifying an image-file, when executed under PowerShell, `nbreplicate` did not seem to know what to do with said file.

The `nbreplicate` command has several key flags we're interested in for this exercise:

• -backupid: The backup-identifier captured via the `bpimmedia` tool
• -cn: The copy-number to replicate — in most circumstances, this should be "1"
• -rcn: The copy-number to assign to the replicated backup-image — in most circumstances, this should be "1"
• -slp: the name of the SLP hosted on the destination NetBackup domain
• -target_sts: the FQDN of the destination storage-server (use `nbemmcmd -listhosts` to verify names - or the replication jobs will fail with a status 191, sub-status 174)
• -target_user: the username of a user that has administrative rights to the destination storage-server
• -target_user: the password of the the -target_user username

 If you don't care about minimizing the number of replication operations, this can all be put together similar to the following:

• For Unix:

  for ID in $(bpimmedia -l | awk '/^IMAGE/{print $4}')
  do
     nbreplicate -backupid ${ID} -cn 1 -slp_name <REMOTE_SLP_NAME> \
       -target_sts <REMOTE_STORAGE_SERVER> -target_user <REMOTE_USER> \
       -target_pwd <REMOTE_USER_PASSWORD>
  done
  

• For Windows:

  @(bpimmedia -l | select-string -pattern "IMAGE *" | \
     ForEach-Object { $data = $_ -split " " ; "{0}" -f $data[3] }) | \
     ForEach-Object { nbreplicate -backupid $_ -cn 1 \
       -slp_name <REMOTE_SLP_NAME> -target_sts <REMOTE_STORAGE_SERVER> \
       -target_user <REMOTE_USER> -target_pwd <REMOTE_USER_PASSWORD> }