Why I Love The UNIX/Linux CLI

There's many reasons, really. Probably too many to list in any one sitting.

That said, some days you pull an old trick out of your bag and you're reminded all over why you love something. Today was such a day.

All day, we've been having sporadic performance issues with AWS's CloudFormation functionality. Some jobs will run in (a few tens of) minutes, others will take an hour or more. Now, this would be understandable if they were markedly different tasks. But that's not the case. It's literally "run this template 'A' and wait" followed by "re-run template 'A' and wait" where the first run goes in about the time you'd expect and the second run of the same automation-template takes 10x as long.

Fun thing with cloudformation is that, when you launch from the command line, you can pass all of your parameters via a file. Some templates, however, don't like it when there's two active copies that contain the same parameter values. The way around this is to generalize your parameters file and then "process" that parameter file for each running. The UNIX/Linux CLI means that you can do all of this inline, like so:

aws --region us-east-1 cloudformation create-stack --stack-name RedMine07 \
  --disable-rollback --capabilities CAPABILITY_NAMED_IAM \
  --template-url https://s3.amazonaws.com/my-automation-bucket/Templates/RedMine.tmplt.json \
  --parameters file://<(sed 's/__NUM__/07/g' RedMine.parms.json)

The nifty part here is in the last line of that command. When you wrap a shell command in "<()", it runs the command within the parentheses and encapsulates it into a file-handle. Thus, if your main command requires an input-file be specified, the output of your encapsulated command gets treated just like it was being read from an on-disk file rather than as an inline-command.

Slick.

In my case, I'd generalized my parameters file to include a substitutable token, "__NUM__". Each time I need to run the command, all I have to do is change that "07" to a new value and bingo, new stack with unique values where needed.

Another fun thing is the shell's editable history function. If I want to run that exact same command, again, changing my values — both those passed as stack-parameters and the name of the resultant stack — I can do:

!!:gs/07/08

Which causes the prior stanza to be re-run as so:

aws --region us-east-1 cloudformation create-stack --stack-name RedMine08 \
  --disable-rollback --capabilities CAPABILITY_NAMED_IAM \
  --template-url https://s3.amazonaws.com/my-automation-bucket/Templates/RedMine.tmplt.json \
  --parameters file://<(sed 's/__NUM__/08/g' RedMine.parms.json)

Similarly, if the command I wanted to change was the 66th in my history rather than the one I just ran, I could do:

!66:gs/07/08

And achieve the same results.