Using Arbitrary SSH Keys With GCP-Hosted Linux Instances (Or: "WTF: This Works on AWS")

 As I start on my knowledge-transition from AWS to GCP (see prior article), one of the first things I wanted to figure out was "how do I re-use my project-related SSH-keys to access Linux-based, GCP-hosted instances similar to how I do it for AWS-hosted instances".

In general, I prefer not to make my cloud-hosted Linux instances directly-reachable via public IP addresses. In the case of AWS, I've spent the last few years leveraging SSH-over-SSM. AWS added the capability to access EC2s' interactive shells via SSM back in 2018. Initially, that access was mediated through a web-browser. However, not long after making the "shell via SSM" capability available through the AWS web consoles, AWS published an article on how you can leverage your local workstation's native SSH client.

Sadly, this article no longer seems to show up in web-searches, at least, not on high up in the search results: there's now a number of people that have published their own blog-entries on the topic and those somehow manage to rank higher than AWS's own guidance. I've even alluded to it in some of my other, SSH-tagged posts. Bleah. I'm not going to link to any of the "usurpers'" articles because I don't want to reward that behavior. However, if you check my article, "So You Work in Private VPCs and Want CLI Access to Your Linux EC2s?", does link to another, less "easy-button" article that is on an AWS URL.

Lack of link-out to reference articles aside, the ability to use my local (OpenSSH) client tunneled through SSM has meant:

• Not having to deal with the AWS web console — and, if your AWS accounts are security-focussed in their setup, you might not even have access to the AWS web consoles — and its clunkiness.
• Not having to make your EC2s available via public IPs — be that an ephemeral IP, an elastic IP or even public-facing elastic loadbalancer. This in turn meant:
• Not having to make the choice of "do I allow the world to bang on my EC2s' SSH daemon or do I have to ass around with maintaining IP whitelists". Given that I'm a remote worker who's both mobile and a habitual user of VPN services, maintaining IP whitelists was always a freaking chore …especially if an account I was working in didn't allow access to the EC2 web console (so I could update my security-groups' whitelists)
• Not even having to sacrifice VPC-security by allowing public ingress — whether directly or via a bastion-host
• Being able to use scp from wherever I was working whenever I needed to transfer files from home to my EC2(s) …and not having to ass with copying shit to an internet-reachable file-share from my workstation and then pulling the file(s) to the EC2(s)

Bonus, transiting SSM meant that there was additional, cloud-layer security-logging (even to the point one can kludge a keystroke-logger — there's a linkout to a how-to from my previously-linked "So You Work…" post). Providing additional logging capabilities generally makes your security folks happy.

 Oof: tangent. Let's try to get back on track…

At any rate, GCP provides a native capability that's analogous to tunneling SSH over SSM. Specifically, the gcloud CLI utility includes the ability to easily leverage GCP's Identity Aware Proxy. The gcloud CLI utility includes an SSH-client wrapper, accessed by using `gcloud compute ssh …` and tacking on the `--tunnel-through-iap`flag. Problem is, its default behavior is to use its own key, rather than one of your keys. Maybe it's just me, but it feels like the documentation for how to override that behavior is somewhere in the "ain't great" neighborhood. Similarly, the various web-searches I did were turning up other, "ain't great" guidances. Now, I'm going to contribute my own, probably "ain't great", guidance. 

Tangent inbound…

One of the things that `gcloud compute ssh …` does is that it takes whatever SSH key you're using and adds it to your GCP project's metadata. You can see any such keys by looking at your project's metadata. If you're in the GCP web-console, this is found by clicking through the service-menus ("Compute Engine → Metadata"). Once you're in the metadata console, you can click on the "SSH Keys" tab. If there are keys attached to the project, you will get a display like:

Similarly, you can get this information from the CLI:

gcloud compute project-info describe --format='json(commonInstanceMetadata.items)' | \
jq -r '.commonInstanceMetadata.items | .[] | select(.key == "ssh-keys") | .value'

The above can probably be done more-efficiently and wholly within the `gcloud` command (i.e., not need to use `jq`), but I'm too lazy to continue banging on it.

</tangent>

With the SSH public-key attached to the project's metadata, the Google agent running within the VM will insert it into a user's "${HOME}/.ssh/authorized_keys" file. This allows the VM-administrator to SSH in with the matching private key (whether ushing `gcloud compute ssh` or other SSH client). Upon logging in, one can see the results of the agent having done this by looking at the logged-in user's "${HOME}/.ssh/authorized_keys" file:

$ gcloud compute ssh \
  --tunnel-through-iap \
  --ssh-key-file="/tmp/rsa_test-20250522" \
  test-user@development-vm
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

Last login: Fri May 23 20:01:48 2025 from 35.235.244.34
[test-user@development-vm ~]$ cat ~/.ssh/authorized_keys
# Added by Google
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyjNt3anbU1XNaUbNLk8sZq+8lOY+WSQ/QHWAN+uzBKxNfZXIi/EnRjvgudMl4tiNVlEGPa+VA+he8TpQAomvSYSTgelNFaCzukiJ0wMJKKCb1u2QXRBV3k8ihbZx8nKE2OBHonOu4lGlMWl7P0mq4m7ir+t/1Pf9lGlbNvx1WgEdp4tnFO/eGIvBupPSwS8ew6p+ulwJBa9Po6KwNWg1UiG5BVLAejWJYZeBZ44dKQCc1i60ziFqr2lC4jktl032ftAGQaT+rA7RhppzErAn53eC5c70skt0EcFVd/y1773f2rjow+9VzSLJ9QKTSMp9meoLyqJpuctiwSLbCb4L2fSdsdXQcn+0ncEkbM4gvvqDWT8l4mL8Ar2xxYcIssEGqJ1uhLQgGPMXlb02PbePU8KIVt2ViW/s3fIwwUdNmewRxIdjPrIa2ddOmTy4SP6Js9lP/Y8yU4et9k9oLbl6eDg95d50uzFCIX5thEgQygWNrqBQjphWcbSvPO3kh1Z0= test-user@development-vm

Unfortunately, the `gcloud` utility's SSH wrapper doesn't — or doesn't reliably do so — know how to interact with any SSH-agent that might be in use. So, while it will do SSH key-forwarding, just fine (any keys in the local SSH-agent will be show if one executes `ssh-add -l` from the GCP-hosted instance), if the key-file you passed with the `--ssh-key-file` option has a password associated with it, you'll likely be prompted for it.

Expanding Expertise to Go "Multi-Cloud"

 

Recently, my employer had some customers ask them, "can you help us go multi-cloud …to Google Cloud". Up until this year, my employer specialized in helping customers with cloud-adoption, and several customers have expressed interest in "multi-cloud". However, the clouds they were all previously interested in were AWS and Azure. We (as a company) have quite a few people that are rather familiar with AWS (e.g., "me": I've been working on AWS for a few months shy of a full decade). We also have a smaller group Azure practitioners. What we didn't have was people with experience with Google Cloud.

Due to a lapse in a contract — currently in re-negotiation — another engineer and I were between active assignments. As a result, our employer asked us to start getting ourselves spun up, enough, on Google Cloud to be able to get certified. So, we each did. Over the past couple months, I trained up on Google Cloud (thank you, CloudSkillsBoost), and then sat exams for an Associate Cloud Engineer (ACE) certification and then a Professional Cloud Architect (PCA) certification. Even gots the proof of having (successfully) sat those exams:

That said, the priority of the exercise was getting the certifications. Actually having the skills the certifications were supposed to represent was secondary.

Having significant AWS experience under my belt, studying for the Google certification(s) was mostly an exercise in figuring out the deltas. The actual tests were fairly easy: to use a poor metaphor, "if you know how to use a claw-hammer to drive nails, it's not that hard to figure out how to use a ball-peen hammer to do the same thing".

Now, I'm using my "spare time" to close the knowledge gaps… Expect a bunch of GCP-related posts as I leave myself breadcrumbs on how to do things.

Curbing `~/.vimrc` Sprawl

I've been using vi-based editors since the late 80s. While an .exrc – or, as I eventually settled on vim as my vi-like editor, a .vimrc – is a generically decent way to pre-configure one's edit sessions, they have their shortcomings. As my work causes me to need to edit files of ever-broadening file-types, I find, with greater frequency, that I want to tailor my editor's behavior along swaths of file-types. In the land of monolithic .exrcs/.vimrcs, setting up customized-behaviors for multiple file-types results in the monolithic .exrcs/.vimrcs becoming stoopid-long. This makes them harder to read, harder to understand a general pain in the ass to maintain.

In general, I'm a fan of smaller config-files and using some kind of inheritance structure or other conditional-read method. Wasn't until today that I decided to see, "do current iterations of vim let me do that?" Boy was I glad I did: it turns out that the answer is, "yes it does". The key is judicious use of not only one's ~/.vimrc but of files in ~/.vim/.

After some digging around in the documentation and experimentation, I was able to solve my most pressing needs with a collection of files:

• ~/.vimrc: the grand-daddy
• ~/.vim/filetype.vim: To handle mapping file-extensions to vim-recognized file-type handlers
• ~/.vim/ftplugin/bash.vim: A file-type handler for files with the .bash extension
• ~/.vim/ftplugin/sh.vim: A file-type handler for files with the .sh extension

…I'll be adding more 

~/.vim/filetype.vim:

This file provides file-extension to file-type mappings. These mappings are used when using file-type files in the ~/.vim/ftplugin/ directory.

The contents of the ~/.vim/filetype.vim file is (currently) fairly simple and straight-foward:

"An easier-to read filetype mapper routine
if exists("did_load_filetypes")
  finish
endif
augroup filetypedetect
  au! BufRead,BufNewFile *.bash        setfiletype bash
  au! BufRead,BufNewFile *.sh          setfiletype sh
augroup END

As the comment-line indicates, this file makes it easy to legibly extend mapping of file-suffixes to file types. This is done by using a grouping-block to apply autocmd-type directives. Such a grouping-block allow line-by-line mapping-extensions. The above replaces content like:

autocmd BufRead,BufNewFile *.sh,*.bash,*.sls,*.json,*.yml <ACTION>

Thus, if one wants to, say, add further file-type mappings (say for YAML files), one could add a line like:

au! BufRead,BufNewFile *.yml           setfiletype yaml

Or even:

au! BufRead,BufNewFile *.yml           setfiletype yaml
au! BufRead,BufNewFile *.yaml          setfiletype yaml

The latter line-pair captures YAML files that might be suffixed .yml or .yaml. As someone who does work for multiple organizations – either concurrently or serially – I've found, much to my consternation, that selected file-suffixes may differ from organization to organization.

~/.vim/ftplugin/*.vim:

The …/ftplugin directory contains the files that contain the actual, per file-type configuration-directives. The previous …/filetype.vim file's `setfiletype` command provides the value of the `*` in the header's wildcarded .vim file-names. Thus a .bash file that was mapped to the bash file-type will cause the ~/.vim/ftplugin/bash.vim file's configuration-options to be read.

Note: while my earlier file-list appears to show two, discrete config-files:

• ~/.vim/ftplugin/bash.vim
• ~/.vim/ftplugin/sh.vim

They are actually just one file that's hard-linked:

$ ls -il *.vim
330115 -rw-r--r-- 2 ferric ferric 453 Feb  5 16:27 bash.vim
330115 -rw-r--r-- 2 ferric ferric 453 Feb  5 16:27 sh.vim

These files (currently) contain:

"Ensure that lines break at 80 characters
set wrap
set textwidth=80
set wrapmargin=0

"Ensure linewraps happen when textwidth is reached
set formatoptions+=t

"Enable auto-indent
set autoindent

"Set 4-char tabstops
set tabstop=4

"Convert tabs to spaces
set expandtab

"Set autoindent to 4-chars
set shiftwidth=4

"Eliminate trailing whitespace
augroup prewrites
   autocmd!
    autocmd BufWritePre,FileWritePre * :%s/\s\+$//e | %s/\r$//e
augroup END

One could create equivalent *.vim files for other filetypes. One would do this if, say, you leveraged different maximum line-widths, indent-paddings or the like on a per-language file-basis.

~/.vimrc:

With the other files in place, this allows for a svelter ~/.vimrc file. Mine now looks like:

"Disable syntax-highlighting
syntax off

"Turn off search-highlighting
set nohlsearch

"Turn off angry fruit-salad
set t_Co=0

I, uh, hate the "angry fruit-salad" appearance that many distro's default vim-configs use.

SSH Problems When Using `sysadm_u` SELinux Confinement

Decided to be proactive on the security-setup for my one project. Opted to confine my default-user to sysadm_u. However, as soon as I did that, I stopped being able to ssh into the resulting EC2 as the default-user. Turns out there's a bit more requirede in order to use that confinement with a user that also needs to be able to SSH into the host.

For those reading and who don't have a Red Hat login, if I want to confine a user to the to sysadm_u, I also need to ensure that my system-configuration automation includes:

setsebool ssh_sysadm_login on
setsebool -P ssh_sysadm_login on

Without the above, doing an ssh -v to the target user will show a spurious:

Authenticated to 0.0.0.0 ([127.0.0.1]:22) using "publickey".
debug1: pkcs11_del_provider: called, provider_id = (null)
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem full
client_loop: send disconnect: Broken pipe

The `pledge: filesystem full` kinda threw me, at first, since I knew that neither my local nor my remote filesystem was full. So, I assumed that it was just a misleading error message (as seems to so often be the case when SELinux is involved). So, I searched for ssh login problems associated with the selected SELinux-confinement, which led me to the previously-linked Red Hat article.

I guess that's why the hardening guidelines show `staff_u` as the recommended confinement for administrator users?

 Ultimately, I opted to use  `staff_u`, instead. Having a cloud-config block like:

user: {
  gecos: "GitLab Provisioning-Account (LOCAL)",
  name: "${PROVISIONING_USER}",
  selinux_user: 'staff_u',
  sudo: ['ALL=(root) TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL']
}

Ensuring to have ROLE and TYPE SELinux transition-mappings defined for my default-user eliminates the confusion that can result when confining a user to staff_u and not supplying a mapping. Without the mapping, if an confined admin-user executes `sudo -i`, they get all sorts of unexpected `permission denied` errors.

Proving a Point: Git-over-SSH Edition

I'm in the process of helping a customer migrate from an on-premises GitLab configuration to an AWS-hosted configuration. The on-premises GitLab is hosted on CentOS 7. The cloud-hosted GitLab will be hosted on Rocky or Alma 9. 

Right now, the customer is doing User Acceptance Testing (UAT). They're running into some issues making legacy projects' repositories and associated automations work with the new GitLab service. One of these problems is that their developers are using keys in excess of five years' age. The OpenSSH server in CentOS 7 is elderly and, as a result, had been ok using these keys. However, because the maintainers of OpenSSH deprecated the use of SHA1-signed RSAv2 keys some time ago (and the OpenSSH versions in EL 8&9 and derivatives updated the shipped OpenSSH server version), the OpenSSH server in RHEL 9-derived distros just is not having it with these elderly keys.

The developer was suspicious of my claim that this was the source of their git-over-ssh problems. I needed to be able to prove things to them, but I haven't had a Linux host capable of creating SHA1-signed RSAv2 SSH keys in quite a while. So, "what to do"?

Turns out, "Docker to the rescue". Not wanting to dick with the overhead of writing a Dockerfile, I simply did it interactively:

1. Login to Dockerhub …to reduce the likelihood of getting errors around too many anonymous fetch-attempts in a given timespan
   
2. Launch an interactive CentOS 6 container with a volume attached so that I could easily save out any generated keys:

   $ docker run -it -v $( pwd ):/save_dir --entrypoint /bin/bash centos:6

3. Install the openssh-clients RPM:

   # yum install -y --disablerepo=* --enablerepo=C6.9* openssh-clients

4. Generate a suitable key:

   # ssh-keygen \
       -t rsa \
       -b 2048 \
       -C "SHA1-signed key generated on CentOS 6" \
       -f /save_dir/id_rsa_sha1-signed

5. Exit from the running container (<CTRL>-D suffices)
6. Fix any ownership/permission problems on the new files
   
7. Register the new key with GitLab
   
8. Attempt a git clone using the SSH URL and ensure that I'm using the newly-generated key

   $ git clone \
       -c core.sshCommand="/usr/bin/ssh -i $( pwd )/id_rsa_sha1-signed" \
       git@<GIT_SERVER_FQD>:<REPOSITORY_PATH>

9. Receive an error like:

   Cloning into '<REPO_NAME>'...
   git@<GIT_SERVER_FQD>: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
   fatal: Could not read from remote repository.
   
   Please make sure you have the correct access rights
   and the repository exists.
   
   

10. Create a SHA2-signed RSAv2 key like:

    $ ssh-keygen -t rsa-sha2-512

11. Register the SHA2-signed key to GitLab
12. Attempt the same `git clone …`operation (minus the `core.sshCommand` stuff) that had peviously failed with the SHA1-signed RSAv2 key.
    
13. This time, the clone operation succeeds
    

Getting the Most Out of EC2 Transfers With Private S3 Endpoints

Recently, I was given a project to help a customer migrate an on-premises GitLab installation into AWS. The current GitLab was pretty large: a full export of the configuration was nearly 500GiB in size.

It turned out a good chunk of that 500GiB was due to disk-hosted artifacts and LFS objects. Since I was putting it all into AWS, I opted to make use of GitLab's ability to store BLOBs in S3. Ultimately, that turned out to be nearly 8,000 LFS objects and nearly 150,000 artifacts (plus several hundred "uploads").

The first challenge was getting the on-premises data into my EC2. Customer didn't want to give me access to their on-premises network, so I needed to have them generate the export TAR-file and upload it to S3. Once in S3, I needed to get it into an EC2.

Wanting to make sure that the S3→EC2 task was as quick as possible, I selected an instance-type rated to 12.5Gbps of network bandwidth and 10Gbps of EBS bandwidth. However, my first attempt at downloading the TAR-file from S3 took nearly an hour to run: it was barely creeping along at 120MiB/s. Abysmal.

I broke out `iostat` and found that my target EBS was reporting 100% utilization and a bit less than 125MiB/s of average throughput. That seemed "off" to me, so I looked at the EBS. Was then that I noticed that the default volume-throughput was only 125MiB/s. So, I upped the setting to its maximum: 1000MiB/s. I re-ran the transfer only to find that, while the transfer-speed had improved, it had only improved to a shade under 150MiB/s. Still abysmal.

So, I started rifling through the AWS documentation to see what CLI settings I could change to improve things. First mods were:

max_concurrent_requests = 40
multipart_chunksize = 10MB
multipart_threshold = 10MB

This didn't really make much difference. `iostat` was showing really variable utilization-numbers, but mostly that my target-disk was all but idle. Similarly, `netstat` was showing only a handful of simultaneous-streams between my EC2 and S3.

Contacted AWS support. They let me know that S3 multi-part upload and download was limited to 10,0000 chunks. So, I did the math (<FILE_SIZE> / <MAX_CHUNKS>) and changed the above to:

max_concurrent_requests = 40
multipart_chunksize = 55MB
multipart_threshold = 64MB

This time, the transfers were running about 220-250MiB/s. While that was a 46% throughput increase, it was still abysmal. While `netstat` was finally showing the expected number of simultaneous connections, my `iostat` was still saying that my EBS was mostly idle.

Reached back out to AWS support. They had the further suggestion of adding:

preferred_transfer_client = crt
target_bandwidth = 10GB/s

To my S3 configuration. Re-ran my test and was getting ≈990MiB/s of continuous throughput for the transfer! This knocked the transfer speed down from fifty-five minutes to a shade over eight minutes. In other words, I was going to be able to knock nearly an hour off the upcoming migration-task.

In digging back through the documentation, it seems that, when one doesn't specify a preferred_transfer_client value, the CLI will select the `classic` (`python`) client. And, depending on your Python version, the performance ranges from merely-horrible to ungodly-bad: using RHEL 9 for my EC2, it was pretty freaking bad, but had been less-bad when using AWS for my EC2's OS. Presumably a difference in the two distro's respective Python versions?

Specifying a preferred_transfer_client value of `crt` (C run-time client) unleashed the full might and fury of my EC2's and GP3's capabilities.

Interestingly, this "use 'classic'" behavior isn't a universal auto-selection. If you've selected an EC2 with any of the instance-types:

• p4d.24xlarge
• p4de.24xlarge
• p5.48xlarge
• trn1n.32xlarge
• trn1.32xlarge

The auto-selection gets you `crt`. Not sure why `crt` isn't the auto-selected value for Nitro-based instance-types. But, "it's what it's". 

Side note: just selecting `crt` probably wouldn't have completely roided-out the transfer. I assume the further setting of `target_bandwidth` to `10GB/s` probably fully-unleashed things. There definitely wasn't much bandwidth leftover for me to actually monitor the transfer. I assume that the `target_bandwidth` parameter has a default value that's less than "all the bandwidth". However, I didn't actually bother to verify that.

Update: 

After asking support "why isn't `crt` the default for more instance-types", I got back the reply:

Thank you for your response. I see that these particular P5, P4d and Trn1 instances are purpose built for high-performance ML training¹. Hence I assume the throughput needed for this ML applications needs to high and CRT is auto enabled for these instance types.

Currently, the CRT transfer client does not support all of the functionality available in the classic transfer client.

These are few limitations for CRT configurations²:

• Region redirects - Transfers fail for requests sent to a region that does not match the region of the targeted S3 bucket.
• max_concurrent_requests, max_queue_size, multipart_threshold, and max_bandwidth configuration values - Ignores these configuration values.
• S3 to S3 copies - Falls back to using the classic transfer client


All of which is to say that, once I set `preferred_transfer_client = crt` all of my other, prior settings got ignored.

Implementing (Psuedo) Profiles in Git (Part 2!)

 As noted in my first Implementing (Psuedo) Profiles in Git post:

I'm an automation consultant for an IT contracting company. Using git is a daily part of my work-life. … Then things started shifting, a bit. Some customers wanted me to use my corporate email address as my ID. Annoying, but not an especially big deal, by itself. Then, some wanted me to use their privately-hosted repositories and wanted me to use identities issued by them.

This led me down a path of setting up multiple git "profiles" that I captured into my first article on this topic. To better support such per-project identities, it's also a good habit to use per-project authentication methods. I generally prefer to do git-over-SSH – rather than git-over-http(s) – when interfacing with remote Git repositories. Because I don't like having to keep re-entering my password, I use an SSH-agent to manage my keys. When one only has one or two projects they regularly interface with, this means a key-agent that is only needing to store a couple of authentication-keys.

Unfortunately, if you have more than one key in your SSH agent, when you attempt to connect to a remote SSH service, the agent will iteratively-present keys until the remote accepts one of them. If you've got three or more keys in your agent, the agent could present 3+ keys to the remote SSH server. By itself, this isn't a problem: the remote logs the earlier-presented keys as an authentication failure, but otherwise let you go about your business. However, if the remote SSH server is hardened, it very likely will be configured to lock your account after the third authentication-failure. As such, if you've got four or more keys in your agent and the remote requires a key that your agent doesn't present in the first three autentication-attempts, you'll find your account for that remote SSH service getting locked out.

What to do? Launch multiple ssh-agent instantiations.

Unfortunately, without modifying the default behavior, when you invoke the ssh-agent service, it will create a (semi) randomly-named UNIX domain-socket to listen for requests on. If you've only got a single ssh-agent instance running, this is a non-problem. If you've got multiple, particularly if you're using a tool like direnv, setting up your SSH_AUTH_SOCKET in your .envrc files is problematic if you don't have predictably-named socket-paths.

How to solve this conundrum? Well, I finally got tired of, every time I rebooted my dev-console, having to run `eval $( ssh-agent )` in per-project Xterms. So, I started googling and ultimately just dug through the man page for ssh-agent. In doing the latter, I found:

DESCRIPTION
     ssh-agent is a program to hold private keys used for public key authentication.  Through use of environment variables the
     agent can be located and automatically used for authentication when logging in to other machines using ssh(1).

     The options are as follows:

     -a bind_address
             Bind the agent to the UNIX-domain socket bind_address.  The default is $TMPDIR/ssh-XXXXXXXXXX/agent..

So, now I can add appropriate command-aliases to my bash profile(s) (which I've already moved to ~/.bash_profile.d/<PROJECT>)  that can be referenced based on where in my dev-console's filesystem hierachy I am and can set up my .envrcs, too. Result: if I'm in <CUSTOMER_1>/<PROJECT>/<TREE>, I get attached to an ssh-agent set up for that customer's project(s); if I'm in <CUSTOMER_2>/<PROJECT>/<TREE>, I get attached to an ssh-agent set up for that customer's project(s); etc.. For example:

$ cd ~/GIT/work/Customer_1/
direnv: loading ~/GIT/work/Customer_1/.envrc
direnv: export +AWS_CONFIG_FILE +AWS_DEFAULT_PROFILE +AWS_DEFAULT_REGION ~SSH_AUTH_SOCK
ferric@fountain:~/GIT/work/Customer_1 $ ssh-add -l
3072 SHA256:oMI+47EiStyAGIPnMfTRNliWrftKBIxMfzwYuxspy2E SH512-signed RSAv2 key for Customer 1's Project (RSA)