ssh-connection-management

name	ssh-connection-management
description	Guides on establishing persistent master SSH multiplexing sockets at ~/.ssh/sockets/<TARGET_NAME>.sock using ssh -f -N -M -S, testing connection liveness, gracefully terminating via -O exit, removing stale control sockets, and recreating sockets when remote user permissions or groups change.

SSH Connection Management

This skill guides you through establishing, managing, and troubleshooting persistent SSH socket multiplexing connections to target GCE VMs or remote VMs. Socket multiplexing speeds up command execution and maintains session resilience across automated workflow steps such as remote testing, benchmarking, or environment management.

Prerequisites & Trigger Conditions

Prerequisites

1. GCP Compute SSH Keys: Local SSH key pair present at ~/.ssh/google_compute_engine (or standard ~/.ssh/id_rsa).
2. GCP Authentication: Local gcloud authenticated with compute viewer/admin permissions.
3. Network Reachability: Internal IP or hostname reachability to target VM (e.g. nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com or <REMOTE_HOST>).
4. OpenSSH Client: Local OpenSSH client supporting ControlMaster options (-M, -S).

Trigger Conditions

• Executed prior to running remote commands, setup scripts, conformance test suites, benchmarking, or environment management on target GCE or remote VMs.
• Triggered when SSH connections fail due to stale socket files ("Control socket connect failed").
• Triggered when user permissions or group memberships on target VM are modified requiring session group ID refresh.

Input/Output Contract

Inputs

• Target Connection Details:
  • Target Name (<TARGET_NAME>, e.g., gce-c4-ssd or target VM identifier).
  • VM Name (<VM_NAME>).
  • Zone (<ZONE>).
  • GCP Project ID (<PROJECT_ID>).
  • SSH User (<SSH_USER>).
• Socket Cache Directory: ~/.ssh/sockets/.

Outputs

• Active Master Socket File: Unix domain socket located at ~/.ssh/sockets/<TARGET_NAME>.sock.
• Background SSH Process: Persistent background ssh -f -N -M process holding the master channel open.

Step-by-Step Procedure

Step 1: Create Socket Cache Directory

Ensure local socket directory exists:

mkdir -p ~/.ssh/sockets

Step 2: Clean Up Lingering Sessions and Stale Sockets

Before starting a master connection, gracefully terminate any active master SSH daemon associated with the target to avoid leaving orphaned background processes in memory, and remove any broken or stale socket file:

ssh -O exit -S ~/.ssh/sockets/<TARGET_NAME>.sock <SSH_USER>@nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com 2>/dev/null || rm -f ~/.ssh/sockets/<TARGET_NAME>.sock

Step 3: Establish Master SSH Connection

Launch the persistent master SSH connection in the background (forks immediately via -f):

ssh -f -N -M -S ~/.ssh/sockets/<TARGET_NAME>.sock -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/google_compute_engine <SSH_USER>@nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com

Key options explained:

• -f: Requests ssh to go to background just before command execution (forks into background).
• -N: Do not execute a remote command (background connection mode).
• -M: Place the SSH client into master mode for connection sharing.
• -S ~/.ssh/sockets/<TARGET_NAME>.sock: Path to the control socket.
• -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null: Prevent interactive host key prompts.

Step 4: Verify Connection Liveness

Test remote command execution over the master socket:

ssh -S ~/.ssh/sockets/<TARGET_NAME>.sock -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/google_compute_engine <SSH_USER>@nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com "echo 'Connection Alive'"

Step 5: Refreshing / Recreating Sockets

If user permissions or group memberships change on the remote VM (e.g., after usermod -aG <group>):

1. Gracefully terminate the active master SSH session and clean up the socket file:

   ssh -O exit -S ~/.ssh/sockets/<TARGET_NAME>.sock <SSH_USER>@nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com 2>/dev/null || rm -f ~/.ssh/sockets/<TARGET_NAME>.sock
   

2. Re-establish the master connection by repeating Step 3.

Failure Modes & Edge Cases

Failure Scenario	Root Cause	Remediation / Recovery Action
Control socket connect failed: Connection refused	Master SSH process died unexpectedly, leaving a dead socket file	Gracefully exit or remove stale socket file (`ssh -O exit -S ~/.ssh/sockets/<TARGET_NAME>.sock <SSH_USER>@ 2>/dev/null
Permission Denied (publickey)	SSH key ~/.ssh/google_compute_engine missing or expired GCP IAM SSH login credentials	Run gcloud compute config-default-ssh-keys or gcloud compute ssh <VM_NAME> --zone=<ZONE> to refresh SSH keys.
Permission / Group Refresh Delay	User added to a new group or permissions modified, but commands fail with permission errors	Active SSH master session retains original user and group IDs. Gracefully exit master session (`ssh -O exit -S ~/.ssh/sockets/<TARGET_NAME>.sock <SSH_USER>@ 2>/dev/null
Connection Drop / Network Disconnect	Remote VM rebooted or network path reset	Gracefully exit master socket if responsive (`ssh -O exit -S ~/.ssh/sockets/<TARGET_NAME>.sock <SSH_USER>@ 2>/dev/null

Verification Checks

1. Local Socket File Check: Confirm socket file exists and is an active socket file:

   test -S ~/.ssh/sockets/<TARGET_NAME>.sock && echo "SOCKET_EXISTS"
   

2. Remote Echo Check: Confirm commands execute over multiplexed socket:

   ssh -S ~/.ssh/sockets/<TARGET_NAME>.sock -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/google_compute_engine <SSH_USER>@nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com "echo 'Connection Alive'"
   

3. Graceful Teardown Check: Confirm graceful termination of the master socket:

   ssh -S ~/.ssh/sockets/<TARGET_NAME>.sock -O exit <SSH_USER>@nic0.<VM_NAME>.<ZONE>.c.<PROJECT_ID>.internal.gcpnode.com