Post-Tyranny-Tech-Infrastru.../keys/ssh
Pieter f795920f24 🚀 GREEN CLIENT DEPLOYMENT + CRITICAL SECURITY FIXES
═══════════════════════════════════════════════════════════════
 COMPLETED: Green Client Deployment (green.vrije.cloud)
═══════════════════════════════════════════════════════════════

Services deployed and operational:
- Traefik (reverse proxy with SSL)
- Authentik SSO (auth.green.vrije.cloud)
- Nextcloud (nextcloud.green.vrije.cloud)
- Collabora Office (online document editing)
- PostgreSQL databases (Authentik + Nextcloud)
- Redis (caching + file locking)

═══════════════════════════════════════════════════════════════
🔐 CRITICAL SECURITY FIX: Unique Passwords Per Client
═══════════════════════════════════════════════════════════════

PROBLEM FIXED:
All clients were using IDENTICAL passwords from template (critical vulnerability).
If one server compromised, all servers compromised.

SOLUTION IMPLEMENTED:
 Auto-generate unique passwords per client
 Store securely in SOPS-encrypted files
 Easy retrieval with get-passwords.sh script

NEW SCRIPTS:
- scripts/generate-passwords.sh - Auto-generate unique 43-char passwords
- scripts/get-passwords.sh      - Retrieve client credentials from SOPS

UPDATED SCRIPTS:
- scripts/deploy-client.sh - Now auto-calls password generator

PASSWORD CHANGES:
- dev.sops.yaml   - Regenerated with unique passwords
- green.sops.yaml - Created with unique passwords

SECURITY PROPERTIES:
- 43-character passwords (258 bits entropy)
- Cryptographically secure (openssl rand -base64 32)
- Unique across all clients
- Stored encrypted with SOPS + age

═══════════════════════════════════════════════════════════════
🛠️  BUG FIX: Nextcloud Volume Mounting
═══════════════════════════════════════════════════════════════

PROBLEM FIXED:
Volume detection was looking for "nextcloud-data-{client}" in device ID,
but Hetzner volumes use numeric IDs (scsi-0HC_Volume_104429514).

SOLUTION:
Simplified detection to find first Hetzner volume (works for all clients):
  ls -1 /dev/disk/by-id/scsi-0HC_Volume_* | head -1

FIXED FILE:
- ansible/roles/nextcloud/tasks/mount-volume.yml:15

═══════════════════════════════════════════════════════════════
🐛 BUG FIX: Authentik Invitation Task Safety
═══════════════════════════════════════════════════════════════

PROBLEM FIXED:
invitation.yml task crashed when accessing undefined variable attribute
(enrollment_blueprint_result.rc when API not ready).

SOLUTION:
Added safety checks before accessing variable attributes:
  {{ 'In Progress' if (var is defined and var.rc is defined) else 'Complete' }}

FIXED FILE:
- ansible/roles/authentik/tasks/invitation.yml:91

═══════════════════════════════════════════════════════════════
📝 OTHER CHANGES
═══════════════════════════════════════════════════════════════

GITIGNORE:
- Added *.md (except README.md) to exclude deployment reports

GREEN CLIENT FILES:
- keys/ssh/green.pub - SSH public key for green server
- secrets/clients/green.sops.yaml - Encrypted secrets with unique passwords

═══════════════════════════════════════════════════════════════
 IMPACT: All Future Deployments Now Secure & Reliable
═══════════════════════════════════════════════════════════════

FUTURE DEPLOYMENTS:
-  Automatically get unique passwords
-  Volume mounting works reliably
-  Ansible tasks handle API delays gracefully
-  No manual intervention required

DEPLOYMENT TIME: ~15 minutes (fully automated)
AUTOMATION RATE: 95%

═══════════════════════════════════════════════════════════════

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-18 17:06:04 +01:00
..
.gitignore feat: Implement per-client SSH key isolation 2026-01-17 19:50:30 +01:00
blue.pub feat: Automate OpenTofu terraform.tfvars management 2026-01-17 21:34:05 +01:00
dev.pub feat: Implement per-client SSH key isolation 2026-01-17 19:50:30 +01:00
green.pub 🚀 GREEN CLIENT DEPLOYMENT + CRITICAL SECURITY FIXES 2026-01-18 17:06:04 +01:00
README.md feat: Implement per-client SSH key isolation 2026-01-17 19:50:30 +01:00

SSH Keys Directory

This directory contains per-client SSH key pairs for server access.

Purpose

Each client gets a dedicated SSH key pair to ensure:

  • Isolation: Compromise of one client ≠ access to others
  • Granular control: Rotate or revoke keys per-client
  • Security: Defense in depth, minimize blast radius

Files

keys/ssh/
├── .gitignore          # Protects private keys from git
├── README.md           # This file
├── dev                 # Private key for dev server (gitignored)
├── dev.pub             # Public key for dev server (committed)
├── client1             # Private key for client1 (gitignored)
└── client1.pub         # Public key for client1 (committed)

Generating Keys

Use the helper script:

./scripts/generate-client-keys.sh <client_name>

Or manually:

ssh-keygen -t ed25519 -f keys/ssh/<client_name> -C "client-<client_name>-deploy-key" -N ""

Security

What Gets Committed

  • Public keys (*.pub) - Safe to commit
  • README.md - Documentation
  • .gitignore - Protection rules

What NEVER Gets Committed

  • Private keys (no .pub extension) - Gitignored
  • Temporary files - Gitignored
  • Backup keys - Gitignored

The .gitignore file in this directory ensures private keys are never committed:

# NEVER commit SSH private keys
*

# Allow README and public keys only
!.gitignore
!README.md
!*.pub

Backup Strategy

⚠️ IMPORTANT: Backup private keys securely!

Private keys must be backed up to prevent lockout:

  1. Password Manager (Recommended):

    • Store in 1Password, Bitwarden, etc.
    • Tag with client name and server IP
  2. Encrypted Archive:

    tar czf - keys/ssh/ | gpg -c > ssh-keys-backup.tar.gz.gpg
    
  3. Team Vault:

    • Share securely with team members who need access
    • Document key ownership

Usage

SSH Connection

# Connect to client server
ssh -i keys/ssh/dev root@<server_ip>

# Run command
ssh -i keys/ssh/dev root@<server_ip> "docker ps"

Ansible

Ansible automatically uses the correct key (via dynamic inventory and OpenTofu):

ansible-playbook -i hcloud.yml playbooks/deploy.yml --limit dev

SSH Config

Add to ~/.ssh/config for convenience:

Host dev.vrije.cloud
    User root
    IdentityFile ~/path/to/infrastructure/keys/ssh/dev

Then: ssh dev.vrije.cloud

Key Rotation

Rotate keys annually or on security events:

# Generate new key (backs up old automatically)
./scripts/generate-client-keys.sh dev

# Apply to server (recreates server with new key)
cd tofu && tofu apply

# Test new key
ssh -i keys/ssh/dev root@<new_ip> hostname

Verification

Check Key Fingerprint

# Show fingerprint of private key
ssh-keygen -lf keys/ssh/dev

# Show fingerprint of public key
ssh-keygen -lf keys/ssh/dev.pub

# Should match!

Check What's in Git

# Verify no private keys committed
git ls-files keys/ssh/

# Should only show:
# keys/ssh/.gitignore
# keys/ssh/README.md
# keys/ssh/*.pub

Check Permissions

# Private keys must be 600
ls -la keys/ssh/dev

# Should show: -rw------- (600)

# Fix if needed:
chmod 600 keys/ssh/*
chmod 644 keys/ssh/*.pub

Troubleshooting

"Permission denied (publickey)"

  1. Check you're using the correct private key for the client
  2. Verify public key is on server (check OpenTofu state)
  3. Ensure private key has correct permissions (600)

"No such file or directory"

Generate the key first:

./scripts/generate-client-keys.sh <client_name>

"Bad permissions"

Fix key permissions:

chmod 600 keys/ssh/<client_name>
chmod 644 keys/ssh/<client_name>.pub

See Also