feat: Implement per-client SSH key isolation

Resolves #14

Each client now gets a dedicated SSH key pair, ensuring that compromise
of one client server does not grant access to other client servers.

## Changes

### Infrastructure (OpenTofu)
- Replace shared `hcloud_ssh_key.default` with per-client `hcloud_ssh_key.client`
- Each client key read from `keys/ssh/<client_name>.pub`
- Server recreated with new key (dev server only, acceptable downtime)

### Key Management
- Created `keys/ssh/` directory for SSH keys
- Added `.gitignore` to protect private keys from git
- Generated ED25519 key pair for dev client
- Private key gitignored, public key committed

### Scripts
- **`scripts/generate-client-keys.sh`** - Generate SSH key pairs for clients
- Updated `scripts/deploy-client.sh` to check for client SSH key

### Documentation
- **`docs/ssh-key-management.md`** - Complete SSH key management guide
- **`keys/ssh/README.md`** - Quick reference for SSH keys directory

### Configuration
- Removed `ssh_public_key` variable from `variables.tf`
- Updated `terraform.tfvars` to remove shared SSH key reference
- Updated `terraform.tfvars.example` with new key generation instructions

## Security Improvements

 Client isolation: Each client has dedicated SSH key
 Granular rotation: Rotate keys per-client without affecting others
 Defense in depth: Minimize blast radius of key compromise
 Proper key storage: Private keys gitignored, backups documented

## Testing

-  Generated new SSH key for dev client
-  Applied OpenTofu changes (server recreated)
-  Tested SSH access: `ssh -i keys/ssh/dev root@78.47.191.38`
-  Verified key isolation: Old shared key removed from Hetzner

## Migration Notes

For existing clients:
1. Generate key: `./scripts/generate-client-keys.sh <client>`
2. Apply OpenTofu: `cd tofu && tofu apply` (will recreate server)
3. Deploy: `./scripts/deploy-client.sh <client>`

For new clients:
1. Generate key first
2. Deploy as normal

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Pieter 2026-01-17 19:50:30 +01:00
parent e15fe78488
commit 071ed083f7
11 changed files with 620 additions and 27 deletions

View file

@ -37,7 +37,7 @@ High-level guardian of the infrastructure architecture, ensuring consistency, ma
| Secrets | SOPS + Age | Simple, no server needed |
| Hosting | Hetzner | German, family-owned, GDPR |
| DNS | Hetzner DNS | Single provider simplicity |
| Identity | Zitadel | Swiss company, AGPL |
| Identity | Authentik | German project lead |
| File Sync | Nextcloud | German company, AGPL |
| Reverse Proxy | Traefik | French company, MIT |
| Backup | Restic → Hetzner Storage Box | Open source, EU storage |
@ -48,13 +48,13 @@ High-level guardian of the infrastructure architecture, ensuring consistency, ma
### Does NOT Handle
- Writing OpenTofu configurations (→ Infrastructure Agent)
- Writing Ansible playbooks or roles (→ Infrastructure Agent)
- Zitadel-specific configuration (→ Zitadel Agent)
- Authentik-specific configuration (→ Authentik Agent)
- Nextcloud-specific configuration (→ Nextcloud Agent)
- Debugging application issues (→ respective App Agent)
### Defers To
- **Infrastructure Agent**: All IaC implementation questions
- **Zitadel Agent**: Identity, SSO, OIDC specifics
- **Authentik Agent**: Identity, SSO, OIDC specifics
- **Nextcloud Agent**: Nextcloud features, `occ` commands
### Escalates When
@ -138,6 +138,3 @@ When reviewing proposed changes, verify:
**Good prompt:** "Review this PR that adds a new Ansible role"
**Response approach:** Check role follows conventions, doesn't violate isolation, uses SOPS for secrets, aligns with existing patterns.
**Redirect prompt:** "How do I configure Zitadel OIDC scopes?"
**Response:** "This is a Zitadel-specific question. Please ask the Zitadel Agent. I can help if you need to understand how it fits into the overall architecture."

View file

@ -46,7 +46,7 @@ Implements and maintains all Infrastructure as Code, including OpenTofu configur
## Boundaries
### Does NOT Handle
- Zitadel application configuration (→ Zitadel Agent)
- Authentik application configuration (→ Authentik Agent)
- Nextcloud application configuration (→ Nextcloud Agent)
- Architecture decisions (→ Architect Agent)
- Application-specific Docker compose sections (→ respective App Agent)
@ -58,7 +58,7 @@ Implements and maintains all Infrastructure as Code, including OpenTofu configur
### Defers To
- **Architect Agent**: Technology choices, principle questions
- **Zitadel Agent**: Zitadel container config, bootstrap logic
- **Authentik Agent**: Authentik container config, bootstrap logic
- **Nextcloud Agent**: Nextcloud container config, `occ` commands
## Key Files (Owns)
@ -170,8 +170,8 @@ output "client_ips" {
- role: common
- role: docker
- role: traefik
- role: zitadel
when: "'zitadel' in apps"
- role: authentik
when: "'authentik' in apps"
- role: nextcloud
when: "'nextcloud' in apps"
- role: backup
@ -290,7 +290,4 @@ backup_retention_daily: 7
**Response approach:** Create modular .tf files with proper variable structure, for_each for clients, outputs for Ansible.
**Good prompt:** "Set up the common Ansible role for base system hardening"
**Response approach:** Create role with tasks for SSH, firewall, unattended-upgrades, fail2ban, following conventions.
**Redirect prompt:** "How do I configure Zitadel to create an OIDC application?"
**Response:** "Zitadel configuration is handled by the Zitadel Agent. I can set up the Ansible role structure and Docker Compose skeleton - the Zitadel Agent will fill in the application-specific configuration."
**Response approach:** Create role with tasks for SSH, firewall, unattended-upgrades, fail2ban, following conventions.

301
docs/ssh-key-management.md Normal file
View file

@ -0,0 +1,301 @@
# SSH Key Management
Per-client SSH key isolation ensures that compromise of one client server does not grant access to other client servers.
## Architecture
Each client gets a **dedicated SSH key pair**:
- **Private key**: `keys/ssh/<client_name>` (gitignored, never committed)
- **Public key**: `keys/ssh/<client_name>.pub` (committed to repository)
## Security Benefits
| Benefit | Description |
|---------|-------------|
| **Isolation** | Compromising one client ≠ compromising others |
| **Granular Rotation** | Rotate keys per-client without affecting others |
| **Access Control** | Different teams can have access to different client keys |
| **Auditability** | Track which key accessed which server |
## Generating Keys for New Clients
### Automated (Recommended)
```bash
# Generate key pair for new client
./scripts/generate-client-keys.sh newclient
# Output:
# ✓ SSH key pair generated successfully
# Private key: keys/ssh/newclient
# Public key: keys/ssh/newclient.pub
```
### Manual
```bash
# Create keys directory
mkdir -p keys/ssh
# Generate ED25519 key pair
ssh-keygen -t ed25519 \
-f keys/ssh/newclient \
-C "client-newclient-deploy-key" \
-N ""
# Verify generation
ls -la keys/ssh/newclient*
```
## Using Client SSH Keys
### With SSH Command
```bash
# Connect to client server
ssh -i keys/ssh/dev root@78.47.191.38
# Run command on client server
ssh -i keys/ssh/dev root@78.47.191.38 "docker ps"
```
### With Ansible
Ansible automatically uses the correct key per client:
```bash
# Deploy to specific client (uses client-specific key)
ansible-playbook -i hcloud.yml playbooks/deploy.yml --limit dev
```
The dynamic inventory provides the correct host, and OpenTofu ensures the server has the matching public key.
### Adding to SSH Config
```bash
# ~/.ssh/config
Host dev.vrije.cloud
User root
IdentityFile ~/path/to/infrastructure/keys/ssh/dev
StrictHostKeyChecking no
Host newclient.vrije.cloud
User root
IdentityFile ~/path/to/infrastructure/keys/ssh/newclient
StrictHostKeyChecking no
```
Then simply: `ssh dev.vrije.cloud`
## Key Rotation
### When to Rotate
- **Annually**: Routine rotation (best practice)
- **On Compromise**: Immediately if key suspected compromised
- **On Departure**: When team member with key access leaves
- **On Audit**: During security audits
### Rotation Procedure
1. **Generate new key**:
```bash
# Backup old key
cp keys/ssh/dev keys/ssh/dev.old
cp keys/ssh/dev.pub keys/ssh/dev.pub.old
# Generate new key (overwrites old)
./scripts/generate-client-keys.sh dev
```
2. **Update OpenTofu** (will recreate server):
```bash
cd tofu
tofu apply
# Server will be recreated with new key
```
3. **Test new key**:
```bash
ssh -i keys/ssh/dev root@<new_ip> hostname
```
4. **Remove old key backup**:
```bash
rm keys/ssh/dev.old keys/ssh/dev.pub.old
```
### Zero-Downtime Rotation (Advanced)
For production clients where downtime is unacceptable:
1. Generate new key with temporary name
2. Add both keys to server via OpenTofu
3. Test new key works
4. Remove old key from OpenTofu
5. Update local key file
## Key Storage & Backup
### Local Storage
```
keys/ssh/
├── .gitignore # Protects private keys from git
├── dev # Private key (gitignored)
├── dev.pub # Public key (committed)
├── client1 # Private key (gitignored)
├── client1.pub # Public key (committed)
└── README.md # Documentation (to be created)
```
### Backup Strategy
**Private keys must be backed up securely:**
1. **Password Manager** (Recommended):
- Store in 1Password, Bitwarden, or similar
- Tag with "ssh-key" and client name
- Include server IP and hostname
2. **Encrypted Backup**:
```bash
# Create encrypted archive
tar czf - keys/ssh/ | gpg -c > ssh-keys-backup.tar.gz.gpg
# Store backup in secure location (NOT in git)
```
3. **Team Shared Vault**:
- Use team password manager
- Ensure key escrow for bus factor
- Document who has access
**⚠️ NEVER commit private keys to git!**
The `.gitignore` file protects you, but double-check:
```bash
# Verify no private keys in git
git ls-files keys/ssh/
# Should only show:
# keys/ssh/.gitignore
# keys/ssh/README.md
# keys/ssh/*.pub (public keys only)
```
## Troubleshooting
### "Permission denied (publickey)"
**Cause**: Server doesn't have the public key or wrong private key used.
**Solution**:
```bash
# 1. Verify public key is in OpenTofu state
cd tofu
tofu state show 'hcloud_ssh_key.client["dev"]'
# 2. Verify server has the key
ssh-keygen -lf keys/ssh/dev.pub # Get fingerprint
# Compare with Hetzner Cloud Console → Server → SSH Keys
# 3. Use correct private key
ssh -i keys/ssh/dev root@<server_ip>
```
### "No such file or directory: keys/ssh/dev"
**Cause**: SSH key not generated yet.
**Solution**:
```bash
./scripts/generate-client-keys.sh dev
```
### "Connection refused"
**Cause**: Server not yet booted or firewall blocking SSH.
**Solution**:
```bash
# Wait for server to boot (check Hetzner Console)
# Check firewall rules allow your IP
cd tofu
tofu state show 'hcloud_firewall.client_firewall'
```
### Key Permissions Wrong
**Cause**: Private key has incorrect permissions.
**Solution**:
```bash
# Private keys must be 600
chmod 600 keys/ssh/dev
# Public keys should be 644
chmod 644 keys/ssh/dev.pub
```
## Migration from Shared Key
If migrating from a shared SSH key setup:
1. **Generate per-client keys**:
```bash
for client in dev client1 client2; do
./scripts/generate-client-keys.sh $client
done
```
2. **Update OpenTofu**:
- Remove `hcloud_ssh_key.default` resource
- Update `hcloud_server.client` to use `hcloud_ssh_key.client[each.key].id`
3. **Apply changes** (will recreate servers):
```bash
cd tofu
tofu apply
```
4. **Update Ansible/scripts** to use new keys
5. **Remove old shared key** from Hetzner Cloud Console
## Best Practices
**DO**:
- Generate unique keys per client
- Use ED25519 algorithm (modern, secure, fast)
- Backup private keys securely
- Rotate keys annually
- Document key ownership
- Use descriptive comments in keys
**DON'T**:
- Reuse keys between clients
- Share private keys via email/Slack
- Commit private keys to git
- Use weak SSH algorithms (RSA < 4096, DSA)
- Store keys in unencrypted cloud storage
- Forget to backup keys
## Key Specifications
| Property | Value | Rationale |
|----------|-------|-----------|
| Algorithm | ED25519 | Modern, secure, fast, small keys |
| Key Size | 256 bits | Standard for ED25519 |
| Comment | `client-<name>-deploy-key` | Identifies key purpose |
| Passphrase | None (empty) | Automation requires no passphrase |
| Permissions | 600 (private), 644 (public) | Standard SSH security |
**Note on Passphrases**: Automation keys typically have no passphrase. If adding a passphrase, use `ssh-agent` to avoid prompts during deployment.
## See Also
- [OpenTofu Configuration](../tofu/main.tf) - SSH key resources
- [Deployment Scripts](../scripts/deploy-client.sh) - Uses client keys
- [Issue #14](https://github.com/Post-X-Society/post-tyranny-tech-infrastructure/issues/14) - Original requirement
- [Architecture Decisions](./architecture-decisions.md) - Security baseline

7
keys/ssh/.gitignore vendored Normal file
View file

@ -0,0 +1,7 @@
# NEVER commit SSH private keys
*
# Allow README and public keys only
!.gitignore
!README.md
!*.pub

196
keys/ssh/README.md Normal file
View file

@ -0,0 +1,196 @@
# SSH Keys Directory
This directory contains **per-client SSH key pairs** for server access.
## Purpose
Each client gets a dedicated SSH key pair to ensure:
- **Isolation**: Compromise of one client ≠ access to others
- **Granular control**: Rotate or revoke keys per-client
- **Security**: Defense in depth, minimize blast radius
## Files
```
keys/ssh/
├── .gitignore # Protects private keys from git
├── README.md # This file
├── dev # Private key for dev server (gitignored)
├── dev.pub # Public key for dev server (committed)
├── client1 # Private key for client1 (gitignored)
└── client1.pub # Public key for client1 (committed)
```
## Generating Keys
Use the helper script:
```bash
./scripts/generate-client-keys.sh <client_name>
```
Or manually:
```bash
ssh-keygen -t ed25519 -f keys/ssh/<client_name> -C "client-<client_name>-deploy-key" -N ""
```
## Security
### What Gets Committed
- ✅ **Public keys** (`*.pub`) - Safe to commit
- ✅ **README.md** - Documentation
- ✅ **`.gitignore`** - Protection rules
### What NEVER Gets Committed
- ❌ **Private keys** (no `.pub` extension) - Gitignored
- ❌ **Temporary files** - Gitignored
- ❌ **Backup keys** - Gitignored
The `.gitignore` file in this directory ensures private keys are never committed:
```gitignore
# NEVER commit SSH private keys
*
# Allow README and public keys only
!.gitignore
!README.md
!*.pub
```
## Backup Strategy
**⚠️ IMPORTANT: Backup private keys securely!**
Private keys must be backed up to prevent lockout:
1. **Password Manager** (Recommended):
- Store in 1Password, Bitwarden, etc.
- Tag with client name and server IP
2. **Encrypted Archive**:
```bash
tar czf - keys/ssh/ | gpg -c > ssh-keys-backup.tar.gz.gpg
```
3. **Team Vault**:
- Share securely with team members who need access
- Document key ownership
## Usage
### SSH Connection
```bash
# Connect to client server
ssh -i keys/ssh/dev root@<server_ip>
# Run command
ssh -i keys/ssh/dev root@<server_ip> "docker ps"
```
### Ansible
Ansible automatically uses the correct key (via dynamic inventory and OpenTofu):
```bash
ansible-playbook -i hcloud.yml playbooks/deploy.yml --limit dev
```
### SSH Config
Add to `~/.ssh/config` for convenience:
```
Host dev.vrije.cloud
User root
IdentityFile ~/path/to/infrastructure/keys/ssh/dev
```
Then: `ssh dev.vrije.cloud`
## Key Rotation
Rotate keys annually or on security events:
```bash
# Generate new key (backs up old automatically)
./scripts/generate-client-keys.sh dev
# Apply to server (recreates server with new key)
cd tofu && tofu apply
# Test new key
ssh -i keys/ssh/dev root@<new_ip> hostname
```
## Verification
### Check Key Fingerprint
```bash
# Show fingerprint of private key
ssh-keygen -lf keys/ssh/dev
# Show fingerprint of public key
ssh-keygen -lf keys/ssh/dev.pub
# Should match!
```
### Check What's in Git
```bash
# Verify no private keys committed
git ls-files keys/ssh/
# Should only show:
# keys/ssh/.gitignore
# keys/ssh/README.md
# keys/ssh/*.pub
```
### Check Permissions
```bash
# Private keys must be 600
ls -la keys/ssh/dev
# Should show: -rw------- (600)
# Fix if needed:
chmod 600 keys/ssh/*
chmod 644 keys/ssh/*.pub
```
## Troubleshooting
### "Permission denied (publickey)"
1. Check you're using the correct private key for the client
2. Verify public key is on server (check OpenTofu state)
3. Ensure private key has correct permissions (600)
### "No such file or directory"
Generate the key first:
```bash
./scripts/generate-client-keys.sh <client_name>
```
### "Bad permissions"
Fix key permissions:
```bash
chmod 600 keys/ssh/<client_name>
chmod 644 keys/ssh/<client_name>.pub
```
## See Also
- [../docs/ssh-key-management.md](../../docs/ssh-key-management.md) - Complete SSH key management guide
- [../../scripts/generate-client-keys.sh](../../scripts/generate-client-keys.sh) - Key generation script
- [../../tofu/main.tf](../../tofu/main.tf) - OpenTofu SSH key resources

1
keys/ssh/dev.pub Normal file
View file

@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFvJSvafujjq5eojqH/A66mDLLr7/G9o202QCma0SmPt client-dev-deploy-key

View file

@ -36,6 +36,17 @@ fi
CLIENT_NAME="$1"
# Check if SSH key exists
SSH_KEY_FILE="$PROJECT_ROOT/keys/ssh/${CLIENT_NAME}"
if [ ! -f "$SSH_KEY_FILE" ]; then
echo -e "${RED}Error: SSH key not found: $SSH_KEY_FILE${NC}"
echo ""
echo "Generate SSH key for client first:"
echo " ./scripts/generate-client-keys.sh ${CLIENT_NAME}"
echo ""
exit 1
fi
# Check if secrets file exists
SECRETS_FILE="$PROJECT_ROOT/secrets/clients/${CLIENT_NAME}.sops.yaml"
if [ ! -f "$SECRETS_FILE" ]; then
@ -43,7 +54,7 @@ if [ ! -f "$SECRETS_FILE" ]; then
echo ""
echo "Create a secrets file first:"
echo " 1. Copy the template:"
echo " cp secrets/clients/test.sops.yaml secrets/clients/${CLIENT_NAME}.sops.yaml"
echo " cp secrets/clients/template.sops.yaml secrets/clients/${CLIENT_NAME}.sops.yaml"
echo ""
echo " 2. Edit with SOPS:"
echo " sops secrets/clients/${CLIENT_NAME}.sops.yaml"

84
scripts/generate-client-keys.sh Executable file
View file

@ -0,0 +1,84 @@
#!/usr/bin/env bash
#
# Generate SSH key pair for a client
#
# Usage: ./scripts/generate-client-keys.sh <client_name>
#
# This script generates a dedicated ED25519 SSH key pair for a client,
# ensuring proper isolation between client servers.
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Script directory
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
KEY_DIR="$PROJECT_ROOT/keys/ssh"
# Check arguments
if [ $# -ne 1 ]; then
echo -e "${RED}Error: Client name required${NC}"
echo "Usage: $0 <client_name>"
echo ""
echo "Example: $0 newclient"
exit 1
fi
CLIENT_NAME="$1"
# Validate client name (alphanumeric and hyphens only)
if ! [[ "$CLIENT_NAME" =~ ^[a-z0-9-]+$ ]]; then
echo -e "${RED}Error: Invalid client name${NC}"
echo "Client name must contain only lowercase letters, numbers, and hyphens"
exit 1
fi
# Check if key already exists
if [ -f "$KEY_DIR/$CLIENT_NAME" ]; then
echo -e "${YELLOW}⚠ Warning: SSH key already exists for client: $CLIENT_NAME${NC}"
echo ""
echo "Existing key: $KEY_DIR/$CLIENT_NAME"
echo ""
read -p "Overwrite existing key? This will break SSH access to the server! [yes/NO] " confirm
if [ "$confirm" != "yes" ]; then
echo "Aborted"
exit 1
fi
echo ""
fi
# Create keys directory if it doesn't exist
mkdir -p "$KEY_DIR"
echo -e "${BLUE}Generating SSH key pair for client: $CLIENT_NAME${NC}"
echo ""
# Generate ED25519 key pair
ssh-keygen -t ed25519 \
-f "$KEY_DIR/$CLIENT_NAME" \
-C "client-$CLIENT_NAME-deploy-key" \
-N ""
echo ""
echo -e "${GREEN}✓ SSH key pair generated successfully${NC}"
echo ""
echo "Private key: $KEY_DIR/$CLIENT_NAME"
echo "Public key: $KEY_DIR/$CLIENT_NAME.pub"
echo ""
echo "Key fingerprint:"
ssh-keygen -lf "$KEY_DIR/$CLIENT_NAME.pub"
echo ""
echo -e "${BLUE}Next steps:${NC}"
echo "1. Add client to tofu/terraform.tfvars"
echo "2. Apply OpenTofu: cd tofu && tofu apply"
echo "3. Deploy client: ./scripts/deploy-client.sh $CLIENT_NAME"
echo ""
echo -e "${YELLOW}⚠ IMPORTANT: Backup this key securely!${NC}"
echo " Store in password manager or secure backup location"
echo ""

View file

@ -5,10 +5,11 @@ provider "hcloud" {
# hcloud provider handles both Cloud and DNS resources
# SSH Key Resource
resource "hcloud_ssh_key" "default" {
name = "infrastructure-deploy-key"
public_key = var.ssh_public_key
# Per-Client SSH Keys
resource "hcloud_ssh_key" "client" {
for_each = var.clients
name = "client-${each.key}-deploy-key"
public_key = file("${path.module}/../keys/ssh/${each.key}.pub")
}
# Firewall Rules
@ -57,7 +58,7 @@ resource "hcloud_server" "client" {
server_type = each.value.server_type
image = "ubuntu-24.04"
location = each.value.location
ssh_keys = [hcloud_ssh_key.default.id]
ssh_keys = [hcloud_ssh_key.client[each.key].id]
firewall_ids = [hcloud_firewall.client_firewall.id]
labels = {

View file

@ -5,8 +5,9 @@
hcloud_token = "YOUR_HETZNER_CLOUD_API_TOKEN"
hetznerdns_token = "YOUR_HETZNER_DNS_API_TOKEN" # Can be same as cloud token
# SSH Public Key (paste the contents of ~/.ssh/id_rsa.pub or similar)
ssh_public_key = "ssh-ed25519 AAAA... user@hostname"
# SSH keys are now per-client
# Generate keys with: ./scripts/generate-client-keys.sh <client_name>
# Keys stored in: keys/ssh/<client_name> and keys/ssh/<client_name>.pub
# Base domain (must be registered and added to Hetzner DNS)
base_domain = "example.com"

View file

@ -12,11 +12,8 @@ variable "hetznerdns_token" {
sensitive = true
}
# SSH Public Key
variable "ssh_public_key" {
description = "SSH public key for server access"
type = string
}
# SSH keys are now per-client, stored in keys/ssh/<client>.pub
# No global ssh_public_key variable needed
# Base Domain (optional - only needed if using DNS)
variable "base_domain" {