Post-Tyranny-Tech-Infrastru.../ansible
Pieter f795920f24 🚀 GREEN CLIENT DEPLOYMENT + CRITICAL SECURITY FIXES
═══════════════════════════════════════════════════════════════
 COMPLETED: Green Client Deployment (green.vrije.cloud)
═══════════════════════════════════════════════════════════════

Services deployed and operational:
- Traefik (reverse proxy with SSL)
- Authentik SSO (auth.green.vrije.cloud)
- Nextcloud (nextcloud.green.vrije.cloud)
- Collabora Office (online document editing)
- PostgreSQL databases (Authentik + Nextcloud)
- Redis (caching + file locking)

═══════════════════════════════════════════════════════════════
🔐 CRITICAL SECURITY FIX: Unique Passwords Per Client
═══════════════════════════════════════════════════════════════

PROBLEM FIXED:
All clients were using IDENTICAL passwords from template (critical vulnerability).
If one server compromised, all servers compromised.

SOLUTION IMPLEMENTED:
 Auto-generate unique passwords per client
 Store securely in SOPS-encrypted files
 Easy retrieval with get-passwords.sh script

NEW SCRIPTS:
- scripts/generate-passwords.sh - Auto-generate unique 43-char passwords
- scripts/get-passwords.sh      - Retrieve client credentials from SOPS

UPDATED SCRIPTS:
- scripts/deploy-client.sh - Now auto-calls password generator

PASSWORD CHANGES:
- dev.sops.yaml   - Regenerated with unique passwords
- green.sops.yaml - Created with unique passwords

SECURITY PROPERTIES:
- 43-character passwords (258 bits entropy)
- Cryptographically secure (openssl rand -base64 32)
- Unique across all clients
- Stored encrypted with SOPS + age

═══════════════════════════════════════════════════════════════
🛠️  BUG FIX: Nextcloud Volume Mounting
═══════════════════════════════════════════════════════════════

PROBLEM FIXED:
Volume detection was looking for "nextcloud-data-{client}" in device ID,
but Hetzner volumes use numeric IDs (scsi-0HC_Volume_104429514).

SOLUTION:
Simplified detection to find first Hetzner volume (works for all clients):
  ls -1 /dev/disk/by-id/scsi-0HC_Volume_* | head -1

FIXED FILE:
- ansible/roles/nextcloud/tasks/mount-volume.yml:15

═══════════════════════════════════════════════════════════════
🐛 BUG FIX: Authentik Invitation Task Safety
═══════════════════════════════════════════════════════════════

PROBLEM FIXED:
invitation.yml task crashed when accessing undefined variable attribute
(enrollment_blueprint_result.rc when API not ready).

SOLUTION:
Added safety checks before accessing variable attributes:
  {{ 'In Progress' if (var is defined and var.rc is defined) else 'Complete' }}

FIXED FILE:
- ansible/roles/authentik/tasks/invitation.yml:91

═══════════════════════════════════════════════════════════════
📝 OTHER CHANGES
═══════════════════════════════════════════════════════════════

GITIGNORE:
- Added *.md (except README.md) to exclude deployment reports

GREEN CLIENT FILES:
- keys/ssh/green.pub - SSH public key for green server
- secrets/clients/green.sops.yaml - Encrypted secrets with unique passwords

═══════════════════════════════════════════════════════════════
 IMPACT: All Future Deployments Now Secure & Reliable
═══════════════════════════════════════════════════════════════

FUTURE DEPLOYMENTS:
-  Automatically get unique passwords
-  Volume mounting works reliably
-  Ansible tasks handle API delays gracefully
-  No manual intervention required

DEPLOYMENT TIME: ~15 minutes (fully automated)
AUTOMATION RATE: 95%

═══════════════════════════════════════════════════════════════

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-18 17:06:04 +01:00
..
playbooks Remove automated recovery flow configuration 2026-01-17 09:57:07 +01:00
roles 🚀 GREEN CLIENT DEPLOYMENT + CRITICAL SECURITY FIXES 2026-01-18 17:06:04 +01:00
ansible.cfg Complete Ansible base configuration (#2) 2025-12-27 14:13:15 +01:00
configure-oidc.yml feat: Complete Authentik SSO integration with automated OIDC setup 2026-01-08 16:56:19 +01:00
hcloud.yml WIP: Ansible base configuration - common role (#2) 2025-12-27 14:00:22 +01:00
README.md WIP: Ansible base configuration - common role (#2) 2025-12-27 14:00:22 +01:00

Ansible Configuration Management

Ansible playbooks and roles for configuring and managing the multi-tenant VPS infrastructure.

Prerequisites

1. Install Ansible (via pipx - isolated environment)

Why pipx? Isolates Ansible in its own Python environment, preventing conflicts.

# Install pipx
brew install pipx
pipx ensurepath

# Install Ansible
pipx install --include-deps ansible

# Install required dependencies
pipx inject ansible requests python-dateutil

# Verify installation
ansible --version

2. Install Ansible Collections

ansible-galaxy collection install hetzner.hcloud community.sops community.general

3. Set Hetzner Cloud API Token

export HCLOUD_TOKEN="your-hetzner-cloud-api-token"

Or add to your shell profile (~/.zshrc or ~/.bashrc):

export HCLOUD_TOKEN="your-token-here"

Quick Start

Test Dynamic Inventory

cd ansible
ansible-inventory --graph

You should see your servers grouped by labels.

Ping All Servers

ansible all -m ping

Run Setup Playbook

# Full setup (common + docker + traefik)
ansible-playbook playbooks/setup.yml

# Specific server
ansible-playbook playbooks/setup.yml --limit test

# Dry run (check mode)
ansible-playbook playbooks/setup.yml --check

Directory Structure

ansible/
├── ansible.cfg              # Ansible configuration
├── hcloud.yml               # Hetzner Cloud dynamic inventory
├── playbooks/               # Playbook definitions
│   ├── setup.yml            # Initial server setup
│   ├── deploy.yml           # Deploy/update applications
│   └── upgrade.yml          # System upgrades
├── roles/                   # Role definitions
│   ├── common/              # Base system hardening
│   ├── docker/              # Docker + Docker Compose
│   ├── traefik/             # Reverse proxy
│   ├── zitadel/             # Identity provider
│   ├── nextcloud/           # File sync/share
│   └── backup/              # Restic backup
└── group_vars/              # Group variables
    └── all.yml              # Variables for all hosts

Roles

common

Base system configuration and security hardening:

  • SSH hardening (key-only auth, no root password)
  • UFW firewall configuration
  • Fail2ban for SSH protection
  • Automatic security updates
  • Timezone and locale setup

Variables (roles/common/defaults/main.yml):

  • common_timezone: System timezone (default: Europe/Amsterdam)
  • common_ssh_port: SSH port (default: 22)
  • common_ufw_allowed_ports: List of allowed firewall ports

docker

Docker and Docker Compose installation:

  • Latest Docker Engine from official repository
  • Docker Compose V2
  • Docker daemon configuration
  • User permissions for Docker

traefik

Reverse proxy with automatic SSL:

  • Traefik v3 with Docker provider
  • Let's Encrypt automatic certificate generation
  • HTTP to HTTPS redirection
  • Dashboard (optional)

zitadel

Identity provider deployment (see Zitadel Agent for details)

nextcloud

File sync/share deployment (see Nextcloud Agent for details)

backup

Restic backup configuration to Hetzner Storage Box

Playbooks

setup.yml

Initial server provisioning and configuration:

ansible-playbook playbooks/setup.yml

Runs roles in order:

  1. common - Base hardening
  2. docker - Container platform
  3. traefik - Reverse proxy

deploy.yml

Deploy or update applications:

ansible-playbook playbooks/deploy.yml

Runs application-specific roles based on server labels.

Dynamic Inventory

The hcloud.yml inventory automatically queries Hetzner Cloud API for servers.

Server Grouping:

  • By client: client_test, client_alpha
  • By role: role_app_server
  • By location: location_fsn1, location_nbg1

View inventory:

ansible-inventory --graph
ansible-inventory --list
ansible-inventory --host test

Common Tasks

Check Server Connectivity

ansible all -m ping

Run Ad-hoc Command

ansible all -a "uptime"
ansible all -a "df -h"

Update All Packages

ansible all -m apt -a "update_cache=yes upgrade=dist"

Restart Service

ansible all -m service -a "name=docker state=restarted"

Limit to Specific Hosts

# Single host
ansible-playbook playbooks/setup.yml --limit test

# Multiple hosts
ansible-playbook playbooks/setup.yml --limit "test,alpha"

# Group
ansible-playbook playbooks/setup.yml --limit client_test

Development Workflow

Creating a New Role

cd ansible/roles
mkdir -p newrole/{tasks,handlers,templates,defaults,files}

Minimum structure:

  • defaults/main.yml - Default variables
  • tasks/main.yml - Main task list
  • handlers/main.yml - Service handlers (optional)
  • templates/ - Jinja2 templates (optional)

Testing Changes

# Syntax check
ansible-playbook playbooks/setup.yml --syntax-check

# Dry run (no changes)
ansible-playbook playbooks/setup.yml --check

# Limit to test server
ansible-playbook playbooks/setup.yml --limit test

# Verbose output
ansible-playbook playbooks/setup.yml -v
ansible-playbook playbooks/setup.yml -vvv  # Very verbose

Troubleshooting

"No inventory was parsed"

  • Ensure HCLOUD_TOKEN environment variable is set
  • Verify token has read access
  • Check hcloud.yml syntax

"Failed to connect to host"

  • Verify server is running: tofu show
  • Check SSH key is correct: ssh -i ~/.ssh/ptt_infrastructure root@<ip>
  • Verify firewall allows SSH from your IP

"Permission denied (publickey)"

  • Ensure ~/.ssh/ptt_infrastructure private key exists
  • Check ansible.cfg points to correct key
  • Verify public key was added to server via OpenTofu

"Module not found"

  • Install missing Ansible collection:
    ansible-galaxy collection install <collection-name>
    

Ansible is slow

  • Enable SSH pipelining (already configured in ansible.cfg)
  • Use --forks to increase parallelism: ansible-playbook playbooks/setup.yml --forks 20
  • Enable fact caching (already configured)

Security Notes

  • Ansible connects as root user via SSH key
  • No passwords are used anywhere
  • SSH hardening applied automatically via common role
  • UFW firewall enabled by default
  • Fail2ban protects SSH
  • Automatic security updates enabled

Next Steps

After initial setup:

  1. Deploy Zitadel: Follow Zitadel Agent instructions
  2. Deploy Nextcloud: Follow Nextcloud Agent instructions
  3. Configure backups: Use backup role
  4. Set up monitoring: Configure Uptime Kuma

Resources