WIP: Ansible base configuration - common role (#2)

Progress on Issue #2: Ansible Base Configuration

Completed:
- ✅ Ansible installed via pipx (isolated Python environment)
- ✅ Hetzner Cloud dynamic inventory configured
- ✅ Ansible configuration (ansible.cfg)
- ✅ Common role for base system hardening:
  - SSH hardening (key-only, no root password)
  - UFW firewall configuration
  - Fail2ban for SSH protection
  - Automatic security updates
  - Timezone and system packages
- ✅ Comprehensive Ansible README with setup guide

Architecture Updates:
- Added Decision #15: pipx for isolated Python environments
- Updated ADR changelog with pipx adoption

Still TODO for #2:
- Docker role
- Traefik role
- Setup playbook
- Deploy playbook
- Testing against live server

Files added:
- ansible/README.md - Complete Ansible guide
- ansible/ansible.cfg - Ansible configuration
- ansible/hcloud.yml - Hetzner dynamic inventory
- ansible/roles/common/* - Base hardening role

Partial progress on #2

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

ansible/README.md

@@ -0,0 +1,285 @@
+# Ansible Configuration Management
+
+Ansible playbooks and roles for configuring and managing the multi-tenant VPS infrastructure.
+
+## Prerequisites
+
+### 1. Install Ansible (via pipx - isolated environment)
+
+**Why pipx?** Isolates Ansible in its own Python environment, preventing conflicts.
+
+```bash
+# Install pipx
+brew install pipx
+pipx ensurepath
+
+# Install Ansible
+pipx install --include-deps ansible
+
+# Install required dependencies
+pipx inject ansible requests python-dateutil
+
+# Verify installation
+ansible --version
+```
+
+### 2. Install Ansible Collections
+
+```bash
+ansible-galaxy collection install hetzner.hcloud community.sops community.general
+```
+
+### 3. Set Hetzner Cloud API Token
+
+```bash
+export HCLOUD_TOKEN="your-hetzner-cloud-api-token"
+```
+
+Or add to your shell profile (`~/.zshrc` or `~/.bashrc`):
+```bash
+export HCLOUD_TOKEN="your-token-here"
+```
+
+## Quick Start
+
+### Test Dynamic Inventory
+
+```bash
+cd ansible
+ansible-inventory --graph
+```
+
+You should see your servers grouped by labels.
+
+### Ping All Servers
+
+```bash
+ansible all -m ping
+```
+
+### Run Setup Playbook
+
+```bash
+# Full setup (common + docker + traefik)
+ansible-playbook playbooks/setup.yml
+
+# Specific server
+ansible-playbook playbooks/setup.yml --limit test
+
+# Dry run (check mode)
+ansible-playbook playbooks/setup.yml --check
+```
+
+## Directory Structure
+
+```
+ansible/
+├── ansible.cfg              # Ansible configuration
+├── hcloud.yml               # Hetzner Cloud dynamic inventory
+├── playbooks/               # Playbook definitions
+│   ├── setup.yml            # Initial server setup
+│   ├── deploy.yml           # Deploy/update applications
+│   └── upgrade.yml          # System upgrades
+├── roles/                   # Role definitions
+│   ├── common/              # Base system hardening
+│   ├── docker/              # Docker + Docker Compose
+│   ├── traefik/             # Reverse proxy
+│   ├── zitadel/             # Identity provider
+│   ├── nextcloud/           # File sync/share
+│   └── backup/              # Restic backup
+└── group_vars/              # Group variables
+    └── all.yml              # Variables for all hosts
+```
+
+## Roles
+
+### common
+Base system configuration and security hardening:
+- SSH hardening (key-only auth, no root password)
+- UFW firewall configuration
+- Fail2ban for SSH protection
+- Automatic security updates
+- Timezone and locale setup
+
+**Variables** (`roles/common/defaults/main.yml`):
+- `common_timezone`: System timezone (default: `Europe/Amsterdam`)
+- `common_ssh_port`: SSH port (default: `22`)
+- `common_ufw_allowed_ports`: List of allowed firewall ports
+
+### docker
+Docker and Docker Compose installation:
+- Latest Docker Engine from official repository
+- Docker Compose V2
+- Docker daemon configuration
+- User permissions for Docker
+
+### traefik
+Reverse proxy with automatic SSL:
+- Traefik v3 with Docker provider
+- Let's Encrypt automatic certificate generation
+- HTTP to HTTPS redirection
+- Dashboard (optional)
+
+### zitadel
+Identity provider deployment (see Zitadel Agent for details)
+
+### nextcloud
+File sync/share deployment (see Nextcloud Agent for details)
+
+### backup
+Restic backup configuration to Hetzner Storage Box
+
+## Playbooks
+
+### setup.yml
+Initial server provisioning and configuration:
+```bash
+ansible-playbook playbooks/setup.yml
+```
+
+Runs roles in order:
+1. `common` - Base hardening
+2. `docker` - Container platform
+3. `traefik` - Reverse proxy
+
+### deploy.yml
+Deploy or update applications:
+```bash
+ansible-playbook playbooks/deploy.yml
+```
+
+Runs application-specific roles based on server labels.
+
+## Dynamic Inventory
+
+The `hcloud.yml` inventory automatically queries Hetzner Cloud API for servers.
+
+**Server Grouping:**
+- By client: `client_test`, `client_alpha`
+- By role: `role_app_server`
+- By location: `location_fsn1`, `location_nbg1`
+
+**View inventory:**
+```bash
+ansible-inventory --graph
+ansible-inventory --list
+ansible-inventory --host test
+```
+
+## Common Tasks
+
+### Check Server Connectivity
+```bash
+ansible all -m ping
+```
+
+### Run Ad-hoc Command
+```bash
+ansible all -a "uptime"
+ansible all -a "df -h"
+```
+
+### Update All Packages
+```bash
+ansible all -m apt -a "update_cache=yes upgrade=dist"
+```
+
+### Restart Service
+```bash
+ansible all -m service -a "name=docker state=restarted"
+```
+
+### Limit to Specific Hosts
+```bash
+# Single host
+ansible-playbook playbooks/setup.yml --limit test
+
+# Multiple hosts
+ansible-playbook playbooks/setup.yml --limit "test,alpha"
+
+# Group
+ansible-playbook playbooks/setup.yml --limit client_test
+```
+
+## Development Workflow
+
+### Creating a New Role
+
+```bash
+cd ansible/roles
+mkdir -p newrole/{tasks,handlers,templates,defaults,files}
+```
+
+Minimum structure:
+- `defaults/main.yml` - Default variables
+- `tasks/main.yml` - Main task list
+- `handlers/main.yml` - Service handlers (optional)
+- `templates/` - Jinja2 templates (optional)
+
+### Testing Changes
+
+```bash
+# Syntax check
+ansible-playbook playbooks/setup.yml --syntax-check
+
+# Dry run (no changes)
+ansible-playbook playbooks/setup.yml --check
+
+# Limit to test server
+ansible-playbook playbooks/setup.yml --limit test
+
+# Verbose output
+ansible-playbook playbooks/setup.yml -v
+ansible-playbook playbooks/setup.yml -vvv  # Very verbose
+```
+
+## Troubleshooting
+
+### "No inventory was parsed"
+- Ensure `HCLOUD_TOKEN` environment variable is set
+- Verify token has read access
+- Check `hcloud.yml` syntax
+
+### "Failed to connect to host"
+- Verify server is running: `tofu show`
+- Check SSH key is correct: `ssh -i ~/.ssh/ptt_infrastructure root@<ip>`
+- Verify firewall allows SSH from your IP
+
+### "Permission denied (publickey)"
+- Ensure `~/.ssh/ptt_infrastructure` private key exists
+- Check `ansible.cfg` points to correct key
+- Verify public key was added to server via OpenTofu
+
+### "Module not found"
+- Install missing Ansible collection:
+  ```bash
+  ansible-galaxy collection install <collection-name>
+  ```
+
+### Ansible is slow
+- Enable SSH pipelining (already configured in `ansible.cfg`)
+- Use `--forks` to increase parallelism: `ansible-playbook playbooks/setup.yml --forks 20`
+- Enable fact caching (already configured)
+
+## Security Notes
+
+- Ansible connects as `root` user via SSH key
+- No passwords are used anywhere
+- SSH hardening applied automatically via `common` role
+- UFW firewall enabled by default
+- Fail2ban protects SSH
+- Automatic security updates enabled
+
+## Next Steps
+
+After initial setup:
+1. Deploy Zitadel: Follow Zitadel Agent instructions
+2. Deploy Nextcloud: Follow Nextcloud Agent instructions
+3. Configure backups: Use `backup` role
+4. Set up monitoring: Configure Uptime Kuma
+
+## Resources
+
+- [Ansible Documentation](https://docs.ansible.com/)
+- [Hetzner Cloud Ansible Collection](https://github.com/ansible-collections/hetzner.hcloud)
+- [Ansible Best Practices](https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html)

ansible/ansible.cfg

@@ -0,0 +1,39 @@
+[defaults]
+# Inventory configuration
+inventory = hcloud.yml
+host_key_checking = False
+interpreter_python = auto_silent
+
+# Performance
+forks = 10
+gathering = smart
+fact_caching = jsonfile
+fact_caching_connection = /tmp/ansible_facts
+fact_caching_timeout = 86400
+
+# Output
+stdout_callback = yaml
+bin_ansible_callbacks = True
+display_skipped_hosts = False
+
+# SSH
+remote_user = root
+private_key_file = ~/.ssh/ptt_infrastructure
+timeout = 30
+
+# Roles
+roles_path = ./roles
+
+[inventory]
+# Enable Hetzner Cloud dynamic inventory plugin
+enable_plugins = hetzner.hcloud.hcloud
+
+[privilege_escalation]
+become = True
+become_method = sudo
+become_user = root
+become_ask_pass = False
+
+[ssh_connection]
+pipelining = True
+ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

ansible/hcloud.yml

@@ -0,0 +1,37 @@
+# Hetzner Cloud Dynamic Inventory Configuration
+# Queries Hetzner Cloud API for server list at runtime
+# Requires: HCLOUD_TOKEN environment variable
+
+plugin: hetzner.hcloud.hcloud
+
+# Optional: Filter by label
+# label_selector: role=app-server
+
+# Group servers by labels
+keyed_groups:
+  # Group by client label
+  - key: labels.client
+    prefix: client
+    separator: _
+
+  # Group by role label
+  - key: labels.role
+    prefix: role
+    separator: _
+
+  # Group by location
+  - key: location
+    prefix: location
+    separator: _
+
+# Compose custom variables
+compose:
+  ansible_host: ipv4_address
+  server_id: id
+  server_type: server_type
+  datacenter: datacenter
+
+# Create groups for all servers
+groups:
+  # All Hetzner Cloud servers
+  hetzner_cloud: True

ansible/roles/common/defaults/main.yml

@@ -0,0 +1,42 @@
+---
+# Default variables for common role
+
+# Timezone
+common_timezone: "Europe/Amsterdam"
+
+# SSH Configuration
+common_ssh_port: 22
+common_ssh_permit_root_login: "prohibit-password"
+common_ssh_password_authentication: "no"
+common_ssh_pubkey_authentication: "yes"
+
+# UFW Firewall
+common_ufw_default_incoming: "deny"
+common_ufw_default_outgoing: "allow"
+common_ufw_allowed_ports:
+  - { port: "22", proto: "tcp", comment: "SSH" }
+  - { port: "80", proto: "tcp", comment: "HTTP" }
+  - { port: "443", proto: "tcp", comment: "HTTPS" }
+
+# Automatic Updates
+common_unattended_upgrades: true
+common_auto_reboot: false  # Manual control over reboots
+
+# Fail2ban
+common_fail2ban_enabled: true
+common_fail2ban_bantime: 3600  # 1 hour
+common_fail2ban_findtime: 600  # 10 minutes
+common_fail2ban_maxretry: 5
+
+# System packages
+common_packages:
+  - curl
+  - wget
+  - git
+  - vim
+  - htop
+  - net-tools
+  - ufw
+  - fail2ban
+  - unattended-upgrades
+  - apt-listchanges

ansible/roles/common/handlers/main.yml

@@ -0,0 +1,17 @@
+---
+# Handlers for common role
+
+- name: Restart SSH
+  service:
+    name: ssh
+    state: restarted
+
+- name: Restart unattended-upgrades
+  service:
+    name: unattended-upgrades
+    state: restarted
+
+- name: Restart fail2ban
+  service:
+    name: fail2ban
+    state: restarted

ansible/roles/common/tasks/fail2ban.yml

@@ -0,0 +1,20 @@
+---
+# Fail2ban configuration for SSH protection
+
+- name: Install fail2ban
+  apt:
+    name: fail2ban
+    state: present
+
+- name: Configure fail2ban defaults
+  template:
+    src: jail.local.j2
+    dest: /etc/fail2ban/jail.local
+    mode: '0644'
+  notify: Restart fail2ban
+
+- name: Ensure fail2ban is running and enabled
+  service:
+    name: fail2ban
+    state: started
+    enabled: yes

ansible/roles/common/tasks/firewall.yml

@@ -0,0 +1,28 @@
+---
+# UFW firewall configuration
+
+- name: Reset UFW to default state
+  community.general.ufw:
+    state: reset
+  when: ansible_facts['distribution'] == 'Ubuntu'
+
+- name: Set UFW default policies
+  community.general.ufw:
+    direction: "{{ item.direction }}"
+    policy: "{{ item.policy }}"
+  loop:
+    - { direction: 'incoming', policy: '{{ common_ufw_default_incoming }}' }
+    - { direction: 'outgoing', policy: '{{ common_ufw_default_outgoing }}' }
+
+- name: Allow specified ports through UFW
+  community.general.ufw:
+    rule: allow
+    port: "{{ item.port }}"
+    proto: "{{ item.proto }}"
+    comment: "{{ item.comment }}"
+  loop: "{{ common_ufw_allowed_ports }}"
+
+- name: Enable UFW
+  community.general.ufw:
+    state: enabled
+    logging: 'on'

ansible/roles/common/tasks/main.yml

@@ -0,0 +1,30 @@
+---
+# Main tasks for common role - base system setup and hardening
+
+- name: Update apt cache
+  apt:
+    update_cache: yes
+    cache_valid_time: 3600
+
+- name: Install common packages
+  apt:
+    name: "{{ common_packages }}"
+    state: present
+
+- name: Set timezone
+  community.general.timezone:
+    name: "{{ common_timezone }}"
+
+- name: Configure SSH hardening
+  include_tasks: ssh.yml
+
+- name: Configure UFW firewall
+  include_tasks: firewall.yml
+
+- name: Configure automatic updates
+  include_tasks: updates.yml
+  when: common_unattended_upgrades
+
+- name: Configure fail2ban
+  include_tasks: fail2ban.yml
+  when: common_fail2ban_enabled

ansible/roles/common/tasks/ssh.yml

@@ -0,0 +1,23 @@
+---
+# SSH hardening configuration
+
+- name: Configure SSH daemon
+  lineinfile:
+    path: /etc/ssh/sshd_config
+    regexp: "{{ item.regexp }}"
+    line: "{{ item.line }}"
+    state: present
+  with_items:
+    - { regexp: '^#?PermitRootLogin', line: 'PermitRootLogin {{ common_ssh_permit_root_login }}' }
+    - { regexp: '^#?PasswordAuthentication', line: 'PasswordAuthentication {{ common_ssh_password_authentication }}' }
+    - { regexp: '^#?PubkeyAuthentication', line: 'PubkeyAuthentication {{ common_ssh_pubkey_authentication }}' }
+    - { regexp: '^#?PermitEmptyPasswords', line: 'PermitEmptyPasswords no' }
+    - { regexp: '^#?X11Forwarding', line: 'X11Forwarding no' }
+    - { regexp: '^#?MaxAuthTries', line: 'MaxAuthTries 3' }
+  notify: Restart SSH
+
+- name: Ensure SSH is running and enabled
+  service:
+    name: ssh
+    state: started
+    enabled: yes

ansible/roles/common/tasks/updates.yml

@@ -0,0 +1,23 @@
+---
+# Automatic security updates configuration
+
+- name: Install unattended-upgrades
+  apt:
+    name:
+      - unattended-upgrades
+      - apt-listchanges
+    state: present
+
+- name: Configure unattended-upgrades
+  template:
+    src: 50unattended-upgrades.j2
+    dest: /etc/apt/apt.conf.d/50unattended-upgrades
+    mode: '0644'
+  notify: Restart unattended-upgrades
+
+- name: Enable automatic updates
+  template:
+    src: 20auto-upgrades.j2
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    mode: '0644'
+  notify: Restart unattended-upgrades

ansible/roles/common/templates/20auto-upgrades.j2

@@ -0,0 +1,7 @@
+// Enable automatic package updates
+// Managed by Ansible - do not edit manually
+
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Download-Upgradeable-Packages "1";
+APT::Periodic::AutocleanInterval "7";
+APT::Periodic::Unattended-Upgrade "1";

ansible/roles/common/templates/50unattended-upgrades.j2

@@ -0,0 +1,28 @@
+// Automatic upgrade configuration
+// Managed by Ansible - do not edit manually
+
+Unattended-Upgrade::Allowed-Origins {
+    "${distro_id}:${distro_codename}";
+    "${distro_id}:${distro_codename}-security";
+    "${distro_id}ESMApps:${distro_codename}-apps-security";
+    "${distro_id}ESM:${distro_codename}-infra-security";
+};
+
+// List of packages to not update
+Unattended-Upgrade::Package-Blacklist {
+};
+
+// Automatically reboot if needed
+Unattended-Upgrade::Automatic-Reboot "{{ 'true' if common_auto_reboot else 'false' }}";
+
+// Reboot time if automatic reboot is enabled
+Unattended-Upgrade::Automatic-Reboot-Time "03:00";
+
+// Email notification (disabled by default)
+// Unattended-Upgrade::Mail "root";
+
+// Remove unused dependencies
+Unattended-Upgrade::Remove-Unused-Dependencies "true";
+
+// Automatically cleanup old kernels
+Unattended-Upgrade::Remove-New-Unused-Dependencies "true";

ansible/roles/common/templates/jail.local.j2

@@ -0,0 +1,18 @@
+# Fail2ban configuration
+# Managed by Ansible - do not edit manually
+
+[DEFAULT]
+bantime  = {{ common_fail2ban_bantime }}
+findtime = {{ common_fail2ban_findtime }}
+maxretry = {{ common_fail2ban_maxretry }}
+
+# Email notifications (disabled by default)
+# destemail = root@localhost
+# sendername = Fail2Ban
+# mta = sendmail
+
+[sshd]
+enabled = true
+port    = {{ common_ssh_port }}
+logpath = %(sshd_log)s
+backend = %(sshd_backend)s

docs/architecture-decisions.md

@@ -798,6 +798,47 @@ infrastructure/
 
 ---
 
+## 15. Development Environment and Tooling
+
+### Decision: Isolated Python Environments with pipx
+
+**Choice:** Use `pipx` for installing Python CLI tools (Ansible) in isolated virtual environments.
+
+**Why pipx:**
+- Prevents dependency conflicts between tools
+- Each tool has its own Python environment
+- No interference with system Python packages
+- Easy to upgrade/rollback individual tools
+- Modern best practice for Python CLI tools
+
+**Implementation:**
+```bash
+# Install pipx
+brew install pipx
+pipx ensurepath
+
+# Install Ansible in isolation
+pipx install --include-deps ansible
+
+# Inject additional dependencies as needed
+pipx inject ansible requests python-dateutil
+```
+
+**Benefits:**
+| Aspect | Benefit |
+|--------|---------|
+| Isolation | No conflicts with other Python tools |
+| Reproducibility | Each team member gets same isolated environment |
+| Maintainability | Easy to upgrade Ansible without breaking other tools |
+| Clean system | No pollution of system Python packages |
+
+**Alternatives Considered:**
+- **Homebrew Ansible** - Rejected: Can conflict with system Python, harder to manage dependencies
+- **System pip install** - Rejected: Pollutes global Python environment
+- **Manual venv** - Rejected: More manual work, pipx automates this
+
+---
+
 ## Changelog
 
 | Date | Change | Author |
@@ -807,4 +848,5 @@ infrastructure/
 | 2024-12 | Switched from Terraform to OpenTofu (licensing concerns) | Pieter / Claude |
 | 2024-12 | Switched from HashiCorp Vault to SOPS + Age (simplicity, open source) | Pieter / Claude |
 | 2024-12 | Switched from Keycloak to Zitadel (Swiss company, GDPR jurisdiction) | Pieter / Claude |
+| 2024-12 | Adopted pipx for isolated Python tool environments (Ansible) | Pieter / Claude |
 ```