Post-Tyranny-Tech-Infrastructure/docs/storage-architecture.md

Storage Architecture

Comprehensive guide to storage architecture using Hetzner Volumes for Nextcloud data.

Overview

The infrastructure uses Hetzner Volumes (block storage) for Nextcloud user data, separating application and data layers:

• Server local disk: Operating system, Docker images, application code
• Hetzner Volume: Nextcloud user files (/var/www/html/data)
• Docker volumes: Database and Redis data (ephemeral, can be rebuilt)

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│ Hetzner Cloud Server (cpx22)                                │
│                                                               │
│  ┌──────────────────────┐       ┌────────────────────────┐  │
│  │ Local Disk (80 GB)   │       │ Hetzner Volume (100GB) │  │
│  │                      │       │                        │  │
│  │ - OS (Ubuntu 24.04)  │       │ Mounted at:            │  │
│  │ - Docker images      │       │ /mnt/nextcloud-data    │  │
│  │ - Application code   │       │                        │  │
│  │ - Config files       │       │ Contains:              │  │
│  │                      │       │ - Nextcloud user files │  │
│  │ Docker volumes:      │       │ - Uploaded documents   │  │
│  │ - postgres-db        │       │ - Photos, videos       │  │
│  │ - redis-cache        │       │ - All user data        │  │
│  │ - nextcloud-app      │       │                        │  │
│  └──────────────────────┘       └────────────────────────┘  │
│           │                                │                  │
│           └────────────────────────────────┘                  │
│                  Both accessible to                          │
│                  Docker containers                           │
└─────────────────────────────────────────────────────────────┘

Benefits

1. Data Independence

• User data survives server rebuilds
• Can detach volume from one server and attach to another
• Easier disaster recovery

2. Flexible Scaling

• Resize storage without touching server
• Pay only for storage you need
• Start small (100 GB), grow as needed

3. Better Separation

• Application layer (ephemeral, can be rebuilt)
• Data layer (persistent, backed up)
• Clear distinction between code and content

4. Simplified Backups

• Snapshot volumes independently
• Smaller, faster snapshots (only data, not OS)
• Point-in-time recovery of user files

5. Cost Optimization

• Small clients: 50 GB (~€2.70/month)
• Medium clients: 100 GB (~€5.40/month)
• Large clients: 250+ GB (~€13.50+/month)
• Only pay for what you use

Volume Specifications

Feature	Value
Minimum size	10 GB
Maximum size	10 TB (10,000 GB)
Pricing	€0.054/GB/month
Performance	Fast NVMe SSD
IOPS	High performance
Filesystem	ext4 (pre-formatted)
Snapshots	Supported
Backups	Via Hetzner API

How It Works

1. OpenTofu Creates Volume

When deploying a client:

# tofu/volumes.tf
resource "hcloud_volume" "nextcloud_data" {
  for_each = var.clients

  name     = "nextcloud-data-${each.key}"
  size     = each.value.nextcloud_volume_size  # e.g., 100 GB
  location = each.value.location
  format   = "ext4"
}

resource "hcloud_volume_attachment" "nextcloud_data" {
  for_each  = var.clients
  volume_id = hcloud_volume.nextcloud_data[each.key].id
  server_id = hcloud_server.client[each.key].id
  automount = false
}

2. Ansible Mounts Volume

During deployment:

# ansible/roles/nextcloud/tasks/mount-volume.yml
- Find volume device at /dev/disk/by-id/scsi-0HC_Volume_*
- Format as ext4 (if not already formatted)
- Mount at /mnt/nextcloud-data
- Create data directory with proper permissions

3. Docker Uses Mount

Docker Compose configuration:

services:
  nextcloud:
    volumes:
      - nextcloud-app:/var/www/html           # Application code (local)
      - /mnt/nextcloud-data/data:/var/www/html/data  # User data (volume)

Directory Structure

On Server Local Disk

/var/lib/docker/volumes/
├── nextcloud-app/           # Nextcloud application code
├── nextcloud-db-data/       # PostgreSQL database
└── nextcloud-redis-data/    # Redis cache

/opt/docker/
├── authentik/               # Authentik configuration
├── nextcloud/               # Nextcloud docker-compose.yml
└── traefik/                 # Traefik configuration

On Hetzner Volume

/mnt/nextcloud-data/
└── data/                    # Nextcloud user data directory
    ├── admin/               # Admin user files
    ├── user1/               # User 1 files
    ├── user2/               # User 2 files
    └── appdata_*/           # Application data

Volume Sizing Guidelines

Small Clients (1-10 users)

• Starting size: 50 GB
• Monthly cost: ~€2.70
• Use case: Personal use, small teams
• Growth: +10 GB increments

Medium Clients (10-50 users)

• Starting size: 100 GB
• Monthly cost: ~€5.40
• Use case: Small businesses, departments
• Growth: +25 GB increments

Large Clients (50-200 users)

• Starting size: 250 GB
• Monthly cost: ~€13.50
• Use case: Medium businesses
• Growth: +50 GB increments

Enterprise Clients (200+ users)

• Starting size: 500 GB+
• Monthly cost: ~€27+
• Use case: Large organizations
• Growth: +100 GB increments

Pro tip: Start conservative and grow as needed. Resizing is online and takes seconds.

Volume Operations

Resize Volume

Increase volume size (cannot decrease):

./scripts/resize-client-volume.sh <client> <new_size_gb>

Example:

# Resize dev client from 100 GB to 200 GB
./scripts/resize-client-volume.sh dev 200

The script will:

1. Resize via Hetzner API
2. Expand filesystem
3. Verify new size
4. Show cost increase

Note: Resizing is online (no downtime) and instant.

Snapshot Volume

Create a point-in-time snapshot:

# Via Hetzner Cloud Console
# Or via API:
hcloud volume create-snapshot nextcloud-data-dev \
    --description "Before major update"

Restore from Snapshot

1. Create new volume from snapshot
2. Attach to server
3. Update mount in Ansible
4. Restart Nextcloud containers

Detach and Move Volume

Move data between servers:

# 1. Stop Nextcloud on old server
ansible old-server -i hcloud.yml -m shell -a "docker stop nextcloud"

# 2. Detach volume via Hetzner Console or API
hcloud volume detach nextcloud-data-client1

# 3. Attach to new server
hcloud volume attach nextcloud-data-client1 --server new-server

# 4. Mount on new server
ansible new-server -i hcloud.yml -m shell -a "mount /dev/disk/by-id/scsi-0HC_Volume_* /mnt/nextcloud-data"

# 5. Start Nextcloud
ansible new-server -i hcloud.yml -m shell -a "docker start nextcloud"

Backup Strategy

Option 1: Hetzner Volume Snapshots

Pros:

• Fast (incremental)
• Integrated with Hetzner
• Point-in-time recovery

Cons:

• Stored in same region
• Not off-site

Implementation:

# Daily snapshots via cron
0 2 * * * hcloud volume create-snapshot nextcloud-data-prod \
    --description "Daily backup $(date +%Y-%m-%d)"

Option 2: Rsync to External Storage

Pros:

• Off-site backup
• Full control
• Can use any storage provider

Cons:

• Slower
• More complex

Implementation:

# Backup to external server
ansible client -i hcloud.yml -m shell -a "\
    rsync -av /mnt/nextcloud-data/data/ \
    backup-server:/backups/client/nextcloud/"

Option 3: Nextcloud Built-in Backup

Pros:

• Uses Nextcloud's own backup tools
• Consistent with application state

Cons:

• Slower than volume snapshots

Implementation:

# Using occ command
docker exec -u www-data nextcloud php occ maintenance:mode --on
rsync -av /mnt/nextcloud-data/ /backup/location/
docker exec -u www-data nextcloud php occ maintenance:mode --off

Performance Considerations

Hetzner Volume Performance

Metric	Specification
Type	NVMe SSD
IOPS	High (exact spec varies)
Throughput	Fast sequential R/W
Latency	Low (local to server)

Optimization Tips

1. Use ext4 filesystem (default, well-tested)
2. Enable discard for SSD optimization (default in our setup)
3. Monitor I/O with iostat -x 1
4. Check volume usage regularly

Monitoring

# Check volume usage
df -h /mnt/nextcloud-data

# Check I/O stats
iostat -x 1 /dev/disk/by-id/scsi-0HC_Volume_*

# Check mount status
mount | grep nextcloud-data

Troubleshooting

Volume Not Mounting

Problem: Volume doesn't mount after server restart

Solutions:

1. Check if volume is attached:

   lsblk
   ls -la /dev/disk/by-id/scsi-0HC_Volume_*
   

2. Check fstab entry:

   cat /etc/fstab | grep nextcloud-data
   

3. Manually mount:

   mount /dev/disk/by-id/scsi-0HC_Volume_* /mnt/nextcloud-data
   

4. Re-run Ansible:

   ansible-playbook -i hcloud.yml playbooks/deploy.yml --limit client --tags volume
   

Volume Full

Problem: Nextcloud reports "not enough space"

Solutions:

1. Check usage:

   df -h /mnt/nextcloud-data
   

2. Resize volume:

   ./scripts/resize-client-volume.sh client 200
   

3. Clean up old files:

   docker exec -u www-data nextcloud php occ files:scan --all
   docker exec -u www-data nextcloud php occ files:cleanup
   

Permission Issues

Problem: Nextcloud can't write to volume

Solutions:

1. Check ownership:

   ls -la /mnt/nextcloud-data/
   

2. Fix permissions:

   chown -R www-data:www-data /mnt/nextcloud-data/data
   chmod -R 750 /mnt/nextcloud-data/data
   

3. Re-run mount tasks:

   ansible-playbook -i hcloud.yml playbooks/deploy.yml --limit client --tags volume
   

Volume Detached Accidentally

Problem: Volume was detached and lost mount

Solutions:

1. Re-attach via Hetzner Console or API
2. Remount:

   ansible client -i hcloud.yml -m shell -a "\
       mount /dev/disk/by-id/scsi-0HC_Volume_* /mnt/nextcloud-data"
   

3. Restart Nextcloud:

   docker restart nextcloud nextcloud-cron
   

Cost Analysis

Example Scenarios

Scenario 1: 10 Clients, 100 GB each

• Volume cost: 10 × 100 GB × €0.054 = €54/month
• Server cost: 10 × €7/month = €70/month
• Total: €124/month

Scenario 2: 5 Small + 3 Medium + 2 Large

• Small (50 GB): 5 × €2.70 = €13.50
• Medium (100 GB): 3 × €5.40 = €16.20
• Large (250 GB): 2 × €13.50 = €27.00
• Volume total: €56.70/month
• Plus server costs

Cost Savings vs Local Disk:

• Can use smaller servers (cheaper compute)
• Pay only for storage needed
• Resize incrementally vs over-provisioning

Migration from Local Volumes

See volume-migration.md for detailed migration procedures.

Related Documentation

• Volume Migration Guide - Migrating existing clients
• Deployment Guide - Full deployment with volumes
• Maintenance Tracking - Monitoring and updates