The Problem

Wächtersbach’s waste calendar: PDF-only distribution. No structured data. No API. Manual transcription required.

We organize everything with Google Calendar in our family. I don’t want another single-purpose app just to check when garbage is due.

The official page offers static PDFs—hostile to calendar automation.

Failed Approaches

Structured PDF parsing: Traditional tools failed. The PDF isn’t tabular data — it’s a visual calendar layout based on images. In fact thats the worst case as this defies what usual OCR can do.

The Solution: LLM-Powered Extraction

I used Gemini (fast tier) with a targeted prompt and attached a screenshot of the PDF:

"You are an expert OCR.
Your task is to extract Garbage Tours out of a PDF containing visual elements. 
The expectation is to provide a structured file containing the tours and the dates. 
Do not add complexity to the file like recurring dates, 
just print the ones which are in the visual elements.

Respond only with this file"

Workflow:

1. Gemini extracts dates → structured JSON
2. Python script generates ICS files per waste type
3. Validation script compares output versions across runs for accuracy

Download Calendars

Not sure which tour you need? Check the official tour map to find your street’s tour number.

Bio Waste:

• Tour 12
• Tours 3+4

Paper (PPK):

• Tour 5
• Tours 1+2
• Tours 3+4

Recycling (DSD):

• Tours 1+2

Residual Waste (RM):

• Tours 1+2
• Tours 3+4

Phase 1: Core Infrastructure and Virtualization

With the Fujitsu server anchoring the rack, the fleet expanded with two HP EliteDesk 600 G3 units. Talos Linux was installed on them to establish a programmable infrastructure fabric. One was commissioned as a Windows 11 Pro VM, a project that forced a deep dive into GPU passthrough on Proxmox. Mastering this was a critical success: GPU acceleration became available inside the virtualized perimeter.

This is also where the baseline architecture locked in: DNS, TLS, and identity.

• DNS: ExternalDNS automates record management, but the zone is only resolvable from inside. Names look public. Reach is not.
• Edge + TLS: Traefik is configured for Let’s Encrypt using the Cloudflare DNS-01 challenge (API token-based). If it has a hostname, it has HTTPS.
• SSO: Authentik sits in front of everything that matters. Once it’s set up, it turns access control from a scattered set of configs into a single system.
• Selective Exposure: A Cloudflare Tunnel Operator exposes only what should be public. This blog is served from the homelab through this tunnel.

Phase 2: The Industrialization of Compute

The EliteDesk experiment revealed the operational limitations of small-form-factor systems under sustained load. The next logical step was to architect a resilient, standardized solution. The design brief was clear: maximize the price-to-performance-to-wattage ratio.

The result is a pair of custom-built 4U servers, the new backbone of the lab.

• Chassis: Inter-Tech 4408 4U. A raw, functional enclosure.
• Processing: 11th Gen Intel Core i5 and i7 CPUs.
• Motherboard: Standard ATX boards for maximum compatibility and cost-effectiveness.
• Storage: Dedicated RAID controllers for data redundancy and performance.

These machines are the primary hypervisors, both running Proxmox VE. They represent a fundamental shift from ad-hoc expansion to a replicable server architecture.

Phase 3: A New Deployment Doctrine

The deployment model has undergone a significant strategic shift, moving away from inefficient and rigid structures.

• Initial State: 1 VM per container. Simple, but resource-intensive and unscalable.
• Interim State: 1 VM for multiple containers, grouped by role. This improved density but introduced scalability bottlenecks and single points of failure.
• Current Doctrine: A cluster of three purpose-built VMs, deployed and configured via a single, unified Ansible role. This provides a balance of isolation, resource management, and automated scalability.

Phase 4: The Automation Singularity

The inflection point was the total adoption of Infrastructure as Code (IaC). Manual configuration was deprecated.

The lab now has a single planning surface: a GitLab-hosted repository containing the Ansible and Terraform code. It has crossed the 3000-commit mark. That number matters because it signals intent: iteration, review, rollback, history. The homelab stopped being “machines I own” and became “a system I can recreate.”

The workflow is codified and automated.

• Proxmox VM Provisioning: VMs are no longer created; they are stamped out from standardized templates. Consistency and speed are enforced through automation.
• Talos Deployment via Terraform: The entire Talos Linux environment is defined in Terraform and managed via GitLab CI/CD. A git push to the repository triggers a pipeline that configures the entire cluster. This is GitOps executed.
• Automated Dependency Management: RenovateBot is integrated into the workflow, automatically scanning for updates, creating merge requests, and keeping the entire software stack current.
• Secrets Handling: HashiCorp Vault is the secrets backbone. Credentials are no longer scattered across shell histories and ad-hoc files; they are stored, accessed, and rotated intentionally.

This trifecta of templating, Terraform, and automated updates was the catalyst. The lab transformed from a manually curated system into a self-provisioning, self-updating organism. The velocity of experimentation and deployment increased by an order of magnitude.

The Kubernetes Question: Storage as the Final Frontier

While the infrastructure is robust, mass adoption of Kubernetes is pending. The critical dependency is a resilient, distributed storage solution. Without it, stateful workloads in Kubernetes are a liability. The current focus is on architecting a storage backend that meets the required performance and reliability specifications.

However, a lean Kubernetes control plane is operational, serving critical ingress and service discovery functions.

• CNI: Cilium provides networking, observability, and security.

Ingress, DNS automation, TLS issuance, and SSO are part of the baseline architecture (see Phase 1). Kubernetes plugs into that foundation; it doesn’t reinvent it.

Up Next: Media, Models, and Acceleration

The next steps are pragmatic. They target consolidation, capability, and GPU acceleration.

• Photos: Consolidate the photo estate into Immich. One system of record.
• GPU Node: Build a dedicated GPU-enabled node. The exact model is secondary; 16GB VRAM is not. Candidates are an RTX 4060 Ti (16GB) or a future 50-series equivalent with the same memory tier.
• Local Inference: Use that node to host smaller local LLMs for the workloads that benefit from proximity, privacy, and low latency.
• Accelerated Workflows: Run audio tooling (“audiomuse”) and use the GPU for Immich’s accelerated tagging.

The homelab is no longer a collection of hardware. It is an engineered system. The next iteration will solve the storage challenge and unlock the full potential of the Kubernetes ecosystem. The blueprint is drawn. Execution is in progress.

The Problem

My original Vault workflow:

export VAULT_TOKEN="hvs.CAESIF..."
vault kv get secret/my-app

Why this is terrible:

• ❌ Root token never expires (unlimited attack window)
• ❌ No audit trail (all operations appear as “root”)
• ❌ Single point of failure (anyone with this token has god-mode)
• ❌ No MFA (compromised laptop = compromised infrastructure)
• ❌ Can’t revoke selectively (revoking root token locks everyone out)

The wake-up call: I accidentally committed .vault-token to Gitlab. Panic-revoking the root token meant:

1. Regenerating via Terraform
2. Updating every automation script
3. Breaking every service that depended on Vault (all of them)

The Solution

Replace root tokens with federated authentication. Users log in once through Authentik SSO, and Vault grants permissions based on group membership.

Architecture

User/Script
    │
    ▼
HashiCorp Vault (OIDC Client)
    │
    │ Redirect
    ▼
Authentik (OIDC Provider)
    │
    │ Groups: vault-admins, vault-readonly
    ▼
User Identity

Why Authentik? I already had it running for GitLab, Grafana, and other services. Adding Vault to the same identity provider gives:

• ✅ Single source of truth for identities
• ✅ Centralized MFA enforcement (TOTP, WebAuthn)
• ✅ Group-based RBAC (manage access in one place)
• ✅ Audit trail (who accessed what, when, from where)
• ✅ Token expiration (force re-authentication every 8 hours)

Implementation with Terraform

1. Enable OIDC Auth Method

# terraform/modules/vault/auth-methods.tf
resource "vault_jwt_auth_backend" "authentik" {
  path               = "oidc"
  type               = "oidc"
  oidc_discovery_url = "https://auth.mittbachweg.de/application/o/vault/"
  oidc_client_id     = var.authentik_client_id
  oidc_client_secret = var.authentik_client_secret
  default_role       = "default"
  
  tune {
    default_lease_ttl = "8h"
    max_lease_ttl     = "24h"
  }
}

Key parameters:

• oidc_discovery_url: Authentik’s OIDC discovery endpoint
• default_lease_ttl: Tokens expire after 8 hours (force re-authentication)

2. Configure OIDC Roles

resource "vault_jwt_auth_backend_role" "default" {
  backend         = vault_jwt_auth_backend.authentik.path
  role_name       = "default"
  bound_audiences = [var.authentik_client_id]
  user_claim      = "sub"
  role_type       = "oidc"
  
  token_policies = [
    "default",
    "read-own-token"
  ]
  
  groups_claim = "groups"
  allowed_redirect_uris = [
    "http://localhost:8250/oidc/callback",
    "https://vault.one.mittbachweg.de/ui/vault/auth/oidc/oidc/callback"
  ]
  
  claim_mappings = {
    preferred_username = "username"
    email              = "email"
  }
}

Role breakdown:

• bound_audiences: Only accept tokens for this Vault instance (prevents token replay)
• groups_claim: Extract Authentik group memberships from JWT
• allowed_redirect_uris: Whitelist for OAuth2 redirect (CLI and Web UI)

3. Group-Based Policies

Map Authentik groups to Vault policies:

# Admin group gets full access
resource "vault_identity_group" "vault_admins" {
  name     = "vault-admins"
  type     = "external"
  policies = [
    "admin",
    "create-tokens",
    "manage-auth"
  ]
}

# Readonly group gets limited access
resource "vault_identity_group" "vault_readonly" {
  name     = "vault-readonly"
  type     = "external"
  policies = [
    "read-secrets",
    "read-own-token"
  ]
}

# Bind to Authentik groups
resource "vault_identity_group_alias" "vault_admins" {
  name           = "vault-admins"
  mount_accessor = vault_jwt_auth_backend.authentik.accessor
  canonical_id   = vault_identity_group.vault_admins.id
}

Policy example (admin.hcl):

# Full access to secrets
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage auth methods
path "auth/*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Authentik Configuration

Create an OAuth2/OpenID provider in Authentik:

# OAuth2 Provider for Vault
resource "authentik_provider_oauth2" "vault" {
  name               = "Vault OIDC Provider"
  client_id          = "vault"
  client_secret      = random_password.vault_oauth_secret.result
  authorization_flow = data.authentik_flow.default-provider-authorization-implicit-consent.id
  invalidation_flow  = data.authentik_flow.default-invalidation-flow.id

  # Redirect URIs for Vault UI, API, and CLI
  allowed_redirect_uris = [
    {
      matching_mode = "strict"
      url           = "https://vault.one.mittbachweg.de/ui/vault/auth/oidc/oidc/callback"
    },
    {
      matching_mode = "strict"
      url           = "https://vault.one.mittbachweg.de/oidc/callback"
    },
    {
      matching_mode = "strict"
      url           = "http://localhost:8250/oidc/callback"
    },
  ]

  # Property mappings for OAuth scopes
  property_mappings = [
    data.authentik_property_mapping_provider_scope.scope-email.id,
    data.authentik_property_mapping_provider_scope.scope-profile.id,
    data.authentik_property_mapping_provider_scope.scope-openid.id,
  ]

  # Additional configuration for proper OIDC flow
  signing_key = data.authentik_certificate_key_pair.default.id
}

# Vault OIDC Application
resource "authentik_application" "vault" {
  name               = "HashiCorp Vault"
  slug               = "vault"
  protocol_provider  = authentik_provider_oauth2.vault.id
  meta_description   = "OIDC authentication for HashiCorp Vault"
  meta_publisher     = "Mittbachweg Infrastructure"
  policy_engine_mode = "any"
}

Usage

CLI Login

$ vault login -method=oidc

# Browser opens for Authentik login
# After success:
Success! You are now authenticated.

$ vault token lookup
Key                 Value
---                 -----
entity_id           8a7b9c1d-2e3f-4a5b-6c7d-8e9f0a1b2c3d
expire_time         2024-02-15T19:20:00Z
policies            [default read-secrets]

Web UI Login

Navigate to https://vault.one.mittbachweg.de → Select “OIDC” method → Redirects to Authentik

Ansible Integration

# playbooks/vault_lookup.yml
- name: Fetch secret from Vault
  set_fact:
    db_password: "{{ lookup('community.hashi_vault.hashi_vault_read',
      'secret=ansible/data/database',
      auth_method='oidc',
      url='https://vault.one.mittbachweg.de'
    ).password }}"

For automation, use service accounts with AppRole auth:

resource "vault_auth_backend" "approle" {
  type = "approle"
}

resource "vault_approle_auth_backend_role" "ansible" {
  backend        = vault_auth_backend.approle.path
  role_name      = "ansible"
  token_policies = ["read-secrets"]
  token_ttl      = 3600
}

Lessons Learned

What worked:

• ✅ OIDC login eliminated root token dependency
• ✅ Group-based RBAC simplified permission management
• ✅ MFA enforcement via Authentik blocked unauthorized access
• ✅ Token expiration forced regular re-authentication

What didn’t:

• ❌ Initial 8-hour token TTL was too short for long-running tasks (extended to 24h max)
• ❌ Forgot to configure service accounts for CI/CD (added AppRole auth)
• ❌ Debugging OIDC callback issues required checking Vault AND Authentik logs

The security improvement is worth the complexity. No more root tokens in plaintext files.

The Problem

My initial architecture:

• Single Stackback (Restic wrapper) container
• Two shared S3 buckets
• Centralized backup configuration via Ansible

The catch: Stackback relies on Docker labels (stack-back.volumes=true) to auto-discover backup targets. When your backup container runs in a separate docker-compose.yml from your application stacks, it can’t see what it’s supposed to back up.

Real-world impact:

• ❌ Inconsistent backup coverage (some volumes discovered, others missed)
• ❌ No visibility into PostgreSQL backups
• ❌ Single point of failure for credentials
• ❌ Resource contention when all backups ran simultaneously

The Solution

Each application stack gets its own dedicated backup container.

Architecture Evolution

Before:

# centralized-stackback/docker-compose.yml
services:
  stackback:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      RESTIC_REPOSITORY: s3:minio.internal/shared-bucket

After:

# mattermost/docker-compose.yml
services:
  mattermost:
    labels:
      stack-back.volumes: "true"
      stack-back.postgres: "true"
  
  stackback:
    image: ghcr.io/lawndoc/stack-back:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      RESTIC_REPOSITORY: s3:minio.internal/mattermost-backup-bucket
      RESTIC_PASSWORD: ${RESTIC_PASSWORD_MATTERMOST}
      BACKUP_CRON: "0 2 * * *"

Key Design Decisions

1. Dedicated S3 Buckets Per Application

Using Terraform’s MinIO provider:

# terraform/modules/minio/main.tf
resource "minio_s3_bucket" "stackback_per_app" {
  for_each = var.applications
  bucket   = "restic-stackback-${each.key}-bucket"
  acl      = "private"
}

resource "minio_ilm_policy" "stackback_lifecycle" {
  for_each = minio_s3_bucket.stackback_per_app
  bucket   = each.value.bucket
  
  rule {
    id         = "delete-old-backups"
    expiration {
      days = 30
    }
  }
}

Why 30 days? GitLab backups alone consumed exponential storage. Automated lifecycle policies prevent the “set-and-forget-until-disk-full” trap.

2. IAM Credential Isolation

Each application receives unique S3 credentials:

resource "minio_iam_user" "stackback_per_app" {
  for_each = var.applications
  name     = "restic-${each.key}-user"
}

resource "minio_iam_policy" "stackback_per_app" {
  policy = jsonencode({
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:*"]
      Resource = [
        "arn:aws:s3:::restic-stackback-${each.key}-bucket/*"
      ]
    }]
  })
}

Security win: A compromised application can only access its own backup bucket.

3. Staggered Backup Schedules

Running all backups simultaneously caused I/O storms on NFS storage. Solution: offset schedules.

4. Vault Integration for Secrets

Backup credentials stored in HashiCorp Vault:

# ansible/roles/platform_one/templates/stackback.env.j2
RESTIC_REPOSITORY=s3:https://{{ minio_endpoint }}/{{ backup_bucket }}
RESTIC_PASSWORD={{ lookup('community.hashi_vault.hashi_vault_read', 
  'secret=ansible/data/stackback_{{ app_name }}').password }}
AWS_ACCESS_KEY_ID={{ lookup('community.hashi_vault.hashi_vault_read', 
  'secret=ansible/data/stackback_{{ app_name }}').access_key }}

Implementation with Ansible

Dynamic template generation per application:

# roles/platform_one/tasks/deploy_application.yml
- name: Generate stackback environment file
  template:
    src: stackback.env.j2
    dest: "{{ container_data }}/{{ app_name }}/stackback.env"
    mode: '0600'
  vars:
    backup_bucket: "restic-stackback-{{ vm_name }}-{{ app_name }}-bucket"
    backup_schedule: "{{ applications[app_name].backup_schedule | default('0 2 * * *') }}"
  when: applications[app_name].backup_enabled | default(false)

Docker Compose integration:

# templates/docker-compose.yml.j2
{% if app.backup_enabled | default(false) %}
  stackback:
    image: ghcr.io/mittbachweg/stack-back:2024.11.1
    container_name: {{ app_name }}_stackback
    env_file: ./stackback.env
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped
{% endif %}

Lessons Learned

What worked:

• ✅ Application-level isolation caught backup failures early
• ✅ Staggered schedules eliminated I/O contention
• ✅ Lifecycle policies prevented storage exhaustion
• ✅ Vault integration centralized credential management

What didn’t:

• ❌ Initial 7-day retention was too short (extended to 30 days)
• ❌ Forgot to monitor backup success (added Prometheus metrics)
• ❌ Manual Restic repository initialization (automated via Ansible)

The modular approach trades simplicity for reliability. Worth it.

The Context

I procure all my domains on Netcup (can only recommend checking them out) and delegate nameservers to wherever I want them managed. For years, that was AWS Route53 for mittbachweg.de and benmatheja.de.

Route53 worked well:

• Reliable DNS resolution
• LetsEncrypt DNS-01 challenges via Traefik (automated cert issuance)
• Terraform-managed records

But it wasn’t free. ~€12/year for basic DNS hosting.

The real trigger: Cloudflare Tunnels. My ISP uses Carrier Grade NAT (DS-Lite), so port forwarding was never an option. Cloudflare Tunnels enabled external access for the first time. Once I was using Cloudflare for tunnels, managing DNS in two places didn’t make sense.

The Solution

Cloudflare’s free plan includes:

• Unlimited DNS queries
• Unlimited DNS records
• DDoS protection
• Universal SSL for all subdomains
• Cloudflare Tunnels support
• Terraform provider

Annual cost: $0

Migration

Export Route53 Records

aws route53 list-resource-record-sets \
  --hosted-zone-id Z1234EXAMPLE \
  --output json > mittbachweg-zone.json

Discovered: 47 DNS records across both domains (A, AAAA, CNAME, MX, TXT)

Terraform Cloudflare Provider

# terraform/3-security/cloudflare.tf
terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.43.0"
    }
  }
}

provider "cloudflare" {
  api_token = var.cloudflare_api_token
}

resource "cloudflare_zone" "mittbachweg" {
  zone = "mittbachweg.de"
  plan = "free"
  jump_start = false
}

resource "cloudflare_record" "gitlab" {
  zone_id = cloudflare_zone.mittbachweg.id
  name    = "git"
  type    = "A"
  value   = "192.168.1.10"
  ttl     = 1
}

resource "cloudflare_record" "wildcard_platform_one" {
  zone_id = cloudflare_zone.mittbachweg.id
  name    = "*.one"
  type    = "CNAME"
  value   = "ruby.mittbachweg.de"
  ttl     = 300
}

Ansible Integration

Instead of managing DNS via Terraform for every service, I integrated Cloudflare DNS into Ansible roles:

# roles/platform_one/tasks/dns.yml
- name: Register Cloudflare DNS record for application
  community.general.cloudflare_dns:
    zone: mittbachweg.de
    record: ".one"
    type: A
    value: ""
    proxied: yes
    account_email: ""
    account_api_token: ""
    state: present
  when: dns_record | default(false)

Benefits:

• DNS records created automatically when deploying services
• No manual Terraform updates for each app
• DNS lifecycle tied to application deployment

Phase 4: Nameserver Cutover

The critical migration moment:

# 1. Verify Cloudflare DNS is fully populated
dig @eva.ns.cloudflare.com git.mittbachweg.de

# Verify Cloudflare DNS populated
dig @eva.ns.cloudflare.com git.mittbachweg.de

# Lower Route53 TTL to 60 seconds
aws route53 change-resource-record-sets --change-batch '...'

# Update nameservers at Netcup
# Old: ns-1234.awsdns-12.org
# New: eva.ns.cloudflare.com, walt.ns.cloudflare.com

# Monitor propagation
watch -n 5 'dig NS mittbachweg.de +short'

Result: Zero downtime. Full propagation in ~45 minutes.

Cloudflare Tunnel: The Real Motivation

Port forwarding was never an option for me. My ISP uses Carrier Grade NAT (DS-Lite), which means I don’t have a public IPv4 address. Traditional port forwarding simply doesn’t work in this setup.

Even if I could forward ports, the security implications always turned me off. Exposing services directly to the internet invites constant scanning, brute force attempts, and potential DDoS attacks.

Cloudflare Tunnels changed everything. For the first time, I could reliably expose services from home without:

• ❌ Public IP address
• ❌ Port forwarding rules
• ❌ Firewall holes
• ❌ Direct exposure to internet threats

The architecture:

Internet → Cloudflare Edge → Encrypted Tunnel → Traefik → Services

Cloudflare establishes an outbound connection from my homelab to their edge network. All inbound traffic is proxied through Cloudflare’s DDoS protection and WAF. My actual infrastructure remains completely hidden.

Terraform Configuration

Tunnel setup:

resource "cloudflare_tunnel" "app_platform" {
  account_id = var.cloudflare_account_id
  name       = "app-platform-tunnel"
  secret     = base64encode(random_password.tunnel_secret.result)
}

resource "cloudflare_tunnel_config" "app_platform" {
  tunnel_id  = cloudflare_tunnel.app_platform.id
  account_id = var.cloudflare_account_id
  
  config {
    ingress_rule {
      hostname = "git.mittbachweg.de"
      service  = "https://ruby.internal:443"
    }
    
    ingress_rule {
      hostname = "*.one.mittbachweg.de"
      service  = "https://ruby.internal:443"
    }
    
    ingress_rule {
      service = "http_status:404"
    }
  }
}

resource "cloudflare_record" "tunnel_gitlab" {
  zone_id = cloudflare_zone.mittbachweg.id
  name    = "git"
  type    = "CNAME"
  value   = "${cloudflare_tunnel.app_platform.id}.cfargotunnel.com"
  proxied = true
}

Deployment

# roles/cloudflare_tunnel/tasks/main.yml
- name: Install cloudflared
  apt:
    name: cloudflared

- name: Configure tunnel credentials
  copy:
    content: \"{{ tunnel_credentials | to_json }}\"
    dest: /etc/cloudflared/credentials.json
    mode: '0600'

- name: Start tunnel service
  systemd:
    name: cloudflared-tunnel@app-platform
    enabled: yes

Why This Matters

Cloudflare Tunnels solved a problem I couldn’t fix any other way:

• Carrier Grade NAT / DualStack Lite: No public IPv4, port forwarding impossible
• Security: Zero exposed ports, all traffic proxied through Cloudflare, adding policies like GeoIP-based blocking or another SSO is possible
• Reliability: Persistent outbound connection, no inbound firewall rules needed
• DDoS protection: Cloudflare’s network absorbs attacks before they reach my connection

This wasn’t a migration from port forwarding. It was the first time I could expose homelab services to the internet reliably and securely.

What I Learned

Ansible vs. Terraform for DNS Management

I use Ansible for DNS record creation during service deployments and vm setups. This caused problems.

The issue: Ansible doesn’t track state. When you remove a DNS record from your playbook, Ansible doesn’t know to delete it from Cloudflare. The record stays there until you manually remove it.

This led to:

• Duplicate records: Deploying the same service twice created multiple DNS entries
• Stale records: Removing services left orphaned DNS records in Cloudflare
• Manual cleanup: Had to manually edit DNS records in Cloudflare’s UI to fix duplicates

The fix: Move DNS records to Terraform where possible.

Terraform tracks state. When you remove a record from your Terraform code:

# Delete this resource
resource "cloudflare_record" "old_service" {
  # ...
}

Run terraform apply, and Terraform automatically removes it from Cloudflare.

Current approach:

• Terraform: Static DNS records (zones, subdomains, tunnel CNAMEs) - the majority
• Ansible: Dynamic records only when deployment requires it (rare)

Terraform’s state management eliminated the duplicate/stale record problem entirely. Everything configured via Terraform. LetsEncrypt DNS-01 challenges still work via Traefik.

The migration to Cloudflare made sense for consistency with Tunnel and eliminated AWS entirely.

• Home Renovation and Automation: We moved and renovated a house, which led to automating shutters with Shelly 2.5s.
• Proxmox Node Setup: I turned my TX120 S3 running Ubuntu into my first Proxmox Node. This setup, featuring Home Assistant, a self-hosted VPN VM, and a GitLab Instance, served me well for months.
• Deploying a Real Server: I acquired an RX2530 M2 rack-mounted server. Setting up VMs on it taught me about IPMI and the key differences between servers and consumer PCs (Noise, Heat, Power Consumption, Redundancy).
• HP EliteDesk Addition: I added two HP EliteDesk 600 G3 units to leverage integrated GPUs for transcoding and overall lab management.
• Central Media Server: Recently, I got an ITX system with an Intel i5-11400, now my central media server due to its superior Intel UHD 730 transcoding capabilities.

This journey has been one of learning, experimenting, and optimizing. Each step brought new insights and capabilities. Stay tuned for more updates as I continue to refine my setup and explore new technologies!

The Problem

Since my first homelab post in November 2020, I’d accumulated a collection of snowflake servers—each one unique, manually configured, and completely undocumented.

The cost:

• ❌ 4-hour recovery times for failed VMs
• ❌ Deployment anxiety (one wrong click = broken service)
• ❌ Zero reproducibility
• ❌ Tribal knowledge locked in my head

The wake-up call: A hard drive failure on my GitLab VM. I had no backup of the VM configuration. Was it 8GB or 16GB of RAM? What VLAN was it on? Where were the mount points?

The Solution

Every component of my infrastructure is now defined in code. No more clicking through UIs. No more manual SSH sessions.

Two-Layer Architecture

Terraform: Manages external/immutable infrastructure.

• Cloudflare DNS records and Tunnel

Ansible: Manages mutable state and VM lifecycle.

• Proxmox VMs (provisioning from cloud-init templates)
• Docker containers
• File system configurations (NFS mounts)
• Service orchestration

Why both? Terraform manages external resources. Ansible handles everything on Proxmox VMs.

Implementation

VM Provisioning with Ansible

Before, I clicked through Proxmox UI screens and hoped I wrote down what I did.

Now, VM specs are defined in YAML:

# vars/vm_specs.yml
vms:
  - name: ruby
    vmId: 920
    target_node: pve5
    cores: 8
    memory: 16384
    disk:
      scsi0: 'nvme-thin:200'
    net:
      net0: 'virtio,bridge=vmbr0,tag=50'
    ipconfig:
      ipconfig0: 'ip=192.168.50.20/24,gw=192.168.50.1'
    tags: ['ansible', 'gitlab-runner']
    clone: 'ubuntu-24.04-server-cloudinit-template'

# roles/hypervisor/tasks/provision_vm.yml
- name: Provision VM from cloud-init template
  community.general.proxmox_kvm:
    api_host: "{{ ansible_host }}"
    node: "{{ inventory_hostname }}"
    name: "{{ vm_name }}"
    vmid: "{{ vm_id }}"
    clone: "{{ vm_clone }}"
    cores: "{{ vm_cores }}"
    memory: "{{ vm_memory }}"
    net: "{{ vm_net }}"
    state: "{{ vm_state }}"

Benefits:

• ✅ Reproducible (spin up identical VMs from templates)
• ✅ Version controlled (VM specs in Git)
• ✅ Self-documenting (vm_specs.yml IS the documentation)
• ✅ Idempotent (run multiple times safely)

Application Deployment

Before: SSH in, manually install packages, hope nothing breaks.

After:

# roles/platform_one/tasks/gitlab.yml
- name: Ensure GitLab directory structure
  file:
    path: "{{ container_data }}/ruby/gitlab"
    state: directory

- name: Template GitLab docker-compose
  template:
    src: gitlab-docker-compose.yml.j2
    dest: "{{ container_data }}/ruby/gitlab/docker-compose.yml"
  notify: restart gitlab

- name: Deploy GitLab container
  community.docker.docker_compose_v2:
    project_src: "{{ container_data }}/ruby/gitlab"
    state: present

Run once: ansible-playbook main_playbook.yml --tags gitlab --limit ruby

Terraform → Ansible Integration

Terraform outputs (VM IPs, bucket names) flow into Ansible via a generated vars file:

# terraform/outputs.tf
resource "local_file" "ansible_vars" {
  filename = "${path.root}/../../vars/tf_ansible_vars_file.yml"
  content  = yamlencode({
    minio_endpoint = minio_s3_bucket.backups.endpoint
    vault_addr     = vault_auth_backend.oidc.path
  })
}

Ansible consumes this automatically:

# playbook.yml
- hosts: all
  vars_files:
    - vars/tf_ansible_vars_file.yml

The Results

Full environment rebuild:

$ cd terraform/1-infrastructure && terraform apply
$ ansible-playbook main_playbook.yml

That’s it. Every VM, every service, every configuration restored from code.

This was the foundation. Terraform came later in December 2024. For now, Ansible handled everything—VM provisioning, Docker deployments, configuration management. The infrastructure repo was born on April 1, 2024. What followed was rapid iteration.

Status Quo

Having the need to run at least the Unifi-Controller within my network to control my Unifi devices I used an old Raspberry Pi 2 before. The RPI 2 used my Synology NAS as NFS as I was aware of file system issues using only the internal SD cards. Still the setup was painfully slow and not extendable.

Overview

With the use of Containers and Orchestration Tools such as docker-compose you can bring up entire stacks within seconds and manage them in a painless way. I procured a used Fujitsu TX-120 S3 with a single Xeon E3-1220 3,1GHz 8GB Ram and 300GB SAS Drives for below 200€. Then I installed Ubuntu 18.04 on it to use it as my primary docker-compose host.

Sidenote: The TX-120 is placed in the hall of our flat and I’m a bit annoyed by the continous noisy write on the Disks. So i’ll opt to switch the SAS Drives towards 2,5” SSDs in the Future. But still i’m really amazed by the size of the case. You’ll find a spot for a server that big. And as a bonus tt features 2 usable Network Interfaces out of the box.

Here is an overview of the current setup.

                          +--------------------------------------+
                          |                       TX120 S3 (io)  |
                          |                       docker-compose |
                          |                                      |
+-----------+             | +-----------+        +-------------+ |
|           |             | |           |        |             | |
| Quaysi.de +--------------->  Traefik  +-------->  Services   | |
|           |             | |           |        |             | |
+-----------+             | +-----------+        +-------------+ |
                          |                                      |
                          +--------------------------------------+

Everything is accessible and exposed via HTTPS below the quaysi.de domain. Each services has its own subdomain e.g. unifi.quaysi.de.

Traefik, Domain and Certificates

Traefik is configured as the central edge router handling incoming requests and forwarding them to the services. If you ever set up local services I assume you encountered the issues regarding non-trusted certificates. With the help of Amazon Route 53, Traefik and the LetsEncrypt Resolver its possible to bypass that.

• Set public Records in a Hosted Zone within Route 53 for your internal services e.g. quaysi.de points to 192.168.1.9, 192.168.1.10 and 192.168.1.11
• Configure Traefik to use the DNS-Challenge and provision AWS Credentials

The effect is: all your internal services receive a trusted LetsEncrypt certificate and you can just work around those annoying “untrusted connection” issues, as everything is provisioned.

This was the first part of the series about my homelab. As always love to hear your feedback about.

Stop Trying to Make Hard Work Easy by Nir Eyal

Nir Eyal hightlights the number one barrier to getting our work done is distraction. And for me quite interesting elaborates on the opposite of distraction which is not focus, it’s traction.

From his perspective the main challenge to remain productive i.e gaining traction is mastering our triggers. They may be internal (our discomfort when pushed to focus on a given task without allowing distraction to ourselves) as well as external triggers (disturbances like a message on our phone).

Here is why!

Why

My old approach had a lot of shortcomings.

The way I was developing on the app and how it ran in production was not the same. There was no real CI/CD process involved and the packaging of the app itself really differed from local development. This caused issues, whenever changes had to be done.

The simple function consisted of too many moving parts which also introduced more complexity to the overall system. I used an apt-get installed Nginx as reverse proxy. There was Gunicorn with it’s own configuration files and last but not least the Python App itself.

The durability of the app was not convincing. From a user’s perspective the performance got worse the longer the application has been running. This resulted in dropped events and made the service not reliable. I remember someone talking about self-driving cars and he said “if it doesn’t work in all circumstances - there is no use for it”. To be honest, the complexity of the Todoist Webhook Integration is trivial compared to the challenges of writing software for self-driving cars, but still the argument holds.

The handling of credentials was far from optimal. With the current app, they were just inserted to an settings.py file on the machine.

What I did

I used both Zappa and Serverless for setting up the AWS Stack and configuring the app to run properly on AWS Lambda. Serverless seemed more mature to me which is the reason I’m still using it that way.

The integration runs at free tier with no hassle. I configured Todoist Webhooks to fire whenever an item on my list is marked as completed

I reduced the moving parts necessary to maintain, it’s just packaging the app, making sure that it’ll run, deploy it, test it on Lambda and you’re done.

I kind of fought with Github Actions to set up a CI/CD but can say that it’ll now deploy to AWS whenever something is pushed to master. This is a real relief, knowing that wherever I commit changes to the repository, the app will get deployed in the same (working) way. So you can say I’m pretty happy with the status quo of the app.

Future

• Let Github Actions run integration tests after deployment on dev (e.g. is the Endpoint responding as expected using newman or something else). If succesfull stage to prod