EZ-Homelab/wiki/action-reports/2026-01-12-ssl-wildcard-certificate-setup.md

Action Report: SSL Wildcard Certificate Setup

Date: January 12, 2026
Status: ✅ Completed Successfully
Impact: All homelab services now have valid Let's Encrypt SSL certificates

---

Problem Statement

Services were showing "not secure" warnings in browsers despite Traefik being configured for Let's Encrypt certificates. Multiple simultaneous certificate requests were failing due to DNS challenge conflicts.

Root Causes Identified

1. Multiple Simultaneous Certificate Requests

• Issue: Each service (dockge, dozzle, glances, pihole, authelia) had traefik.http.routers.*.tls.certresolver=letsencrypt labels
• Impact: Traefik attempted to request individual certificates for each subdomain simultaneously
• Consequence: DuckDNS DNS challenge can only handle ONE TXT record at _acme-challenge.kelin-hass.duckdns.org at a time
• Result: All certificate requests failed with "Incorrect TXT record" errors

2. DNS TXT Record Conflicts

• Issue: Multiple services tried to create different TXT records at the same DNS location
• Example:
  • Service A creates: _acme-challenge.kelin-hass.duckdns.org = "token1"
  • Service B overwrites: _acme-challenge.kelin-hass.duckdns.org = "token2"
  • Let's Encrypt validates Service A but finds "token2" → validation fails
• DuckDNS Limitation: Can only maintain ONE TXT record per domain

3. Authelia Configuration Error

• Issue: Environment variable AUTHELIA_NOTIFIER_SMTP_PASSWORD was set without corresponding SMTP configuration
• Impact: Authelia crashed on startup with "please ensure only one of the 'smtp' or 'filesystem' notifier is configured"
• Consequence: Services requiring Authelia authentication were inaccessible

4. Stale DNS Records

• Issue: Old TXT records from failed attempts persisted in DNS
• Impact: New certificate attempts validated against old, incorrect TXT records

Solution Implemented

Phase 1: Identify Certificate Request Pattern

Actions:

1. Discovered Traefik logs at /var/log/traefik/traefik.log (not stdout)
2. Analyzed logs showing multiple simultaneous DNS-01 challenges
3. Confirmed DuckDNS TXT record conflicts

Command Used:

docker exec traefik tail -f /var/log/traefik/traefik.log

Phase 2: Configure Wildcard Certificate

Actions:

1. Removed certresolver labels from all services except Traefik
2. Configured wildcard certificate on Traefik router only
3. Added DNS propagation skip for faster validation

Changes Made:

File: /home/kelin/AI-Homelab/docker-compose/core.yml

# Traefik - Only service with certresolver
traefik:
  labels:
    - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
    - "traefik.http.routers.traefik.tls.domains[0].main=${DOMAIN}"
    - "traefik.http.routers.traefik.tls.domains[0].sans=*.${DOMAIN}"

# Authelia - No certresolver, just tls=true
authelia:
  labels:
    - "traefik.http.routers.authelia.tls=true"

File: /home/kelin/AI-Homelab/docker-compose/infrastructure.yml

# All infrastructure services - No certresolver
dockge:
  labels:
    - "traefik.http.routers.dockge.tls=true"

dozzle:
  labels:
    - "traefik.http.routers.dozzle.tls=true"

glances:
  labels:
    - "traefik.http.routers.glances.tls=true"

pihole:
  labels:
    - "traefik.http.routers.pihole.tls=true"

File: /opt/stacks/core/traefik/traefik.yml

certificatesResolvers:
  letsencrypt:
    acme:
      email: kelinfoxy@gmail.com
      storage: /acme.json
      dnsChallenge:
        provider: duckdns
        disablePropagationCheck: true  # Added to skip DNS propagation wait
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

Phase 3: Clear DNS and Reset Certificates

Actions:

1. Stopped all services to clear DNS TXT records
2. Reset acme.json to force fresh certificate request
3. Waited 60 seconds for DNS to fully clear
4. Restarted services with wildcard-only configuration

Commands Executed:

# Stop services
cd /opt/stacks/core && docker compose down

# Reset certificate storage
rm /opt/stacks/core/traefik/acme.json
touch /opt/stacks/core/traefik/acme.json
chmod 600 /opt/stacks/core/traefik/acme.json
chown kelin:kelin /opt/stacks/core/traefik/acme.json

# Wait for DNS to clear
sleep 60
dig +short TXT _acme-challenge.kelin-hass.duckdns.org  # Verified empty

# Deploy updated configuration
cp /home/kelin/AI-Homelab/docker-compose/core.yml /opt/stacks/core/docker-compose.yml
cd /opt/stacks/core && docker compose up -d

Phase 4: Fix Authelia Configuration

Issue Found: Environment variable triggering SMTP configuration check

File: /opt/stacks/core/docker-compose.yml

Removed:

environment:
  - AUTHELIA_NOTIFIER_SMTP_PASSWORD=${SMTP_PASSWORD}  # ❌ Removed

Command:

cd /opt/stacks/core && docker compose up -d authelia

Phase 5: Fix Infrastructure Services

Issue: Missing networks: header in compose file

File: /opt/stacks/infrastructure/infrastructure.yml

Fixed:

# Before (incorrect):
  traefik-network:
    external: true

# After (correct):
networks:
  traefik-network:
    external: true
  homelab-network:
    driver: bridge
  dockerproxy-network:
    driver: bridge

Command:

cd /opt/stacks/infrastructure && docker compose -f infrastructure.yml up -d

Results

Certificate Obtained Successfully ✅

acme.json Contents:

{
  "letsencrypt": {
    "Account": {
      "Email": "kelinfoxy@gmail.com",
      "Registration": {
        "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/2958966636"
      }
    },
    "Certificates": [
      {
        "domain": {
          "main": "dockge.kelin-hass.duckdns.org"
        }
      },
      {
        "domain": {
          "main": "kelin-hass.duckdns.org",
          "sans": ["*.kelin-hass.duckdns.org"]
        }
      }
    ]
  }
}

Certificate Details:

• Subject: CN=kelin-hass.duckdns.org
• Issuer: C=US, O=Let's Encrypt, CN=R12
• Coverage: Wildcard certificate covering all subdomains
• File Size: 23KB (up from 0 bytes)

Services Status

All services running with valid SSL certificates:

Service	Status	URL	Certificate
Traefik	✅ Up	https://traefik.kelin-hass.duckdns.org	Valid
Authelia	✅ Up	https://auth.kelin-hass.duckdns.org	Valid
Dockge	✅ Up	https://dockge.kelin-hass.duckdns.org	Valid
Dozzle	✅ Up	https://dozzle.kelin-hass.duckdns.org	Valid
Glances	✅ Up	https://glances.kelin-hass.duckdns.org	Valid
Pi-hole	✅ Up	https://pihole.kelin-hass.duckdns.org	Valid

Best Practices & Prevention

1. ✅ Use Wildcard Certificates with DuckDNS

Rule: Only ONE service should request certificates with DuckDNS DNS challenge

Configuration:

# ✅ CORRECT: Only Traefik requests wildcard cert
traefik:
  labels:
    - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
    - "traefik.http.routers.traefik.tls.domains[0].main=${DOMAIN}"
    - "traefik.http.routers.traefik.tls.domains[0].sans=*.${DOMAIN}"

# ✅ CORRECT: Other services just enable TLS
other-service:
  labels:
    - "traefik.http.routers.service.tls=true"  # Uses wildcard automatically

# ❌ WRONG: Multiple services requesting certs
other-service:
  labels:
    - "traefik.http.routers.service.tls.certresolver=letsencrypt"  # DON'T DO THIS

2. ✅ DuckDNS DNS Challenge Limitations

Understand the Constraint:

• DuckDNS can only maintain ONE TXT record at _acme-challenge.kelin-hass.duckdns.org
• Multiple simultaneous challenges WILL fail
• Use wildcard certificate to avoid this limitation

Alternative Providers (if needed):

• Cloudflare: Supports multiple simultaneous DNS challenges
• Route53: Supports multiple TXT records
• Use HTTP challenge if DNS challenge isn't required

3. ✅ Traefik Logging Configuration

Enable File Logging for Debugging:

File: /opt/stacks/core/traefik/traefik.yml

log:
  level: DEBUG  # Use DEBUG for troubleshooting, INFO for production
  filePath: /var/log/traefik/traefik.log  # Easier to tail than docker logs

# Mount in docker-compose.yml:
volumes:
  - /var/log/traefik:/var/log/traefik

Useful Commands:

# Monitor certificate acquisition
docker exec traefik tail -f /var/log/traefik/traefik.log | grep -E "acme|certificate|DNS"

# Check for errors
docker exec traefik tail -100 /var/log/traefik/traefik.log | grep -E "error|Unable"

# View specific domain
docker exec traefik tail -200 /var/log/traefik/traefik.log | grep "kelin-hass.duckdns.org"

4. ✅ Certificate Troubleshooting Workflow

When certificates aren't working:

# 1. Check acme.json status
cat /opt/stacks/core/traefik/acme.json | python3 -m json.tool | grep -A5 "Certificates"

# 2. Check certificate count
python3 -c "import json; d=json.load(open('/opt/stacks/core/traefik/acme.json')); print(f'Certificates: {len(d[\"letsencrypt\"][\"Certificates\"])}')"

# 3. Test certificate being served
echo | openssl s_client -connect auth.kelin-hass.duckdns.org:443 -servername auth.kelin-hass.duckdns.org 2>/dev/null | openssl x509 -noout -subject -issuer

# 4. Check DNS TXT records
dig +short TXT _acme-challenge.kelin-hass.duckdns.org

# 5. Check Traefik logs
docker exec traefik tail -50 /var/log/traefik/traefik.log

5. ✅ Environment Variable Hygiene

Principle: Only set environment variables that are actually used

Example - Authelia:

# ✅ CORRECT: Only variables for configured features
environment:
  - AUTHELIA_JWT_SECRET=${AUTHELIA_JWT_SECRET}
  - AUTHELIA_SESSION_SECRET=${AUTHELIA_SESSION_SECRET}
  - AUTHELIA_STORAGE_ENCRYPTION_KEY=${AUTHELIA_STORAGE_ENCRYPTION_KEY}

# ❌ WRONG: SMTP variable without SMTP configuration
environment:
  - AUTHELIA_NOTIFIER_SMTP_PASSWORD=${SMTP_PASSWORD}  # Causes crash if SMTP not in config.yml

6. ✅ Docker Compose File Validation

Before deploying:

# Validate syntax
docker compose -f /path/to/file.yml config

# Check for common errors
grep -n "^  [a-z]" file.yml  # Networks should have "networks:" header

7. ✅ Certificate Renewal Strategy

Automatic Renewal:

• Traefik automatically renews certificates 30 days before expiration
• Wildcard certificate covers all subdomains (no individual renewals needed)
• Monitor acme.json for certificate expiration dates

Backup acme.json:

# Regular backup (e.g., daily cron)
cp /opt/stacks/core/traefik/acme.json /opt/backups/acme.json.$(date +%Y%m%d)

# Keep last 7 days
find /opt/backups -name "acme.json.*" -mtime +7 -delete

Key Learnings

Technical Insights

1. DuckDNS Limitation: Single TXT record constraint requires wildcard certificate approach
2. DNS Propagation: disablePropagationCheck: true speeds up validation but relies on fast DNS updates
3. Traefik Labels: tls=true vs tls.certresolver=letsencrypt - use former for wildcard coverage
4. Environment Variables: Can trigger configuration validation even without corresponding config file entries

Process Insights

1. Log Discovery: Traefik logs to files by default, not always visible via docker logs
2. DNS Clearing: Stopping services and waiting 60s ensures DNS records fully clear
3. Incremental Debugging: Monitor logs during certificate acquisition to catch issues early
4. Configuration Synchronization: Repository files must be copied to deployment locations

Documentation Updates

Files Modified

Repository:

• /home/kelin/AI-Homelab/docker-compose/core.yml
• /home/kelin/AI-Homelab/docker-compose/infrastructure.yml

Deployed:

• /opt/stacks/core/docker-compose.yml
• /opt/stacks/core/traefik/traefik.yml
• /opt/stacks/core/traefik/acme.json
• /opt/stacks/infrastructure/infrastructure.yml

Configuration Templates

Wildcard Certificate Template:

services:
  traefik:
    labels:
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.tls.domains[0].main=${DOMAIN}"
      - "traefik.http.routers.traefik.tls.domains[0].sans=*.${DOMAIN}"
  
  any-other-service:
    labels:
      - "traefik.http.routers.service.tls=true"  # No certresolver!

Future Recommendations

Short-term (Next Week)

1. ✅ Monitor certificate auto-renewal (should happen automatically)
2. ✅ Test browser access from different devices to verify SSL
3. ✅ Update homelab documentation with wildcard certificate pattern
4. ⚠️ Consider adding certificate monitoring alerts

Medium-term (Next Month)

1. Set up automated acme.json backups
2. Document certificate troubleshooting runbook
3. Consider migrating to Cloudflare if more services are added
4. Implement certificate expiration monitoring

Long-term (Next Quarter)

1. Evaluate alternative DNS providers for better DNS challenge support
2. Consider setting up staging Let's Encrypt for testing
3. Implement centralized logging for all services
4. Add Prometheus/Grafana monitoring for SSL certificate expiration

Quick Reference

Emergency Certificate Reset

# 1. Stop all services
cd /opt/stacks/core && docker compose down
cd /opt/stacks/infrastructure && docker compose -f infrastructure.yml down

# 2. Reset acme.json
rm /opt/stacks/core/traefik/acme.json
touch /opt/stacks/core/traefik/acme.json
chmod 600 /opt/stacks/core/traefik/acme.json

# 3. Wait for DNS to clear
sleep 60

# 4. Restart
cd /opt/stacks/core && docker compose up -d
cd /opt/stacks/infrastructure && docker compose -f infrastructure.yml up -d

# 5. Monitor
docker exec traefik tail -f /var/log/traefik/traefik.log

Verify Certificate Command

echo | openssl s_client -connect ${SUBDOMAIN}.kelin-hass.duckdns.org:443 -servername ${SUBDOMAIN}.kelin-hass.duckdns.org 2>/dev/null | openssl x509 -noout -subject -issuer -dates

Check All Service Certificates

for subdomain in auth traefik dockge dozzle glances pihole; do
  echo "=== $subdomain.kelin-hass.duckdns.org ==="
  echo | openssl s_client -connect $subdomain.kelin-hass.duckdns.org:443 -servername $subdomain.kelin-hass.duckdns.org 2>/dev/null | openssl x509 -noout -subject -issuer
  echo
done

---

Summary

Successfully implemented wildcard SSL certificate for all homelab services using Let's Encrypt DNS challenge via DuckDNS. Key success factor was recognizing DuckDNS's limitation of one TXT record at a time and configuring Traefik to request a single wildcard certificate instead of individual certificates per service. All services now accessible via HTTPS with valid certificates.

Status: ✅ Production Ready
Next Review: 30 days before certificate expiration (March 13, 2026)