EZ-Homelab/docs/Ondemand-Remote-Services.md

On Demand Remote Services with Authelia, Sablier & Traefik

Overview

This guide explains how to set up lazy-loading services on remote servers (like Raspberry Pi) that start automatically when accessed via Traefik. The core server runs Sablier, which connects to remote Docker daemons via TLS to manage container lifecycle.

Prerequisites

• Core server with Traefik, Authelia, and Sablier deployed
• Remote server with Docker installed
• Shared TLS CA configured between core and remote servers

Automated Setup

For new remote servers, use the automated script:

1. On the remote server, run ez-homelab.sh and select option 3 (Infrastructure Only)
2. When prompted, enter the core server IP for shared TLS CA
3. The script will automatically:
   • Copy shared CA from core server via SSH
   • Configure Docker TLS with shared certificates
   • Generate server certificates signed by shared CA
   • Set up Docker daemon for TLS on port 2376

Important: The script will fail if it cannot copy the shared CA from the core server. Ensure SSH access is configured between servers before running option 3.

Manual Setup (if automated fails)

If the automated setup fails, manually configure TLS:

On Core Server:

# Generate server certificates for remote server
cd /opt/stacks/core/shared-ca
openssl genrsa -out server-key.pem 4096
openssl req -subj "/CN=<REMOTE_IP>" -new -key server-key.pem -out server.csr
echo "subjectAltName = DNS:<REMOTE_IP>,IP:<REMOTE_IP>,IP:127.0.0.1" > extfile.cnf
openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf

On Remote Server:

# Copy certificates
scp user@core-server:/opt/stacks/core/shared-ca/ca.pem /opt/stacks/core/shared-ca/
scp user@core-server:/opt/stacks/core/shared-ca/server-cert.pem /opt/stacks/core/shared-ca/
scp user@core-server:/opt/stacks/core/shared-ca/server-key.pem /opt/stacks/core/shared-ca/

# Update Docker daemon
sudo tee /etc/docker/daemon.json > /dev/null <<EOF
{
  "tls": true,
  "tlsverify": true,
  "tlscacert": "/opt/stacks/core/shared-ca/ca.pem",
  "tlscert": "/opt/stacks/core/shared-ca/server-cert.pem",
  "tlskey": "/opt/stacks/core/shared-ca/server-key.pem"
}
EOF

sudo systemctl restart docker

4 Step Process for Adding Services

1. Add route & service in Traefik external hosts file
2. Add middleware in Sablier config file (sablier.yml)
3. Add labels to compose files on Remote Host
4. Restart services

Required Information

<server> - the hostname of the remote server

<service> - the application/container name

<full domain> - the base url for your proxy host (my-subdomain.duckdns.org)

<ip address> - the ip address of the remote server

<port> - the external port exposed by the service

<service display name> - how it will appear on the now loading page

<group name> - use <service name> for a single service, or something descriptive for the group of services that will start together.
  

Step 1: Add route & service in Traefik external hosts file

In /opt/stacks/core/traefik/dynamic/external-host-server_name.yml

http:
    routers:
        # Add a section under routers for each Route (Proxy Host)
        <service>-<server>:
            rule: "Host(`<service>.<full domain>`)"
            entryPoints:
                - websecure
            service: <service>-<server>
            tls:
                certResolver: letsencrypt
            middlewares:
                - sablier-<server>-<service>@file
                - authelia@docker   # comment this line to disable SSO login

        # next route goes here

# All Routes go above this line
# Services section defines each service used above
services:
    <service>-<server>:
    loadBalancer:
        servers:
        - url: "http://<ip address>:<port>"
        passHostHeader: true
    
    # next service goes here

Step 2: Add middlware to sablier config file

In /opt/stacks/core/traefik/dynamic/sablier.yml

http:
    middlwares:
        # Add a section under middlewares for each Route (Proxy Host)
        sablier-<server>-<service>:
            plugin:
                sablier:
                sablierUrl: http://sablier-service:10000
                group: <server>-<group name>
                sessionDuration: 2m     # Increase this for convience
                ignoreUserAgent: curl   # Don't wake the service for a curl command
                dynamic:
                    displayName: <service display name>
                    theme: ghost        # This can be changed
                    show-details-by-default: true   # May want to disable for production

        # Next middleware goes here

Step 3: Add labels to compose files on Remote Host

On the Remote Server

Apply lables to the services in the compose files

    labels:
      - sablier.enable=true
      - sablier.group=<server>-<group name>
      - sablier.start-on-demand=true

Note

:
Traefik & Authelia labels are not used in the compose file for Remote Hosts

Step 4: Restart services

On host system

docker restart traefik
docker restart sablier-service

On the Remote Host

cd /opt/stacks/<service>
docker compose down && docker compose up -d
docker stop <service>

Setup Complete

Access your service by the proxy url.

---

Deployment Plan for Multi-Server Setup

This section provides a complete deployment plan for scenarios where the core infrastructure (Traefik, Authelia, Sablier) runs on one server, and application services run on remote servers. This setup enables centralized control and routing while maintaining service isolation.

Architecture Overview

• Core Server: Hosts Traefik (reverse proxy), Authelia (SSO), Sablier (lazy loading controller)
• Remote/Media Servers: Host application containers controlled by Sablier
• Communication: TLS-secured Docker API calls between servers

Prerequisites

• Both servers must be on the same network and able to communicate
• SSH access configured between servers (passwordless recommended for automation)
• Domain configured with DuckDNS or similar
• The EZ-Homelab script handles most Docker TLS and certificate setup automatically
• Basic understanding of Docker concepts (optional - script guides you through setup)

Step 1: Configure Docker TLS on All Servers

On Each Server (Core and Remote)

1. Install Docker (if not already installed):

   curl -fsSL https://get.docker.com | sh
   usermod -aG docker $USER
   systemctl enable docker
   systemctl start docker
   # Log out and back in for group changes
   

2. Generate TLS Certificates:

   mkdir -p ~/EZ-Homelab/docker-tls
   cd ~/EZ-Homelab/docker-tls
   
   # Generate CA
   openssl genrsa -out ca-key.pem 4096
   openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=US/ST=State/L=City/O=Organization/CN=Docker-CA"
   
   # Generate server key and cert (replace SERVER_IP with actual IP)
   openssl genrsa -out server-key.pem 4096
   openssl req -subj "/CN=<SERVER_IP>" -new -key server-key.pem -out server.csr
   echo "subjectAltName = DNS:<SERVER_IP>,IP:<SERVER_IP>,IP:127.0.0.1" > extfile.cnf
   openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf
   
   # Generate client key and cert
   openssl genrsa -out client-key.pem 4096
   openssl req -subj "/CN=client" -new -key client-key.pem -out client.csr
   openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem
   

3. Configure Docker Daemon: Create /etc/docker/daemon.json:

   {
     "tls": true,
     "tlsverify": true,
     "tlscacert": "/home/$USER/EZ-Homelab/docker-tls/ca.pem",
     "tlscert": "/home/$USER/EZ-Homelab/docker-tls/server-cert.pem",
     "tlskey": "/home/$USER/EZ-Homelab/docker-tls/server-key.pem"
   }
   

4. Update Systemd Service:

   sudo sed -i 's|-H fd://|-H fd:// -H tcp://0.0.0.0:2376|' /lib/systemd/system/docker.service
   sudo systemctl daemon-reload
   sudo systemctl restart docker
   

5. Configure Firewall:

   sudo ufw allow 2376/tcp
   sudo ufw --force enable
   

Certificate and Secret Sharing

The EZ-Homelab script automatically handles certificate and secret sharing for infrastructure-only deployments:

Automatic Process (Recommended)

1. On Remote Server: Run ./scripts/ez-homelab.sh and select option 3
2. Script Actions:
   • Prompts for core server IP
   • Tests SSH connectivity
   • Copies Docker TLS certificates for remote control
   • Sets up certificates in the correct location

Manual Process (Fallback)

If automatic sharing fails, manually share certificates:

1. On Core Server:

   # Copy client certificates to remote server
   scp /opt/stacks/core/docker-tls/ca.pem /opt/stacks/core/docker-tls/client-cert.pem /opt/stacks/core/docker-tls/client-key.pem user@remote-server:/opt/stacks/infrastructure/docker-tls/
   

2. On Remote Server:

   # Ensure certificates are in the correct location
   ls -la /opt/stacks/infrastructure/docker-tls/
   # Should contain: ca.pem, client-cert.pem, client-key.pem
   

Step 3: Deploy Core Infrastructure

On Core Server

1. Run the EZ-Homelab script with core deployment:

   cd ~/EZ-Homelab
   ./scripts/ez-homelab.sh
   # Select option 1 (Default Setup) or 2 (Core Only)
   

   The script will:

   • Generate Authelia secrets automatically
   • Configure TLS for Docker API
   • Deploy Traefik, Authelia, and Sablier
   • Set up certificates for secure communication

2. Verify core deployment:

   # Check services are running
   docker ps --filter "label=com.docker.compose.project=core"
   
   # Test Authelia access
   curl -k https://auth.<your-domain>
   

Step 4: Deploy Remote Infrastructure

On Remote/Media Server

1. Run the EZ-Homelab script with infrastructure-only deployment:

   cd ~/EZ-Homelab
   ./scripts/ez-homelab.sh
   # Select option 3 (Infrastructure Only)
   

   The script will automatically:

   • Prompt for core server IP address
   • Establish SSH connection to core server
   • Copy Authelia secrets and TLS certificates
   • Configure Docker TLS for remote control
   • Set up required networks and directories

2. Manual certificate sharing (if automatic fails): If SSH connection fails, manually copy certificates:

   # On core server, copy certs to remote server
   scp /opt/stacks/core/docker-tls/ca.pem /opt/stacks/core/docker-tls/client-cert.pem /opt/stacks/core/docker-tls/client-key.pem user@remote-server:/opt/stacks/infrastructure/docker-tls/
   
   # On remote server, copy Authelia secrets
   scp /home/kelin/EZ-Homelab/.env user@remote-server:/home/kelin/EZ-Homelab/.env.core
   

Step 5: Configure Sablier for Remote Control

On Core Server

Update Sablier configuration to control remote servers:

1. Edit core docker-compose.yml:

   sablier-service:
     environment:
       - DOCKER_HOST=tcp://<REMOTE_SERVER_IP>:2376
       - DOCKER_TLS_VERIFY=1
       - DOCKER_CERT_PATH=/certs
     volumes:
       - ./docker-tls/ca.pem:/certs/ca.pem:ro
       - ./docker-tls/client-cert.pem:/certs/cert.pem:ro
       - ./docker-tls/client-key.pem:/certs/key.pem:ro
   

2. Restart core stack:

   cd /opt/stacks/core
   docker compose down
   docker compose up -d
   

Step 6: Deploy Application Services

On Remote Server

1. Deploy application stacks with Sablier labels:

   # Example: /opt/stacks/media-management/docker-compose.yml
   services:
     sonarr:
       labels:
         - sablier.enable=true
         - sablier.group=<REMOTE_HOSTNAME>-media
         - sablier.start-on-demand=true
   

2. Deploy and stop services for lazy loading:

   cd /opt/stacks/media-management
   docker compose up -d
   docker compose stop
   

Step 5: Configure Traefik Routing

On Core Server

Since Traefik cannot auto-discover labels from remote Docker hosts, use the file provider method:

1. Create external host configuration: /opt/stacks/core/traefik/dynamic/external-host-<remote_server>.yml

   http:
     routers:
       sonarr-remote:
         rule: "Host(`sonarr.<DOMAIN>`)"
         entrypoints:
           - websecure
         service: sonarr-remote
         tls:
           certResolver: letsencrypt
         middlewares:
           - sablier-<remote_hostname>-arr@file
           - authelia@docker
   
     services:
       sonarr-remote:
         loadBalancer:
           servers:
             - url: "http://<REMOTE_IP>:8989"
           passHostHeader: true
   

2. Create Sablier middleware configuration: /opt/stacks/core/traefik/dynamic/sablier.yml

   http:
     middlewares:
       sablier-<remote_hostname>-arr:
         plugin:
           sablier:
             sablierUrl: http://sablier-service:10000
             group: <remote_hostname>-arr
             sessionDuration: 2m
             ignoreUserAgent: curl
             dynamic:
               displayName: "Media Management Services"
               theme: ghost
               show-details-by-default: true
   

3. Restart Traefik:

   docker restart traefik
   

Step 6: Verification and Testing

1. Check Sablier connection:

   # On core server
   docker logs sablier-service
   # Should show groups from remote server
   

2. Test lazy loading:

   • Access https://sonarr.<DOMAIN>
   • Should show Sablier loading page
   • Container should start on remote server

3. Verify Traefik routes:

   curl -k https://localhost:8080/api/http/routers | jq
   

Troubleshooting

• TLS Connection Issues: Check certificate validity and paths
• Sablier Not Detecting Groups: Verify DOCKER_HOST and certificates
• Traefik Routing Problems: Check external host YAML syntax
• Network Connectivity: Ensure ports 2376, 80, 443 are open between servers

Security Considerations

• TLS certificates expire after 365 days - monitor and renew
• Limit Docker API access to trusted networks
• Use strong firewall rules
• Regularly update all components

This setup provides centralized management with distributed execution, optimal for resource management and security.