Merge branch 'main' of https://github.com/kelinfoxy/EZ-Homelab

2026-01-26 23:45:56 -05:00
parent 916e970994 ea91151829
commit ee93f26d52
17 changed files with 1137 additions and 278 deletions
--- a/docs/Ondemand-Remote-Services.md
+++ b/docs/Ondemand-Remote-Services.md
@@ -1,6 +1,66 @@
 # On Demand Remote Services with Authelia, Sablier & Traefik

-## 4 Step Process
+## Overview
+
+This guide explains how to set up lazy-loading services on remote servers (like Raspberry Pi) that start automatically when accessed via Traefik. The core server runs Sablier, which connects to remote Docker daemons via TLS to manage container lifecycle.
+
+## Prerequisites
+
+- Core server with Traefik, Authelia, and Sablier deployed
+- Remote server with Docker installed
+- Shared TLS CA configured between core and remote servers
+
+## Automated Setup
+
+For new remote servers, use the automated script:
+
+1. On the remote server, run `ez-homelab.sh` and select option 3 (Infrastructure Only)
+2. When prompted, enter the core server IP for shared TLS CA
+3. The script will automatically:
+   - Copy shared CA from core server via SSH
+   - Configure Docker TLS with shared certificates
+   - Generate server certificates signed by shared CA
+   - Set up Docker daemon for TLS on port 2376
+
+**Important**: The script will fail if it cannot copy the shared CA from the core server. Ensure SSH access is configured between servers before running option 3.
+
+## Manual Setup (if automated fails)
+
+If the automated setup fails, manually configure TLS:
+
+### On Core Server:
+```bash
+# Generate server certificates for remote server
+cd /opt/stacks/core/shared-ca
+openssl genrsa -out server-key.pem 4096
+openssl req -subj "/CN=<REMOTE_IP>" -new -key server-key.pem -out server.csr
+echo "subjectAltName = DNS:<REMOTE_IP>,IP:<REMOTE_IP>,IP:127.0.0.1" > extfile.cnf
+openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf
+```
+
+### On Remote Server:
+```bash
+# Copy certificates
+scp user@core-server:/opt/stacks/core/shared-ca/ca.pem /opt/stacks/core/shared-ca/
+scp user@core-server:/opt/stacks/core/shared-ca/server-cert.pem /opt/stacks/core/shared-ca/
+scp user@core-server:/opt/stacks/core/shared-ca/server-key.pem /opt/stacks/core/shared-ca/
+
+# Update Docker daemon
+sudo tee /etc/docker/daemon.json > /dev/null <<EOF
+{
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/opt/stacks/core/shared-ca/ca.pem",
+  "tlscert": "/opt/stacks/core/shared-ca/server-cert.pem",
+  "tlskey": "/opt/stacks/core/shared-ca/server-key.pem"
+}
+EOF
+
+sudo systemctl restart docker
+```
+
+## 4 Step Process for Adding Services
+
 1. Add route & service in Traefik external hosts file
 2. Add middleware in Sablier config file (sablier.yml)
 3. Add labels to compose files on Remote Host
@@ -115,3 +175,302 @@ docker stop <service>

 Access your service by the proxy url.

+---
+
+# Deployment Plan for Multi-Server Setup
+
+This section provides a complete deployment plan for scenarios where the core infrastructure (Traefik, Authelia, Sablier) runs on one server, and application services run on remote servers. This setup enables centralized control and routing while maintaining service isolation.
+
+## Architecture Overview
+
+- **Core Server**: Hosts Traefik (reverse proxy), Authelia (SSO), Sablier (lazy loading controller)
+- **Remote/Media Servers**: Host application containers controlled by Sablier
+- **Communication**: TLS-secured Docker API calls between servers
+
+## Prerequisites
+
+- Both servers must be on the same network and able to communicate
+- SSH access configured between servers (key-based or password authentication supported)
+- Domain configured with DuckDNS or similar
+- The EZ-Homelab script handles most Docker TLS and certificate setup automatically
+- Basic understanding of Docker concepts (optional - script guides you through setup)
+
+## Step 1: Configure Docker TLS on All Servers
+
+### On Each Server (Core and Remote)
+
+1. **Install Docker** (if not already installed):
+   ```bash
+   curl -fsSL https://get.docker.com | sh
+   usermod -aG docker $USER
+   systemctl enable docker
+   systemctl start docker
+   # Log out and back in for group changes
+   ```
+
+2. **Generate TLS Certificates**:
+   ```bash
+   mkdir -p ~/EZ-Homelab/docker-tls
+   cd ~/EZ-Homelab/docker-tls
+
+   # Generate CA
+   openssl genrsa -out ca-key.pem 4096
+   openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=US/ST=State/L=City/O=Organization/CN=Docker-CA"
+
+   # Generate server key and cert (replace SERVER_IP with actual IP)
+   openssl genrsa -out server-key.pem 4096
+   openssl req -subj "/CN=<SERVER_IP>" -new -key server-key.pem -out server.csr
+   echo "subjectAltName = DNS:<SERVER_IP>,IP:<SERVER_IP>,IP:127.0.0.1" > extfile.cnf
+   openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf
+
+   # Generate client key and cert
+   openssl genrsa -out client-key.pem 4096
+   openssl req -subj "/CN=client" -new -key client-key.pem -out client.csr
+   openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem
+   ```
+
+3. **Configure Docker Daemon**:
+   Create `/etc/docker/daemon.json`:
+   ```json
+   {
+     "tls": true,
+     "tlsverify": true,
+     "tlscacert": "/home/$USER/EZ-Homelab/docker-tls/ca.pem",
+     "tlscert": "/home/$USER/EZ-Homelab/docker-tls/server-cert.pem",
+     "tlskey": "/home/$USER/EZ-Homelab/docker-tls/server-key.pem"
+   }
+   ```
+
+4. **Update Systemd Service**:
+   ```bash
+   sudo sed -i 's|-H fd://|-H fd:// -H tcp://0.0.0.0:2376|' /lib/systemd/system/docker.service
+   sudo systemctl daemon-reload
+   sudo systemctl restart docker
+   ```
+
+5. **Configure Firewall**:
+   ```bash
+   sudo ufw allow 2376/tcp
+   sudo ufw --force enable
+   ```
+
+## Certificate and Secret Sharing
+
+The EZ-Homelab script automatically handles certificate and secret sharing for infrastructure-only deployments:
+
+### Automatic Process (Recommended)
+
+1. **On Remote Server**: Run `./scripts/ez-homelab.sh` and select option 3
+2. **Script Actions**:
+   - Prompts for core server IP
+   - Tests SSH connectivity
+   - Copies Docker TLS certificates for remote control
+   - Sets up certificates in the correct location
+
+### Manual Process (Fallback)
+
+If automatic sharing fails, manually share certificates:
+
+1. **On Core Server**:
+   ```bash
+   # Copy client certificates to remote server
+   scp /opt/stacks/core/docker-tls/ca.pem /opt/stacks/core/docker-tls/client-cert.pem /opt/stacks/core/docker-tls/client-key.pem user@remote-server:/opt/stacks/infrastructure/docker-tls/
+   ```
+
+2. **On Remote Server**:
+   ```bash
+   # Ensure certificates are in the correct location
+   ls -la /opt/stacks/infrastructure/docker-tls/
+   # Should contain: ca.pem, client-cert.pem, client-key.pem
+   ```
+
+## Step 3: Deploy Core Infrastructure
+
+### On Core Server
+
+1. **Run the EZ-Homelab script** with core deployment:
+   ```bash
+   cd ~/EZ-Homelab
+   ./scripts/ez-homelab.sh
+   # Select option 1 (Default Setup) or 2 (Core Only)
+   ```
+
+   The script will:
+   - Generate Authelia secrets automatically
+   - Configure TLS for Docker API
+   - Deploy Traefik, Authelia, and Sablier
+   - Set up certificates for secure communication
+
+2. **Verify core deployment**:
+   ```bash
+   # Check services are running
+   docker ps --filter "label=com.docker.compose.project=core"
+   
+   # Test Authelia access
+   curl -k https://auth.<your-domain>
+   ```
+
+## Step 4: Deploy Remote Infrastructure
+
+### On Remote/Media Server
+
+1. **Run the EZ-Homelab script** with infrastructure-only deployment:
+   ```bash
+   cd ~/EZ-Homelab
+   ./scripts/ez-homelab.sh
+   # Select option 3 (Infrastructure Only)
+   ```
+
+   The script will automatically:
+   - Prompt for core server IP address
+   - Establish SSH connection to core server
+   - Copy Authelia secrets and TLS certificates
+   - Configure Docker TLS for remote control
+   - Set up required networks and directories
+
+2. **Manual certificate sharing** (if automatic fails):
+   If SSH connection fails, manually copy certificates:
+   ```bash
+   # On core server, copy certs to remote server
+   scp /opt/stacks/core/docker-tls/ca.pem /opt/stacks/core/docker-tls/client-cert.pem /opt/stacks/core/docker-tls/client-key.pem user@remote-server:/opt/stacks/infrastructure/docker-tls/
+   
+   # On remote server, copy Authelia secrets
+   scp /home/kelin/EZ-Homelab/.env user@remote-server:/home/kelin/EZ-Homelab/.env.core
+   ```
+
+## Step 5: Configure Sablier for Remote Control
+
+### On Core Server
+
+Update Sablier configuration to control remote servers:
+
+1. **Edit core docker-compose.yml**:
+   ```yaml
+   sablier-service:
+     environment:
+       - DOCKER_HOST=tcp://<REMOTE_SERVER_IP>:2376
+       - DOCKER_TLS_VERIFY=1
+       - DOCKER_CERT_PATH=/certs
+     volumes:
+       - ./docker-tls/ca.pem:/certs/ca.pem:ro
+       - ./docker-tls/client-cert.pem:/certs/cert.pem:ro
+       - ./docker-tls/client-key.pem:/certs/key.pem:ro
+   ```
+
+2. **Restart core stack**:
+   ```bash
+   cd /opt/stacks/core
+   docker compose down
+   docker compose up -d
+   ```
+
+## Step 6: Deploy Application Services
+
+### On Remote Server
+
+1. **Deploy application stacks** with Sablier labels:
+   ```yaml
+   # Example: /opt/stacks/media-management/docker-compose.yml
+   services:
+     sonarr:
+       labels:
+         - sablier.enable=true
+         - sablier.group=<REMOTE_HOSTNAME>-media
+         - sablier.start-on-demand=true
+   ```
+
+2. **Deploy and stop services** for lazy loading:
+   ```bash
+   cd /opt/stacks/media-management
+   docker compose up -d
+   docker compose stop
+   ```
+
+## Step 5: Configure Traefik Routing
+
+### On Core Server
+
+Since Traefik cannot auto-discover labels from remote Docker hosts, use the file provider method:
+
+1. **Create external host configuration**:
+   `/opt/stacks/core/traefik/dynamic/external-host-<remote_server>.yml`
+   ```yaml
+   http:
+     routers:
+       sonarr-remote:
+         rule: "Host(`sonarr.<DOMAIN>`)"
+         entrypoints:
+           - websecure
+         service: sonarr-remote
+         tls:
+           certResolver: letsencrypt
+         middlewares:
+           - sablier-<remote_hostname>-arr@file
+           - authelia@docker
+
+     services:
+       sonarr-remote:
+         loadBalancer:
+           servers:
+             - url: "http://<REMOTE_IP>:8989"
+           passHostHeader: true
+   ```
+
+2. **Create Sablier middleware configuration**:
+   `/opt/stacks/core/traefik/dynamic/sablier.yml`
+   ```yaml
+   http:
+     middlewares:
+       sablier-<remote_hostname>-arr:
+         plugin:
+           sablier:
+             sablierUrl: http://sablier-service:10000
+             group: <remote_hostname>-arr
+             sessionDuration: 2m
+             ignoreUserAgent: curl
+             dynamic:
+               displayName: "Media Management Services"
+               theme: ghost
+               show-details-by-default: true
+   ```
+
+3. **Restart Traefik**:
+   ```bash
+   docker restart traefik
+   ```
+
+## Step 6: Verification and Testing
+
+1. **Check Sablier connection**:
+   ```bash
+   # On core server
+   docker logs sablier-service
+   # Should show groups from remote server
+   ```
+
+2. **Test lazy loading**:
+   - Access `https://sonarr.<DOMAIN>`
+   - Should show Sablier loading page
+   - Container should start on remote server
+
+3. **Verify Traefik routes**:
+   ```bash
+   curl -k https://localhost:8080/api/http/routers | jq
+   ```
+
+## Troubleshooting
+
+- **TLS Connection Issues**: Check certificate validity and paths
+- **Sablier Not Detecting Groups**: Verify DOCKER_HOST and certificates
+- **Traefik Routing Problems**: Check external host YAML syntax
+- **Network Connectivity**: Ensure ports 2376, 80, 443 are open between servers
+
+## Security Considerations
+
+- TLS certificates expire after 365 days - monitor and renew
+- Limit Docker API access to trusted networks
+- Use strong firewall rules
+- Regularly update all components
+
+This setup provides centralized management with distributed execution, optimal for resource management and security.
+
--- a/docs/README.md
+++ b/docs/README.md
@@ -2,6 +2,42 @@

 Welcome to the AI-Homelab documentation! This is your comprehensive guide to deploying and managing a production-ready homelab infrastructure with 50+ pre-configured services.

+# Mission Statement
+
+>  ### Provide a fast, convenient, and free way to deploy homelab servers with popular features  
+> *  DuckDNS subdomains with LetsEncrypt SSL wildcard certificates  
+> *  Security First - Secure Single Sign On with Authelia  
+> *  Traefik - Proxy Host for an entire homelab  
+> *  Sablier - Lazy Loading of services  
+> *  TLS Certificates for docker proxy  
+>&nbsp;
+
+# Deployment Scenarios 
+
+1. **Single server**
+
+    **Select Option 1**
+    * Use Dockge to start the desired stacks
+    * Use Homepage to explore your services
+
+
+2. **Core Server + Remote Server**
+
+    **Core Server**
+    * A small low power device, like a Raspberry Pi 4 4GB.
+    * Deploy core stack. (Authelia, DuckDNS, Traefik, Sablier, Homepage) 
+    * Functions as a gateway between the internet and your servers.  
+    * Configure a remote host file for each Remote Server.  
+    * Configure sablier.yml for services to lazyload.  
+    * Generate TLS certificate, all servers use the same TLS certificate for dockerproxy
+
+    **Remote Server(s)** 
+    * Your old office or gaming PC, SBCs, second hand PCs, etc
+    * Select Option 3 to install Infrastructure only
+    * Ensure the TLS certificate was copied correctly and docker daemon configured
+    * Use Dockge to start the desired stacks
+    * Use Homepage to explore your services
+
 ## 📚 Documentation Structure

 ### 🚀 Getting Started
--- a/docs/manual-setup.md
+++ b/docs/manual-setup.md
@@ -150,6 +150,63 @@ cd /opt/stacks/dashboards
 docker compose up -d
 ```

+## Step 10.5: Multi-Server TLS Setup (Optional)
+
+If you plan to deploy services on remote servers (like Raspberry Pi) that will be managed by Sablier for lazy loading, set up shared TLS certificates.
+
+### On Core Server (where Traefik/Authelia run):
+
+```bash
+# Create shared CA directory
+sudo mkdir -p /opt/stacks/core/shared-ca
+sudo chown $USER:$USER /opt/stacks/core/shared-ca
+
+# Generate shared CA certificate
+cd /opt/stacks/core/shared-ca
+openssl genrsa -out ca-key.pem 4096
+openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem -subj "/C=US/ST=State/L=City/O=Homelab/CN=Homelab-CA"
+
+# Set proper permissions
+chmod 600 ca-key.pem
+chmod 644 ca.pem
+```
+
+### On Remote Servers:
+
+```bash
+# Create TLS directory
+sudo mkdir -p /opt/stacks/core/shared-ca
+sudo chown $USER:$USER /opt/stacks/core/shared-ca
+
+# Copy shared CA from core server (replace CORE_IP with your core server IP)
+scp user@CORE_IP:/opt/stacks/core/shared-ca/ca.pem /opt/stacks/core/shared-ca/
+scp user@CORE_IP:/opt/stacks/core/shared-ca/ca-key.pem /opt/stacks/core/shared-ca/
+
+# Generate client certificate for Docker client connections
+openssl genrsa -out client-key.pem 4096
+openssl req -subj "/CN=client" -new -key client-key.pem -out client.csr
+openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem
+
+# Configure Docker TLS
+sudo tee /etc/docker/daemon.json > /dev/null <<EOF
+{
+  "tls": true,
+  "tlsverify": true,
+  "tlscacert": "/opt/stacks/core/shared-ca/ca.pem",
+  "tlscert": "/opt/stacks/core/shared-ca/server-cert.pem",
+  "tlskey": "/opt/stacks/core/shared-ca/server-key.pem"
+}
+EOF
+
+# Update Docker service to listen on TLS port
+sudo sed -i 's|-H fd://|-H fd:// -H tcp://0.0.0.0:2376|' /lib/systemd/system/docker.service
+sudo systemctl daemon-reload
+sudo systemctl restart docker
+
+# Test TLS connection from core server
+# On core server, run: docker --tlsverify --tlscacert /opt/stacks/core/shared-ca/ca.pem --tlscert /opt/stacks/core/shared-ca/client-cert.pem --tlskey /opt/stacks/core/shared-ca/client-key.pem -H tcp://REMOTE_IP:2376 ps
+```
+
 ## Step 11: Verify Deployment

 ```bash