Fix: Resolve password hash corruption in Authelia users_database.yml

Critical fix for argon2 password hash preservation:
- Root cause: Bash variable expansion of $ characters in argon2id hashes
- Solution: Write hash directly from Docker output to file, bypass bash variables entirely
- setup-homelab.sh: Stream Docker output directly to /tmp/authelia_password_hash.tmp
- deploy-homelab.sh: Read hash file in Python to avoid any bash expansion
- Result: Password hash correctly preserved with full $argon2id$v=19$m=... format

Other changes:
- Added DOCKER_API_VERSION=1.44 env var for watchtower (API compatibility)
- Watchtower still has issues with Docker 29.1.4 - keeping version pinned for investigation

Tested on Debian 12 with Docker 29.1.4:
✅ All 11 critical containers healthy
✅ Authelia authentication working correctly
✅ Password hash preserved through entire deployment workflow
⚠️  Watchtower restart loop (non-critical, under investigation)

docker-compose/infrastructure.yml

@@ -67,9 +67,9 @@ services:
       - "traefik.http.services.pihole.loadbalancer.server.port=80"
 
   # Watchtower - Automatic container updates
-  # Runs silently in background, no UI
+  # Runs silently in background, no UI  
   watchtower:
-    image: containrrr/watchtower:1.7.1
+    image: containrrr/watchtower:1.7.2
     container_name: watchtower
     restart: unless-stopped
     networks:
@@ -77,6 +77,7 @@ services:
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
     environment:
+      - DOCKER_API_VERSION=1.44
       - WATCHTOWER_CLEANUP=true
       - WATCHTOWER_INCLUDE_RESTARTING=true
       - WATCHTOWER_SCHEDULE=0 0 4 * * *  # 4 AM daily

docs/getting-started.md

@@ -19,29 +19,18 @@ For most users, the automated setup script handles everything:
    ```bash
       sudo apt update && sudo apt upgrade -y && sudo apt install git
 
-4. **Run the setup script**:
+3. **Clone the rep**:
    ```bash
    git clone https://github.com/kelinfoxy/AI-Homelab.git
    cd AI-Homelab
-   sudo ./scripts/setup-homelab.sh
+
-   ```
+4. **Configure environment**:
-5. **Log out and back in** (or run `newgrp docker`)
-6. **Generate Authelia Secrets**:
-   ```bash
-   # Generate three required secrets for Authelia (128 characters each)
-   echo "AUTHELIA_JWT_SECRET=$(openssl rand -hex 64)"
-   echo "AUTHELIA_SESSION_SECRET=$(openssl rand -hex 64)"
-   echo "AUTHELIA_STORAGE_ENCRYPTION_KEY=$(openssl rand -hex 64)"
-   
-   # Copy these values and add them to your .env file
-   ```
-7. **Configure environment**:
    ```bash
    cp .env.example .env
    nano .env  # Edit with your settings and paste the Authelia secrets
    ```
    
-   **IMPORTANT: .env File Location**
+   **Testing considerations: .env File Location**
    - The `.env` file should remain in the **repository folder** (`~/AI-Homelab/.env`)
    - The deploy script will automatically copy it to `/opt/stacks/*/` as needed
    - Always edit the repo copy, not the deployed copies
@@ -56,6 +45,23 @@ For most users, the automated setup script handles everything:
    - `AUTHELIA_STORAGE_ENCRYPTION_KEY` - Generated in step 6
    - `SURFSHARK_USERNAME` and `SURFSHARK_PASSWORD` - If using VPN
 
+5. **Run the setup script**   
+    ```bash
+      sudo ./scripts/setup-homelab.sh
+   
+6. **Log out and back in** (or run `newgrp docker`)
+   >Don't skip this step!
+
+7. **Generate Authelia Secrets**:
+   ```bash
+   # Generate three required secrets for Authelia (128 characters each)
+   echo "AUTHELIA_JWT_SECRET=$(openssl rand -hex 64)"
+   echo "AUTHELIA_SESSION_SECRET=$(openssl rand -hex 64)"
+   echo "AUTHELIA_STORAGE_ENCRYPTION_KEY=$(openssl rand -hex 64)"
+   
+   # Copy these values and add them to your .env file
+   ```
+
 8. **Deploy homelab**:
    ```bash
    ./scripts/deploy-homelab.sh

scripts/deploy-homelab.sh

@@ -179,11 +179,11 @@ log_info "Configuring Authelia for domain: $DOMAIN..."
 sed -i "s/your-domain.duckdns.org/${DOMAIN}/g" /opt/stacks/core/authelia/configuration.yml
 
 # Configure Authelia admin user from setup script
-if [ -f /tmp/authelia_admin_credentials.tmp ]; then
+if [ -f /tmp/authelia_admin_credentials.tmp ] && [ -f /tmp/authelia_password_hash.tmp ]; then
     log_info "Loading Authelia admin credentials from setup script..."
     source /tmp/authelia_admin_credentials.tmp
     
-    if [ -n "$PASSWORD_HASH" ] && [ -n "$ADMIN_USER" ] && [ -n "$ADMIN_EMAIL" ]; then
+    if [ -n "$ADMIN_USER" ] && [ -n "$ADMIN_EMAIL" ]; then
         log_success "Using credentials: $ADMIN_USER ($ADMIN_EMAIL)"
         
         # Create users_database.yml with credentials from setup
@@ -203,13 +203,36 @@ users:
       - users
 EOF
         # Now safely replace placeholders
-        # Escape special characters in PASSWORD_HASH for sed
+        # Read hash from file (not bash variable) to avoid shell expansion
-        ESCAPED_HASH=$(printf '%s\n' "$PASSWORD_HASH" | sed 's:[\/&]:\\&:g;$!s/$/\\/')
+        # The hash file was written directly from Docker output in setup script
-        ESCAPED_HASH="${ESCAPED_HASH%\\}"
+        export ADMIN_USER
-        
+        export ADMIN_EMAIL
-        sed -i "s/ADMIN_USER_PLACEHOLDER/${ADMIN_USER}/g" /opt/stacks/core/authelia/users_database.yml
+        python3 << 'PYTHON_EOF'
-        sed -i "s|PASSWORD_HASH_PLACEHOLDER|${ESCAPED_HASH}|g" /opt/stacks/core/authelia/users_database.yml  
+# Read password hash from file to completely avoid bash variable expansion
-        sed -i "s/ADMIN_EMAIL_PLACEHOLDER/${ADMIN_EMAIL}/g" /opt/stacks/core/authelia/users_database.yml
+with open('/tmp/authelia_password_hash.tmp', 'r') as f:
+    password_hash = f.read().strip()
+
+import os
+admin_user = os.environ['ADMIN_USER']
+admin_email = os.environ['ADMIN_EMAIL']
+
+content = f"""###############################################################
+#                         Users Database                      #
+###############################################################
+
+users:
+  {admin_user}:
+    displayname: "Admin User"
+    password: "{password_hash}"
+    email: {admin_email}
+    groups:
+      - admins
+      - users
+"""
+
+with open('/opt/stacks/core/authelia/users_database.yml', 'w') as f:
+    f.write(content)
+PYTHON_EOF
         
         log_success "Authelia admin user configured from setup script"
         echo ""

scripts/setup-homelab.sh

@@ -313,8 +313,11 @@ fi
 echo ""
 log_info "Generating password hash..."
 
-# Generate hash with timeout and better error capture
+# Generate hash and write DIRECTLY to file to avoid bash variable expansion of $ characters
-HASH_OUTPUT=$(timeout 60 docker run --rm authelia/authelia:4.37 authelia crypto hash generate argon2 --password "$ADMIN_PASSWORD" 2>&1)
+# The argon2 hash contains multiple $ characters that bash would try to expand as variables
+timeout 60 docker run --rm authelia/authelia:4.37 authelia crypto hash generate argon2 --password "$ADMIN_PASSWORD" 2>&1 | \
+    grep -oP 'Digest: \K\$argon2.*' > /tmp/authelia_password_hash.tmp
+
 HASH_EXIT_CODE=$?
 
 if [ $HASH_EXIT_CODE -eq 124 ]; then
@@ -324,27 +327,15 @@ if [ $HASH_EXIT_CODE -eq 124 ]; then
     log_info "  2. Docker status: docker ps"
     log_info "  3. Try manually: docker run --rm authelia/authelia:4.37 authelia crypto hash generate argon2"
     exit 1
-elif [ $HASH_EXIT_CODE -ne 0 ]; then
+elif [ $HASH_EXIT_CODE -ne 0 ] || [ ! -s /tmp/authelia_password_hash.tmp ]; then
-    log_error "Failed to generate password hash (exit code: $HASH_EXIT_CODE)"
+    log_error "Failed to generate password hash"
-    log_info "Error output:"
-    echo "$HASH_OUTPUT"
-    exit 1
-fi
-
-# Extract hash - format is "Digest: $argon2id$..."
-PASSWORD_HASH=$(echo "$HASH_OUTPUT" | grep -oP 'Digest: \K\$argon2.*' || echo "$HASH_OUTPUT" | grep '^\$argon2')
-
-if [ -z "$PASSWORD_HASH" ]; then
-    log_error "Failed to extract password hash from output"
-    log_info "Command output:"
-    echo "$HASH_OUTPUT"
-    log_info ""
     log_info "You can generate the hash manually after setup:"
     log_info "  docker run --rm authelia/authelia:4.37 authelia crypto hash generate argon2"
     log_info "  Then edit: /opt/stacks/core/authelia/users_database.yml"
     exit 1
 fi
 
+chmod 600 /tmp/authelia_password_hash.tmp
 log_success "Password hash generated successfully"
 
 # Read admin email from .env or prompt
@@ -358,13 +349,13 @@ log_success "Admin user configured: $ADMIN_USER"
 log_success "Password hash generated and will be applied during deployment"
 
 # Store the admin credentials for the deployment script
-# Include both password and hash so deploy can show the password to user
+# Password hash is already in /tmp/authelia_password_hash.tmp (written directly from Docker)
-cat > /tmp/authelia_admin_credentials.tmp << EOF
+# This avoids bash variable expansion issues with $ characters in argon2 hashes
-ADMIN_USER=$ADMIN_USER
+{
-ADMIN_EMAIL=$ADMIN_EMAIL
+    echo "ADMIN_USER=$ADMIN_USER"
-ADMIN_PASSWORD=$ADMIN_PASSWORD
+    echo "ADMIN_EMAIL=$ADMIN_EMAIL"
-PASSWORD_HASH=$PASSWORD_HASH
+    echo "ADMIN_PASSWORD=$ADMIN_PASSWORD"
-EOF
+} > /tmp/authelia_admin_credentials.tmp
 chmod 600 /tmp/authelia_admin_credentials.tmp
 
 log_info "Credentials saved for deployment script"