EZ-Homelab/config-templates/dokuwiki/data/pages/services/infrastructure/watchtower.txt

====== Watchtower ======

Watchtower is an automated container update service that monitors running Docker containers and automatically updates them when new images are available. It ensures your homelab services stay up-to-date with the latest security patches and features.

===== Overview =====

**Purpose:** Automated container updates
**Deployment:** Infrastructure stack (currently disabled)
**Monitoring:** Passive background service
**Update Strategy:** Rolling updates with health checks

===== Key Features =====

**Automated Updates:**
  * **Image Monitoring**: Checks for new image versions
  * **Scheduled Updates**: Configurable update intervals
  * **Rolling Updates**: Updates containers one by one
  * **Health Checks**: Waits for container health before proceeding

**Update Control:**
  * **Include/Exclude**: Control which containers to update
  * **Update Notifications**: Email/Slack notifications
  * **Rollback Support**: Revert to previous versions
  * **Dry Run Mode**: Test updates without applying

**Safety Features:**
  * **Health Monitoring**: Ensures containers start successfully
  * **Timeout Control**: Prevents hanging updates
  * **Resource Limits**: Controls update resource usage
  * **Backup Integration**: Coordinates with backup services

===== Configuration =====

**Container Configuration:**
```yaml
services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_POLL_INTERVAL=3600
      - WATCHTOWER_TIMEOUT=30s
      - WATCHTOWER_NOTIFICATIONS=shoutrrr
      - WATCHTOWER_NOTIFICATION_URL=discord://token@webhook
      - WATCHTOWER_INCLUDE_STOPPED=false
      - WATCHTOWER_REVIVE_STOPPED=false
      - WATCHTOWER_REMOVE_VOLUMES=false
      - WATCHTOWER_LABEL_ENABLE=true
      - WATCHTOWER_MONITOR_ONLY=false
      - WATCHTOWER_RUN_ONCE=false
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command: --interval 3600 --cleanup --label-enable
    deploy:
      resources:
        limits:
          cpus: '0.2'
          memory: 64M
        reservations:
          cpus: '0.01'
          memory: 16M
```

**Environment Variables:**
```bash
# Update interval (seconds)
WATCHTOWER_POLL_INTERVAL=3600

# Update timeout
WATCHTOWER_TIMEOUT=30s

# Cleanup old images
WATCHTOWER_CLEANUP=true

# Notification settings
WATCHTOWER_NOTIFICATIONS=shoutrrr
WATCHTOWER_NOTIFICATION_URL=discord://token@webhook

# Container control
WATCHTOWER_INCLUDE_STOPPED=false
WATCHTOWER_REVIVE_STOPPED=false
WATCHTOWER_REMOVE_VOLUMES=false

# Label-based control
WATCHTOWER_LABEL_ENABLE=true

# Monitoring mode
WATCHTOWER_MONITOR_ONLY=false

# One-time run
WATCHTOWER_RUN_ONCE=false
```

===== Update Process =====

**Monitoring Phase:**
  1. **Image Check**: Queries Docker registry for new versions
  2. **Version Comparison**: Compares current vs latest versions
  3. **Update Decision**: Determines if update is needed
  4. **Schedule Planning**: Plans update timing

**Update Execution:**
  1. **Container Stop**: Gracefully stops current container
  2. **Image Pull**: Downloads new image version
  3. **Container Start**: Starts container with new image
  4. **Health Check**: Verifies container health
  5. **Cleanup**: Removes old images (if enabled)

**Post-Update:**
  * **Notification**: Sends update notifications
  * **Logging**: Records update details
  * **Monitoring**: Continues monitoring for next updates
  * **Error Handling**: Handles update failures

===== Container Selection =====

**Label-Based Control:**
```yaml
# Enable updates for specific containers
labels:
  - "com.centurylinklabs.watchtower.enable=true"

# Disable updates for specific containers
labels:
  - "com.centurylinklabs.watchtower.enable=false"
```

**Include/Exclude Patterns:**
```bash
# Include only specific containers
command: --include=traefik,authelia,dockge

# Exclude specific containers
command: --exclude=plex,jellyfin

# Use regex patterns
command: --include="^media-.*"
```

**Scope Control:**
  * **All Containers**: Update all running containers
  * **Specific Services**: Update only selected services
  * **Stack-Based**: Update containers in specific stacks
  * **Label-Based**: Use Docker labels for control

===== Notification System =====

**Supported Notifications:**
  * **Email**: SMTP email notifications
  * **Slack**: Slack channel notifications
  * **Discord**: Discord webhook notifications
  * **Gotify**: Gotify push notifications
  * **Telegram**: Telegram bot notifications

**Notification Configuration:**
```yaml
environment:
  - WATCHTOWER_NOTIFICATIONS=shoutrrr
  - WATCHTOWER_NOTIFICATION_URL=slack://token@channel
  # Or for Discord
  - WATCHTOWER_NOTIFICATION_URL=discord://token@webhook
  # Or for email
  - WATCHTOWER_NOTIFICATION_URL=smtp://user:pass@host:port
```

**Notification Content:**
  * **Update Started**: Container update beginning
  * **Update Completed**: Successful update confirmation
  * **Update Failed**: Error details and troubleshooting
  * **Rollback Performed**: Automatic rollback notifications

===== Safety & Reliability =====

**Health Checks:**
```yaml
# Wait for health checks
command: --interval 3600 --cleanup --label-enable --enable-healthchecks
```

**Timeout Management:**
```yaml
# Set update timeouts
environment:
  - WATCHTOWER_TIMEOUT=60s
```

**Rollback Capability:**
```yaml
# Enable automatic rollback on failure
command: --rollback-on-failure
```

**Resource Protection:**
  * **CPU Limits**: Prevent update resource exhaustion
  * **Memory Limits**: Control memory usage during updates
  * **Network Limits**: Manage download bandwidth
  * **Concurrent Updates**: Limit simultaneous updates

===== Scheduling =====

**Update Intervals:**
```bash
# Check every hour
command: --interval 3600

# Check every 24 hours
command: --interval 86400

# Check at specific times
command: --schedule "0 0 4 * * *"  # Daily at 4 AM
```

**Maintenance Windows:**
  * **Off-hours Updates**: Schedule updates during low-usage times
  * **Weekend Updates**: Perform updates on weekends
  * **Manual Control**: Trigger updates manually when needed
  * **Holiday Scheduling**: Avoid updates during holidays

===== Troubleshooting =====

**Update Failures:**
```bash
# Check Watchtower logs
docker logs watchtower

# Manual update test
docker pull image:latest
docker stop container
docker rm container
docker run -d --name container image:latest
```

**Permission Issues:**
  * **Docker Socket**: Verify socket access permissions
  * **Registry Access**: Check Docker registry authentication
  * **Network Issues**: Verify internet connectivity
  * **Disk Space**: Ensure sufficient space for image downloads

**Notification Problems:**
  * **Webhook URLs**: Verify notification endpoint URLs
  * **Authentication**: Check API tokens and credentials
  * **Network Access**: Ensure outbound connectivity
  * **Rate Limits**: Check service rate limiting

**Performance Issues:**
  * **Resource Usage**: Monitor CPU/memory during updates
  * **Update Frequency**: Adjust polling intervals
  * **Concurrent Updates**: Limit simultaneous container updates
  * **Network Bandwidth**: Control download speeds

**Troubleshooting Steps:**
  1. **Check logs**: `docker logs watchtower`
  2. **Test manually**: Perform manual container updates
  3. **Verify configuration**: Check environment variables
  4. **Test notifications**: Send test notifications
  5. **Restart service**: `docker restart watchtower`

===== Advanced Configuration =====

**Custom Update Logic:**
```bash
# Use custom update script
command: --script /path/to/update-script.sh
```

**Lifecycle Hooks:**
```yaml
# Pre/post update hooks
command: --pre-check /path/to/pre-check.sh --post-check /path/to/post-check.sh
```

**Advanced Filtering:**
```bash
# Complex filtering rules
command: --filter-by-label=com.example.version=latest --filter-by-label=com.example.tier=frontend
```

**Monitoring Integration:**
```yaml
# Export metrics
command: --metrics
environment:
  - WATCHTOWER_METRICS_PORT=8080
```

===== Security Considerations =====

**Access Control:**
  * **Docker Socket Security**: Read-only socket access
  * **Registry Authentication**: Secure registry credentials
  * **Network Security**: Secure update traffic
  * **Audit Logging**: Track all update activities

**Update Security:**
  * **Image Verification**: Verify image authenticity
  * **Vulnerability Scanning**: Check for security issues
  * **Trusted Sources**: Only update from trusted registries
  * **Rollback Security**: Secure rollback procedures

===== Integration with Backup =====

**Backup Coordination:**
```yaml
# Coordinate with backup services
command: --pre-check /scripts/backup-check.sh --post-check /scripts/backup-verify.sh
```

**Backup Scripts:**
```bash
#!/bin/bash
# Pre-update backup
docker exec backup-service backup-now

# Post-update verification
docker exec backup-service verify-backup
```

**Automated Backup:**
  * **Pre-update Backup**: Backup before each update
  * **Post-update Verification**: Verify backup integrity
  * **Rollback Backup**: Maintain rollback capability
  * **Retention Policy**: Manage backup retention

===== Best Practices =====

**Update Strategy:**
  * **Staged Updates**: Update non-critical services first
  * **Monitoring**: Monitor updates closely initially
  * **Testing**: Test updates in development first
  * **Documentation**: Document update procedures

**Safety Measures:**
  * **Health Checks**: Always enable health checks
  * **Timeouts**: Set appropriate update timeouts
  * **Notifications**: Enable comprehensive notifications
  * **Rollback**: Have rollback procedures ready

**Performance:**
  * **Resource Limits**: Appropriate CPU/memory limits
  * **Update Windows**: Schedule during low-usage times
  * **Concurrent Limits**: Limit simultaneous updates
  * **Network Management**: Control bandwidth usage

**Monitoring:**
  * **Update Tracking**: Monitor update success/failure
  * **Performance Impact**: Track update performance impact
  * **Error Analysis**: Analyze update failure patterns
  * **Success Metrics**: Track update success rates

===== Use Cases =====

**Production Environments:**
  * **Security Updates**: Automatic security patch deployment
  * **Feature Updates**: Deploy new features automatically
  * **Compliance**: Maintain compliance with update policies
  * **Stability**: Ensure service stability through updates

**Development Environments:**
  * **Testing Updates**: Test update procedures safely
  * **CI/CD Integration**: Integrate with development pipelines
  * **Version Control**: Manage container versions
  * **Rollback Testing**: Test rollback capabilities

**Homelab Management:**
  * **Convenience**: Hands-off update management
  * **Security**: Maintain security through updates
  * **Stability**: Prevent version drift issues
  * **Monitoring**: Track update status and health

**Enterprise Deployments:**
  * **Policy Compliance**: Enforce update policies
  * **Change Management**: Manage change through updates
  * **Audit Trails**: Maintain update audit logs
  * **Reporting**: Generate update compliance reports

===== Manual Update Process =====

**When Watchtower is Disabled:**
```bash
# Manual update procedure
# 1. Identify containers to update
docker ps --format "table {{.Names}}\t{{.Image}}"

# 2. Check for updates
docker pull image:latest

# 3. Backup current state
docker tag current-image backup-image

# 4. Stop and update container
docker stop container
docker rm container
docker run -d --name container image:latest

# 5. Verify update
docker logs container
docker ps | grep container
```

Watchtower provides automated container updates with safety features and monitoring, ensuring your homelab services remain current and secure.

**Next:** Learn about [[services:infrastructure:code-server|Code Server]] or explore [[architecture:backup|Backup Architecture]].