EZ-Homelab/docs/service-docs/watchtower.md

Watchtower - Automated Container Updates

Table of Contents

• Overview
• What is Watchtower?
• Why Use Watchtower?
• How It Works
• Configuration in AI-Homelab
• Official Resources
• Educational Resources
• Docker Configuration
• Controlling Updates
• Advanced Topics
• Troubleshooting

Overview

Category: Infrastructure Automation
Docker Image: containrrr/watchtower
Default Stack: infrastructure.yml
Web UI: None (runs as automated service)
Default Behavior: Checks for updates daily at 4 AM

What is Watchtower?

Watchtower is an automated Docker container update service that monitors running containers and automatically updates them when new images are available. It pulls new images, stops old containers, and starts new ones while preserving volumes and configuration.

Key Features

• Automatic Updates: Keeps containers up-to-date automatically
• Schedule Control: Cron-based update scheduling
• Selective Monitoring: Choose which containers to update
• Notifications: Email, Slack, Discord, etc. on updates
• Update Strategies: Rolling updates, one-time runs
• Cleanup: Automatically removes old images
• Dry Run Mode: Test without actually updating
• Label-Based Control: Fine-grained update control per container
• Zero Downtime: Seamless container recreation

Why Use Watchtower?

1. Security: Automatic security patches
2. Convenience: No manual image updates needed
3. Consistency: All containers stay updated
4. Time Savings: Automates tedious update process
5. Minimal Downtime: Fast container recreation
6. Safe Updates: Preserves volumes and configuration
7. Rollback Support: Keep old images for fallback
8. Notification: Get notified of updates

How It Works

Watchtower (scheduled check)
     ↓
Check Docker Hub for new images
     ↓
Compare with local image digests
     ↓
If new version found:
     ├─ Pull new image
     ├─ Stop old container
     ├─ Remove old container
     ├─ Create new container (same config)
     ├─ Start new container
     └─ Remove old image (optional)

Update Process

1. Scheduled Check: Watchtower wakes up (e.g., 4 AM daily)
2. Scan Containers: Finds all monitored containers
3. Check Registries: Queries Docker Hub/registries for new images
4. Compare Digests: Checks if image hash changed
5. Pull New Image: Downloads latest version
6. Recreate Container: Stops old, starts new with same config
7. Cleanup: Removes old images (if configured)
8. Notify: Sends notification (if configured)

What Gets Preserved

✅ Preserved:

• Volumes and data
• Networks
• Environment variables
• Labels
• Port mappings
• Restart policy

❌ Not Preserved:

• Running processes inside container
• Temporary files in container filesystem
• In-memory data

Configuration in AI-Homelab

Directory Structure

# Watchtower doesn't need persistent storage
# All configuration via environment variables

Environment Variables

# Schedule (cron format)
WATCHTOWER_SCHEDULE=0 0 4 * * *  # Daily at 4 AM

# Cleanup old images
WATCHTOWER_CLEANUP=true

# Include stopped containers
WATCHTOWER_INCLUDE_STOPPED=false

# Include restarting containers
WATCHTOWER_INCLUDE_RESTARTING=false

# Debug logging
WATCHTOWER_DEBUG=false

# Notifications (optional)
WATCHTOWER_NOTIFICATIONS=email
WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@yourdomain.com
WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com
WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=your-email@gmail.com
WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=your-app-password

Official Resources

• GitHub: https://github.com/containrrr/watchtower
• Docker Hub: https://hub.docker.com/r/containrrr/watchtower
• Documentation: https://containrrr.dev/watchtower/
• Arguments Reference: https://containrrr.dev/watchtower/arguments/

Educational Resources

Videos

• Watchtower - Auto Update Docker Containers (Techno Tim)
• Keep Docker Containers Updated Automatically
• Watchtower Setup Tutorial (DB Tech)

Articles & Guides

• Watchtower Official Documentation
• Docker Update Strategies
• Notification Configuration
• Cron Schedule Examples

Concepts to Learn

• Container Updates: How Docker updates work
• Image Digests: SHA256 hashes for images
• Cron Scheduling: Time-based task automation
• Rolling Updates: Zero-downtime deployment
• Image Tags vs Digests: Differences and implications
• Semantic Versioning: Understanding version numbers
• Docker Labels: Metadata for containers

Docker Configuration

Complete Service Definition

watchtower:
  image: containrrr/watchtower:latest
  container_name: watchtower
  restart: unless-stopped
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - WATCHTOWER_SCHEDULE=0 0 4 * * *  # 4 AM daily
    - WATCHTOWER_CLEANUP=true
    - WATCHTOWER_INCLUDE_STOPPED=false
    - WATCHTOWER_INCLUDE_RESTARTING=true
    - WATCHTOWER_ROLLING_RESTART=true
    - WATCHTOWER_TIMEOUT=30s
    - TZ=America/New_York
    # Optional: Notifications
    # - WATCHTOWER_NOTIFICATIONS=email
    # - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
    # - WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@yourdomain.com

Schedule Formats

# Cron format: second minute hour day month weekday
# * * * * * *  = every second
# 0 * * * * *  = every minute
# 0 0 * * * *  = every hour
# 0 0 4 * * *  = daily at 4 AM
# 0 0 4 * * 0  = weekly on Sunday at 4 AM
# 0 0 4 1 * *  = monthly on 1st at 4 AM

# Common schedules:
WATCHTOWER_SCHEDULE="0 0 4 * * *"        # Daily 4 AM
WATCHTOWER_SCHEDULE="0 0 4 * * 0"        # Weekly Sunday 4 AM
WATCHTOWER_SCHEDULE="0 0 2 1,15 * *"     # 1st and 15th at 2 AM
WATCHTOWER_SCHEDULE="0 */6 * * * *"      # Every 6 hours

Use https://crontab.guru/ to test cron expressions.

Controlling Updates

Label-Based Control

Enable/Disable per Container:

myservice:
  image: myapp:latest
  labels:
    # Enable updates (default)
    - "com.centurylinklabs.watchtower.enable=true"
    
    # OR disable updates
    - "com.centurylinklabs.watchtower.enable=false"

Monitor Specific Containers

Run Watchtower with container names:

docker run -d \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  container1 container2 container3

Exclude Containers

watchtower:
  image: containrrr/watchtower
  environment:
    - WATCHTOWER_LABEL_ENABLE=true  # Only update labeled containers

Then add label to containers you want updated:

myservice:
  labels:
    - "com.centurylinklabs.watchtower.enable=true"

Update Scope

Scope Options:

environment:
  # Only update containers with specific scope
  - WATCHTOWER_SCOPE=myscope

# Then label containers:
myservice:
  labels:
    - "com.centurylinklabs.watchtower.scope=myscope"

Advanced Topics

Notification Configuration

Email Notifications

environment:
  - WATCHTOWER_NOTIFICATIONS=email
  - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
  - WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@yourdomain.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=your-email@gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=your-app-password
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY=false

Slack Notifications

environment:
  - WATCHTOWER_NOTIFICATIONS=slack
  - WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
  - WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER=watchtower

Discord Notifications

environment:
  - WATCHTOWER_NOTIFICATIONS=discord
  - WATCHTOWER_NOTIFICATION_URL=https://discord.com/api/webhooks/YOUR/WEBHOOK

One-Time Run

Run once and exit (useful for testing):

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --run-once

Dry Run Mode

Test without actually updating:

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --run-once \
  --debug \
  --dry-run

Rolling Restart

Update containers one at a time (safer):

environment:
  - WATCHTOWER_ROLLING_RESTART=true

Monitor Only Mode

Check for updates but don't apply:

environment:
  - WATCHTOWER_MONITOR_ONLY=true
  - WATCHTOWER_NOTIFICATIONS=email  # Get notified of available updates

Custom Update Commands

Run commands after update:

myservice:
  labels:
    - "com.centurylinklabs.watchtower.lifecycle.post-update=/scripts/post-update.sh"

Private Registry

Update containers from private registries:

# Docker Hub
docker login

# Private registry
docker login registry.example.com

# Watchtower will use stored credentials

Or configure explicitly:

environment:
  - REPO_USER=username
  - REPO_PASS=password

Troubleshooting

Check Watchtower Status

# View logs
docker logs watchtower

# Follow logs in real-time
docker logs -f watchtower

# Check last run
docker logs watchtower | grep "Session done"

Force Manual Update

# Run Watchtower once
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --run-once \
  --debug

Container Not Updating

# Check if container is monitored
docker logs watchtower | grep container-name

# Verify image has updates
docker pull image:tag

# Check for update labels
docker inspect container-name | grep watchtower

# Force update specific container
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower \
  --run-once \
  container-name

Updates Breaking Services

# Stop Watchtower
docker stop watchtower

# Rollback manually
docker pull image:old-tag
docker compose up -d --force-recreate service-name

# Or restore from backup
# If you kept old images (no cleanup):
docker images | grep image-name
docker tag image:old-sha image:latest
docker compose up -d service-name

High Resource Usage

# Check Watchtower stats
docker stats watchtower

# Increase check interval
# Change from hourly to daily:
WATCHTOWER_SCHEDULE="0 0 4 * * *"

# Enable cleanup to free disk space
WATCHTOWER_CLEANUP=true

Notification Not Working

# Test email manually
docker exec watchtower wget --spider https://smtp.gmail.com

# Check credentials
docker logs watchtower | grep -i notification

# Verify SMTP settings
# For Gmail, use App Password, not account password

Best Practices

Recommended Configuration

watchtower:
  image: containrrr/watchtower:latest
  container_name: watchtower
  restart: unless-stopped
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    # Daily at 4 AM
    - WATCHTOWER_SCHEDULE=0 0 4 * * *
    
    # Clean up old images
    - WATCHTOWER_CLEANUP=true
    
    # Rolling restart (safer)
    - WATCHTOWER_ROLLING_RESTART=true
    
    # 30 second timeout
    - WATCHTOWER_TIMEOUT=30s
    
    # Enable notifications
    - WATCHTOWER_NOTIFICATIONS=email
    - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
    - WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@yourdomain.com
    
    - TZ=America/New_York

Update Strategy

1. Start Conservative:

   • Weekly updates initially
   • Monitor for issues
   • Gradually increase frequency

2. Use Labels:

   • Disable updates for critical services
   • Update testing services first
   • Separate production from testing

3. Enable Notifications:

   • Know when updates happen
   • Track update history
   • Alert on failures

4. Test Regularly:

   • Dry run before enabling
   • Manual one-time runs
   • Verify services after updates

5. Backup Strategy:

   • Keep old images (disable cleanup initially)
   • Backup volumes before major updates
   • Document rollback procedures

Selective Updates

# Core infrastructure: Manual updates only
traefik:
  labels:
    - "com.centurylinklabs.watchtower.enable=false"

authelia:
  labels:
    - "com.centurylinklabs.watchtower.enable=false"

# Media services: Auto-update
plex:
  labels:
    - "com.centurylinklabs.watchtower.enable=true"

sonarr:
  labels:
    - "com.centurylinklabs.watchtower.enable=true"

Security Considerations

1. Docker Socket Access: Watchtower needs full Docker access (security risk)
2. Automatic Updates: Can break things unexpectedly
3. Test Environment: Test updates in dev before production
4. Backup First: Always backup critical services
5. Monitor Logs: Watch for failed updates
6. Rollback Plan: Know how to revert updates
7. Pin Versions: Use specific tags for critical services
8. Update Windows: Schedule during low-usage times
9. Notification: Always enable notifications
10. Review Updates: Check changelog before auto-updating

Summary

Watchtower automates container updates, providing:

• Automatic security patches
• Consistent update schedule
• Minimal manual intervention
• Clean, simple configuration
• Notification support
• Label-based control

Use Watchtower for:

• Non-critical services
• Media apps (Plex, Sonarr, etc.)
• Utilities and tools
• Development environments

Avoid for:

• Core infrastructure (Traefik, Authelia)
• Databases (require careful updates)
• Custom applications
• Services requiring migration steps

Remember:

• Start with conservative schedule
• Enable cleanup after testing
• Use labels for fine control
• Monitor notifications
• Have rollback plan
• Test in dry-run mode first
• Backup before major updates