copilot Instructions improvements
and documentation updates
This commit is contained in:
kelinfoxy
@@ -5,8 +5,7 @@ You are an AI agent specialized in managing Docker-based homelab infrastructure
|
||||
|
||||
## Repository Context
|
||||
- **Repository Location**: `~/AI-Homelab/`
|
||||
- **Purpose**: Development and testing of automated homelab management via GitHub Copilot
|
||||
- **Testing Phase**: Round 6 - Focus on script reliability, error handling, and deployment robustness
|
||||
- **Purpose**: Production-ready Docker homelab infrastructure managed through GitHub Copilot in VS Code
|
||||
- **User**: `kelin` (PUID=1000, PGID=1000)
|
||||
- **Critical**: All file operations must respect user ownership - avoid permission escalation issues
|
||||
|
||||
@@ -34,6 +33,7 @@ You are an AI agent specialized in managing Docker-based homelab infrastructure
|
||||
│ ├── setup-homelab.sh # First-run setup
|
||||
│ └── deploy-homelab.sh # Automated deployment
|
||||
├── .env.example # Environment template
|
||||
├── .env # User-created environment file (not in git)
|
||||
├── AGENT_INSTRUCTIONS.md # This file
|
||||
└── README.md # Project overview
|
||||
```
|
||||
@@ -75,7 +75,7 @@ services:
|
||||
service-name:
|
||||
image: image:tag # Always pin versions
|
||||
container_name: service-name
|
||||
restart: unless-stopped
|
||||
restart: unless-stopped # Use 'no' if ondemand (Sablier) is enabled
|
||||
networks:
|
||||
- homelab-network
|
||||
ports:
|
||||
@@ -115,21 +115,84 @@ networks:
|
||||
- Use **absolute paths** (`/mnt/media`) only for large shared data on separate drives
|
||||
- This allows stacks to be portable and work correctly in Dockge's `/opt/stacks/` structure
|
||||
|
||||
## Service Creation with Traefik on a different server Template
|
||||
|
||||
For services not running on the same server as Traefik, use this template. Does not require Traefik labels in compose file.
|
||||
|
||||
>From Traekif's perspective the service is on a remote host
|
||||
|
||||
**Remote Server Setup:**
|
||||
1. Deploy service on the target server using standard Docker Compose
|
||||
2. Note the target server's IP address and service port
|
||||
3. Create/update Traefik dynamic configuration file: `/opt/stacks/core/traefik/dynamic/external-host-servername.yml` on the server with Traefik
|
||||
|
||||
**Traefik Dynamic Configuration Template:**
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
# Remote service on servername
|
||||
remote-service:
|
||||
rule: "Host(`remote-service.${DOMAIN}`)"
|
||||
entryPoints:
|
||||
- websecure
|
||||
service: remote-service
|
||||
tls:
|
||||
certResolver: letsencrypt
|
||||
# Optional: Add Authelia protection
|
||||
middlewares:
|
||||
- authelia@docker
|
||||
|
||||
services:
|
||||
remote-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://remote-server-ip:service-port" # Replace with actual IP/port
|
||||
passHostHeader: true
|
||||
|
||||
middlewares:
|
||||
# Optional: Add headers for WebSocket support if needed
|
||||
remote-service-headers:
|
||||
headers:
|
||||
customRequestHeaders:
|
||||
X-Forwarded-Proto: "https"
|
||||
customResponseHeaders:
|
||||
X-Frame-Options: "SAMEORIGIN"
|
||||
```
|
||||
|
||||
**File Naming Convention:**
|
||||
- External host files named: `external-host-servername.yml`
|
||||
- Example: `external-host-raspberry-pi.yml`, `external-host-nas.yml`
|
||||
- See `docs/proxying-external-hosts.md` for detailed examples
|
||||
|
||||
## Critical Deployment Order
|
||||
|
||||
>The core stack should only be deployed on one server in the homelab
|
||||
The dashboards stack is intended to be deployed on the same server as the core stack, for better UX
|
||||
|
||||
Skip step 1 if the homelab already has the core stack on another server.
|
||||
|
||||
1. **Core Stack First**: Deploy `/opt/stacks/core/docker-compose.yml`
|
||||
- DuckDNS, Traefik, Authelia, Gluetun
|
||||
- DuckDNS, Traefik, Authelia
|
||||
- All other services depend on this
|
||||
2. **Infrastructure**: Dockge, Portainer, monitoring
|
||||
3. **Applications**: Media services, dashboards, etc.
|
||||
2. **VPN Stack**: Deploy `/opt/stacks/vpn/docker-compose.yml`
|
||||
- Gluetun VPN client, qBittorrent
|
||||
- Optional but recommended for secure downloads
|
||||
3. **Infrastructure**: Dockge, docker-proxy, code-server, etc.
|
||||
4. **Applications**: Media services, dashboards, etc.
|
||||
|
||||
## VPN Integration Rules
|
||||
|
||||
**All VPN-related services must be in the VPN stack** (`/opt/stacks/vpn/`), including Gluetun itself. This ensures proper network isolation and management.
|
||||
|
||||
Use Gluetun for services requiring VPN:
|
||||
```yaml
|
||||
services:
|
||||
# Gluetun VPN client (must be in vpn stack)
|
||||
gluetun:
|
||||
# VPN configuration here
|
||||
|
||||
download-client:
|
||||
network_mode: "service:gluetun"
|
||||
network_mode: "service:gluetun" # Routes through VPN
|
||||
depends_on:
|
||||
- gluetun
|
||||
# No ports section - use Gluetun's ports
|
||||
@@ -142,6 +205,8 @@ gluetun:
|
||||
- 8080:8080 # Download client web UI
|
||||
```
|
||||
|
||||
**Important**: Never place VPN-dependent services in other stacks. All VPN routing must happen within the dedicated VPN stack.
|
||||
|
||||
## SSO Management
|
||||
|
||||
### Enable SSO (Default)
|
||||
@@ -158,7 +223,7 @@ labels:
|
||||
|
||||
## Agent Actions Checklist
|
||||
|
||||
### Permission Safety (CRITICAL - Established in Round 4)
|
||||
### Permission Safety (CRITICAL)
|
||||
- [ ] **NEVER** use sudo for file operations in user directories
|
||||
- [ ] Always check file ownership before modifying: `ls -la`
|
||||
- [ ] Respect existing ownership - files should be owned by `kelin:kelin`
|
||||
@@ -196,8 +261,8 @@ labels:
|
||||
|
||||
## Common Agent Tasks
|
||||
|
||||
### Development Workflow (Current Focus - Round 6)
|
||||
1. **Repository Testing**
|
||||
### Development Workflow
|
||||
1. **Repository Maintenance**
|
||||
- Test deployment scripts: `./scripts/setup-homelab.sh`, `./scripts/deploy-homelab.sh`
|
||||
- Verify compose file syntax across all stacks
|
||||
- Validate `.env.example` completeness
|
||||
@@ -214,38 +279,111 @@ labels:
|
||||
- Document new features or configuration patterns
|
||||
|
||||
### Deploy New Service
|
||||
**Research Protocol:**
|
||||
1. **Check existing services**: Search `docs/services-overview.md` and compose files for similar services
|
||||
2. **Read official documentation**: Visit Docker Hub, LinuxServer.io, or official project docs
|
||||
3. **Check awesome-docker-compose**: Review https://awesome-docker-compose.com/apps for compose templates
|
||||
4. **Prefer LinuxServer.io**: Use lscr.io images when available for consistency
|
||||
5. **Check port availability**: Review `docs/ports-in-use.md` for conflicts
|
||||
6. **Determine SSO requirements**: Media services (Plex/Jellyfin) bypass SSO, admin tools require it
|
||||
7. **Check VPN needs**: Download clients need `network_mode: "service:gluetun"`
|
||||
8. **Review resource limits**: Apply CPU/memory limits following existing patterns
|
||||
|
||||
**Deployment Steps:**
|
||||
1. Create stack directory: `/opt/stacks/stack-name/`
|
||||
2. Write docker-compose.yml with template
|
||||
3. Create `.env` file for secrets
|
||||
4. Deploy: `docker compose up -d`
|
||||
5. Verify Traefik routing
|
||||
6. Test SSO protection
|
||||
2. Write docker-compose.yml using LinuxServer.io template with Traefik labels
|
||||
3. Create `.env` file for secrets (copy from `~/AI-Homelab/.env`)
|
||||
4. **Ask user**: Enable SSO protection? (Default: Yes, unless media service like Plex/Jellyfin)
|
||||
5. **Ask user**: Enable lazy loading (Sablier)? (Default: Yes for resource conservation) - **Note**: Requires `restart: no` instead of `unless-stopped`
|
||||
6. Add service to Homepage dashboard config
|
||||
7. Deploy: `docker compose up -d`
|
||||
8. Verify Traefik routing at `https://service.${DOMAIN}`
|
||||
9. Test SSO protection (or bypass for media services)
|
||||
10. Check logs: `docker compose logs -f`
|
||||
|
||||
### Update Existing Service
|
||||
1. Read current configuration
|
||||
2. Make minimal necessary changes
|
||||
3. Validate dependencies still work
|
||||
4. Redeploy: `docker compose up -d service-name`
|
||||
5. Check logs for errors
|
||||
**Verification Protocol:**
|
||||
1. **Read current config**: `docker compose -f /opt/stacks/stack/docker-compose.yml config`
|
||||
2. **Check dependencies**: Identify linked services, networks, volumes
|
||||
3. **Backup current state**: Copy compose file before changes
|
||||
4. **Test in isolation**: Use `docker run` for image testing if needed
|
||||
5. **Check breaking changes**: Review changelog for new versions
|
||||
6. **Validate environment**: Ensure `.env` has required variables
|
||||
|
||||
**Update Steps:**
|
||||
1. Read current configuration and dependencies
|
||||
2. **If modifying core services**: Verify integrity of `/opt/stacks/core/traefik/dynamic/` files and `sablier.yml`
|
||||
3. Make minimal necessary changes (version, config, ports)
|
||||
4. Validate YAML syntax: `docker compose config`
|
||||
5. Check for port conflicts with `docs/ports-in-use.md`
|
||||
6. Pull new image: `docker compose pull service-name`
|
||||
7. Redeploy: `docker compose up -d service-name`
|
||||
8. Monitor logs: `docker compose logs -f service-name`
|
||||
9. Test functionality and verify dependent services still work
|
||||
10. Update documentation if behavior changed
|
||||
|
||||
### Enable/Disable VPN
|
||||
1. For VPN: Add `network_mode: "service:gluetun"`
|
||||
2. Move port mapping to Gluetun service
|
||||
3. Add `depends_on: gluetun`
|
||||
4. For no VPN: Remove network_mode, add ports directly
|
||||
**Enable VPN Routing:**
|
||||
1. **Check Gluetun status**: Verify VPN stack is running with VPN configured
|
||||
2. **Move service to VPN stack**: All VPN-dependent services must be in `/opt/stacks/vpn/` with Gluetun
|
||||
3. **Modify service config**: Change from `ports:` to `network_mode: "service:gluetun"`
|
||||
4. **Move port mappings**: Add exposed ports to Gluetun service in VPN stack
|
||||
5. **Add dependency**: Include `depends_on: - gluetun`
|
||||
6. **Remove direct ports**: Delete any `ports:` section from service
|
||||
7. **Update documentation**: Note VPN routing in service docs
|
||||
8. **Test connectivity**: Verify service routes through VPN IP
|
||||
|
||||
**Disable VPN Routing:**
|
||||
1. **Check port availability**: Review `docs/ports-in-use.md` for conflicts
|
||||
2. **Move service out of VPN stack**: Relocate to appropriate non-VPN stack
|
||||
3. **Remove network_mode**: Delete `network_mode: "service:gluetun"`
|
||||
4. **Add direct ports**: Add appropriate `ports:` mapping
|
||||
5. **Remove dependency**: Delete `depends_on: gluetun`
|
||||
6. **Remove Gluetun ports**: Clean up port mappings from Gluetun service
|
||||
7. **Test direct access**: Verify service accessible without VPN
|
||||
8. **Update documentation**: Remove VPN routing notes
|
||||
|
||||
### Toggle SSO
|
||||
1. Enable: Add authelia middleware label
|
||||
2. Disable: Comment out middleware label
|
||||
3. Redeploy service
|
||||
4. Verify access works as expected
|
||||
**Enable Authelia SSO (Default for Security):**
|
||||
1. **Verify Authelia running**: Check core stack status
|
||||
2. **Uncomment middleware label**: Change `# - "traefik.http.routers.service.middlewares=authelia@docker"` to active
|
||||
3. **Remove bypass rules**: Delete domain exception from `authelia/configuration.yml` if present
|
||||
4. **Test authentication**: Access service and verify login prompt
|
||||
5. **Document protection**: Note SSO requirement in service docs
|
||||
|
||||
**Disable Authelia SSO (Media Services Only):**
|
||||
1. **Justify bypass**: Confirm service needs direct app access (Plex, Jellyfin, etc.)
|
||||
2. **Comment middleware**: Change to `# - "traefik.http.routers.service.middlewares=authelia@docker"`
|
||||
3. **Add bypass rule**: Update `authelia/configuration.yml` with domain exception
|
||||
4. **Test direct access**: Verify no authentication required
|
||||
5. **Document exception**: Explain why SSO bypassed in service docs
|
||||
6. **Monitor security**: Regular review of bypass justifications
|
||||
|
||||
**Note**: Authelia middleware label should ALWAYS be present in compose files - comment out to disable, never remove entirely.
|
||||
|
||||
## Error Prevention
|
||||
|
||||
### Port Conflicts
|
||||
- Check existing services before assigning ports
|
||||
- Use Traefik instead of port mapping when possible
|
||||
- Document port usage in comments
|
||||
**Prevention Protocol:**
|
||||
1. **Check existing ports**: Review `docs/ports-in-use.md` before assigning new ports
|
||||
2. **Use Traefik routing**: Prefer host-based routing over direct port exposure
|
||||
3. **Standardize ports**: Follow LinuxServer.io defaults when possible
|
||||
4. **Document usage**: Add port comments in compose files
|
||||
5. **Test conflicts**: Use `netstat -tlnp | grep :PORT` to check availability
|
||||
|
||||
**AI Usage of ports-in-use.md:**
|
||||
- Always consult `docs/ports-in-use.md` before suggesting new port assignments
|
||||
- Update the document whenever ports are changed or new services added
|
||||
- Cross-reference with service documentation links in the ports file
|
||||
- Verify port availability across all stacks before deployment
|
||||
|
||||
**Resolution Steps:**
|
||||
1. **Identify conflict**: Check `docker ps --format "table {{.Names}}\t{{.Ports}}"` for port usage
|
||||
2. **Reassign port**: Choose unused port following service conventions
|
||||
3. **Update Traefik labels**: Modify `loadbalancer.server.port` if changed
|
||||
4. **Update documentation**: Reflect port changes in `docs/ports-in-use.md`
|
||||
5. **Test access**: Verify service accessible at new port
|
||||
6. **Update dependent services**: Check for hardcoded port references
|
||||
|
||||
### Permission Issues
|
||||
- Always set PUID=1000, PGID=1000
|
||||
@@ -259,6 +397,19 @@ labels:
|
||||
|
||||
## Key Configurations to Monitor
|
||||
|
||||
### Sablier Configurations
|
||||
- Lazy loading enabled: `sablier.enable=true`
|
||||
- Proper group naming: `sablier.group=stack-service`
|
||||
- On-demand startup: `sablier.start-on-demand=true`
|
||||
- **Critical**: Set `restart: no` when ondemand is enabled (cannot use `unless-stopped`)
|
||||
- Group consistency across related services
|
||||
|
||||
### Traefik's Dynamic Folder Configurations
|
||||
- File provider configuration in `/opt/stacks/core/traefik/dynamic/`
|
||||
- External host routing rules in `external.yml`
|
||||
- Middleware definitions and routing rules
|
||||
- Certificate resolver settings for wildcard domains
|
||||
|
||||
### Traefik Labels
|
||||
- Correct hostname format: `service.${DOMAIN}`
|
||||
- Proper entrypoint: `websecure`
|
||||
@@ -344,6 +495,7 @@ labels:
|
||||
- **Blindly escalate privileges when encountering errors**
|
||||
|
||||
### Always Do
|
||||
- Read documentation and service-docs files and follow established practices
|
||||
- Read existing configurations first
|
||||
- Test changes in isolation when possible
|
||||
- Document complex configurations
|
||||
@@ -354,11 +506,10 @@ labels:
|
||||
- **Respect user ownership boundaries**
|
||||
- **Ask before modifying system directories**
|
||||
|
||||
## Testing and Development Guidelines (Round 6)
|
||||
## Repository Management Guidelines
|
||||
|
||||
### Repository Development
|
||||
- Work within `~/AI-Homelab/` for all development
|
||||
- Test scripts in isolated environment before production
|
||||
### Repository Maintenance
|
||||
- Work within `~/AI-Homelab/` for all operations
|
||||
- Validate all YAML files before committing
|
||||
- Ensure `.env.example` stays updated with new variables
|
||||
- Document breaking changes in commit messages
|
||||
@@ -384,7 +535,7 @@ bash -n scripts/deploy-homelab.sh
|
||||
ls -la ~/AI-Homelab/
|
||||
```
|
||||
|
||||
### Deployment Testing Checklist
|
||||
### Deployment Checklist
|
||||
- [ ] Fresh system: Test `setup-homelab.sh`
|
||||
- [ ] Core stack: Deploy and verify DuckDNS, Traefik, Authelia, Gluetun
|
||||
- [ ] Infrastructure: Deploy Dockge and verify web UI access
|
||||
@@ -394,7 +545,7 @@ ls -la ~/AI-Homelab/
|
||||
- [ ] VPN: Test Gluetun routing
|
||||
- [ ] Documentation: Validate all steps in docs/
|
||||
|
||||
### Round 6 Success Criteria
|
||||
### Production Success Criteria
|
||||
- [ ] No permission-related crashes
|
||||
- [ ] All deployment scripts work on fresh Debian install
|
||||
- [ ] Documentation matches actual implementation
|
||||
|
||||
@@ -72,11 +72,15 @@ Individual service documentation is available in [`docs/service-docs/`](docs/ser
|
||||
- **Traefik** - Reverse proxy with automatic HTTPS termination
|
||||
- **Authelia** - Single sign-on (SSO) authentication
|
||||
- **DuckDNS** - Dynamic DNS with wildcard SSL certificates
|
||||
- **Gluetun** - VPN client for secure downloads
|
||||
- **Sablier** - Lazy loading service for on-demand containers
|
||||
|
||||
### VPN Services
|
||||
- **Gluetun** - VPN client for secure downloads
|
||||
- **qBittorrent** - Torrent client routed through VPN
|
||||
|
||||
### Service Categories
|
||||
- **Media** - Plex, Jellyfin, Sonarr, Radarr, qBittorrent
|
||||
- **Media** - Plex, Jellyfin, Sonarr, Radarr
|
||||
- **VPN** - qBittorrent (VPN-routed downloads)
|
||||
- **Productivity** - Nextcloud, Gitea, BookStack, OnlyOffice
|
||||
- **Monitoring** - Grafana, Prometheus, Uptime Kuma
|
||||
- **Home Automation** - Home Assistant, Node-RED, Zigbee2MQTT
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
# Downloaders Stack
|
||||
# VPN-routed download clients
|
||||
# Place in /opt/stacks/downloaders/docker-compose.yml
|
||||
|
||||
# Service Access URLs:
|
||||
# - qBittorrent: https://qbit.${DOMAIN}
|
||||
|
||||
services:
|
||||
# Gluetun - VPN client (Surfshark)
|
||||
# Routes download clients through VPN for security
|
||||
gluetun:
|
||||
image: qmcgaw/gluetun:latest
|
||||
container_name: gluetun
|
||||
restart: unless-stopped
|
||||
cap_add:
|
||||
- NET_ADMIN
|
||||
devices:
|
||||
- /dev/net/tun:/dev/net/tun
|
||||
networks:
|
||||
- homelab-network
|
||||
- traefik-network
|
||||
ports:
|
||||
- "8888:8888/tcp" # HTTP proxy
|
||||
- "8388:8388/tcp" # Shadowsocks
|
||||
- "8388:8388/udp" # Shadowsocks
|
||||
- "8081:8080" # qBittorrent web UI
|
||||
- "6881:6881" # qBittorrent
|
||||
- "6881:6881/udp" # qBittorrent
|
||||
volumes:
|
||||
- ./gluetun:/gluetun
|
||||
environment:
|
||||
- VPN_SERVICE_PROVIDER=surfshark
|
||||
- VPN_TYPE=openvpn
|
||||
- OPENVPN_USER=${SURFSHARK_USERNAME}
|
||||
- OPENVPN_PASSWORD=${SURFSHARK_PASSWORD}
|
||||
- SERVER_COUNTRIES=${VPN_SERVER_COUNTRIES:-Netherlands}
|
||||
- TZ=${TZ}
|
||||
labels:
|
||||
- "homelab.category=downloaders"
|
||||
- "homelab.description=VPN client for secure downloads"
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.qbittorrent.rule=Host(`qbit.${DOMAIN}`)"
|
||||
- "traefik.http.routers.qbittorrent.entrypoints=websecure"
|
||||
- "traefik.http.routers.qbittorrent.tls=true"
|
||||
- "traefik.http.routers.qbittorrent.middlewares=authelia@docker"
|
||||
- "traefik.http.services.qbittorrent.loadbalancer.server.port=8080"
|
||||
|
||||
# qBittorrent - Torrent client
|
||||
# Routes through Gluetun VPN
|
||||
qbittorrent:
|
||||
image: lscr.io/linuxserver/qbittorrent:latest
|
||||
container_name: qbittorrent
|
||||
restart: unless-stopped
|
||||
network_mode: "service:gluetun" # Routes through VPN in same compose file
|
||||
volumes:
|
||||
- ./qbittorrent/config:/config
|
||||
- /mnt/downloads:/downloads
|
||||
environment:
|
||||
- PUID=${PUID:-1000}
|
||||
- PGID=${PGID:-1000}
|
||||
- TZ=${TZ}
|
||||
- WEBUI_PORT=8080
|
||||
depends_on:
|
||||
- gluetun
|
||||
|
||||
networks:
|
||||
homelab-network:
|
||||
external: true
|
||||
traefik-network:
|
||||
external: true
|
||||
@@ -1,6 +1,6 @@
|
||||
# Downloaders Stack
|
||||
# VPN-routed download clients
|
||||
# Place in /opt/stacks/downloaders/docker-compose.yml
|
||||
# VPN Stack
|
||||
# VPN client and VPN-routed download clients
|
||||
# Place in /opt/stacks/vpn/docker-compose.yml
|
||||
|
||||
# Service Access URLs:
|
||||
# - qBittorrent: https://qbit.${DOMAIN}
|
||||
98
docs/ports-in-use.md
Normal file
98
docs/ports-in-use.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Ports in Use
|
||||
|
||||
This document tracks all ports used by services in the AI-Homelab. Update this document whenever services are added or ports are changed.
|
||||
|
||||
## Core Stack ([core.yml](../docker-compose/core.yml))
|
||||
|
||||
| Service | Port | Protocol | Purpose | Internal Port |
|
||||
|---------|------|----------|---------|---------------|
|
||||
| [Traefik](../service-docs/traefik.md) | 80 | TCP | HTTP (redirects to HTTPS) | 80 |
|
||||
| [Traefik](../service-docs/traefik.md) | 443 | TCP | HTTPS | 443 |
|
||||
| [Traefik](../service-docs/traefik.md) | 8080 | TCP | Dashboard (protected) | 8080 |
|
||||
|
||||
## Infrastructure Stack ([infrastructure.yml](../docker-compose/infrastructure.yml))
|
||||
|
||||
| Service | Port | Protocol | Purpose | Internal Port |
|
||||
|---------|------|----------|---------|---------------|
|
||||
| [Dockge](../service-docs/dockge.md) | 5001 | TCP | Web UI | 5001 |
|
||||
| [Pi-hole](../service-docs/pihole.md) | 53 | TCP/UDP | DNS | 53 |
|
||||
| [Docker Proxy](../service-docs/docker-proxy.md) | 127.0.0.1:2375 | TCP | Docker API proxy | 2375 |
|
||||
|
||||
## Development Stack ([development.yml](../docker-compose/development.yml))
|
||||
|
||||
| Service | Port | Protocol | Purpose | Internal Port |
|
||||
|---------|------|----------|---------|---------------|
|
||||
| [PostgreSQL](../service-docs/postgresql.md) | 5432 | TCP | Database | 5432 |
|
||||
| [Redis](../service-docs/redis.md) | 6379 | TCP | Cache/Database | 6379 |
|
||||
|
||||
## Home Assistant Stack ([homeassistant.yml](../docker-compose/homeassistant.yml))
|
||||
|
||||
| Service | Port | Protocol | Purpose | Internal Port |
|
||||
|---------|------|----------|---------|---------------|
|
||||
| [MotionEye](../service-docs/motioneye.md) | 8765 | TCP | Web UI | 8765 |
|
||||
| [Mosquitto](../service-docs/mosquitto.md) | 1883 | TCP | MQTT | 1883 |
|
||||
| [Mosquitto](../service-docs/mosquitto.md) | 9001 | TCP | MQTT Websockets | 9001 |
|
||||
|
||||
## Monitoring Stack ([monitoring.yml](../docker-compose/monitoring.yml))
|
||||
|
||||
| Service | Port | Protocol | Purpose | Internal Port |
|
||||
|---------|------|----------|---------|---------------|
|
||||
| [Prometheus](../service-docs/prometheus.md) | 9090 | TCP | Web UI/Metrics | 9090 |
|
||||
|
||||
## VPN Stack ([vpn.yml](../docker-compose/vpn.yml))
|
||||
|
||||
| Service | Port | Protocol | Purpose | Internal Port |
|
||||
|---------|------|----------|---------|---------------|
|
||||
| [Gluetun](../service-docs/gluetun.md) | 8888 | TCP | HTTP proxy | 8888 |
|
||||
| [Gluetun](../service-docs/gluetun.md) | 8388 | TCP/UDP | Shadowsocks | 8388 |
|
||||
| [Gluetun](../service-docs/gluetun.md) | 8081 | TCP | qBittorrent Web UI | 8080 |
|
||||
| [Gluetun](../service-docs/gluetun.md) | 6881 | TCP/UDP | qBittorrent | 6881 |
|
||||
|
||||
## Port Range Reference
|
||||
|
||||
| Range | Usage |
|
||||
|-------|-------|
|
||||
| 1-1023 | System ports (well-known) |
|
||||
| 1024-49151 | Registered ports |
|
||||
| 49152-65535 | Dynamic/private ports |
|
||||
|
||||
## Common Port Conflicts
|
||||
|
||||
- **Port 80/443**: Used by Traefik for HTTP/HTTPS
|
||||
- **Port 53**: Used by Pi-hole for DNS
|
||||
- **Port 2375**: Used by Docker Proxy (localhost only)
|
||||
- **Port 5001**: Used by Dockge
|
||||
- **Port 5432**: Used by PostgreSQL
|
||||
- **Port 6379**: Used by Redis
|
||||
- **Port 8080**: Used by Traefik dashboard
|
||||
- **Port 9090**: Used by Prometheus
|
||||
|
||||
## Adding New Services
|
||||
|
||||
When adding new services:
|
||||
|
||||
1. Check this document for available ports
|
||||
2. Choose ports that don't conflict with existing services
|
||||
3. Update this document with new port mappings
|
||||
4. Consider using Traefik labels instead of direct port exposure for web services
|
||||
|
||||
## Port Planning Guidelines
|
||||
|
||||
- **Web services**: Use Traefik labels (no direct ports needed)
|
||||
- **Databases**: Use internal networking only (no external ports)
|
||||
- **VPN services**: Route through Gluetun for security
|
||||
- **Development tools**: Consider localhost-only binding (127.0.0.1:port)
|
||||
- **Monitoring**: Use high-numbered ports (9000+ range)
|
||||
|
||||
## Updating This Document
|
||||
|
||||
This document should be updated whenever:
|
||||
- New services are added to any stack
|
||||
- Existing services change their port mappings
|
||||
- Services are removed from stacks
|
||||
- Network configurations change
|
||||
|
||||
Run this command to find all port mappings in compose files:
|
||||
```bash
|
||||
grep -r "ports:" docker-compose/ | grep -v "^#" | sort
|
||||
```
|
||||
@@ -14,7 +14,7 @@ Traefik can proxy services that aren't running in Docker, such as:
|
||||
|
||||
### Step 1: Create External Service Configuration
|
||||
|
||||
Create a file in `/opt/stacks/traefik/dynamic/external-hosts.yml`:
|
||||
Create a file in `/opt/stacks/traefik/dynamic/` with the format 'external-host-servername.yml'
|
||||
|
||||
```yaml
|
||||
http:
|
||||
@@ -71,7 +71,11 @@ Visit `https://ha.yourdomain.duckdns.org` - Traefik will:
|
||||
|
||||
## Method 2: Using Docker Labels (Dummy Container)
|
||||
|
||||
If you prefer managing routes via Docker labels (so the AI can modify them), create a dummy container:
|
||||
If you prefer managing routes via Docker labels, create a dummy container:
|
||||
|
||||
>This can be resource intensive with serveral services running.
|
||||
Not recomended due to unnecessary resource/power consumption.
|
||||
Don't try it on a Raspberry Pi
|
||||
|
||||
### Create a Label Container
|
||||
|
||||
@@ -108,13 +112,6 @@ cd /opt/stacks/external-proxies
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Method 3: Hybrid Approach (File + Docker Discovery)
|
||||
|
||||
Combine both methods for maximum flexibility:
|
||||
- Use file provider for static external hosts
|
||||
- Use Docker labels for frequently changing services
|
||||
- AI can manage both!
|
||||
|
||||
## Common External Services to Proxy
|
||||
|
||||
### Home Assistant (Raspberry Pi)
|
||||
@@ -134,34 +131,6 @@ router-admin:
|
||||
- authelia@docker # Add SSO protection
|
||||
```
|
||||
|
||||
### Proxmox Server
|
||||
```yaml
|
||||
proxmox:
|
||||
rule: "Host(`proxmox.yourdomain.duckdns.org`)"
|
||||
service: https://192.168.1.100:8006
|
||||
middlewares:
|
||||
- authelia@docker
|
||||
# Note: Use https:// if backend uses HTTPS
|
||||
```
|
||||
|
||||
### TrueNAS/FreeNAS
|
||||
```yaml
|
||||
truenas:
|
||||
rule: "Host(`nas.yourdomain.duckdns.org`)"
|
||||
service: http://192.168.1.200:80
|
||||
middlewares:
|
||||
- authelia@docker
|
||||
```
|
||||
|
||||
### Security Camera NVR
|
||||
```yaml
|
||||
nvr:
|
||||
rule: "Host(`cameras.yourdomain.duckdns.org`)"
|
||||
service: http://192.168.1.10:80
|
||||
middlewares:
|
||||
- authelia@docker # Definitely protect cameras!
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### WebSocket Support
|
||||
|
||||
@@ -139,7 +139,7 @@ calibre_web: # Ebook manager
|
||||
template: web_service
|
||||
notes: "Light web app with database"
|
||||
|
||||
# Downloaders Stack
|
||||
# VPN Stack
|
||||
qbittorrent: # Torrent client
|
||||
template: downloader_service
|
||||
notes: "Network I/O heavy, moderate CPU for hashing"
|
||||
|
||||
@@ -6,11 +6,13 @@ This document provides a comprehensive overview of all 50+ pre-configured servic
|
||||
|
||||
| Stacks (10) | Services (70 + 6db) | SSO | Storage | Access URLs |
|
||||
|-------|----------|-----|---------|-------------|
|
||||
| **📦 core.yaml (4)** | **Deploy First** | | | |
|
||||
| **📦 core.yaml (3)** | **Deploy First** | | | |
|
||||
| ├─ DuckDNS | Dynamic DNS updater | - | /opt/stacks/core/duckdns | No UI |
|
||||
| ├─ Traefik | Reverse proxy + SSL | ✓ | /opt/stacks/core/traefik | traefik.${DOMAIN} |
|
||||
| ├─ Authelia | SSO authentication | - | /opt/stacks/core/authelia | auth.${DOMAIN} |
|
||||
| └─ Gluetun | VPN (Surfshark) | - | /opt/stacks/core/gluetun | No UI |
|
||||
| └─ Authelia | SSO authentication | - | /opt/stacks/core/authelia | auth.${DOMAIN} |
|
||||
| **🔒 vpn.yaml (2)** | **VPN Services** | | | |
|
||||
| ├─ Gluetun | VPN (Surfshark) | - | /opt/stacks/vpn/gluetun | No UI |
|
||||
| └─ qBittorrent | Torrent (via VPN) | ✓ | /mnt/downloads | qbit.${DOMAIN} |
|
||||
| **🔧 infrastructure.yaml** (12) | | | | |
|
||||
| ├─ Dockge | Stack manager (PRIMARY) | ✓ | /opt/stacks/infrastructure | dockge.${DOMAIN} |
|
||||
| ├─ Portainer | Container management | ✓ | /opt/stacks/infrastructure | portainer.${DOMAIN} |
|
||||
@@ -31,8 +33,7 @@ This document provides a comprehensive overview of all 50+ pre-configured servic
|
||||
| ├─ Jellyfin | Media server (OSS) | ✗ | /mnt/media, /mnt/transcode | jellyfin.${DOMAIN} |
|
||||
| ├─ Sonarr | TV automation | ✓ | /opt/stacks/media, /mnt/media | sonarr.${DOMAIN} |
|
||||
| ├─ Radarr | Movie automation | ✓ | /opt/stacks/media, /mnt/media | radarr.${DOMAIN} |
|
||||
| ├─ Prowlarr | Indexer manager | ✓ | /opt/stacks/media | prowlarr.${DOMAIN} |
|
||||
| └─ qBittorrent | Torrent (via VPN) | ✓ | /mnt/downloads | qbit.${DOMAIN} |
|
||||
| └─ Prowlarr | Indexer manager | ✓ | /opt/stacks/media | prowlarr.${DOMAIN} |
|
||||
| **📚 media-extended.yaml** (10) | | | | |
|
||||
| ├─ Readarr | Ebooks/Audiobooks | ✓ | /opt/stacks/media-ext, /mnt/media | readarr.${DOMAIN} |
|
||||
| ├─ Lidarr | Music manager | ✓ | /opt/stacks/media-ext, /mnt/media | lidarr.${DOMAIN} |
|
||||
|
||||
@@ -89,7 +89,7 @@ Automated deployment script that deploys the core infrastructure and Dockge. Run
|
||||
1. **Validate Prerequisites** - Checks for Docker, .env file, and proper configuration
|
||||
2. **Create Directories** - Sets up `/opt/stacks/core` and `/opt/stacks/infrastructure`
|
||||
3. **Create Docker Networks** - Ensures homelab-network, traefik-network, and media-network exist
|
||||
4. **Deploy Core Stack** - Deploys DuckDNS, Traefik, Authelia, and Gluetun
|
||||
4. **Deploy Core Stack** - Deploys DuckDNS, Traefik, and Authelia
|
||||
5. **Deploy Infrastructure Stack** - Deploys Dockge, Portainer, Pi-hole, and monitoring tools
|
||||
6. **Wait for Dockge** - Waits for Dockge web UI to become accessible
|
||||
7. **Open Browser** - Automatically opens Dockge in your default browser
|
||||
|
||||
15
tasks.txt
15
tasks.txt
@@ -1,14 +1,9 @@
|
||||
Tasks
|
||||
|
||||
.env.example
|
||||
✅ create new variables
|
||||
DAFAULT_USER: admin
|
||||
DEFAULT_PASSWORD: changeme
|
||||
DEFAULT_EMAIL: admin@example.com
|
||||
Completed:
|
||||
- Created VPN stack and moved Gluetun & qBittorrent from downloaders stack
|
||||
- Removed downloaders stack entirely (directory and .yml file)
|
||||
- Updated all documentation and AGENT_INSTRUCTIONS.md for VPN stack structure
|
||||
- Updated resource limits template to reflect VPN stack
|
||||
|
||||
✅ modify existing variables to use these 3 new variables like
|
||||
NEXTCLOUD_ADMIN_USER=${DEFAULT_USER}
|
||||
except for QBITTORRENT_USER=admin should stay as is
|
||||
the goal is ease of setup for the user
|
||||
|
||||
✅ Then update env-configuration.md
|
||||
Reference in New Issue
Block a user