diff --git a/config-templates/README.md b/config-templates/README.md new file mode 100644 index 0000000..c508b20 --- /dev/null +++ b/config-templates/README.md @@ -0,0 +1,196 @@ +# Configuration Templates + +This directory contains example configuration files for various services. These templates provide sensible defaults and are ready to use with minimal modifications. + +## Usage + +1. **Create your config directory** (if it doesn't exist): + ```bash + mkdir -p config/service-name + ``` + +2. **Copy the template** to your config directory: + ```bash + cp config-templates/service-name/* config/service-name/ + ``` + +3. **Edit the configuration** as needed for your environment + +4. **Start the service** using Docker Compose + +## Available Templates + +### Prometheus (`prometheus/prometheus.yml`) +Metrics collection and monitoring system configuration. + +**Features:** +- Pre-configured to scrape Node Exporter and cAdvisor +- 15-second scrape interval +- Ready for additional service monitoring + +**Setup:** +```bash +mkdir -p config/prometheus +cp config-templates/prometheus/prometheus.yml config/prometheus/ +docker compose -f docker-compose/monitoring.yml up -d prometheus +``` + +### Loki (`loki/loki-config.yml`) +Log aggregation system configuration. + +**Features:** +- Filesystem-based storage +- 30-day log retention +- Automatic log compaction +- Pre-configured for Promtail + +**Setup:** +```bash +mkdir -p config/loki +cp config-templates/loki/loki-config.yml config/loki/ +docker compose -f docker-compose/monitoring.yml up -d loki +``` + +### Promtail (`promtail/promtail-config.yml`) +Log shipper for Loki. + +**Features:** +- Automatically ships Docker container logs +- Parses Docker JSON format +- Extracts container IDs and names +- Optional system log collection + +**Setup:** +```bash +mkdir -p config/promtail +cp config-templates/promtail/promtail-config.yml config/promtail/ +docker compose -f docker-compose/monitoring.yml up -d promtail +``` + +### Redis (`redis/redis.conf`) +In-memory data store configuration. + +**Features:** +- Both AOF and RDB persistence enabled +- 256MB memory limit with LRU eviction +- Sensible defaults for homelab use +- Security options (password protection available) + +**Setup:** +```bash +mkdir -p config/redis +cp config-templates/redis/redis.conf config/redis/ +# Optional: Edit redis.conf to set a password +docker compose -f docker-compose/development.yml up -d redis +``` + +## Customization Tips + +### Prometheus +- Add more scrape targets to monitor additional services +- Adjust `scrape_interval` based on your needs (lower = more frequent, more data) +- Configure alerting by uncommenting the alertmanager section + +### Loki +- Adjust `retention_period` to keep logs longer or shorter +- Change storage from filesystem to S3 for better scalability +- Configure multiple tenants if needed + +### Promtail +- Add more scrape configs for system logs, application logs, etc. +- Customize pipeline stages to extract more labels +- Filter logs based on patterns + +### Redis +- Set `maxmemory` based on your available RAM +- Choose appropriate `maxmemory-policy` for your use case +- Enable password protection by uncommenting `requirepass` + +## Service-Specific Notes + +### Services That Don't Need Config Templates + +Many services work perfectly with just environment variables and don't require separate config files: + +- **Plex, Jellyfin**: Configure via web UI +- **Sonarr, Radarr, Prowlarr**: Configure via web UI +- **Portainer**: Configure via web UI +- **Grafana**: Can use provisioning or web UI +- **Most LinuxServer.io images**: Configured via environment variables + +### Services That Benefit from Config Files + +- **Prometheus**: Requires `prometheus.yml` for scrape configuration +- **Loki**: Requires config for storage and retention +- **Promtail**: Requires config for log sources +- **Redis**: Benefits from custom config for persistence and security +- **Nginx**: Needs config for proxy rules (use Nginx Proxy Manager UI instead) + +## Best Practices + +1. **Version Control**: Keep your config templates in git +2. **Secrets**: Never commit passwords or API keys +3. **Comments**: Add comments explaining custom settings +4. **Backups**: Backup config directories regularly +5. **Testing**: Test config changes in a separate environment first + +## Creating New Templates + +When creating templates for other services: + +1. Start with the official documentation +2. Use sensible defaults for homelab use +3. Add comments explaining important settings +4. Include examples for common customizations +5. Test the template before committing + +## Getting Help + +- Check the official documentation for each service +- Ask GitHub Copilot in VS Code for configuration help +- Review the [Docker Guidelines](../docs/docker-guidelines.md) +- Consult service-specific community forums + +## Example: Full Monitoring Stack Setup + +```bash +# Create all config directories +mkdir -p config/{prometheus,loki,promtail,grafana} + +# Copy templates +cp config-templates/prometheus/prometheus.yml config/prometheus/ +cp config-templates/loki/loki-config.yml config/loki/ +cp config-templates/promtail/promtail-config.yml config/promtail/ + +# Start the monitoring stack +docker compose -f docker-compose/monitoring.yml up -d + +# Access services +# Prometheus: http://server-ip:9090 +# Grafana: http://server-ip:3000 +# Loki: http://server-ip:3100 +``` + +## Troubleshooting + +### Config file not found +Ensure you copied the template to the correct location referenced in the docker-compose file. + +### Permission errors +Fix ownership: +```bash +sudo chown -R 1000:1000 config/service-name +``` + +### Syntax errors +Validate YAML files: +```bash +# For YAML files +python3 -c "import yaml; yaml.safe_load(open('config/service/config.yml'))" +``` + +### Service won't start +Check logs for configuration errors: +```bash +docker compose -f docker-compose/file.yml logs service-name +``` diff --git a/config-templates/loki/loki-config.yml b/config-templates/loki/loki-config.yml new file mode 100644 index 0000000..2d7c57c --- /dev/null +++ b/config-templates/loki/loki-config.yml @@ -0,0 +1,46 @@ +# Loki Configuration Template +# Copy this file to ./config/loki/loki-config.yml + +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + +common: + path_prefix: /loki + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + replication_factor: 1 + ring: + instance_addr: 127.0.0.1 + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2020-10-24 + store: boltdb-shipper + object_store: filesystem + schema: v11 + index: + prefix: index_ + period: 24h + +ruler: + alertmanager_url: http://localhost:9093 + +# Retention configuration (delete logs older than 30 days) +limits_config: + retention_period: 720h # 30 days + +# Compactor to delete old data +compactor: + working_directory: /loki/compactor + shared_store: filesystem + compaction_interval: 10m + retention_enabled: true + retention_delete_delay: 2h + retention_delete_worker_count: 150 diff --git a/config-templates/prometheus/prometheus.yml b/config-templates/prometheus/prometheus.yml new file mode 100644 index 0000000..ab3ee50 --- /dev/null +++ b/config-templates/prometheus/prometheus.yml @@ -0,0 +1,49 @@ +# Prometheus Configuration Template +# Copy this file to ./config/prometheus/prometheus.yml + +global: + scrape_interval: 15s + evaluation_interval: 15s + external_labels: + monitor: 'homelab' + +# Alertmanager configuration (optional) +# alerting: +# alertmanagers: +# - static_configs: +# - targets: +# - alertmanager:9093 + +# Load rules once and periodically evaluate them +# rule_files: +# - "alerts/*.yml" + +# Scrape configurations +scrape_configs: + # Prometheus itself + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + # Node Exporter - System metrics + - job_name: 'node-exporter' + static_configs: + - targets: ['node-exporter:9100'] + labels: + instance: 'homelab-server' + + # cAdvisor - Container metrics + - job_name: 'cadvisor' + static_configs: + - targets: ['cadvisor:8080'] + labels: + instance: 'homelab-server' + + # Add your own services here + # Example: Monitor a service with /metrics endpoint + # - job_name: 'my-service' + # static_configs: + # - targets: ['my-service:8080'] + # labels: + # instance: 'homelab-server' + # service: 'my-service' diff --git a/config-templates/promtail/promtail-config.yml b/config-templates/promtail/promtail-config.yml new file mode 100644 index 0000000..19f4f24 --- /dev/null +++ b/config-templates/promtail/promtail-config.yml @@ -0,0 +1,53 @@ +# Promtail Configuration Template +# Copy this file to ./config/promtail/promtail-config.yml + +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + # Docker container logs + - job_name: docker + static_configs: + - targets: + - localhost + labels: + job: docker + __path__: /var/lib/docker/containers/*/*-json.log + + pipeline_stages: + # Parse Docker JSON logs + - json: + expressions: + output: log + stream: stream + attrs: attrs + + # Extract container name from path + - regex: + expression: '/var/lib/docker/containers/(?P[^/]+)/.*' + source: filename + + # Add labels + - labels: + stream: + container_id: + + # Output the log line + - output: + source: output + + # System logs (optional) + # - job_name: system + # static_configs: + # - targets: + # - localhost + # labels: + # job: varlogs + # __path__: /var/log/*.log diff --git a/config-templates/redis/redis.conf b/config-templates/redis/redis.conf new file mode 100644 index 0000000..d6990bb --- /dev/null +++ b/config-templates/redis/redis.conf @@ -0,0 +1,42 @@ +# Redis Configuration Template +# Copy this file to ./config/redis/redis.conf + +# Network +bind 0.0.0.0 +protected-mode yes +port 6379 + +# General +daemonize no +supervised no +pidfile /var/run/redis_6379.pid +loglevel notice +logfile "" + +# Persistence - AOF (Append Only File) +appendonly yes +appendfilename "appendonly.aof" +appendfsync everysec +no-appendfsync-on-rewrite no +auto-aof-rewrite-percentage 100 +auto-aof-rewrite-min-size 64mb + +# Persistence - RDB (Snapshotting) +save 900 1 +save 300 10 +save 60 10000 +stop-writes-on-bgsave-error yes +rdbcompression yes +rdbchecksum yes +dbfilename dump.rdb +dir /data + +# Memory Management +maxmemory 256mb +maxmemory-policy allkeys-lru + +# Security +# requirepass yourpassword # Uncomment and set a strong password + +# Limits +maxclients 10000 diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..36522b4 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,506 @@ +# Getting Started Guide + +This guide will walk you through setting up your AI-powered homelab from scratch. + +## Prerequisites + +Before you begin, ensure you have: + +- [ ] A Linux server (Ubuntu 22.04+ recommended) +- [ ] Docker Engine 24.0+ installed +- [ ] Docker Compose V2 installed +- [ ] Git installed +- [ ] At least 8GB RAM (16GB+ recommended) +- [ ] Sufficient disk space (100GB+ recommended) +- [ ] Static IP address for your server +- [ ] VS Code with GitHub Copilot extension (for AI assistance) + +## Step 1: Verify Docker Installation + +```bash +# Check Docker version +docker --version +# Should show: Docker version 24.0.0 or higher + +# Check Docker Compose version +docker compose version +# Should show: Docker Compose version v2.x.x + +# Test Docker works +docker run --rm hello-world +# Should download and run successfully +``` + +## Step 2: Clone the Repository + +```bash +# Navigate to your home directory +cd ~ + +# Clone the repository +git clone https://github.com/kelinfoxy/AI-Homelab.git + +# Enter the directory +cd AI-Homelab +``` + +## Step 3: Configure Environment Variables + +```bash +# Copy the example environment file +cp .env.example .env + +# Get your user and group IDs +id -u # This is your PUID +id -g # This is your PGID + +# Edit the .env file +nano .env +``` + +**Update these values in `.env`:** +```bash +# Your user/group IDs +PUID=1000 # Replace with your user ID +PGID=1000 # Replace with your group ID + +# Your timezone (find yours: timedatectl list-timezones) +TZ=America/New_York + +# Your server's IP address +SERVER_IP=192.168.1.100 # Replace with your actual IP + +# Directory paths +USERDIR=/home/yourusername/homelab # Update username +MEDIADIR=/mnt/media # Update if different +DOWNLOADDIR=/mnt/downloads # Update if different + +# Set secure passwords for services +GRAFANA_ADMIN_PASSWORD=your-secure-password-here +CODE_SERVER_PASSWORD=your-secure-password-here +POSTGRES_PASSWORD=your-secure-password-here +PGADMIN_PASSWORD=your-secure-password-here +JUPYTER_TOKEN=your-secure-token-here +PIHOLE_PASSWORD=your-secure-password-here +``` + +**Save and exit** (Ctrl+X, Y, Enter in nano) + +## Step 4: Create Docker Networks + +```bash +# Create the main homelab network +docker network create homelab-network + +# Create additional networks for better security +docker network create media-network +docker network create monitoring-network +docker network create database-network + +# Verify networks were created +docker network ls +``` + +## Step 5: Create Configuration Directories + +```bash +# Create the main config directory +mkdir -p config + +# Create config directories for services you plan to use +mkdir -p config/{nginx-proxy-manager,pihole,portainer} +mkdir -p config/{plex,sonarr,radarr,prowlarr,qbittorrent,jellyfin} +mkdir -p config/{prometheus,grafana,loki,promtail} +mkdir -p config/{code-server,postgres,redis} + +# Set proper permissions +sudo chown -R $(id -u):$(id -g) config/ +``` + +## Step 6: Copy Configuration Templates (Optional) + +For services that need config files: + +```bash +# Prometheus +mkdir -p config/prometheus +cp config-templates/prometheus/prometheus.yml config/prometheus/ + +# Loki +mkdir -p config/loki +cp config-templates/loki/loki-config.yml config/loki/ + +# Promtail +mkdir -p config/promtail +cp config-templates/promtail/promtail-config.yml config/promtail/ + +# Redis +mkdir -p config/redis +cp config-templates/redis/redis.conf config/redis/ +``` + +## Step 7: Start Your First Service (Portainer) + +Portainer provides a web UI for managing Docker containers. It's a great first service to deploy. + +```bash +# Start Portainer +docker compose -f docker-compose/infrastructure.yml up -d portainer + +# Check if it's running +docker compose -f docker-compose/infrastructure.yml ps + +# View logs +docker compose -f docker-compose/infrastructure.yml logs -f portainer +``` + +**Access Portainer:** +1. Open your browser +2. Navigate to `http://YOUR_SERVER_IP:9000` +3. Create an admin account on first login +4. Select "Get Started" and choose local Docker environment + +## Step 8: Deploy Infrastructure Services + +```bash +# Start all infrastructure services +docker compose -f docker-compose/infrastructure.yml up -d + +# Check status +docker compose -f docker-compose/infrastructure.yml ps + +# Services now running: +# - Nginx Proxy Manager: http://YOUR_SERVER_IP:81 +# - Pi-hole: http://YOUR_SERVER_IP:8080/admin +# - Portainer: http://YOUR_SERVER_IP:9000 +# - Watchtower: (runs in background, no UI) +``` + +### Configure Nginx Proxy Manager (Optional) + +1. Access: `http://YOUR_SERVER_IP:81` +2. Login with default credentials: + - Email: `admin@example.com` + - Password: `changeme` +3. Change password immediately +4. Add proxy hosts for your services + +### Configure Pi-hole (Optional) + +1. Access: `http://YOUR_SERVER_IP:8080/admin` +2. Login with password from `.env` (PIHOLE_PASSWORD) +3. Configure DNS settings +4. Update your router to use Pi-hole as DNS server + +## Step 9: Deploy Media Services (Optional) + +If you want a media server setup: + +```bash +# Start media services +docker compose -f docker-compose/media.yml up -d + +# Check status +docker compose -f docker-compose/media.yml ps + +# Services now running: +# - Plex: http://YOUR_SERVER_IP:32400/web +# - Jellyfin: http://YOUR_SERVER_IP:8096 +# - Sonarr: http://YOUR_SERVER_IP:8989 +# - Radarr: http://YOUR_SERVER_IP:7878 +# - Prowlarr: http://YOUR_SERVER_IP:9696 +# - qBittorrent: http://YOUR_SERVER_IP:8081 +``` + +### Initial Media Service Setup + +**Plex:** +1. Access: `http://YOUR_SERVER_IP:32400/web` +2. Sign in with your Plex account +3. Add your media libraries + +**Sonarr/Radarr:** +1. Access the web UI +2. Go to Settings → Profiles → Quality +3. Configure your quality preferences +4. Add Prowlarr as an indexer +5. Add qBittorrent as a download client + +## Step 10: Deploy Monitoring Services (Optional) + +For system and service monitoring: + +```bash +# Start monitoring services +docker compose -f docker-compose/monitoring.yml up -d + +# Check status +docker compose -f docker-compose/monitoring.yml ps + +# Services now running: +# - Prometheus: http://YOUR_SERVER_IP:9090 +# - Grafana: http://YOUR_SERVER_IP:3000 +# - Node Exporter: http://YOUR_SERVER_IP:9100 +# - cAdvisor: http://YOUR_SERVER_IP:8082 +# - Uptime Kuma: http://YOUR_SERVER_IP:3001 +``` + +### Configure Grafana + +1. Access: `http://YOUR_SERVER_IP:3000` +2. Login with credentials from `.env`: + - Username: `admin` + - Password: `GRAFANA_ADMIN_PASSWORD` from .env +3. Add Prometheus as a data source: + - URL: `http://prometheus:9090` +4. Import dashboards: + - Dashboard ID 1860 for Node Exporter + - Dashboard ID 893 for Docker metrics + +### Configure Uptime Kuma + +1. Access: `http://YOUR_SERVER_IP:3001` +2. Create an account on first login +3. Add monitors for your services + +## Step 11: Deploy Development Services (Optional) + +If you need development tools: + +```bash +# Start development services +docker compose -f docker-compose/development.yml up -d + +# Check status +docker compose -f docker-compose/development.yml ps + +# Services now running: +# - Code Server: http://YOUR_SERVER_IP:8443 +# - PostgreSQL: localhost:5432 +# - Redis: localhost:6379 +# - pgAdmin: http://YOUR_SERVER_IP:5050 +# - Jupyter Lab: http://YOUR_SERVER_IP:8888 +# - Node-RED: http://YOUR_SERVER_IP:1880 +``` + +## Step 12: Set Up VS Code with GitHub Copilot + +1. **Install VS Code** on your local machine (if not already installed) + +2. **Install GitHub Copilot extension:** + - Open VS Code + - Go to Extensions (Ctrl+Shift+X) + - Search for "GitHub Copilot" + - Click Install + - Sign in with your GitHub account + +3. **Clone your repository in VS Code:** + ```bash + # On your local machine + git clone https://github.com/kelinfoxy/AI-Homelab.git + cd AI-Homelab + code . + ``` + +4. **Start using AI assistance:** + - Open Copilot Chat (Ctrl+Shift+I or click the chat icon) + - The AI assistant automatically follows the guidelines in `.github/copilot-instructions.md` + - Ask questions like: + - "Help me add Home Assistant to my homelab" + - "Create a backup script for my Docker volumes" + - "How do I configure GPU support for Plex?" + +## Step 13: Verify Everything is Running + +```bash +# Check all running containers +docker ps + +# Check container health +docker ps --format "table {{.Names}}\t{{.Status}}" + +# View resource usage +docker stats --no-stream + +# Check disk usage +docker system df +``` + +## Step 14: Set Up Backups + +Create a backup script: + +```bash +# Create a backup directory +mkdir -p ~/backups + +# Create a simple backup script +cat > ~/backup-homelab.sh << 'EOF' +#!/bin/bash +BACKUP_DIR=~/backups +DATE=$(date +%Y%m%d) + +# Backup config directories +tar czf $BACKUP_DIR/config-$DATE.tar.gz ~/AI-Homelab/config/ + +# Backup .env file +cp ~/AI-Homelab/.env $BACKUP_DIR/.env-$DATE + +# Backup Docker volumes (example for Portainer) +docker run --rm \ + -v portainer-data:/data \ + -v $BACKUP_DIR:/backup \ + busybox tar czf /backup/portainer-data-$DATE.tar.gz /data + +echo "Backup completed: $DATE" +EOF + +# Make it executable +chmod +x ~/backup-homelab.sh + +# Test the backup +~/backup-homelab.sh +``` + +Set up a cron job for automated backups: + +```bash +# Open crontab +crontab -e + +# Add this line to run backup daily at 2 AM +0 2 * * * /home/yourusername/backup-homelab.sh +``` + +## Step 15: Configure Firewall (Optional but Recommended) + +If using UFW: + +```bash +# Allow SSH (if not already allowed) +sudo ufw allow 22/tcp + +# Allow web traffic (if exposing services to internet) +sudo ufw allow 80/tcp +sudo ufw allow 443/tcp + +# Allow specific services from local network only +# Replace 192.168.1.0/24 with your network +sudo ufw allow from 192.168.1.0/24 to any port 9000 proto tcp # Portainer +sudo ufw allow from 192.168.1.0/24 to any port 81 proto tcp # Nginx Proxy Manager +sudo ufw allow from 192.168.1.0/24 to any port 3000 proto tcp # Grafana + +# Enable firewall +sudo ufw enable +``` + +## Next Steps + +Now that your homelab is running: + +1. **Explore Services:** + - Access each service's web UI + - Configure settings as needed + - Set up integrations between services + +2. **Add More Services:** + - Ask GitHub Copilot for help adding new services + - Follow the patterns in existing compose files + - Check [awesome-selfhosted](https://github.com/awesome-selfhosted/awesome-selfhosted) for ideas + +3. **Optimize:** + - Review logs for errors + - Adjust resource limits + - Set up proper monitoring and alerts + +4. **Secure:** + - Change all default passwords + - Set up SSL certificates (use Nginx Proxy Manager) + - Enable 2FA where available + - Keep services updated + +5. **Learn:** + - Read the [Docker Guidelines](docker-guidelines.md) + - Experiment with new services + - Use AI assistance to understand and modify configurations + +## Troubleshooting + +### Can't access services + +1. **Check if service is running:** + ```bash + docker ps + ``` + +2. **Check service logs:** + ```bash + docker compose -f docker-compose/file.yml logs service-name + ``` + +3. **Verify network connectivity:** + ```bash + ping YOUR_SERVER_IP + ``` + +4. **Check firewall:** + ```bash + sudo ufw status + ``` + +### Permission errors + +```bash +# Fix config directory permissions +sudo chown -R $(id -u):$(id -g) config/ + +# Verify PUID/PGID in .env match your user +id -u # Should match PUID in .env +id -g # Should match PGID in .env +``` + +### Port already in use + +```bash +# Find what's using the port +sudo netstat -tlnp | grep PORT_NUMBER + +# Stop the conflicting service or change the port in docker-compose +``` + +### Out of disk space + +```bash +# Check disk usage +df -h + +# Clean up Docker resources +docker system prune -a + +# Remove old logs +docker compose logs --tail=0 service-name +``` + +## Getting Help + +- **Documentation:** Check the `docs/` directory for comprehensive guides +- **AI Assistance:** Use GitHub Copilot in VS Code for real-time help +- **Community:** Search for service-specific help in respective communities +- **Issues:** Open an issue on GitHub for problems with this repository + +## Success Checklist + +- [ ] Docker and Docker Compose installed +- [ ] Repository cloned +- [ ] `.env` file configured +- [ ] Networks created +- [ ] Config directories created +- [ ] Portainer running and accessible +- [ ] Infrastructure services deployed +- [ ] At least one service category deployed (media/monitoring/dev) +- [ ] VS Code with GitHub Copilot set up +- [ ] Backup strategy in place +- [ ] Firewall configured (if applicable) +- [ ] All services accessible and working + +Congratulations! Your AI-powered homelab is now running! 🎉