From 31309046d21942dd28c6ecca399457478cea4e30 Mon Sep 17 00:00:00 2001 From: Top1055 <123alexfeetham@gmail.com> Date: Sun, 1 Feb 2026 18:44:36 +0000 Subject: [PATCH] removed current roadmap --- PRODUCTION_ROADMAP.md | 849 ------------------------------------------ 1 file changed, 849 deletions(-) delete mode 100644 PRODUCTION_ROADMAP.md diff --git a/PRODUCTION_ROADMAP.md b/PRODUCTION_ROADMAP.md deleted file mode 100644 index 8cefbd5..0000000 --- a/PRODUCTION_ROADMAP.md +++ /dev/null @@ -1,849 +0,0 @@ -# Groovy-Zilean Production Roadmap - -**Goal:** Transform groovy-zilean from a personal project into a production-ready Discord music bot with a web dashboard. - -**Philosophy:** Production-quality architecture with manageable complexity for a solo developer. No overkill, no shortcuts. - ---- - -## Table of Contents -1. [Architecture Overview](#architecture-overview) -2. [Tech Stack Decisions](#tech-stack-decisions) -3. [Development Phases](#development-phases) -4. [Python Environment Setup](#python-environment-setup) -5. [Why This Approach](#why-this-approach) -6. [Quick Reference](#quick-reference) - ---- - -## Architecture Overview - -### The Winning Design: Database-Mediated Architecture - -``` -┌─────────────┐ HTTP/SSE ┌──────────────┐ -│ Browser │◄────────────►│ FastAPI │ -│ (HTMX HTML) │ │ Backend │ -└─────────────┘ └──────┬───────┘ - │ - ┌──────▼───────┐ ┌─────────────┐ - │ PostgreSQL │◄────────│ Discord Bot │ - │ Database │ │ (discord.py)│ - └──────────────┘ └─────────────┘ -``` - -### Key Principles - -1. **Decoupled Services** - - Bot and web can restart independently - - No tight coupling via IPC - - Database is the single source of truth - -2. **Simple Frontend (Initially)** - - HTMX for interactivity (no build step) - - Jinja2 templates (server-side rendering) - - Tailwind CSS via CDN (beautiful, no npm) - - **Optional:** Upgrade to React later when ready - -3. **Production-Ready Backend** - - PostgreSQL for reliability - - FastAPI for modern Python web - - Discord OAuth2 for authentication - - Connection pooling, proper error handling - ---- - -## Tech Stack Decisions - -### Backend - -| Component | Choice | Why | -|-----------|--------|-----| -| **Bot Framework** | discord.py 2.6.4+ | Industry standard, hybrid commands | -| **Web Framework** | FastAPI | Modern, async, auto-docs, large community | -| **Database** | PostgreSQL 14+ | Production-ready, ACID compliance, better than SQLite | -| **Music Extraction** | yt-dlp | Actively maintained, multi-platform support | -| **Auth** | Discord OAuth2 | Native Discord integration | -| **ORM** | SQLAlchemy (optional) | Or use asyncpg directly for simplicity | - -### Frontend (Phase 1) - -| Component | Choice | Why | -|-----------|--------|-----| -| **Templates** | Jinja2 | Server-side rendering, no build step | -| **Interactivity** | HTMX | Modern interactivity without React complexity | -| **Styling** | Tailwind CSS (CDN) | Beautiful UI, no build process | -| **Real-time** | Server-Sent Events | Simple polling/updates, no WebSocket complexity | - -### Frontend (Phase 2 - Optional) - -| Component | Choice | Why | -|-----------|--------|-----| -| **Framework** | React 18 | If you need more complex UI later | -| **Real-time** | WebSocket | For true real-time when needed | -| **State** | Zustand/Context | Simpler than Redux | - -### Infrastructure - -| Component | Choice | Why | -|-----------|--------|-----| -| **Python Version** | 3.11 or 3.12 | Modern features, better performance | -| **Environment** | venv | Isolated dependencies, no PATH conflicts | -| **Process Manager** | systemd | Reliable, built into Linux | -| **Reverse Proxy** | nginx | Standard, handles SSL, static files | - ---- - -## Development Phases - -### Phase 0: Current State ✅ -- Working Discord bot with music playback -- SQLite database -- Hybrid commands (slash + prefix) -- 17 audio effects -- Queue, loop, shuffle functionality -- Spotify integration - -### Phase 1: Quick Wins (1-2 hours) - -**Goal:** Immediate improvements for better UX - -Tasks: -1. **Lower default volume from 100% to 25%** - - Change default in `queue.py:119` - - Scale volume command display (user sees 0-200%, internally 0-50%) - - Prevents earrape for new users - -2. **Add .mp3 and .mp4 file support** - - Extend `translate.py` to detect direct file URLs - - Support HTTP/HTTPS links to audio files - - Validate file type before processing - -**Files to modify:** -- `cogs/music/queue.py` -- `cogs/music/translate.py` - ---- - -### Phase 2: Code Refactoring (4-6 hours) - -**Goal:** Clean, maintainable, documented codebase - -#### 2.1 Database Abstraction -Create `cogs/music/db_manager.py`: -- Database connection class with context manager -- Connection pooling preparation -- Centralize all SQL queries -- Remove scattered `sqlite3.connect()` calls - -#### 2.2 Configuration Management -Update `config.py`: -- Use environment variables for secrets -- Create `.env.example` template -- Remove hardcoded credentials -- Add validation for required config - -#### 2.3 Code Organization -``` -groovy-zilean/ -├── bot/ -│ ├── __init__.py -│ ├── bot.py # Main bot class -│ └── cogs/ -│ └── music/ -│ ├── __init__.py -│ ├── commands.py # User-facing commands -│ ├── player.py # Playback logic -│ ├── queue.py # Queue management -│ ├── effects.py # Audio effects -│ ├── db_manager.py # Database abstraction -│ └── translate.py # URL/playlist parsing -├── web/ -│ ├── __init__.py # (Future web app) -├── shared/ -│ ├── __init__.py -│ ├── config.py # Shared configuration -│ └── models.py # Data models -├── main.py # Entry point -├── requirements.txt -├── .env.example -└── README.md -``` - -#### 2.4 Error Handling & Logging -- Wrap all commands in try/except -- User-friendly error messages -- Proper logging setup (rotating file logs) -- Debug mode toggle - -#### 2.5 Type Hints & Documentation -- Add type hints to all functions -- Docstrings for all classes/methods -- Inline comments for complex logic - -**Expected outcome:** -- Easy to navigate codebase -- No secrets in code -- Consistent patterns throughout - ---- - -### Phase 3: PostgreSQL Migration (3-4 hours) - -**Goal:** Production-ready database layer - -#### 3.1 Local PostgreSQL Setup -```bash -# Install PostgreSQL -sudo apt update -sudo apt install postgresql postgresql-contrib - -# Create database and user -sudo -u postgres psql -CREATE DATABASE groovy_zilean; -CREATE USER groovy WITH PASSWORD 'your_password'; -GRANT ALL PRIVILEGES ON DATABASE groovy_zilean TO groovy; -``` - -#### 3.2 Database Schema Design -```sql --- servers table -CREATE TABLE servers ( - server_id BIGINT PRIMARY KEY, - is_playing BOOLEAN DEFAULT FALSE, - song_name TEXT, - song_url TEXT, - song_thumbnail TEXT, - loop_mode VARCHAR(10) DEFAULT 'off', - volume INTEGER DEFAULT 25, -- NEW default! - effect VARCHAR(20) DEFAULT 'none', - song_start_time DOUBLE PRECISION DEFAULT 0, - song_duration INTEGER DEFAULT 0, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP -); - --- songs/queue table -CREATE TABLE songs ( - id SERIAL PRIMARY KEY, - server_id BIGINT NOT NULL REFERENCES servers(server_id) ON DELETE CASCADE, - song_link TEXT, - queued_by TEXT, - position INTEGER NOT NULL, - title TEXT, - thumbnail TEXT, - duration INTEGER, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, - UNIQUE(server_id, position) -); - --- Future: users table for web auth -CREATE TABLE users ( - discord_id BIGINT PRIMARY KEY, - username TEXT, - avatar TEXT, - access_token TEXT, - refresh_token TEXT, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, - last_login TIMESTAMP -); - --- Future: permissions table -CREATE TABLE permissions ( - id SERIAL PRIMARY KEY, - server_id BIGINT REFERENCES servers(server_id), - user_id BIGINT, - role_id BIGINT, - can_play BOOLEAN DEFAULT TRUE, - can_skip BOOLEAN DEFAULT FALSE, - can_clear BOOLEAN DEFAULT FALSE, - can_modify_settings BOOLEAN DEFAULT FALSE -); -``` - -#### 3.3 Migration Script -Create `scripts/migrate_to_postgres.py`: -- Read all data from SQLite -- Insert into PostgreSQL -- Validate migration -- Backup SQLite file - -#### 3.4 Update Database Code -Replace all `sqlite3` calls with `asyncpg` or `psycopg3`: -```python -# Old (SQLite) -conn = sqlite3.connect(db_path) -cursor = conn.cursor() - -# New (PostgreSQL with asyncpg) -async with pool.acquire() as conn: - result = await conn.fetch("SELECT * FROM servers WHERE server_id = $1", server_id) -``` - -#### 3.5 Connection Pooling -```python -# In bot startup -self.db_pool = await asyncpg.create_pool( - host='localhost', - database='groovy_zilean', - user='groovy', - password=os.getenv('DB_PASSWORD'), - min_size=5, - max_size=20 -) -``` - -**Expected outcome:** -- Reliable database with ACID guarantees -- Better concurrent access handling -- Ready for multi-server production load - ---- - -### Phase 4: Web Dashboard (20-30 hours) - -**Goal:** User-friendly web interface for bot control - -#### 4.1 FastAPI Backend Setup - -**Project structure:** -``` -web/ -├── __init__.py -├── main.py # FastAPI app -├── routes/ -│ ├── __init__.py -│ ├── auth.py # Discord OAuth2 -│ ├── servers.py # Server list/select -│ └── playback.py # Queue/controls -├── templates/ -│ ├── base.html -│ ├── index.html -│ ├── dashboard.html -│ └── components/ -│ ├── queue.html -│ └── controls.html -├── static/ -│ ├── css/ -│ └── js/ -└── dependencies.py # Auth dependencies -``` - -**Core dependencies:** -```bash -pip install fastapi uvicorn jinja2 python-multipart httpx -``` - -#### 4.2 Discord OAuth2 Authentication - -**Flow:** -1. User clicks "Login with Discord" -2. Redirect to Discord OAuth -3. Discord redirects back with code -4. Exchange code for token -5. Fetch user info + guilds -6. Create session -7. Show dashboard with user's servers - -**Implementation:** -```python -# routes/auth.py -@router.get("/login") -async def login(): - # Redirect to Discord OAuth - discord_auth_url = ( - f"https://discord.com/api/oauth2/authorize" - f"?client_id={DISCORD_CLIENT_ID}" - f"&redirect_uri={REDIRECT_URI}" - f"&response_type=code" - f"&scope=identify guilds" - ) - return RedirectResponse(discord_auth_url) - -@router.get("/callback") -async def callback(code: str): - # Exchange code for token - # Fetch user info - # Create session - # Redirect to dashboard -``` - -#### 4.3 HTMX Frontend - -**Example dashboard with HTMX:** -```html - -
- - - - -
- -
- - -
- -
- - -
- - - -
- - -
- - -
-
-``` - -**Why HTMX is perfect here:** -- No JavaScript needed for interactivity -- Server renders everything (simpler) -- Auto-updates with `hx-trigger="every Xs"` -- Progressive enhancement (works without JS) - -#### 4.4 API Endpoints - -**Read endpoints:** -```python -GET /api/servers # User's servers (with bot) -GET /api/servers/{id}/queue # Current queue -GET /api/servers/{id}/status # Now playing, volume, etc. -``` - -**Write endpoints:** -```python -POST /api/servers/{id}/play # Add song (body: {query: "..."}) -POST /api/servers/{id}/skip # Skip current song -POST /api/servers/{id}/volume # Set volume (body: {volume: 150}) -POST /api/servers/{id}/effect # Set effect (body: {effect: "nightcore"}) -POST /api/servers/{id}/loop # Set loop mode (body: {mode: "queue"}) -POST /api/servers/{id}/shuffle # Shuffle queue -POST /api/servers/{id}/clear # Clear queue -``` - -**Implementation example:** -```python -# routes/playback.py -@router.post("/api/servers/{server_id}/skip") -async def skip_song( - server_id: int, - user: User = Depends(get_current_user), - db = Depends(get_db) -): - # 1. Check user is in server - if server_id not in user.guild_ids: - raise HTTPException(403, "Not in this server") - - # 2. Check permissions (future) - # if not has_permission(user, server_id, "can_skip"): - # raise HTTPException(403, "No permission") - - # 3. Write command to database - await db.execute( - "INSERT INTO commands (server_id, action, user_id) VALUES ($1, $2, $3)", - server_id, "skip", user.id - ) - - # 4. Return updated queue - return await get_queue(server_id, db) -``` - -#### 4.5 Bot Integration (Command Processing) - -**Add to bot:** -```python -# In bot.py or new cogs/web_commands.py -@tasks.loop(seconds=1) -async def process_web_commands(self): - """Process commands from web dashboard""" - async with self.db_pool.acquire() as conn: - # Fetch unprocessed commands - commands = await conn.fetch( - "SELECT * FROM commands WHERE processed = FALSE" - ) - - for cmd in commands: - server_id = cmd['server_id'] - action = cmd['action'] - data = cmd['data'] - - guild = self.get_guild(server_id) - if not guild or not guild.voice_client: - continue - - # Execute command - if action == "skip": - guild.voice_client.stop() - elif action == "volume": - # Set volume in database, next song picks it up - await queue.set_volume(server_id, data['volume']) - elif action == "play": - # Queue song from web - # ... (use existing play logic) - - # Mark as processed - await conn.execute( - "UPDATE commands SET processed = TRUE WHERE id = $1", - cmd['id'] - ) -``` - -#### 4.6 Permissions System - -**Basic implementation:** -```python -# Check if user can control bot -async def can_control_bot(user_id: int, server_id: int, action: str) -> bool: - # Check if user is in voice channel with bot - # Check if user has DJ role (configurable per server) - # Check specific permission for action - # Default: anyone in VC can control - pass -``` - -**Advanced (Phase 5):** -- Role-based permissions -- User-specific permissions -- Configurable via web dashboard - -**Expected outcome:** -- Beautiful, functional web dashboard -- Discord OAuth login -- Real-time queue display -- Full playback control from browser -- Works on mobile - ---- - -### Phase 5: Permissions & Production (4-6 hours) - -**Goal:** Production-ready deployment - -#### 5.1 Permission System -- DJ role configuration -- Per-server permission settings -- Web UI for permission management - -#### 5.2 Rate Limiting -```python -from slowapi import Limiter -limiter = Limiter(key_func=get_remote_address) - -@app.post("/api/play") -@limiter.limit("10/minute") # Max 10 songs per minute -async def play_song(...): - ... -``` - -#### 5.3 Logging & Monitoring -```python -import logging -from logging.handlers import RotatingFileHandler - -# Setup logging -handler = RotatingFileHandler( - 'logs/bot.log', - maxBytes=10_000_000, # 10MB - backupCount=5 -) -logging.basicConfig( - handlers=[handler], - level=logging.INFO, - format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' -) -``` - -#### 5.4 Systemd Services -```ini -# /etc/systemd/system/groovy-bot.service -[Unit] -Description=Groovy Zilean Discord Bot -After=network.target postgresql.service - -[Service] -Type=simple -User=groovy -WorkingDirectory=/home/groovy/groovy-zilean -Environment="PATH=/home/groovy/groovy-zilean/venv/bin" -ExecStart=/home/groovy/groovy-zilean/venv/bin/python main.py -Restart=always -RestartSec=10 - -[Install] -WantedBy=multi-user.target -``` - -```ini -# /etc/systemd/system/groovy-web.service -[Unit] -Description=Groovy Zilean Web Dashboard -After=network.target postgresql.service - -[Service] -Type=simple -User=groovy -WorkingDirectory=/home/groovy/groovy-zilean -Environment="PATH=/home/groovy/groovy-zilean/venv/bin" -ExecStart=/home/groovy/groovy-zilean/venv/bin/uvicorn web.main:app --host 0.0.0.0 --port 8000 -Restart=always -RestartSec=10 - -[Install] -WantedBy=multi-user.target -``` - -#### 5.5 Nginx Configuration -```nginx -server { - listen 80; - server_name groovy.yourdomain.com; - - # Redirect to HTTPS - return 301 https://$host$request_uri; -} - -server { - listen 443 ssl http2; - server_name groovy.yourdomain.com; - - ssl_certificate /etc/letsencrypt/live/groovy.yourdomain.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/groovy.yourdomain.com/privkey.pem; - - location / { - proxy_pass http://localhost:8000; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - } - - location /static { - alias /home/groovy/groovy-zilean/web/static; - expires 30d; - } -} -``` - -#### 5.6 Database Backups -```bash -#!/bin/bash -# scripts/backup_db.sh - -BACKUP_DIR="/home/groovy/backups" -DATE=$(date +%Y%m%d_%H%M%S) - -pg_dump groovy_zilean > "$BACKUP_DIR/groovy_zilean_$DATE.sql" - -# Keep only last 7 days -find $BACKUP_DIR -name "groovy_zilean_*.sql" -mtime +7 -delete -``` - -Add to crontab: -```bash -0 2 * * * /home/groovy/groovy-zilean/scripts/backup_db.sh -``` - -#### 5.7 Environment Variables -```bash -# .env (NEVER commit this!) -DISCORD_TOKEN=your_bot_token_here -DISCORD_CLIENT_ID=your_client_id -DISCORD_CLIENT_SECRET=your_client_secret -SPOTIFY_CLIENT_ID=your_spotify_id -SPOTIFY_CLIENT_SECRET=your_spotify_secret -DB_PASSWORD=your_db_password -SECRET_KEY=random_secret_for_sessions -ENVIRONMENT=production -``` - -**Expected outcome:** -- Production-ready deployment -- Automatic restarts on failure -- HTTPS enabled -- Automated backups -- Proper logging - ---- - -## Python Environment Setup - -### Avoiding Python Version Hell - -**Step 1: Install Python 3.12** -```bash -# On Debian/Ubuntu -sudo apt update -sudo apt install python3.12 python3.12-venv python3.12-dev - -# Verify installation -python3.12 --version -``` - -**Step 2: Create Virtual Environment** -```bash -cd ~/coding/groovy-zilean - -# Create venv (only once) -python3.12 -m venv venv - -# Activate (every time you work on project) -source venv/bin/activate - -# Your prompt should change to show (venv) -``` - -**Step 3: Install Dependencies** -```bash -# Make sure venv is activated! -pip install --upgrade pip -pip install -r requirements.txt -``` - -**Step 4: Add to .gitignore** -``` -venv/ -__pycache__/ -*.pyc -.env -*.db -logs/ -``` - -**Helpful Aliases (add to ~/.bashrc)** -```bash -alias groovy='cd ~/coding/groovy-zilean && source venv/bin/activate' -``` - -Then just type `groovy` to activate your environment! - ---- - -## Why This Approach? - -### Rejected: discord-ext-ipc + Quart -❌ **discord-ext-ipc is unmaintained** (last update 2+ years ago) -❌ Tight coupling between bot and web -❌ Both processes must run together -❌ Quart less popular than FastAPI -❌ Still need polling for real-time updates - -### Rejected: FastAPI + React + Redis + WebSocket -❌ **Overkill for solo developer** -❌ Too many moving parts (4+ services) -❌ npm/node_modules complexity -❌ Requires learning React well -❌ WebSocket complexity for minimal gain - -### Chosen: FastAPI + PostgreSQL + HTMX ✅ -✅ **Production-quality architecture** -✅ Decoupled services (independent restarts) -✅ Modern, well-maintained tools -✅ No frontend build step (initially) -✅ Easy upgrade path to React later -✅ Manageable complexity for solo dev -✅ Database as source of truth -✅ All Python backend - ---- - -## Quick Reference - -### Daily Development Workflow -```bash -# Activate environment -cd ~/coding/groovy-zilean -source venv/bin/activate - -# Run bot -python main.py - -# Run web (in another terminal) -source venv/bin/activate -uvicorn web.main:app --reload --port 8000 - -# Run tests (future) -pytest - -# Database migrations (future) -alembic upgrade head -``` - -### Project Commands -```bash -# Install new package -pip install package_name -pip freeze > requirements.txt - -# Database -psql groovy_zilean # Connect to database -pg_dump groovy_zilean > backup.sql # Backup - -# Systemd -sudo systemctl start groovy-bot -sudo systemctl status groovy-bot -sudo journalctl -u groovy-bot -f # View logs -``` - -### File Locations -- **Bot entry point:** `main.py` -- **Config:** `shared/config.py` + `.env` -- **Database:** PostgreSQL (not file-based) -- **Logs:** `logs/bot.log`, `logs/web.log` -- **Web templates:** `web/templates/` - ---- - -## Success Metrics - -By the end of this roadmap, you'll have: - -✅ Clean, maintainable codebase -✅ Production-ready database -✅ Web dashboard with Discord auth -✅ Mobile-friendly interface -✅ Automated deployment -✅ Backup system -✅ Proper error handling & logging -✅ Permission system -✅ Rate limiting -✅ No earrape (25% default volume!) -✅ .mp3/.mp4 file support - -**Most importantly:** A bot that real servers can use, that you can maintain solo, and that you're proud of! - ---- - -## Next Steps - -1. **Today:** Quick wins (volume + file support) -2. **This week:** Refactoring -3. **Next week:** PostgreSQL migration -4. **Week after:** Web dashboard MVP -5. **Final week:** Production deployment - -Let's build something awesome! 🎵⏱️