groovy-zilean/PRODUCTION_ROADMAP.md

# Groovy-Zilean Production Roadmap

**Goal:** Transform groovy-zilean from a personal project into a production-ready Discord music bot with a web dashboard.

**Philosophy:** Production-quality architecture with manageable complexity for a solo developer. No overkill, no shortcuts.

---

## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Tech Stack Decisions](#tech-stack-decisions)
3. [Development Phases](#development-phases)
4. [Python Environment Setup](#python-environment-setup)
5. [Why This Approach](#why-this-approach)
6. [Quick Reference](#quick-reference)

---

## Architecture Overview

### The Winning Design: Database-Mediated Architecture

```
┌─────────────┐    HTTP/SSE  ┌──────────────┐
│   Browser   │◄────────────►│   FastAPI    │
│ (HTMX HTML) │              │   Backend    │
└─────────────┘              └──────┬───────┘
                                    │
                             ┌──────▼───────┐         ┌─────────────┐
                             │  PostgreSQL  │◄────────│ Discord Bot │
                             │   Database   │         │ (discord.py)│
                             └──────────────┘         └─────────────┘
```

### Key Principles

1. **Decoupled Services**
   - Bot and web can restart independently
   - No tight coupling via IPC
   - Database is the single source of truth

2. **Simple Frontend (Initially)**
   - HTMX for interactivity (no build step)
   - Jinja2 templates (server-side rendering)
   - Tailwind CSS via CDN (beautiful, no npm)
   - **Optional:** Upgrade to React later when ready

3. **Production-Ready Backend**
   - PostgreSQL for reliability
   - FastAPI for modern Python web
   - Discord OAuth2 for authentication
   - Connection pooling, proper error handling

---

## Tech Stack Decisions

### Backend

| Component | Choice | Why |
|-----------|--------|-----|
| **Bot Framework** | discord.py 2.6.4+ | Industry standard, hybrid commands |
| **Web Framework** | FastAPI | Modern, async, auto-docs, large community |
| **Database** | PostgreSQL 14+ | Production-ready, ACID compliance, better than SQLite |
| **Music Extraction** | yt-dlp | Actively maintained, multi-platform support |
| **Auth** | Discord OAuth2 | Native Discord integration |
| **ORM** | SQLAlchemy (optional) | Or use asyncpg directly for simplicity |

### Frontend (Phase 1)

| Component | Choice | Why |
|-----------|--------|-----|
| **Templates** | Jinja2 | Server-side rendering, no build step |
| **Interactivity** | HTMX | Modern interactivity without React complexity |
| **Styling** | Tailwind CSS (CDN) | Beautiful UI, no build process |
| **Real-time** | Server-Sent Events | Simple polling/updates, no WebSocket complexity |

### Frontend (Phase 2 - Optional)

| Component | Choice | Why |
|-----------|--------|-----|
| **Framework** | React 18 | If you need more complex UI later |
| **Real-time** | WebSocket | For true real-time when needed |
| **State** | Zustand/Context | Simpler than Redux |

### Infrastructure

| Component | Choice | Why |
|-----------|--------|-----|
| **Python Version** | 3.11 or 3.12 | Modern features, better performance |
| **Environment** | venv | Isolated dependencies, no PATH conflicts |
| **Process Manager** | systemd | Reliable, built into Linux |
| **Reverse Proxy** | nginx | Standard, handles SSL, static files |

---

## Development Phases

### Phase 0: Current State ✅
- Working Discord bot with music playback
- SQLite database
- Hybrid commands (slash + prefix)
- 17 audio effects
- Queue, loop, shuffle functionality
- Spotify integration

### Phase 1: Quick Wins (1-2 hours)

**Goal:** Immediate improvements for better UX

Tasks:
1. **Lower default volume from 100% to 25%**
   - Change default in `queue.py:119`
   - Scale volume command display (user sees 0-200%, internally 0-50%)
   - Prevents earrape for new users

2. **Add .mp3 and .mp4 file support**
   - Extend `translate.py` to detect direct file URLs
   - Support HTTP/HTTPS links to audio files
   - Validate file type before processing

**Files to modify:**
- `cogs/music/queue.py`
- `cogs/music/translate.py`

---

### Phase 2: Code Refactoring (4-6 hours)

**Goal:** Clean, maintainable, documented codebase

#### 2.1 Database Abstraction
Create `cogs/music/db_manager.py`:
- Database connection class with context manager
- Connection pooling preparation
- Centralize all SQL queries
- Remove scattered `sqlite3.connect()` calls

#### 2.2 Configuration Management
Update `config.py`:
- Use environment variables for secrets
- Create `.env.example` template
- Remove hardcoded credentials
- Add validation for required config

#### 2.3 Code Organization
```
groovy-zilean/
├── bot/
│   ├── __init__.py
│   ├── bot.py              # Main bot class
│   └── cogs/
│       └── music/
│           ├── __init__.py
│           ├── commands.py     # User-facing commands
│           ├── player.py       # Playback logic
│           ├── queue.py        # Queue management
│           ├── effects.py      # Audio effects
│           ├── db_manager.py   # Database abstraction
│           └── translate.py    # URL/playlist parsing
├── web/
│   ├── __init__.py         # (Future web app)
├── shared/
│   ├── __init__.py
│   ├── config.py           # Shared configuration
│   └── models.py           # Data models
├── main.py                 # Entry point
├── requirements.txt
├── .env.example
└── README.md
```

#### 2.4 Error Handling & Logging
- Wrap all commands in try/except
- User-friendly error messages
- Proper logging setup (rotating file logs)
- Debug mode toggle

#### 2.5 Type Hints & Documentation
- Add type hints to all functions
- Docstrings for all classes/methods
- Inline comments for complex logic

**Expected outcome:**
- Easy to navigate codebase
- No secrets in code
- Consistent patterns throughout

---

### Phase 3: PostgreSQL Migration (3-4 hours)

**Goal:** Production-ready database layer

#### 3.1 Local PostgreSQL Setup
```bash
# Install PostgreSQL
sudo apt update
sudo apt install postgresql postgresql-contrib

# Create database and user
sudo -u postgres psql
CREATE DATABASE groovy_zilean;
CREATE USER groovy WITH PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE groovy_zilean TO groovy;
```

#### 3.2 Database Schema Design
```sql
-- servers table
CREATE TABLE servers (
    server_id BIGINT PRIMARY KEY,
    is_playing BOOLEAN DEFAULT FALSE,
    song_name TEXT,
    song_url TEXT,
    song_thumbnail TEXT,
    loop_mode VARCHAR(10) DEFAULT 'off',
    volume INTEGER DEFAULT 25,  -- NEW default!
    effect VARCHAR(20) DEFAULT 'none',
    song_start_time DOUBLE PRECISION DEFAULT 0,
    song_duration INTEGER DEFAULT 0,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- songs/queue table
CREATE TABLE songs (
    id SERIAL PRIMARY KEY,
    server_id BIGINT NOT NULL REFERENCES servers(server_id) ON DELETE CASCADE,
    song_link TEXT,
    queued_by TEXT,
    position INTEGER NOT NULL,
    title TEXT,
    thumbnail TEXT,
    duration INTEGER,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(server_id, position)
);

-- Future: users table for web auth
CREATE TABLE users (
    discord_id BIGINT PRIMARY KEY,
    username TEXT,
    avatar TEXT,
    access_token TEXT,
    refresh_token TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    last_login TIMESTAMP
);

-- Future: permissions table
CREATE TABLE permissions (
    id SERIAL PRIMARY KEY,
    server_id BIGINT REFERENCES servers(server_id),
    user_id BIGINT,
    role_id BIGINT,
    can_play BOOLEAN DEFAULT TRUE,
    can_skip BOOLEAN DEFAULT FALSE,
    can_clear BOOLEAN DEFAULT FALSE,
    can_modify_settings BOOLEAN DEFAULT FALSE
);
```

#### 3.3 Migration Script
Create `scripts/migrate_to_postgres.py`:
- Read all data from SQLite
- Insert into PostgreSQL
- Validate migration
- Backup SQLite file

#### 3.4 Update Database Code
Replace all `sqlite3` calls with `asyncpg` or `psycopg3`:
```python
# Old (SQLite)
conn = sqlite3.connect(db_path)
cursor = conn.cursor()

# New (PostgreSQL with asyncpg)
async with pool.acquire() as conn:
    result = await conn.fetch("SELECT * FROM servers WHERE server_id = $1", server_id)
```

#### 3.5 Connection Pooling
```python
# In bot startup
self.db_pool = await asyncpg.create_pool(
    host='localhost',
    database='groovy_zilean',
    user='groovy',
    password=os.getenv('DB_PASSWORD'),
    min_size=5,
    max_size=20
)
```

**Expected outcome:**
- Reliable database with ACID guarantees
- Better concurrent access handling
- Ready for multi-server production load

---

### Phase 4: Web Dashboard (20-30 hours)

**Goal:** User-friendly web interface for bot control

#### 4.1 FastAPI Backend Setup

**Project structure:**
```
web/
├── __init__.py
├── main.py              # FastAPI app
├── routes/
│   ├── __init__.py
│   ├── auth.py          # Discord OAuth2
│   ├── servers.py       # Server list/select
│   └── playback.py      # Queue/controls
├── templates/
│   ├── base.html
│   ├── index.html
│   ├── dashboard.html
│   └── components/
│       ├── queue.html
│       └── controls.html
├── static/
│   ├── css/
│   └── js/
└── dependencies.py      # Auth dependencies
```

**Core dependencies:**
```bash
pip install fastapi uvicorn jinja2 python-multipart httpx
```

#### 4.2 Discord OAuth2 Authentication

**Flow:**
1. User clicks "Login with Discord"
2. Redirect to Discord OAuth
3. Discord redirects back with code
4. Exchange code for token
5. Fetch user info + guilds
6. Create session
7. Show dashboard with user's servers

**Implementation:**
```python
# routes/auth.py
@router.get("/login")
async def login():
    # Redirect to Discord OAuth
    discord_auth_url = (
        f"https://discord.com/api/oauth2/authorize"
        f"?client_id={DISCORD_CLIENT_ID}"
        f"&redirect_uri={REDIRECT_URI}"
        f"&response_type=code"
        f"&scope=identify guilds"
    )
    return RedirectResponse(discord_auth_url)

@router.get("/callback")
async def callback(code: str):
    # Exchange code for token
    # Fetch user info
    # Create session
    # Redirect to dashboard
```

#### 4.3 HTMX Frontend

**Example dashboard with HTMX:**
```html
<!-- templates/dashboard.html -->
<div class="container">
    <!-- Server Selector -->
    <select hx-get="/api/servers/{value}/queue"
            hx-target="#queue-container"
            hx-trigger="change">
        {% for server in user_servers %}
            <option value="{{ server.id }}">{{ server.name }}</option>
        {% endfor %}
    </select>

    <!-- Now Playing (auto-updates every 5s) -->
    <div id="now-playing"
         hx-get="/api/now-playing"
         hx-trigger="every 5s">
        <!-- Server renders this -->
    </div>

    <!-- Queue (auto-updates) -->
    <div id="queue-container"
         hx-get="/api/queue"
         hx-trigger="every 3s">
        <!-- Queue items here -->
    </div>

    <!-- Controls -->
    <div class="controls">
        <button hx-post="/api/skip"
                hx-target="#queue-container">
            ⏭️ Skip
        </button>

        <input type="range"
               min="0" max="200"
               hx-post="/api/volume"
               hx-trigger="change"
               hx-vals='{"volume": this.value}'>
    </div>

    <!-- Add Song Form -->
    <form hx-post="/api/play"
          hx-target="#queue-container">
        <input name="query" placeholder="YouTube URL or search">
        <button type="submit">Add to Queue</button>
    </form>
</div>
```

**Why HTMX is perfect here:**
- No JavaScript needed for interactivity
- Server renders everything (simpler)
- Auto-updates with `hx-trigger="every Xs"`
- Progressive enhancement (works without JS)

#### 4.4 API Endpoints

**Read endpoints:**
```python
GET  /api/servers              # User's servers (with bot)
GET  /api/servers/{id}/queue   # Current queue
GET  /api/servers/{id}/status  # Now playing, volume, etc.
```

**Write endpoints:**
```python
POST /api/servers/{id}/play    # Add song (body: {query: "..."})
POST /api/servers/{id}/skip    # Skip current song
POST /api/servers/{id}/volume  # Set volume (body: {volume: 150})
POST /api/servers/{id}/effect  # Set effect (body: {effect: "nightcore"})
POST /api/servers/{id}/loop    # Set loop mode (body: {mode: "queue"})
POST /api/servers/{id}/shuffle # Shuffle queue
POST /api/servers/{id}/clear   # Clear queue
```

**Implementation example:**
```python
# routes/playback.py
@router.post("/api/servers/{server_id}/skip")
async def skip_song(
    server_id: int,
    user: User = Depends(get_current_user),
    db = Depends(get_db)
):
    # 1. Check user is in server
    if server_id not in user.guild_ids:
        raise HTTPException(403, "Not in this server")

    # 2. Check permissions (future)
    # if not has_permission(user, server_id, "can_skip"):
    #     raise HTTPException(403, "No permission")

    # 3. Write command to database
    await db.execute(
        "INSERT INTO commands (server_id, action, user_id) VALUES ($1, $2, $3)",
        server_id, "skip", user.id
    )

    # 4. Return updated queue
    return await get_queue(server_id, db)
```

#### 4.5 Bot Integration (Command Processing)

**Add to bot:**
```python
# In bot.py or new cogs/web_commands.py
@tasks.loop(seconds=1)
async def process_web_commands(self):
    """Process commands from web dashboard"""
    async with self.db_pool.acquire() as conn:
        # Fetch unprocessed commands
        commands = await conn.fetch(
            "SELECT * FROM commands WHERE processed = FALSE"
        )

        for cmd in commands:
            server_id = cmd['server_id']
            action = cmd['action']
            data = cmd['data']

            guild = self.get_guild(server_id)
            if not guild or not guild.voice_client:
                continue

            # Execute command
            if action == "skip":
                guild.voice_client.stop()
            elif action == "volume":
                # Set volume in database, next song picks it up
                await queue.set_volume(server_id, data['volume'])
            elif action == "play":
                # Queue song from web
                # ... (use existing play logic)

            # Mark as processed
            await conn.execute(
                "UPDATE commands SET processed = TRUE WHERE id = $1",
                cmd['id']
            )
```

#### 4.6 Permissions System

**Basic implementation:**
```python
# Check if user can control bot
async def can_control_bot(user_id: int, server_id: int, action: str) -> bool:
    # Check if user is in voice channel with bot
    # Check if user has DJ role (configurable per server)
    # Check specific permission for action
    # Default: anyone in VC can control
    pass
```

**Advanced (Phase 5):**
- Role-based permissions
- User-specific permissions
- Configurable via web dashboard

**Expected outcome:**
- Beautiful, functional web dashboard
- Discord OAuth login
- Real-time queue display
- Full playback control from browser
- Works on mobile

---

### Phase 5: Permissions & Production (4-6 hours)

**Goal:** Production-ready deployment

#### 5.1 Permission System
- DJ role configuration
- Per-server permission settings
- Web UI for permission management

#### 5.2 Rate Limiting
```python
from slowapi import Limiter
limiter = Limiter(key_func=get_remote_address)

@app.post("/api/play")
@limiter.limit("10/minute")  # Max 10 songs per minute
async def play_song(...):
    ...
```

#### 5.3 Logging & Monitoring
```python
import logging
from logging.handlers import RotatingFileHandler

# Setup logging
handler = RotatingFileHandler(
    'logs/bot.log',
    maxBytes=10_000_000,  # 10MB
    backupCount=5
)
logging.basicConfig(
    handlers=[handler],
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
```

#### 5.4 Systemd Services
```ini
# /etc/systemd/system/groovy-bot.service
[Unit]
Description=Groovy Zilean Discord Bot
After=network.target postgresql.service

[Service]
Type=simple
User=groovy
WorkingDirectory=/home/groovy/groovy-zilean
Environment="PATH=/home/groovy/groovy-zilean/venv/bin"
ExecStart=/home/groovy/groovy-zilean/venv/bin/python main.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

```ini
# /etc/systemd/system/groovy-web.service
[Unit]
Description=Groovy Zilean Web Dashboard
After=network.target postgresql.service

[Service]
Type=simple
User=groovy
WorkingDirectory=/home/groovy/groovy-zilean
Environment="PATH=/home/groovy/groovy-zilean/venv/bin"
ExecStart=/home/groovy/groovy-zilean/venv/bin/uvicorn web.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

#### 5.5 Nginx Configuration
```nginx
server {
    listen 80;
    server_name groovy.yourdomain.com;

    # Redirect to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name groovy.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/groovy.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/groovy.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /static {
        alias /home/groovy/groovy-zilean/web/static;
        expires 30d;
    }
}
```

#### 5.6 Database Backups
```bash
#!/bin/bash
# scripts/backup_db.sh

BACKUP_DIR="/home/groovy/backups"
DATE=$(date +%Y%m%d_%H%M%S)

pg_dump groovy_zilean > "$BACKUP_DIR/groovy_zilean_$DATE.sql"

# Keep only last 7 days
find $BACKUP_DIR -name "groovy_zilean_*.sql" -mtime +7 -delete
```

Add to crontab:
```bash
0 2 * * * /home/groovy/groovy-zilean/scripts/backup_db.sh
```

#### 5.7 Environment Variables
```bash
# .env (NEVER commit this!)
DISCORD_TOKEN=your_bot_token_here
DISCORD_CLIENT_ID=your_client_id
DISCORD_CLIENT_SECRET=your_client_secret
SPOTIFY_CLIENT_ID=your_spotify_id
SPOTIFY_CLIENT_SECRET=your_spotify_secret
DB_PASSWORD=your_db_password
SECRET_KEY=random_secret_for_sessions
ENVIRONMENT=production
```

**Expected outcome:**
- Production-ready deployment
- Automatic restarts on failure
- HTTPS enabled
- Automated backups
- Proper logging

---

## Python Environment Setup

### Avoiding Python Version Hell

**Step 1: Install Python 3.12**
```bash
# On Debian/Ubuntu
sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev

# Verify installation
python3.12 --version
```

**Step 2: Create Virtual Environment**
```bash
cd ~/coding/groovy-zilean

# Create venv (only once)
python3.12 -m venv venv

# Activate (every time you work on project)
source venv/bin/activate

# Your prompt should change to show (venv)
```

**Step 3: Install Dependencies**
```bash
# Make sure venv is activated!
pip install --upgrade pip
pip install -r requirements.txt
```

**Step 4: Add to .gitignore**
```
venv/
__pycache__/
*.pyc
.env
*.db
logs/
```

**Helpful Aliases (add to ~/.bashrc)**
```bash
alias groovy='cd ~/coding/groovy-zilean && source venv/bin/activate'
```

Then just type `groovy` to activate your environment!

---

## Why This Approach?

### Rejected: discord-ext-ipc + Quart
❌ **discord-ext-ipc is unmaintained** (last update 2+ years ago)
❌ Tight coupling between bot and web
❌ Both processes must run together
❌ Quart less popular than FastAPI
❌ Still need polling for real-time updates

### Rejected: FastAPI + React + Redis + WebSocket
❌ **Overkill for solo developer**
❌ Too many moving parts (4+ services)
❌ npm/node_modules complexity
❌ Requires learning React well
❌ WebSocket complexity for minimal gain

### Chosen: FastAPI + PostgreSQL + HTMX ✅
✅ **Production-quality architecture**
✅ Decoupled services (independent restarts)
✅ Modern, well-maintained tools
✅ No frontend build step (initially)
✅ Easy upgrade path to React later
✅ Manageable complexity for solo dev
✅ Database as source of truth
✅ All Python backend

---

## Quick Reference

### Daily Development Workflow
```bash
# Activate environment
cd ~/coding/groovy-zilean
source venv/bin/activate

# Run bot
python main.py

# Run web (in another terminal)
source venv/bin/activate
uvicorn web.main:app --reload --port 8000

# Run tests (future)
pytest

# Database migrations (future)
alembic upgrade head
```

### Project Commands
```bash
# Install new package
pip install package_name
pip freeze > requirements.txt

# Database
psql groovy_zilean  # Connect to database
pg_dump groovy_zilean > backup.sql  # Backup

# Systemd
sudo systemctl start groovy-bot
sudo systemctl status groovy-bot
sudo journalctl -u groovy-bot -f  # View logs
```

### File Locations
- **Bot entry point:** `main.py`
- **Config:** `shared/config.py` + `.env`
- **Database:** PostgreSQL (not file-based)
- **Logs:** `logs/bot.log`, `logs/web.log`
- **Web templates:** `web/templates/`

---

## Success Metrics

By the end of this roadmap, you'll have:

✅ Clean, maintainable codebase
✅ Production-ready database
✅ Web dashboard with Discord auth
✅ Mobile-friendly interface
✅ Automated deployment
✅ Backup system
✅ Proper error handling & logging
✅ Permission system
✅ Rate limiting
✅ No earrape (25% default volume!)
✅ .mp3/.mp4 file support

**Most importantly:** A bot that real servers can use, that you can maintain solo, and that you're proud of!

---

## Next Steps

1. **Today:** Quick wins (volume + file support)
2. **This week:** Refactoring
3. **Next week:** PostgreSQL migration
4. **Week after:** Web dashboard MVP
5. **Final week:** Production deployment

Let's build something awesome! 🎵⏱️