grimm/groovy-zilean
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

removed current roadmap

Browse Source
This commit is contained in:
Top1055
2026-02-01 18:44:36 +00:00
parent fecb4947e1
commit 31309046d2
1 changed files with 0 additions and 849 deletions
Download Patch File Download Diff File Expand all files Collapse all files

849
PRODUCTION_ROADMAP.md
View File

@@ -1,849 +0,0 @@
# Groovy-Zilean Production Roadmap
**Goal:** Transform groovy-zilean from a personal project into a production-ready Discord music bot with a web dashboard.
**Philosophy:** Production-quality architecture with manageable complexity for a solo developer. No overkill, no shortcuts.
---
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Tech Stack Decisions](#tech-stack-decisions)
3. [Development Phases](#development-phases)
4. [Python Environment Setup](#python-environment-setup)
5. [Why This Approach](#why-this-approach)
6. [Quick Reference](#quick-reference)
---
## Architecture Overview
### The Winning Design: Database-Mediated Architecture
```
┌─────────────┐ HTTP/SSE ┌──────────────┐
│ Browser │◄────────────►│ FastAPI │
│ (HTMX HTML) │ │ Backend │
└─────────────┘ └──────┬───────┘
│
┌──────▼───────┐ ┌─────────────┐
│ PostgreSQL │◄────────│ Discord Bot │
│ Database │ │ (discord.py)│
└──────────────┘ └─────────────┘
```
### Key Principles
1. **Decoupled Services**
- Bot and web can restart independently
- No tight coupling via IPC
- Database is the single source of truth
2. **Simple Frontend (Initially)**
- HTMX for interactivity (no build step)
- Jinja2 templates (server-side rendering)
- Tailwind CSS via CDN (beautiful, no npm)
- **Optional:** Upgrade to React later when ready
3. **Production-Ready Backend**
- PostgreSQL for reliability
- FastAPI for modern Python web
- Discord OAuth2 for authentication
- Connection pooling, proper error handling
---
## Tech Stack Decisions
### Backend
| Component | Choice | Why |
|-----------|--------|-----|
| **Bot Framework** | discord.py 2.6.4+ | Industry standard, hybrid commands |
| **Web Framework** | FastAPI | Modern, async, auto-docs, large community |
| **Database** | PostgreSQL 14+ | Production-ready, ACID compliance, better than SQLite |
| **Music Extraction** | yt-dlp | Actively maintained, multi-platform support |
| **Auth** | Discord OAuth2 | Native Discord integration |
| **ORM** | SQLAlchemy (optional) | Or use asyncpg directly for simplicity |
### Frontend (Phase 1)
| Component | Choice | Why |
|-----------|--------|-----|
| **Templates** | Jinja2 | Server-side rendering, no build step |
| **Interactivity** | HTMX | Modern interactivity without React complexity |
| **Styling** | Tailwind CSS (CDN) | Beautiful UI, no build process |
| **Real-time** | Server-Sent Events | Simple polling/updates, no WebSocket complexity |
### Frontend (Phase 2 - Optional)
| Component | Choice | Why |
|-----------|--------|-----|
| **Framework** | React 18 | If you need more complex UI later |
| **Real-time** | WebSocket | For true real-time when needed |
| **State** | Zustand/Context | Simpler than Redux |
### Infrastructure
| Component | Choice | Why |
|-----------|--------|-----|
| **Python Version** | 3.11 or 3.12 | Modern features, better performance |
| **Environment** | venv | Isolated dependencies, no PATH conflicts |
| **Process Manager** | systemd | Reliable, built into Linux |
| **Reverse Proxy** | nginx | Standard, handles SSL, static files |
---
## Development Phases
### Phase 0: Current State ✅
- Working Discord bot with music playback
- SQLite database
- Hybrid commands (slash + prefix)
- 17 audio effects
- Queue, loop, shuffle functionality
- Spotify integration
### Phase 1: Quick Wins (1-2 hours)
**Goal:** Immediate improvements for better UX
Tasks:
1. **Lower default volume from 100% to 25%**
- Change default in `queue.py:119`
- Scale volume command display (user sees 0-200%, internally 0-50%)
- Prevents earrape for new users
2. **Add .mp3 and .mp4 file support**
- Extend `translate.py` to detect direct file URLs
- Support HTTP/HTTPS links to audio files
- Validate file type before processing
**Files to modify:**
- `cogs/music/queue.py`
- `cogs/music/translate.py`
---
### Phase 2: Code Refactoring (4-6 hours)
**Goal:** Clean, maintainable, documented codebase
#### 2.1 Database Abstraction
Create `cogs/music/db_manager.py`:
- Database connection class with context manager
- Connection pooling preparation
- Centralize all SQL queries
- Remove scattered `sqlite3.connect()` calls
#### 2.2 Configuration Management
Update `config.py`:
- Use environment variables for secrets
- Create `.env.example` template
- Remove hardcoded credentials
- Add validation for required config
#### 2.3 Code Organization
```
groovy-zilean/
├── bot/
│ ├── __init__.py
│ ├── bot.py # Main bot class
│ └── cogs/
│ └── music/
│ ├── __init__.py
│ ├── commands.py # User-facing commands
│ ├── player.py # Playback logic
│ ├── queue.py # Queue management
│ ├── effects.py # Audio effects
│ ├── db_manager.py # Database abstraction
│ └── translate.py # URL/playlist parsing
├── web/
│ ├── __init__.py # (Future web app)
├── shared/
│ ├── __init__.py
│ ├── config.py # Shared configuration
│ └── models.py # Data models
├── main.py # Entry point
├── requirements.txt
├── .env.example
└── README.md
```
#### 2.4 Error Handling & Logging
- Wrap all commands in try/except
- User-friendly error messages
- Proper logging setup (rotating file logs)
- Debug mode toggle
#### 2.5 Type Hints & Documentation
- Add type hints to all functions
- Docstrings for all classes/methods
- Inline comments for complex logic
**Expected outcome:**
- Easy to navigate codebase
- No secrets in code
- Consistent patterns throughout
---
### Phase 3: PostgreSQL Migration (3-4 hours)
**Goal:** Production-ready database layer
#### 3.1 Local PostgreSQL Setup
```bash
# Install PostgreSQL
sudo apt update
sudo apt install postgresql postgresql-contrib
# Create database and user
sudo -u postgres psql
CREATE DATABASE groovy_zilean;
CREATE USER groovy WITH PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE groovy_zilean TO groovy;
```
#### 3.2 Database Schema Design
```sql
-- servers table
CREATE TABLE servers (
server_id BIGINT PRIMARY KEY,
is_playing BOOLEAN DEFAULT FALSE,
song_name TEXT,
song_url TEXT,
song_thumbnail TEXT,
loop_mode VARCHAR(10) DEFAULT 'off',
volume INTEGER DEFAULT 25, -- NEW default!
effect VARCHAR(20) DEFAULT 'none',
song_start_time DOUBLE PRECISION DEFAULT 0,
song_duration INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- songs/queue table
CREATE TABLE songs (
id SERIAL PRIMARY KEY,
server_id BIGINT NOT NULL REFERENCES servers(server_id) ON DELETE CASCADE,
song_link TEXT,
queued_by TEXT,
position INTEGER NOT NULL,
title TEXT,
thumbnail TEXT,
duration INTEGER,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(server_id, position)
);
-- Future: users table for web auth
CREATE TABLE users (
discord_id BIGINT PRIMARY KEY,
username TEXT,
avatar TEXT,
access_token TEXT,
refresh_token TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
last_login TIMESTAMP
);
-- Future: permissions table
CREATE TABLE permissions (
id SERIAL PRIMARY KEY,
server_id BIGINT REFERENCES servers(server_id),
user_id BIGINT,
role_id BIGINT,
can_play BOOLEAN DEFAULT TRUE,
can_skip BOOLEAN DEFAULT FALSE,
can_clear BOOLEAN DEFAULT FALSE,
can_modify_settings BOOLEAN DEFAULT FALSE
);
```
#### 3.3 Migration Script
Create `scripts/migrate_to_postgres.py`:
- Read all data from SQLite
- Insert into PostgreSQL
- Validate migration
- Backup SQLite file
#### 3.4 Update Database Code
Replace all `sqlite3` calls with `asyncpg` or `psycopg3`:
```python
# Old (SQLite)
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# New (PostgreSQL with asyncpg)
async with pool.acquire() as conn:
result = await conn.fetch("SELECT * FROM servers WHERE server_id = $1", server_id)
```
#### 3.5 Connection Pooling
```python
# In bot startup
self.db_pool = await asyncpg.create_pool(
host='localhost',
database='groovy_zilean',
user='groovy',
password=os.getenv('DB_PASSWORD'),
min_size=5,
max_size=20
)
```
**Expected outcome:**
- Reliable database with ACID guarantees
- Better concurrent access handling
- Ready for multi-server production load
---
### Phase 4: Web Dashboard (20-30 hours)
**Goal:** User-friendly web interface for bot control
#### 4.1 FastAPI Backend Setup
**Project structure:**
```
web/
├── __init__.py
├── main.py # FastAPI app
├── routes/
│ ├── __init__.py
│ ├── auth.py # Discord OAuth2
│ ├── servers.py # Server list/select
│ └── playback.py # Queue/controls
├── templates/
│ ├── base.html
│ ├── index.html
│ ├── dashboard.html
│ └── components/
│ ├── queue.html
│ └── controls.html
├── static/
│ ├── css/
│ └── js/
└── dependencies.py # Auth dependencies
```
**Core dependencies:**
```bash
pip install fastapi uvicorn jinja2 python-multipart httpx
```
#### 4.2 Discord OAuth2 Authentication
**Flow:**
1. User clicks "Login with Discord"
2. Redirect to Discord OAuth
3. Discord redirects back with code
4. Exchange code for token
5. Fetch user info + guilds
6. Create session
7. Show dashboard with user's servers
**Implementation:**
```python
# routes/auth.py
@router.get("/login")
async def login():
# Redirect to Discord OAuth
discord_auth_url = (
f"https://discord.com/api/oauth2/authorize"
f"?client_id={DISCORD_CLIENT_ID}"
f"&redirect_uri={REDIRECT_URI}"
f"&response_type=code"
f"&scope=identify guilds"
)
return RedirectResponse(discord_auth_url)
@router.get("/callback")
async def callback(code: str):
# Exchange code for token
# Fetch user info
# Create session
# Redirect to dashboard
```
#### 4.3 HTMX Frontend
**Example dashboard with HTMX:**
```html
<!-- templates/dashboard.html -->
<div class="container">
<!-- Server Selector -->
<select hx-get="/api/servers/{value}/queue"
hx-target="#queue-container"
hx-trigger="change">
{% for server in user_servers %}
<option value="{{ server.id }}">{{ server.name }}</option>
{% endfor %}
</select>
<!-- Now Playing (auto-updates every 5s) -->
<div id="now-playing"
hx-get="/api/now-playing"
hx-trigger="every 5s">
<!-- Server renders this -->
</div>
<!-- Queue (auto-updates) -->
<div id="queue-container"
hx-get="/api/queue"
hx-trigger="every 3s">
<!-- Queue items here -->
</div>
<!-- Controls -->
<div class="controls">
<button hx-post="/api/skip"
hx-target="#queue-container">
⏭️ Skip
</button>
<input type="range"
min="0" max="200"
hx-post="/api/volume"
hx-trigger="change"
hx-vals='{"volume": this.value}'>
</div>
<!-- Add Song Form -->
<form hx-post="/api/play"
hx-target="#queue-container">
<input name="query" placeholder="YouTube URL or search">
<button type="submit">Add to Queue</button>
</form>
</div>
```
**Why HTMX is perfect here:**
- No JavaScript needed for interactivity
- Server renders everything (simpler)
- Auto-updates with `hx-trigger="every Xs"`
- Progressive enhancement (works without JS)
#### 4.4 API Endpoints
**Read endpoints:**
```python
GET /api/servers # User's servers (with bot)
GET /api/servers/{id}/queue # Current queue
GET /api/servers/{id}/status # Now playing, volume, etc.
```
**Write endpoints:**
```python
POST /api/servers/{id}/play # Add song (body: {query: "..."})
POST /api/servers/{id}/skip # Skip current song
POST /api/servers/{id}/volume # Set volume (body: {volume: 150})
POST /api/servers/{id}/effect # Set effect (body: {effect: "nightcore"})
POST /api/servers/{id}/loop # Set loop mode (body: {mode: "queue"})
POST /api/servers/{id}/shuffle # Shuffle queue
POST /api/servers/{id}/clear # Clear queue
```
**Implementation example:**
```python
# routes/playback.py
@router.post("/api/servers/{server_id}/skip")
async def skip_song(
server_id: int,
user: User = Depends(get_current_user),
db = Depends(get_db)
):
# 1. Check user is in server
if server_id not in user.guild_ids:
raise HTTPException(403, "Not in this server")
# 2. Check permissions (future)
# if not has_permission(user, server_id, "can_skip"):
# raise HTTPException(403, "No permission")
# 3. Write command to database
await db.execute(
"INSERT INTO commands (server_id, action, user_id) VALUES ($1, $2, $3)",
server_id, "skip", user.id
)
# 4. Return updated queue
return await get_queue(server_id, db)
```
#### 4.5 Bot Integration (Command Processing)
**Add to bot:**
```python
# In bot.py or new cogs/web_commands.py
@tasks.loop(seconds=1)
async def process_web_commands(self):
"""Process commands from web dashboard"""
async with self.db_pool.acquire() as conn:
# Fetch unprocessed commands
commands = await conn.fetch(
"SELECT * FROM commands WHERE processed = FALSE"
)
for cmd in commands:
server_id = cmd['server_id']
action = cmd['action']
data = cmd['data']
guild = self.get_guild(server_id)
if not guild or not guild.voice_client:
continue
# Execute command
if action == "skip":
guild.voice_client.stop()
elif action == "volume":
# Set volume in database, next song picks it up
await queue.set_volume(server_id, data['volume'])
elif action == "play":
# Queue song from web
# ... (use existing play logic)
# Mark as processed
await conn.execute(
"UPDATE commands SET processed = TRUE WHERE id = $1",
cmd['id']
)
```
#### 4.6 Permissions System
**Basic implementation:**
```python
# Check if user can control bot
async def can_control_bot(user_id: int, server_id: int, action: str) -> bool:
# Check if user is in voice channel with bot
# Check if user has DJ role (configurable per server)
# Check specific permission for action
# Default: anyone in VC can control
pass
```
**Advanced (Phase 5):**
- Role-based permissions
- User-specific permissions
- Configurable via web dashboard
**Expected outcome:**
- Beautiful, functional web dashboard
- Discord OAuth login
- Real-time queue display
- Full playback control from browser
- Works on mobile
---
### Phase 5: Permissions & Production (4-6 hours)
**Goal:** Production-ready deployment
#### 5.1 Permission System
- DJ role configuration
- Per-server permission settings
- Web UI for permission management
#### 5.2 Rate Limiting
```python
from slowapi import Limiter
limiter = Limiter(key_func=get_remote_address)
@app.post("/api/play")
@limiter.limit("10/minute") # Max 10 songs per minute
async def play_song(...):
...
```
#### 5.3 Logging & Monitoring
```python
import logging
from logging.handlers import RotatingFileHandler
# Setup logging
handler = RotatingFileHandler(
'logs/bot.log',
maxBytes=10_000_000, # 10MB
backupCount=5
)
logging.basicConfig(
handlers=[handler],
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
```
#### 5.4 Systemd Services
```ini
# /etc/systemd/system/groovy-bot.service
[Unit]
Description=Groovy Zilean Discord Bot
After=network.target postgresql.service
[Service]
Type=simple
User=groovy
WorkingDirectory=/home/groovy/groovy-zilean
Environment="PATH=/home/groovy/groovy-zilean/venv/bin"
ExecStart=/home/groovy/groovy-zilean/venv/bin/python main.py
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
```
```ini
# /etc/systemd/system/groovy-web.service
[Unit]
Description=Groovy Zilean Web Dashboard
After=network.target postgresql.service
[Service]
Type=simple
User=groovy
WorkingDirectory=/home/groovy/groovy-zilean
Environment="PATH=/home/groovy/groovy-zilean/venv/bin"
ExecStart=/home/groovy/groovy-zilean/venv/bin/uvicorn web.main:app --host 0.0.0.0 --port 8000
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
```
#### 5.5 Nginx Configuration
```nginx
server {
listen 80;
server_name groovy.yourdomain.com;
# Redirect to HTTPS
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name groovy.yourdomain.com;
ssl_certificate /etc/letsencrypt/live/groovy.yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/groovy.yourdomain.com/privkey.pem;
location / {
proxy_pass http://localhost:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /static {
alias /home/groovy/groovy-zilean/web/static;
expires 30d;
}
}
```
#### 5.6 Database Backups
```bash
#!/bin/bash
# scripts/backup_db.sh
BACKUP_DIR="/home/groovy/backups"
DATE=$(date +%Y%m%d_%H%M%S)
pg_dump groovy_zilean > "$BACKUP_DIR/groovy_zilean_$DATE.sql"
# Keep only last 7 days
find $BACKUP_DIR -name "groovy_zilean_*.sql" -mtime +7 -delete
```
Add to crontab:
```bash
0 2 * * * /home/groovy/groovy-zilean/scripts/backup_db.sh
```
#### 5.7 Environment Variables
```bash
# .env (NEVER commit this!)
DISCORD_TOKEN=your_bot_token_here
DISCORD_CLIENT_ID=your_client_id
DISCORD_CLIENT_SECRET=your_client_secret
SPOTIFY_CLIENT_ID=your_spotify_id
SPOTIFY_CLIENT_SECRET=your_spotify_secret
DB_PASSWORD=your_db_password
SECRET_KEY=random_secret_for_sessions
ENVIRONMENT=production
```
**Expected outcome:**
- Production-ready deployment
- Automatic restarts on failure
- HTTPS enabled
- Automated backups
- Proper logging
---
## Python Environment Setup
### Avoiding Python Version Hell
**Step 1: Install Python 3.12**
```bash
# On Debian/Ubuntu
sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev
# Verify installation
python3.12 --version
```
**Step 2: Create Virtual Environment**
```bash
cd ~/coding/groovy-zilean
# Create venv (only once)
python3.12 -m venv venv
# Activate (every time you work on project)
source venv/bin/activate
# Your prompt should change to show (venv)
```
**Step 3: Install Dependencies**
```bash
# Make sure venv is activated!
pip install --upgrade pip
pip install -r requirements.txt
```
**Step 4: Add to .gitignore**
```
venv/
__pycache__/
*.pyc
.env
*.db
logs/
```
**Helpful Aliases (add to ~/.bashrc)**
```bash
alias groovy='cd ~/coding/groovy-zilean && source venv/bin/activate'
```
Then just type `groovy` to activate your environment!
---
## Why This Approach?
### Rejected: discord-ext-ipc + Quart
❌ **discord-ext-ipc is unmaintained** (last update 2+ years ago)
❌ Tight coupling between bot and web
❌ Both processes must run together
❌ Quart less popular than FastAPI
❌ Still need polling for real-time updates
### Rejected: FastAPI + React + Redis + WebSocket
❌ **Overkill for solo developer**
❌ Too many moving parts (4+ services)
❌ npm/node_modules complexity
❌ Requires learning React well
❌ WebSocket complexity for minimal gain
### Chosen: FastAPI + PostgreSQL + HTMX ✅
✅ **Production-quality architecture**
✅ Decoupled services (independent restarts)
✅ Modern, well-maintained tools
✅ No frontend build step (initially)
✅ Easy upgrade path to React later
✅ Manageable complexity for solo dev
✅ Database as source of truth
✅ All Python backend
---
## Quick Reference
### Daily Development Workflow
```bash
# Activate environment
cd ~/coding/groovy-zilean
source venv/bin/activate
# Run bot
python main.py
# Run web (in another terminal)
source venv/bin/activate
uvicorn web.main:app --reload --port 8000
# Run tests (future)
pytest
# Database migrations (future)
alembic upgrade head
```
### Project Commands
```bash
# Install new package
pip install package_name
pip freeze > requirements.txt
# Database
psql groovy_zilean # Connect to database
pg_dump groovy_zilean > backup.sql # Backup
# Systemd
sudo systemctl start groovy-bot
sudo systemctl status groovy-bot
sudo journalctl -u groovy-bot -f # View logs
```
### File Locations
- **Bot entry point:** `main.py`
- **Config:** `shared/config.py` + `.env`
- **Database:** PostgreSQL (not file-based)
- **Logs:** `logs/bot.log`, `logs/web.log`
- **Web templates:** `web/templates/`
---
## Success Metrics
By the end of this roadmap, you'll have:
✅ Clean, maintainable codebase
✅ Production-ready database
✅ Web dashboard with Discord auth
✅ Mobile-friendly interface
✅ Automated deployment
✅ Backup system
✅ Proper error handling & logging
✅ Permission system
✅ Rate limiting
✅ No earrape (25% default volume!)
✅ .mp3/.mp4 file support
**Most importantly:** A bot that real servers can use, that you can maintain solo, and that you're proud of!
---
## Next Steps
1. **Today:** Quick wins (volume + file support)
2. **This week:** Refactoring
3. **Next week:** PostgreSQL migration
4. **Week after:** Web dashboard MVP
5. **Final week:** Production deployment
Let's build something awesome! 🎵⏱️