Github Backup Architecture

GitHub Backup Automation System — Architecture Guide

This skill provides an overview of the system architecture, data models, communication protocols, and deployment environments.

1. Branch-First Development

[!IMPORTANT] CREATE A LOCAL BRANCH FIRST: Always start by creating a local branch from main:

git switch -c MishraShardendu22/main/<feature-name>

Never make changes directly on main.

2. System Topology

┌─────────────────────────────────────────────────────────────┐
│                    Next.js 16 Frontend                      │
│                  (Deployed on Vercel)                       │
└───────────────┬─────────────────────────────┬───────────────┘
                │ REST / SSE                  │ REST / WebSocket
                ▼                             ▼
┌─────────────────────────────┐ ┌─────────────────────────────┐
│     Python Observatory      │ │         Go Backend          │
│    (Deployed on Vercel)     │ │    (Deployed on Render)     │
│   FastAPI · LangChain AI    │ │  Fiber v2 · Live WS Stream  │
└───────────────┬─────────────┘ └─────────────┬───────────────┘
                │                             │
                │     ┌─────────────────┐     │
                ├────►│ Neon PostgreSQL │◄────┤
                │     │ (pgvector + FTS)│     │
                │     └────────┬────────┘     │
                │              ▲              │
                │              │ Sync         │
                │     ┌────────┴────────┐     │
                │     │ Backup Worker   │     │
                │     │ (CLI / Cron)    │     │
                │     └─────────────────┘     │
                ▼                             ▼
┌─────────────────────────────┐ ┌─────────────────────────────┐
│     OpenRouter AI APIs      │ │     SMTP Email Service      │
│ (Multi-Key Failover Pool)   │ │  (Human-In-The-Loop Alerts) │
└─────────────────────────────┘ └─────────────────────────────┘

2. Service Responsibilities

Next.js Frontend (frontend/)

• Framework: Next.js 16 App Router with Turbopack, Tailwind CSS, Biome linter.
• Responsibilities: Unified Dashboard, AI Chat Interface, Vector Search Playground, Real-time WebSocket Log Streaming, Human-in-the-Loop Action Approvals.
• Config: frontend/src/config/env.ts.

Python Observatory (agentic-observatory/)

• Framework: FastAPI, LangChain, asyncpg, SQLAlchemy, httpx, Jinja2.
• Responsibilities:
  • Multi-turn AI Agent reasoning loop (invoke_agent, stream_agent).
  • Hybrid Search: PostgreSQL Full-Text Search + pgvector cosine similarity + Reciprocal Rank Fusion (RRF).
  • Automated and interactive embedding generation pipeline (embedding_generations, embedding_jobs, embedding_chunks).
  • Human-in-the-loop report generation and SMTP dispatch (send_report_email).
  • JWT Authentication for dashboard chat.
• Config: agentic-observatory/config/settings.py.

Go Backend (backend/)

• Framework: Go Fiber v2, pgxpool.
• Responsibilities:
  • Ingesting backup execution runs, repository results, and structured logs from the worker.
  • Serving real-time WebSocket hub for active backup runs (/ws).
  • Exposing database metrics and telemetry.
• Config: backend/config/config.go.

Backup Worker (backup-worker/)

• Framework: Go CLI (backup-worker/main.go).
• Responsibilities:
  • Discovering repositories from GitHub Organizations & Personal accounts.
  • Cloning / pulling mirrors locally into backup-worker/_Repos/.
  • Caching remote HEAD commit hashes in backup-worker/app.db.
  • Recording telemetry, logs, and failure fixes to PostgreSQL.
• Config: backup-worker/config/data.config.go.

3. Database Schema

1. backup_runs: Stores each backup batch (ID, status, total repos, duration, timestamps, error_message).
2. backup_results: Per-repository outcome (status, error_message, sizes, commit_hash).
3. execution_logs: Structured step-by-step logs.
4. analytics_snapshots: Aggregated metrics and commit snapshots over time (1-to-1 unique with backup_runs).
5. backup_fixes & backup_run_fixes: Historical failure resolutions and commit tags.
6. ai_chat_sessions & ai_session_metadata: Normalized conversation sessions and key-value metadata.
7. ai_chat_messages & ai_tool_calls: Chat history and granular tool execution telemetry.
8. investigations: Saved agent investigation traces, tool calls, and results.
9. embedding_generations, embedding_jobs, embedding_chunks: Vector index and chunk storage with pgvector and deterministic blue-green lifecycle management.