aradotso ---
name: friday-tony-stark-voice-assistant
description: Build and extend FRIDAY, a Tony Stark-inspired voice AI assistant using FastMCP, LiveKit Agents, Gemini LLM, Sarvam STT, and OpenAI TTS.
triggers:
- set up friday tony stark voice assistant
- add a new tool to the friday mcp server
- configure friday voice agent providers
- connect friday to livekit room
- troubleshoot friday mcp server connection
- switch stt or tts provider in friday agent
- run friday voice agent with mcp tools
- extend friday with custom mcp tools
---
# F.R.I.D.A.Y. — Tony Stark Voice Assistant
> Skill by [ara.so](https://ara.so) — Daily 2026 Skills collection.
FRIDAY is a two-process AI voice assistant: a **FastMCP server** exposes tools (web search, news, system info) over SSE, and a **LiveKit voice agent** connects microphone → STT → LLM → TTS → speaker in real time, calling MCP tools as needed.
---
## Architecture
Microphone ──► STT (Sarvam Saaras v3) │ ▼ LLM (Gemini 2.5 Flash) ◄──────► MCP Server (FastMCP / SSE :8000) │ ├─ get_world_news ▼ ├─ open_world_monitor TTS (OpenAI nova) ├─ search_web │ └─ …more tools ▼ Speaker / LiveKit room
Both processes must run simultaneously. The voice agent connects to the MCP server at `http://127.0.0.1:8000/sse`.
---
## Installation
### Prerequisites
- Python ≥ 3.11
- [`uv`](https://github.com/astral-sh/uv)
- A [LiveKit Cloud](https://cloud.livekit.io) project (free tier works)
```bash
git clone https://github.com/SAGAR-TAMANG/friday-tony-stark-demo.git
cd friday-tony-stark-demo
uv sync # creates .venv and installs all dependencies
cp .env.example .env
# Fill in your API keys in .env
Key Commands
| Command | What it does |
|---|---|
uv run friday |
Start the FastMCP server on http://127.0.0.1:8000/sse |
uv run friday_voice |
Start the LiveKit voice agent in dev mode |
Always start the MCP server first, then the voice agent.
# Terminal 1
uv run friday
# Terminal 2
uv run friday_voice
Then open LiveKit Agents Playground and connect to your room.
Environment Variables
Copy .env.example to .env and set:
# LiveKit (required)
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=$LIVEKIT_API_KEY
LIVEKIT_API_SECRET=$LIVEKIT_API_SECRET
# LLM — Gemini (default, required)
GOOGLE_API_KEY=$GOOGLE_API_KEY
# STT — Sarvam (default, required)
SARVAM_API_KEY=$SARVAM_API_KEY
# TTS — OpenAI (default, required)
OPENAI_API_KEY=$OPENAI_API_KEY
# Optional providers
GROQ_API_KEY=$GROQ_API_KEY
DEEPGRAM_API_KEY=$DEEPGRAM_API_KEY
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# Optional integrations
SUPABASE_URL=$SUPABASE_URL
SUPABASE_API_KEY=$SUPABASE_API_KEY
Project Structure
friday-tony-stark-demo/
├── server.py # Entry: starts FastMCP server
├── agent_friday.py # Entry: starts LiveKit voice agent
├── pyproject.toml
├── .env.example
└── friday/
├── config.py # Env-var loading & app-wide settings
├── tools/
│ ├── __init__.py # Registers all tools with mcp instance
│ ├── web.py # search_web, fetch_url, get_world_news, open_world_monitor
│ ├── system.py # get_current_time, get_system_info
│ └── utils.py # format_json, word_count
├── prompts/ # MCP prompt templates
└── resources/ # MCP resources (friday://info)
Adding a New MCP Tool
Step 1 — Create or open a file in friday/tools/
# friday/tools/weather.py
from fastmcp import FastMCP
import httpx
def register(mcp: FastMCP) -> None:
@mcp.tool()
async def get_weather(city: str) -> str:
"""Get current weather for a city."""
async with httpx.AsyncClient() as client:
resp = await client.get(
"https://wttr.in/{city}?format=3",
params={"city": city},
timeout=10,
)
resp.raise_for_status()
return resp.text
Step 2 — Register in friday/tools/__init__.py
# friday/tools/__init__.py
from friday.tools import web, system, utils, weather # add your module
def register_all(mcp):
web.register(mcp)
system.register(mcp)
utils.register(mcp)
weather.register(mcp) # add this line
The MCP server picks up the new tool on next uv run friday.
Switching Providers
Edit the constants at the top of agent_friday.py:
STT_PROVIDER = "sarvam" # "sarvam" | "whisper"
LLM_PROVIDER = "gemini" # "gemini" | "openai"
TTS_PROVIDER = "openai" # "openai" | "sarvam"
Voice Agent Internals
The agent follows the LiveKit Agents pattern:
# agent_friday.py (simplified)
from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
from livekit.agents.voice_assistant import VoiceAssistant
from livekit.plugins import openai, google, sarvam
from livekit.agents.mcp import MCPClient
async def entrypoint(ctx: JobContext):
await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
# Connect to the FastMCP server
mcp_client = MCPClient("http://127.0.0.1:8000/sse")
tools = await mcp_client.list_tools()
initial_ctx = llm.ChatContext().append(
role="system",
text="You are FRIDAY, Tony Stark's AI assistant. Be concise and helpful.",
)
assistant = VoiceAssistant(
vad=silero.VAD.load(),
stt=sarvam.STT(model="saaras:v3"),
llm=google.LLM(model="gemini-2.5-flash"),
tts=openai.TTS(voice="nova"),
chat_ctx=initial_ctx,
tools=tools,
)
assistant.start(ctx.room)
await asyncio.sleep(1)
await assistant.say("Hello. I am FRIDAY. How can I assist you?", allow_interruptions=True)
def dev():
# Injects 'dev' CLI flag so you don't type it manually
import sys
sys.argv = [sys.argv[0], "dev"]
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
MCP Server Internals
# server.py (simplified)
from fastmcp import FastMCP
from friday.tools import register_all
mcp = FastMCP("FRIDAY", description="Tony Stark's AI backend")
register_all(mcp)
def main():
mcp.run(transport="sse", host="127.0.0.1", port=8000)
Common Patterns
Pattern: Tool with structured output
# friday/tools/stocks.py
from fastmcp import FastMCP
from pydantic import BaseModel
class StockInfo(BaseModel):
symbol: str
price: float
change_pct: float
def register(mcp: FastMCP) -> None:
@mcp.tool()
async def get_stock_price(symbol: str) -> StockInfo:
"""Fetch current stock price and daily change for a ticker symbol."""
# ... fetch from API ...
return StockInfo(symbol=symbol.upper(), price=182.34, change_pct=1.2)
Pattern: Tool with error handling
@mcp.tool()
async def search_web(query: str, max_results: int = 5) -> str:
"""Search the web and return summarised results."""
try:
async with httpx.AsyncClient(timeout=15) as client:
resp = await client.get(
"https://api.search-provider.com/search",
params={"q": query, "n": max_results},
headers={"Authorization": f"Bearer {os.environ['SEARCH_API_KEY']}"},
)
resp.raise_for_status()
results = resp.json()
return "\n".join(f"- {r['title']}: {r['snippet']}" for r in results["items"])
except httpx.TimeoutException:
return "Search timed out. Please try again."
except httpx.HTTPStatusError as e:
return f"Search failed with status {e.response.status_code}."
Pattern: Tool that opens a local app or URL
@mcp.tool()
def open_world_monitor() -> str:
"""Open the system monitor application."""
import subprocess, sys
if sys.platform == "win32":
subprocess.Popen(["taskmgr"])
elif sys.platform == "darwin":
subprocess.Popen(["open", "-a", "Activity Monitor"])
else:
subprocess.Popen(["gnome-system-monitor"])
return "World monitor opened."
Pattern: MCP prompt template
# friday/prompts/summarize.py
from fastmcp import FastMCP
def register(mcp: FastMCP) -> None:
@mcp.prompt()
def summarize(text: str, max_words: int = 100) -> str:
"""Prompt to summarise a block of text."""
return (
f"Summarise the following text in {max_words} words or fewer, "
f"maintaining key points:\n\n{text}"
)
WSL / Windows Host IP
When running inside WSL, the MCP server URL resolves automatically to the Windows host IP. No extra config needed — agent_friday.py handles this:
import os, socket
def get_mcp_url() -> str:
if "microsoft" in open("/proc/version").read().lower():
# Inside WSL — use Windows host IP
host = socket.gethostbyname(socket.gethostname() + ".local")
return f"http://{host}:8000/sse"
return "http://127.0.0.1:8000/sse"
Troubleshooting
| Symptom | Fix |
|---|---|
Connection refused on port 8000 |
Start uv run friday before uv run friday_voice |
| Voice agent can't find tools | Verify MCP server is running: curl http://127.0.0.1:8000/sse |
LIVEKIT_URL error |
Set LIVEKIT_URL=wss://your-project.livekit.cloud in .env |
| STT not transcribing | Check SARVAM_API_KEY is valid; try switching to STT_PROVIDER = "whisper" |
| Gemini LLM errors | Ensure GOOGLE_API_KEY is set and has Gemini 2.5 Flash access |
| TTS silent output | Verify OPENAI_API_KEY is valid and has TTS quota |
| Tool not appearing in LLM | Confirm register(mcp) is called in friday/tools/__init__.py |
| WSL: MCP URL unreachable | Check Windows Firewall allows port 8000; use host IP instead of 127.0.0.1 |
pyproject.toml Entry Points
The uv run shortcuts are defined as script entry points:
[project.scripts]
friday = "server:main"
friday_voice = "agent_friday:dev"
Quick Reference
# Install
uv sync
# Start MCP server (Terminal 1)
uv run friday
# Start voice agent (Terminal 2)
uv run friday_voice
# Verify MCP server tools are registered
curl -N http://127.0.0.1:8000/sse
# Add a tool: create friday/tools/mytool.py → register(mcp) → add to __init__.py
# Switch providers: edit STT_PROVIDER / LLM_PROVIDER / TTS_PROVIDER in agent_friday.py