Browzeit Agent System
The Browzeit agent system orchestrates browsers with AI agents, supporting MoE (Mixture of Experts), MoA (Mixture of Agents), subagents, swarms, and session/chat forking. The system lives in a Rust workspace (crates/) with a pywebview desktop GUI shell.
Trigger
- User asks to build or extend the Browzeit browser's agent system.
- User asks for MoE/MoA routing, subagent spawning, session forking, or agent swarms.
- User asks for the desktop GUI scaffold (pywebview/Tauri) for a Rust agentic browser.
- User asks for API/CLI/MCP/ACP interfaces to control the browser agent.
Project Layout + Current State
crates/
traits/ # Trait contracts (Parser, LayoutEngine, PaintBackend, NetworkClient, Agent)
types/ # Complete DOM, render tree, agent types, browser types
core/ # SKELETON: dom.rs (v0.1, doc-only), parser.rs (FIXED, COMPILING), style.rs (v0.1, doc-only)
engine/ # agent.rs + agent/ module files (moa, moi, moe, session, subagent, swarm, canvas, chat, protocol), layout.rs, moa.rs, moi.rs
renderer/ # SKELETON: paint.rs, compositing.rs, bridge.rs, v8_embed.rs
network/ # SKELETON: client.rs (stub returning error), cache.rs, cookies.rs (stub)
driver/ # shell.rs (BrowserWindow), webview.rs (duplicate BrowserWindow), agent.rs (orchestrator)
api/ # lib.rs (types + router) + main.rs (duplicate types + full server)
gui_shell/ # pywebview shell (browzeit_shell.py + index.html)
custode-browzeit/ # Custode agent implementation for Browzeit (agent, router, perceptor, verifier, registry, bridge)
Known issues to resolve before building:
core/src/dom.rsandcore/src/style.rsare doc-only skeletons (v0.1), not implementations.network/src/client.rsis a stub returning error — the HTTP client is not implemented.types::browser::BrowserWindow(driver) is a skeleton with 3 methods;driver::webview::BrowserWindowhas the full 20+ method implementation. The driver's agent.rs uses the skeleton type — this is intentional for the orchestrator but means tab/navigation APIs are not wired through.api/src/lib.rsandapi/src/main.rsboth defineApiResponse,BrowserTab,AgentTaskRequest, etc. — duplicate type definitions that will conflict.- The entire workspace has NEVER been built end-to-end.
cargo check --workspacehas never been run to completion.
Workload Phase 1: Agent Orchestrator Core
AgentOrchestrator::new()creates expert configs:default(gpt-4, 4096 max tokens, 0.7 temp, [text, reasoning]),fast(gpt-3.5-turbo, 2048 max tokens, 0.5 temp, [text]),creative(claude-3, 8192 max tokens, 1.0 temp, [text, creative]).register_llm_client()wraps the LLM client behind a trait (LLMClient: Send + Sync) withgenerate(prompt, tools) -> Stringandset_simulated_response(response).execute(task)createsAgentState(wrapped inArc<...>), runs the agent loop viarun_agent_loop(), records telemetry, returnsSnapshot.run_agent_loop()iterates up tomax_steps, callsllm_client.generate(), parses response viaparse_action(), executes the action (Navigate, Extract, Click, Fill, Evaluate, SpawnSubagent, UseMoE, UseMoA, ForkSession, Done, Wait).parse_action()maps LLM responses to actions by keyword:done/exit-> Done,navigate-> Navigate (extract URL),extract-> Extract (CSS selector),click-> Click,fill-> Fill,eval/evaluate-> Evaluate (extract code),spawn-> SpawnSubagent,moe/ensemble-> UseMoE,moa/committee-> UseMoA,fork-> ForkSession,think/observe/wait-> Wait, default -> Done with text.computethe snapshot: buildSnapshotfromSession(url, title, body_text), create a render tree, return viabuild_snapshot().
AgentState (interior mutability) — for thread-safe API access
#[derive(Debug, Clone)]
struct AgentState {
id: String,
task: Task,
status: RwLock<AgentStatus>,
logs: RwLock<Vec<String>>,
start_time: Instant,
max_steps: u32,
steps_taken: RwLock<u32>,
subagents: RwLock<HashMap<String, Arc<AgentState>>>,
}
CRITICAL: When the API server exposes endpoints that concurrently access AgentOrchestrator, AgentState fields must use RwLock (not RefCell). The API wraps DriverAgentOrchestrator in Mutex<>, and each handler locks it, calling methods that mutate AgentState. RefCell panics on concurrent borrows. Sources: build failures at execute() on line 146, engine_orchestrator_mut() on driver crate.
- All mutable fields:
status: RwLock<AgentStatus>,logs: RwLock<Vec<String>>,steps_taken: RwLock<u32>,subagents: RwLock<HashMap<String, Arc<AgentState>>>. - In struct literals:
RwLock::new(...)for each mutable field.
Re-export of Task to engine crate
The engine crate must re-export Task from types::agent for downstream crates (e.g., api crate) to use. This is done via pub use types::agent::Task; in crates/engine/src/lib.rs. Without this, browzeit_engine::Task does not exist and downstream crates using browzeit_engine::agent::Task fail to compile.
Workload Phase 2: MoE / MoA / Subagents / Session
MoE (Mixture of Experts)
moe.rsexportsMoERouterwithexperts: HashMap<String, ExpertConfig>.route(task) -> (String, ExpertConfig)selects best expert byscore_expert():- +10 per matching capability (capability check:
config.capabilities.contains(&cap)) - +10 if task has no tools (cheaper expert)
- +10 *
(1.0 - latency_estimate / 5.0)for faster experts
- +10 per matching capability (capability check:
score_expert()returns u32.ExpertConfigfields:name,model,max_tokens,temperature,capabilities: Vec<String>,cost_per_token: f64,latency_estimate: f32.dynamic_loadingcreates newexpert-{n}if no expert scores > 0.
MoA (Mixture of Agents)
moa.rsexportsMoAAggregatorwithconfig: MoAConfigandoutputs: HashMap<String, Vec<String>>.run(task, &agents, llm_client) -> String: iterateconfig.iteration_count(3) times:- Generate response from each agent via
llm_client.generate(prompt, tools) - Check convergence (all agents discuss same topic)
- Build consensus prompt from responses
- Generate consensus response
- If converged, break
- Generate response from each agent via
- Aggregation:
weighted_vote(highest score),weighted_avg(weighted by agent type),consensus(most common response) MoAConfigfields:name,agents: Vec<String>,iteration_count: u32,aggregation_strategy,temperature,max_tokens.
Subagent Spawning
spawn_subagent(prompt, tools) -> Arc<AgentState>:- Create
AgentStatewithid: format!("sub-{}", self.session.next_id()) task: Task::new(&prompt)withtask.tools = toolsmax_steps: 5- Insert into
self.agents(Arc clone returned)
- Create
AgentAction::SpawnSubagent { prompt, tools }triggers this path in the agent loop.
Session/chat Forking
fork_session(session_id, prompt) -> Session:- Clone parent session via
new_session.fork_from(&parent) new_session.add_chat(prompt.to_string())
- Clone parent session via
Sessiontracks:chats: Vec<String>,urls: Vec<String>,body_texts: Vec<String>,title,chat_id: u64.next_id()incrementschat_id.current_url()returns last URL orabout:blank.
Workload Phase 3: Driver Wrapper
driver/src/agent.rs provides DriverAgentOrchestrator:
- Use
types::browser::BrowserWindow(NOTdriver::shell::BrowserWindow)types::browser::BrowserWindowis a SKELETON with onlynew(),set_state(),add_tab()— 3 methods.driver::webview::BrowserWindowhas the full 20+ method implementation (navigate, go_back, go_forward, refresh, history, evaluate_js, etc.).- The driver's
agent.rsimportstypes::browser::BrowserWindowfor the orchestrator. Actual tab/navigation work is done ondriver::webview::BrowserWindow.
pub fn new() -> Self(0 args, createsBrowserWindow::new()internally)- Fields:
id: u64,browser: BrowserWindow,engine_orchestrator: AgentOrchestrator,task_queue: Vec<(String, String)>,results: HashMap<String, Snapshot>,shutdown_tx: Option<mpsc::SyncSender<()>>. result(&self, task_id: &str) -> Option<&Snapshot>returnsself.results.get(task_id).- The engine's
AgentOrchestratoris NOT wrapped inArc<Mutex<>>. The API crate wraps it externally.
Workload Phase 4: API Server (axum)
api/src/lib.rs exports DriverAgentOrchestrator + types module.
pub mod typescontainsApiResponse,BrowserTab,BrowserState,AgentTaskRequest,AgentTaskResponse,ThoughtEvent.pub static mut GLOBAL_ORCHESTRATOR: Option<Arc<std::sync::Mutex<DriverAgentOrchestrator>>>must bepubformain.rsto access it.- Routes:
GET /or/api/health->"ok"GET /api/browser/tabs->Json(ApiResponse)withBrowserTabfixturesPOST /api/browser/navigate->Json(ApiResponse<()>)withurlPOST /api/agent/task->Json(ApiResponse<AgentTaskResponse>)withtask_id,status,modelGET /api/agent/results/{task_id}->Json(ApiResponse<()>)viaGLOBAL_ORCHESTRATORlockGET /api/health-> health check
pub async fn run():- Create
Arc<Mutex<DriverAgentOrchestrator>> - Set
unsafe { GLOBAL_ORCHESTRATOR = Some(...) } - Build
axum::Routerwith routes let listener = TcpListener::bind("127.0.0.1:8767")axum::serve(listener, app).await.expect("Server failed")
- Create
NOTE: This host has KotMoE API occupying port 8766. Browzeit's API server binds to 8767 (not 8766 as the earlier stub draft said).
gui_shell/browzeit_shell.pysetsAPI_PORT = 8767. The default port 8766 will fail withaddr in use.
Port conventions on this host:
- API server (Rust axum):
127.0.0.1:8767 - GUI shell (Python pywebview + HTTP):
127.0.0.1:8765— servesindex.html+ static assets, proxies/api/*to the Rust server - SSE streaming for live thought: returned by the API server at
/api/agent/stream/{task_id}
async fn get_agent_results(Path(task_id): Path<String>) -> Json<api_types::ApiResponse<()>> {
let orchestrator = unsafe { browzeit_api::GLOBAL_ORCHESTRATOR.as_ref().and_then(|o| o.lock().ok()) };
if let Some(orchestrator) = orchestrator {
if let Some(_result) = orchestrator.result(&task_id) {
Json(api_types::ApiResponse {
success: true, data: None, error: None, task_id: Some(task_id.clone()),
})
} else {
Json(api_types::ApiResponse { success: false, data: None, error: Some("Task not found".into()), task_id: Some(task_id), })
}
} else {
Json(api_types::ApiResponse { success: false, data: None, error: Some("Orchestrator not initialized".into()), task_id: None, })
}
}
Workload Phase 5: GUI Shell (pywebview + SSE)
gui_shell/browzeit_shell.py:
BrowzeitHandler(SimpleHTTPRequestHandler)serves local HTTP server on127.0.0.1:8765.- Endpoints:
GET /api/browser/tabs->Json({"tabs": [...]})GET /api/agent/task->Json({"tasks": [...]})GET /api/agent/stream-> SSE (Content-Type: text/event-stream,Cache-Control: no-cache,Connection: keep-alive). Sendevent: connected+ JSON.POST /api/browser/navigate->Json({"url": url, "success": true})POST /api/agent/task-> creates task, simulates streaming (threaded), returns{"task_id": ..., "status": "queued", "result": None}POST /api/agent/stream-> SSE for thought streaming
webview.create_window("Browzeit - Next-Gen Browser", "http://127.0.0.1:8765/index.html", ...)for native shell.PYTHONPATH=src pytestfor verification of Python modules.
Verification Gate (F1)
Run in order; stop on first failure:
cargo check --workspace 2>&1 | grep "^error" | wc -l-> must be 0cargo check --workspace-> must complete with only warnings (no errors)cargo fmt --all -- --check-> must report FMT_OKcargo clippy --workspace --all-targets -- -D warnings-> 0 errorscargo test --workspace-> all crates green
Pitfalls
CRLF line endings break patch tool: Files in the repo have CRLF endings. Before patching, run
sed -i 's/\r$//' <file>to convert to LF. The patch tool silently fails on CRLF files.Agent module files are stubs: Files like
moi.rs,subagent.rs,swarm.rs,session.rsmay start as 20-25 byte stubs that need full implementation. Do NOT leave them as placeholders.Clone before move: When you need a value both for side effects and for later use, clone it before moving. E.g.,
responses.push((agent_id.clone(), response.clone()));— if you moveresponseinto the tuple, you cannot use it again for a second push.Mutable borrow through shared reference: When passing
&mut Sessionthrough&crate::agent::Session, you get a compile error. Either take&mutdirectly, or refactor the function to receive the mutable reference at the right level.Patch file must be re-read after external modification: After terminal commands modify a file (e.g.,
sed -i), the patching tool may report "was modified since last read." Re-read the file, then patch again.Right-side file not written: After a file write (cargo patch/terminal), verify with
cargo check— changes may not land if the file was not properly flushed.Heredoc corruption in bash: Writing Rust files via
cat << 'EOF'in bash can silently corrupt content (unbalanced quotes, escape issues). If the file does not compile, re-read it and re-write it using thewrite_filetool orpatchtool instead of heredocs. Thewrite_filetool is safe and does not mangle content.Rust 2024: unexpected closing delimiter: If
cargo checkreports^ unexpected closing delimiter: }, check the file for a duplicate closing brace — a struct/enum definition with a stray}after the final}is a common cause. Read the file around the reported line and remove the duplicate. This happened incrates/types/src/layout.rswhere an extra}after the final}ofimpl LayoutDocumentbroke compilation.API crate duplicate type definitions: The
api/src/lib.rsandapi/src/main.rsboth defineApiResponse,BrowserTab,AgentTaskRequest,AgentTaskResponse, etc. This is a merge-conflict waiting to happen. Consolidate the duplicate definitions into onetypesmodule (keep one source of truth).Workspace verification before declaring done: Always run
cargo check --workspaceand confirm exit 0 with 0 errors. Warnings are acceptable (they are usually unused variable/lint warnings). The Browzeit workspace has NEVER been built end-to-end — this is the target state, not the current state.
Verification Checklist
-
cargo check --workspace-> exit 0 -
cargo test --workspace-> all tests green - API server starts:
cargo run -p browzeit-api-> port 8767 listening - GUI shell starts:
python gui_shell/browzeit_shell.py-> window opens -
/api/healthreturns "ok" -
/api/browser/tabsreturns tabs list -
/api/agent/taskcreates task and streams SSE thoughts -
/api/agent/results/{task_id}returns results - SSE stream shows live thought events
- GUI chat panel displays tool call results
- Canvas shows agent-generated components
GUI verification
Start the GUI shell:
python gui_shell/browzeit_shell.pyVerify the window opens and serves
http://127.0.0.1:8765/. Do NOT useterminal(background=true)— pywebview requires an interactive session. Instead create a.batlaunch script for the user.Verify API endpoints respond:
curl http://127.0.0.1:8767/api/healthVerify SSE streaming:
curl -N http://127.0.0.1:8767/api/agent/streamAgent module files live in a directory, not a mod.rs:
engine/src/agent.rsdeclarespub mod moa; pub mod moi; ...— but these modules are separate files inengine/src/agent/. Each must be patched as a standalone file, not written as a nested mod.rs.AgentState requires interior mutability: All mutable fields in
AgentStateuseRefCell. In struct literals:RefCell::new(...)for every mutable field (status, logs, steps_taken, subagents).API crate thread-safety: The
DriverAgentOrchestratorcontainsAgentOrchestratorwhich hasRefCellfields. The API server wraps it inMutex<DriverAgentOrchestrator>for thread safety.GLOBAL_ORCHESTRATORmust bepubformain.rsto access it.Do NOT use
driver::shell::BrowserWindow: Usetypes::browser::BrowserWindowin the driver's agent.rs. The shell'sBrowserWindowis a different type.LLMClient import path:
LLMClienttrait is defined inengine/src/agent.rs(not inagent/protocol.rs). Import asuse crate::agent::LLMClient;, notuse crate::agent::protocol::LLMClient;.Agent module files are stubs: Files like
moi.rs,subagent.rs,swarm.rs,session.rsmay start as 20-25 byte stubs that need full implementation. Do NOT leave them as placeholders.Patch file must be re-read after external modification: After terminal commands modify a file (e.g.,
sed -i), the patching tool may report "was modified since last read." Re-read the file, then patch again.Workspace verification before declaring done: Always run
cargo check --workspaceand confirm exit 0 with 0 errors. Warnings are acceptable (they are usually unused variable/lint warnings).Rust 2024: static_mut_refs warning:
static mutreferences requireunsafeblocks. Warnings are acceptable — only errors block the build.Right-side file not written: After a file write (cargo patch/terminal), verify with
cargo check— changes may not land if the file was not properly flushed.
Verification Checklist
-
cargo check --workspace-> exit 0 -
cargo test --workspace-> all tests green - API server starts:
cargo run -p browzeit-api-> port 8767 listening - GUI shell starts:
python gui_shell/browzeit_shell.py-> window opens -
/api/healthreturns "ok" -
/api/browser/tabsreturns tabs list -
/api/agent/taskcreates task and streams SSE thoughts -
/api/agent/results/{task_id}returns results - SSE stream shows live thought events
- GUI chat panel displays tool call results
- Canvas shows agent-generated components
GUI verification
- Start the GUI shell:
python gui_shell/browzeit_shell.py - Verify the window opens and serves
http://127.0.0.1:8765/. Do NOT useterminal(background=true)— pywebview requires an interactive session. Instead create a.batlaunch script for the user. - Verify API endpoints respond:
curl http://127.0.0.1:8767/api/health - Verify SSE streaming:
curl -N http://127.0.0.1:8767/api/agent/stream
References
references/rust-workspace-pitfalls.md— Module paths, derives, imports, and crate-specific issues across the Browzeit workspace includingcustode-browzeit.
User's Preferred Workflow
- "Properly proceed with all optimally" = full implementation, never stubs, never placeholder text
- "What are next steps?" = implementation, not release advice
- "Thoroughness demanded: zero tolerance for errors/bugs, full implementation never stubs"
- Desktop GUI = scaffold pywebview/Tauri, never browser dashboard
- Generative UI = agents must create, modify, and optimize UI components in real-time
- Model picker, live stream of thought, tokens per second, tokens used metrics
- Model-toggle profiles (save/load/export/import)
- Cross-app collaboration: KotMoE + Agentic Workbench share provider registry, v1.1.0 profiles
Verification Checklist
-
cargo check --workspace-> exit 0 -
cargo test --workspace-> all tests green - API server starts:
cargo run -p browzeit-api-> port 8767 listening - GUI shell starts:
python gui_shell/browzeit_shell.py-> window opens -
/api/healthreturns "ok" -
/api/browser/tabsreturns tabs list -
/api/agent/taskcreates task and streams SSE thoughts -
/api/agent/results/{task_id}returns results - SSE stream shows live thought events
- GUI chat panel displays tool call results
- Canvas shows agent-generated components