LLM App Builder

LLM App Builder — LLM App Development Pipeline

An agent team collaborates to develop LLM apps through prompt engineering > RAG pipeline > evaluation > optimization > deployment.

Execution Mode

Agent Team — Five agents communicate directly via SendMessage and perform cross-validation.

Agent Composition

Workflow

Phase 1: Preparation (performed directly by the orchestrator)

1. Extract the following from user input:
   • App purpose: What the LLM app does
   • Data sources: Documents/data for RAG (optional)
   • LLM model: Model to use (default: Claude/GPT-4o)
   • Deployment environment: API/web app/chatbot/internal tool
   • Budget: Monthly API cost budget
   • Constraints (optional): Security, regulatory, performance requirements
2. Create the _workspace/ directory at the project root
3. Organize the input and save it to _workspace/00_input.md
4. Create the _workspace/src/ directory
5. Determine the execution mode based on the scope of the request

Phase 2: Team Assembly and Execution

Tasks 1a (prompt) and 1b (RAG) run in parallel.

Inter-agent communication flow:

• prompt completes > passes context injection format to rag, passes expected outputs to eval
• rag completes > passes retrieval test data to eval, passes vector DB infra requirements to deploy
• eval completes > passes performance baseline to optimizer, passes weakness feedback to prompt
• optimizer completes > passes cache/routing config to deploy
• deploy integrates all components to complete the production deployment configuration

Phase 3: Integration and Final Deliverables

1. Verify that the code in _workspace/src/ is executable
2. Confirm that evaluation metrics meet the standards
3. Validate that deployment configuration is complete
4. Report the final summary to the user:
   • Prompt design — 01_prompt_design.md
   • RAG pipeline — 02_rag_pipeline.md
   • Evaluation framework — 03_eval_framework.md
   • Optimization strategy — 04_optimization.md
   • Deployment config — 05_deploy_config.md
   • Source code — src/

Execution Modes by Request Scope

When RAG is not needed: If the user specifies there is no external data source, skip the rag agent.

Reusing existing files: If the user provides existing prompts, RAG configs, or code, copy them to the appropriate numbered location in _workspace/ and skip the corresponding agent. Example: Existing prompt provided > copy to _workspace/01_prompt_design.md > skip prompt and deploy remaining agents.

Data Transfer Protocol

Error Handling

Test Scenarios

Normal Flow

Prompt: "Build me an employee Q&A chatbot based on internal company documents. About 500 Confluence docs. Monthly budget $200" Expected result:

• Prompt: Q&A system prompt, forced source citation, hallucination prevention guardrails
• RAG: Confluence > markdown conversion > semantic chunking > text-embedding-3-small > Chroma
• Evaluation: 20 golden Q&A sets, Recall@5, faithfulness LLM-as-Judge
• Optimization: Semantic caching (expected 40% hit rate), small model routing (simple questions)
• Deployment: FastAPI + Docker + cost ceiling $200/month

Existing File Reuse Flow

Prompt: "Based on this RAG code, just do evaluation framework and optimization" + RAG code attached Expected result:

• Copy existing code to _workspace/src/, copy RAG design to _workspace/02_rag_pipeline.md
• Skip rag; deploy eval + optimizer + deploy
• Extract prompt from existing code

Error Flow

Prompt: "Build me an AI app" (purpose/data unclear) Expected result:

• Request user to clarify app purpose and data sources
• Ask questions to determine RAG necessity
• Proceed with appropriate mode after confirmation

Agent Extension Skills

Extension skills that enhance agent domain expertise: