Feature Map Maintenance
You are maintaining /feature-map.json — the canonical information architecture for LangWatch. Every platform feature is defined here with its surfaces (how it's accessed) and sync state (how code and platform relate). All implementations (sidebar, docs, skills, MCP tools) derive from this map.
/FEATURE_MAP.md is the human-readable companion — a coverage table derived from the JSON. Keep it in sync when you edit the JSON. Use the underscore form (FEATURE_MAP.md), matching GitHub's convention for uppercase root files like CODE_OF_CONDUCT.md and PULL_REQUEST_TEMPLATE.md. Never create a hyphenated FEATURE-MAP.md variant.
Information Architecture
The hierarchy represents the product's mental model, not the code structure:
observability/ — Tracing, Analytics, User Events, Annotations
evaluations/ — Experiments, Online Evaluation (includes guardrails via code)
agent-simulations/ — Scenarios, Runs
prompt-management/ — Prompts, Prompt Playground
library/ — Agents, Workflows, Evaluators, Datasets
settings/ — Model Providers
Key Design Decisions
- No "integrations" category — SDKs/frameworks enable features, they aren't features themselves. Each feature declares its own SDK surface.
- Library contains reusable components (evaluators, datasets, agents, workflows) — NOT "platform" catch-all.
- Annotations live in Observability (they annotate traces).
- Guardrails = online-evaluation accessed via code (
as_guardrail=True), not a separate concept.
- Evaluators and Datasets are in Library, not Evaluations — they're shared components used by experiments, online evaluation, and simulations.
The Surfaces Model
Each feature has two main access paths:
code — developer writes files in their project (SDK, CLI, skill)
platform — no-code via UI or MCP tools (UI route, MCP tool, platform skill)
Plus cross-cutting:
api — REST/Hono API endpoint namespace (used by both code and platform)
docs — canonical documentation URL
Fields point to namespaces, not individual methods. E.g., "python": "langwatch.experiment" means the whole experiment module, not just init().
The Sync Model
How code and platform relate for each feature:
| sync value |
meaning |
example |
null |
separate or one-mode only |
annotations (platform only) |
"bidirectional" |
code ↔ platform, synced |
prompts (via prompt sync) |
"code-to-platform" |
code generates, platform displays |
tracing, experiments |
"platform-to-code" |
platform configures, code consumes |
— (none currently) |
plannedSync captures known future intent (e.g., scenarios will become "bidirectional").
Where to Find Things in the Codebase
API Endpoints
- Hono routes (current):
platform/app/src/app/api/ — each [[...route]]/app.ts is a Hono app
- traces:
platform/app/src/app/api/traces/[[...route]]/app.ts
- scenarios:
platform/app/src/app/api/scenarios/[[...route]]/app.ts
- prompts:
platform/app/src/app/api/prompts/[[...route]]/app.ts
- evaluators:
platform/app/src/app/api/evaluators/[[...route]]/app.ts
- datasets:
platform/app/src/app/api/dataset/[[...route]]/
- analytics:
platform/app/src/app/api/analytics/
- model-providers:
platform/app/src/app/api/model-providers/[[...route]]/
- Legacy Next.js routes (being migrated):
platform/app/src/pages/api/
- tRPC routers:
platform/app/src/server/api/routers/ registered in platform/app/src/server/api/root.ts
Platform UI
- Route definitions:
platform/app/src/utils/routes.ts — projectRoutes object has every page route
- Sidebar menu:
platform/app/src/components/MainMenu.tsx — sections: Observe, Evaluate, Library
- Feature icons:
platform/app/src/utils/featureIcons.ts
MCP Tools
- All tools:
mcp/typescript/src/index.ts — every server.tool() call
- Tool handlers:
mcp/typescript/src/tools/*.ts
- Currently 21 tools: 2 docs, 1 discovery, 3 observability, 4 prompt, 5 scenario, 4 evaluator, 2 model-provider
CLI Commands
- Entry point:
sdks/typescript/src/cli/index.ts
- Command implementations:
sdks/typescript/src/cli/commands/
- Currently:
login + prompt subcommands (init, create, add, remove, list, sync, pull, push)
SDKs
- Python:
sdks/python/src/langwatch/__init__.py (top-level exports), modules: experiment, evaluation, dataset, evaluators, prompts, dspy
- TypeScript:
sdks/typescript/src/index.ts — LangWatch class with .prompts, .experiments, .evaluations, .evaluators, .datasets, .traces
- Scenario SDK (separate):
@langwatch/scenario (TS) / langwatch-scenario (Python)
Skills (external, for users)
- Location:
skills/*/SKILL.md
- Feature skills: tracing, evaluations, scenarios, prompts (each handles both code and platform approaches)
- Meta skills: level-up (orchestrates all feature skills)
- Cross-cutting: analytics
Documentation
- LangWatch docs: served via
fetch_langwatch_docs MCP tool, index at https://langwatch.ai/docs/llms.txt
- Scenario docs: served via
fetch_scenario_docs MCP tool, index at https://langwatch.ai/scenario/llms.txt
How to Update the Feature Map
When a new API endpoint is added
- Read
feature-map.json
- Find the feature entry by
id
- Update
surfaces.api with the route namespace
- If it's a new feature, create a new entry under the right category
When a new MCP tool is added
- Verify the tool exists in
mcp/typescript/src/index.ts
- Add the tool name to
surfaces.platform.mcp array
When a new skill is created
- Verify the skill exists in
skills/{name}/SKILL.md
- Add to
surfaces.code.skill (for code-path skills) or surfaces.platform.skill (for platform-path skills)
When a new CLI command is added
- Verify it exists in
sdks/typescript/src/cli/commands/
- Add to
surfaces.code.cli array
When SDK surface changes
- Update
surfaces.code.sdk with the namespace
When sync capability changes
- Move value from
plannedSync to sync
- Or set new
plannedSync for future plans
When a completely new feature is added
- Decide which category it belongs to based on the hierarchy rules above
- Create a new entry with all known surfaces
- Set
sync appropriately
- Consider: does it need a skill? A docs page? MCP tool?
Validation
After any change, verify:
- Every
api value corresponds to a route in platform/app/src/app/api/ or platform/app/src/pages/api/
- Every
mcp tool name appears in mcp/typescript/src/index.ts
- Every
skill name has a skills/{name}/SKILL.md
- Every
cli command exists in sdks/typescript/src/cli/
- Every
ui route exists in platform/app/src/utils/routes.ts
- No aspirational entries (use
plannedSync for future intent)
Task
$ARGUMENTS
1---2name: feature-map3description: "Maintain the canonical LangWatch feature map (/feature-map.json). Use when adding features, APIs, MCP tools, CLI commands, or skills — to update the central registry and keep surfaces in sync."4user-invocable: true5argument-hint: "[what changed, e.g. 'added dataset MCP tools']"6metadata:7 source: https://skillmd.com/skills/langwatch/feature-map8---910# Feature Map Maintenance1112You are maintaining `/feature-map.json` — the **canonical information architecture** for LangWatch. Every platform feature is defined here with its **surfaces** (how it's accessed) and **sync** state (how code and platform relate). All implementations (sidebar, docs, skills, MCP tools) derive from this map.1314`/FEATURE_MAP.md` is the human-readable companion — a coverage table derived from the JSON. Keep it in sync when you edit the JSON. Use the **underscore** form (`FEATURE_MAP.md`), matching GitHub's convention for uppercase root files like `CODE_OF_CONDUCT.md` and `PULL_REQUEST_TEMPLATE.md`. Never create a hyphenated `FEATURE-MAP.md` variant.1516## Information Architecture1718The hierarchy represents the product's mental model, not the code structure:1920```21observability/ — Tracing, Analytics, User Events, Annotations22evaluations/ — Experiments, Online Evaluation (includes guardrails via code)23agent-simulations/ — Scenarios, Runs24prompt-management/ — Prompts, Prompt Playground25library/ — Agents, Workflows, Evaluators, Datasets26settings/ — Model Providers27```2829### Key Design Decisions30311. **No "integrations" category** — SDKs/frameworks enable features, they aren't features themselves. Each feature declares its own SDK surface.322. **Library** contains reusable components (evaluators, datasets, agents, workflows) — NOT "platform" catch-all.333. **Annotations** live in Observability (they annotate traces).344. **Guardrails** = online-evaluation accessed via code (`as_guardrail=True`), not a separate concept.355. **Evaluators** and **Datasets** are in Library, not Evaluations — they're shared components used by experiments, online evaluation, and simulations.3637### The Surfaces Model3839Each feature has two main access paths:4041- **`code`** — developer writes files in their project (SDK, CLI, skill)42- **`platform`** — no-code via UI or MCP tools (UI route, MCP tool, platform skill)4344Plus cross-cutting:45- **`api`** — REST/Hono API endpoint namespace (used by both code and platform)46- **`docs`** — canonical documentation URL4748Fields point to **namespaces**, not individual methods. E.g., `"python": "langwatch.experiment"` means the whole experiment module, not just `init()`.4950### The Sync Model5152How code and platform relate for each feature:5354| sync value | meaning | example |55|---|---|---|56| `null` | separate or one-mode only | annotations (platform only) |57| `"bidirectional"` | code ↔ platform, synced | prompts (via `prompt sync`) |58| `"code-to-platform"` | code generates, platform displays | tracing, experiments |59| `"platform-to-code"` | platform configures, code consumes | — (none currently) |6061`plannedSync` captures known future intent (e.g., scenarios will become `"bidirectional"`).6263## Where to Find Things in the Codebase6465### API Endpoints66- **Hono routes** (current): `platform/app/src/app/api/` — each `[[...route]]/app.ts` is a Hono app67 - traces: `platform/app/src/app/api/traces/[[...route]]/app.ts`68 - scenarios: `platform/app/src/app/api/scenarios/[[...route]]/app.ts`69 - prompts: `platform/app/src/app/api/prompts/[[...route]]/app.ts`70 - evaluators: `platform/app/src/app/api/evaluators/[[...route]]/app.ts`71 - datasets: `platform/app/src/app/api/dataset/[[...route]]/`72 - analytics: `platform/app/src/app/api/analytics/`73 - model-providers: `platform/app/src/app/api/model-providers/[[...route]]/`74- **Legacy Next.js routes** (being migrated): `platform/app/src/pages/api/`75- **tRPC routers**: `platform/app/src/server/api/routers/` registered in `platform/app/src/server/api/root.ts`7677### Platform UI78- **Route definitions**: `platform/app/src/utils/routes.ts` — `projectRoutes` object has every page route79- **Sidebar menu**: `platform/app/src/components/MainMenu.tsx` — sections: Observe, Evaluate, Library80- **Feature icons**: `platform/app/src/utils/featureIcons.ts`8182### MCP Tools83- **All tools**: `mcp/typescript/src/index.ts` — every `server.tool()` call84- **Tool handlers**: `mcp/typescript/src/tools/*.ts`85- Currently 21 tools: 2 docs, 1 discovery, 3 observability, 4 prompt, 5 scenario, 4 evaluator, 2 model-provider8687### CLI Commands88- **Entry point**: `sdks/typescript/src/cli/index.ts`89- **Command implementations**: `sdks/typescript/src/cli/commands/`90- Currently: `login` + `prompt` subcommands (init, create, add, remove, list, sync, pull, push)9192### SDKs93- **Python**: `sdks/python/src/langwatch/__init__.py` (top-level exports), modules: `experiment`, `evaluation`, `dataset`, `evaluators`, `prompts`, `dspy`94- **TypeScript**: `sdks/typescript/src/index.ts` — `LangWatch` class with `.prompts`, `.experiments`, `.evaluations`, `.evaluators`, `.datasets`, `.traces`95- **Scenario SDK** (separate): `@langwatch/scenario` (TS) / `langwatch-scenario` (Python)9697### Skills (external, for users)98- **Location**: `skills/*/SKILL.md`99- **Feature skills**: tracing, evaluations, scenarios, prompts (each handles both code and platform approaches)100- **Meta skills**: level-up (orchestrates all feature skills)101- **Cross-cutting**: analytics102103### Documentation104- **LangWatch docs**: served via `fetch_langwatch_docs` MCP tool, index at `https://langwatch.ai/docs/llms.txt`105- **Scenario docs**: served via `fetch_scenario_docs` MCP tool, index at `https://langwatch.ai/scenario/llms.txt`106107## How to Update the Feature Map108109### When a new API endpoint is added1101. Read `feature-map.json`1112. Find the feature entry by `id`1123. Update `surfaces.api` with the route namespace1134. If it's a new feature, create a new entry under the right category114115### When a new MCP tool is added1161. Verify the tool exists in `mcp/typescript/src/index.ts`1172. Add the tool name to `surfaces.platform.mcp` array118119### When a new skill is created1201. Verify the skill exists in `skills/{name}/SKILL.md`1212. Add to `surfaces.code.skill` (for code-path skills) or `surfaces.platform.skill` (for platform-path skills)122123### When a new CLI command is added1241. Verify it exists in `sdks/typescript/src/cli/commands/`1252. Add to `surfaces.code.cli` array126127### When SDK surface changes1281. Update `surfaces.code.sdk` with the namespace129130### When sync capability changes1311. Move value from `plannedSync` to `sync`1322. Or set new `plannedSync` for future plans133134### When a completely new feature is added1351. Decide which category it belongs to based on the hierarchy rules above1362. Create a new entry with all known surfaces1373. Set `sync` appropriately1384. Consider: does it need a skill? A docs page? MCP tool?139140## Validation141142After any change, verify:143- Every `api` value corresponds to a route in `platform/app/src/app/api/` or `platform/app/src/pages/api/`144- Every `mcp` tool name appears in `mcp/typescript/src/index.ts`145- Every `skill` name has a `skills/{name}/SKILL.md`146- Every `cli` command exists in `sdks/typescript/src/cli/`147- Every `ui` route exists in `platform/app/src/utils/routes.ts`148- No aspirational entries (use `plannedSync` for future intent)149150## Task151152$ARGUMENTS