How do I install the Choose Observability Stack skill?

Run npx skillmds@latest add contextjet-ai/choose-observability-stack in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Choose Observability Stack safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Choose Observability Stack?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Choose Observability Stack free to use?

Yes. Installing skills from SkillMD is free. This skill is licensed under CC0-1.

Who published Choose Observability Stack?

ContextJet-ai (@contextjet-ai) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Choose Observability Stack by contextjet-ai

Name: Choose Observability Stack
Author: ContextJet-ai

Choose Observability Stack

Use this to recommend an LLM observability / evaluation tool or stack for a specific situation. Trigger on "which observability tool should I use", "compare Langfuse vs Phoenix vs LangSmith", "what's the best LLM monitoring for us", or picking an eval/tracing/gateway tool given constraints (self-hosting, budget, compliance, existing stack). Ask about constraints, then recommend from the curated list - don't just name the most popular tool.

File contents

Choose an LLM observability stack

There's no single best tool - the right choice depends on constraints. Gather them, then map to a recommendation. Base recommendations on this repo's curated list (verified tools + licenses), not on hype.

Ask these constraints first

Deployment: SaaS OK, or must self-host / on-prem (data residency, regulated industry)?
Primary need: tracing/cost, evaluation (quality testing), or both? Prompt management too?
Existing stack: already on Datadog/Grafana/OTel? On LangChain? Using a gateway?
Budget/licensing: need a permissive OSS license (MIT/Apache), or is a commercial tier fine? (Note AGPL/Elastic-license implications for embedding.)
Code-change tolerance: want zero-code (proxy) or fine to add an SDK?
Team: engineers, or also non-technical PMs who need a UI?

Map constraints → recommendation

Must self-host, permissive license, want everything → Langfuse (MIT core: tracing + evals + prompts) or Comet Opik (Apache-2.0). For eval-heavy local work, Arize Phoenix.
Zero code changes, just want cost + logs → a gateway/proxy: Helicone (change base URL), LiteLLM or Portkey (also routing).
Already on OTel / want vendor-neutral, future-proof → emit OpenTelemetry GenAI semantic conventions via OpenLLMetry or OpenInference; export to your existing backend.
Deep in the LangChain ecosystem → LangSmith (tightest integration; SDK OSS, backend commercial).
Enterprise APM already (Datadog/New Relic) → use their LLM Observability product to keep one pane of glass.
Primary need is evaluation/testing, not dashboards → promptfoo (prompt/RAG + CI), DeepEval (pytest-style), Ragas (RAG metrics). Pair with a tracing tool for online scoring.
Regulated / finance / must audit + guardrail → self-hosted tracing (Langfuse/Phoenix) + guardrails (Guardrails AI, LLM Guard for PII/prompt-injection) + strict prompt/PII redaction.

Common production shape

A gateway (cost + routing) + an evaluation framework (quality) + an OTel-native tracing backbone. This keeps cost, quality, and traces decoupled and swappable.

Deliver the recommendation

Name a primary tool + a runner-up, each with a one-line why it fits these constraints.
Call out license/self-hosting implications explicitly (especially AGPL / Elastic-license for embedding, and SaaS data-egress for regulated data).
Link to the tool's row in this repo's README so they can compare stars/license.

Anti-pattern

Recommending the highest-star tool by default. LiteLLM has the most stars but is a gateway - it's the wrong answer for someone who asked for an evaluation framework. Match the tool category to the stated need.

---name: choose-observability-stackdescription: Use this to recommend an LLM observability / evaluation tool or stack for a specific situation. Trigger on "which observability tool should I use", "compare Langfuse vs Phoenix vs LangSmith", "what's the best LLM monitoring for us", or picking an eval/tracing/gateway tool given constraints (self-hosting, budget, compliance, existing stack). Ask about constraints, then recommend from the curated list - don't just name the most popular tool.license: CC0-1.0---# Choose an LLM observability stackThere's no single best tool - the right choice depends on constraints. Gather them, then map to a recommendation. Base recommendations on this repo's curated list (verified tools + licenses), not on hype.## Ask these constraints first1. **Deployment**: SaaS OK, or must self-host / on-prem (data residency, regulated industry)?2. **Primary need**: tracing/cost, *evaluation* (quality testing), or both? Prompt management too?3. **Existing stack**: already on Datadog/Grafana/OTel? On LangChain? Using a gateway?4. **Budget/licensing**: need a permissive OSS license (MIT/Apache), or is a commercial tier fine? (Note AGPL/Elastic-license implications for embedding.)5. **Code-change tolerance**: want zero-code (proxy) or fine to add an SDK?6. **Team**: engineers, or also non-technical PMs who need a UI?## Map constraints → recommendation- **Must self-host, permissive license, want everything** → **Langfuse** (MIT core: tracing + evals + prompts) or **Comet Opik** (Apache-2.0). For eval-heavy local work, **Arize Phoenix**.- **Zero code changes, just want cost + logs** → a **gateway/proxy**: **Helicone** (change base URL), **LiteLLM** or **Portkey** (also routing).- **Already on OTel / want vendor-neutral, future-proof** → emit **OpenTelemetry GenAI semantic conventions** via **OpenLLMetry** or **OpenInference**; export to your existing backend.- **Deep in the LangChain ecosystem** → **LangSmith** (tightest integration; SDK OSS, backend commercial).- **Enterprise APM already (Datadog/New Relic)** → use their **LLM Observability** product to keep one pane of glass.- **Primary need is *evaluation*/testing, not dashboards** → **promptfoo** (prompt/RAG + CI), **DeepEval** (pytest-style), **Ragas** (RAG metrics). Pair with a tracing tool for online scoring.- **Regulated / finance / must audit + guardrail** → self-hosted tracing (Langfuse/Phoenix) + **guardrails** (Guardrails AI, LLM Guard for PII/prompt-injection) + strict prompt/PII redaction.## Common production shapeA gateway (cost + routing) **+** an evaluation framework (quality) **+** an OTel-native tracing backbone. This keeps cost, quality, and traces decoupled and swappable.## Deliver the recommendation- Name a **primary** tool + a **runner-up**, each with a one-line *why it fits these constraints*.- Call out license/self-hosting implications explicitly (especially AGPL / Elastic-license for embedding, and SaaS data-egress for regulated data).- Link to the tool's row in this repo's README so they can compare stars/license.## Anti-patternRecommending the highest-star tool by default. LiteLLM has the most stars but is a *gateway* - it's the wrong answer for someone who asked for an *evaluation* framework. Match the tool category to the stated need.

Frequently asked questions