Langsmith

LangSmith Skill

Expert assistance for the LangSmith Python SDK: observability, evaluation, and dataset management for LLM applications.

Install: pip install langsmith
Setup: export LANGSMITH_API_KEY="ls__..." and export LANGSMITH_TRACING=true

Reference: references/api.md (500 KB — full API reference).

When to Use This Skill

Activate when:

• Tracing functions — adding @traceable to instrument agent steps, LLM calls, or tool executions
• Manual tracing — using trace context manager or tracing_context() for fine-grained control
• Running evaluations — calling evaluate() or aevaluate() on a dataset with custom evaluators
• Comparative A/B evaluation — passing a tuple of experiment IDs to compare two runs
• Writing custom evaluators — using @run_evaluator or summary_evaluators for dataset-level scoring
• Managing datasets — creating, updating, or querying examples via Client
• Attaching files to traces — using Attachment for images, audio, or binary data
• Tracing ASGI/WSGI apps — using TracingMiddleware for FastAPI/Starlette/Django
• Pytest integration — using LangSmithPlugin for test-level tracing
• Prompt caching — using PromptCache or AsyncPromptCache
• Handling LangSmith errors — catching LangSmithAPIError, LangSmithRateLimitError, etc.

Quick Reference

Instrument a function with @traceable

from langsmith import traceable

@traceable(name="my_llm_call", run_type="llm")
def call_llm(prompt: str) -> str:
    return llm.invoke(prompt)

@traceable(name="my_tool", tags=["tool", "search"])
def search(query: str) -> list[str]:
    return search_index.query(query)

# Async works too
@traceable
async def async_agent(inputs: dict) -> dict:
    result = await llm.ainvoke(inputs["prompt"])
    return {"output": result}

Manual trace context manager

from langsmith.run_helpers import trace, tracing_context

# Explicit span with full control
with trace(name="my_pipeline", run_type="chain") as run:
    run.metadata["version"] = "v2"
    result = run_pipeline(inputs)
    run.end(outputs={"result": result})

# Set tracing context for a block
with tracing_context(project_name="my-project", tags=["prod"]):
    result = agent.invoke(inputs)

Add metadata to the current run

from langsmith.run_helpers import get_current_run_tree, set_run_metadata

@traceable
def my_step(inputs: dict) -> dict:
    # Attach metadata to whatever run is active
    set_run_metadata({"user_id": inputs["user_id"], "model": "claude-sonnet-4-6"})

    run = get_current_run_tree()
    run.name = f"step-{inputs['step_id']}"

    return process(inputs)

Run evaluation on a dataset

from langsmith import Client

client = Client()

def target(inputs: dict) -> dict:
    return {"answer": my_agent.invoke(inputs["question"])}

def correctness_evaluator(run, example) -> dict:
    score = llm_judge(run.outputs["answer"], example.outputs["expected"])
    return {"key": "correctness", "score": score}

def length_summary_evaluator(runs, examples) -> dict:
    avg_len = sum(len(r.outputs["answer"]) for r in runs) / len(runs)
    return {"key": "avg_length", "score": avg_len}

results = client.evaluate(
    target,
    data="my-dataset-name",          # dataset name, ID, or list of Examples
    evaluators=[correctness_evaluator],
    summary_evaluators=[length_summary_evaluator],
    experiment_prefix="my-exp",
    max_concurrency=4,               # None = unlimited, 0 = sequential
    num_repetitions=3,               # run each example 3x
    blocking=True,                   # wait for completion
    error_handling="log",            # or "ignore"
)

Async evaluation

from langsmith.evaluation import aevaluate

results = await aevaluate(
    async_target,
    data="my-dataset",
    evaluators=[correctness_evaluator],
    max_concurrency=10,
)

Comparative A/B evaluation

# Pass two experiment IDs to compare them with the same evaluators
results = client.evaluate(
    (experiment_id_a, experiment_id_b),   # tuple of two existing experiments
    evaluators=[correctness_evaluator],
    # summary_evaluators must be omitted for comparative mode
)

# Or use evaluate_comparative() for custom side-by-side evaluators
from langsmith.evaluation import evaluate_comparative

def compare(runs_a, runs_b) -> dict:
    return {"key": "preference", "score": judge_preference(runs_a, runs_b)}

evaluate_comparative([exp_id_a, exp_id_b], evaluators=[compare])

Custom evaluator decorator

from langsmith.evaluation import run_evaluator

@run_evaluator
def my_evaluator(run, example) -> dict:
    prediction = run.outputs.get("answer", "")
    expected = example.outputs.get("expected", "")
    return {
        "key": "exact_match",
        "score": int(prediction.strip() == expected.strip()),
        "comment": f"Got: {prediction!r}",
    }

results = client.evaluate(target, data="dataset", evaluators=[my_evaluator])

Attach files to a trace

from langsmith import traceable
from langsmith.schemas import Attachment
from pathlib import Path

@traceable
def analyze_image(image_path: Path) -> dict:
    attachment = Attachment(
        mime_type="image/png",
        data=image_path.read_bytes(),
    )
    # Attachment is automatically linked to the active run
    return {"result": vision_model.invoke(image_path)}

ASGI middleware (FastAPI / Starlette)

from fastapi import FastAPI
from langsmith.middleware import TracingMiddleware

app = FastAPI()
app.add_middleware(TracingMiddleware)  # traces every request as a LangSmith run

@app.post("/chat")
async def chat(request: ChatRequest):
    return {"response": await agent.ainvoke(request.message)}

Pytest plugin

# conftest.py — enable LangSmith tracing for all tests
# Install: pip install langsmith[pytest]
# Run: pytest --langsmith  (or set LANGSMITH_TEST_TRACKING=true)
# Tests appear as experiments in LangSmith UI

def test_my_agent():
    result = my_agent.invoke({"question": "What is 2+2?"})
    assert result["answer"] == "4"

API Reference

Tracing

Evaluation

Client

Error types

Reference Files

Source: https://reference.langchain.com/python/langsmith
GitHub: https://github.com/langchain-ai/langsmith-sdk