Promptic

Promptic Python SDK

SDK and CLI for the Promptic platform — LLM tracing, prompt optimization, and Agent Optimization.

Installation

pip install promptic-sdk

Install extras for auto-instrumentation:

# LLM providers
pip install promptic-sdk[openai]         # OpenAI
pip install promptic-sdk[anthropic]      # Anthropic
pip install promptic-sdk[bedrock]        # AWS Bedrock
pip install promptic-sdk[vertexai]       # Google Vertex AI
pip install promptic-sdk[mistralai]      # Mistral

# Agent frameworks
pip install promptic-sdk[langchain]      # LangChain / LangGraph / create_agent / deepagents
pip install promptic-sdk[openai-agents]  # OpenAI Agents SDK
pip install promptic-sdk[claude-agent]   # Claude Agent SDK

pip install promptic-sdk[all]            # Everything above

Pydantic AI ships its own OpenTelemetry emitter — enable with Agent(..., instrument=True), no extras needed.

Authentication

# Browser login (local dev)
promptic login

# CI/CD
export PROMPTIC_API_KEY="ptc_..."

Config resolution: explicit args > env vars (PROMPTIC_API_KEY, PROMPTIC_ENDPOINT) > ~/.promptic/config.toml.

Tracing

Call promptic_sdk.init() once at startup. All LLM calls from installed providers are auto-instrumented via OpenTelemetry.

import promptic_sdk
from openai import OpenAI

promptic_sdk.init(service_name="my-agent")
client = OpenAI()

with promptic_sdk.ai_component("my-agent"):
    response = client.chat.completions.create(
        model="gpt-4.1-nano",
        messages=[{"role": "user", "content": "Hello!"}],
    )

init() parameters

The API key determines the owning AI Application. service_name identifies the emitting workload within that application; it is discovered from telemetry and does not need to be created in the dashboard. Set the deployment environment separately with the standard OpenTelemetry resource attribute so Tracing can filter production from development traffic:

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"

Tracing surfaces service and environment as telemetry-derived filters. The Python SDK's list_traces() and get_stats() do not yet expose these as arguments — filter by them with the service and environment query parameters on the REST API when you need to query programmatically.

Auto-detected instrumentors: OpenAI, Anthropic, Google Generative AI, Vertex AI, Bedrock, Mistral, Cohere, LangChain (with LangGraph / create_agent / deepagents), OpenAI Agents SDK, Claude Agent SDK. All emit the official OpenTelemetry GenAI semantic conventions (gen_ai.*).

File and media artifacts

Promptic automatically offloads inline base64 media and large file-like content from auto-instrumented spans into trace artifacts. The span keeps a lightweight promptic-artifact://... reference, and the UI/API/CLI can fetch the bytes on demand.

Artifact uploads should avoid routing file bytes through the Promptic app server when the platform supports direct storage uploads. The SDK should prefer the storage-object flow: request a presigned upload from Promptic, upload bytes directly to object storage, then register the artifact metadata with Promptic. Only fall back to server-side contentBase64 uploads for older servers or temporary compatibility. Do not ask users or coding agents to manually keep large base64 payloads in span attributes.

Do not silently read local filesystem paths from span attributes. If the user wants local file contents in a trace, attach them explicitly:

file_ref = promptic_sdk.artifact("/tmp/report.pdf")
span.set_attribute("retrieval.input_file", file_ref.ref)

The artifact's name is used as the default download filename. It defaults to a local file's base name; pass name= to override it (or to set it for bytes or text content), for example promptic_sdk.artifact(pdf_bytes, name="report.pdf"), and read it back from file_ref.name.

Use this helper for unsupported custom file payloads. External HTTP(S) URLs can remain as URLs.

When debugging image/file previews in the UI, distinguish ingestion from browser rendering. If traces contain promptic-artifact://... references and artifact metadata exists, but images do not render, check that the frontend Content Security Policy allows the object-storage origin. Self-hosted or custom storage setups should provide APP_STORAGE_CSP_ORIGINS with the browser-facing storage origin, and production Docker/Next builds must receive that value at build time.

ai_component context manager

Attribute spans to an existing AI Component by name. Attribution is optional and recorded per span, so a single trace may span several components (or none) — wrap a call in ai_component(...) only when you want those spans connected to a specific component. The platform links each tagged span to the matching component.

with promptic_sdk.ai_component("customer-support-agent"):
    # All LLM calls here are attributed to this component
    ...

# With dataset and run tagging for evaluation.
# dataset_id is the UUID of an existing dataset (create it first via the API/CLI):
with promptic_sdk.ai_component("my-agent", dataset_id="<dataset-uuid>", run="v1-baseline"):
    agent.run(test_input)

Parameters:

• name (str): AI Component name in the AI Application
• dataset_id (str | UUID, optional): UUID of an existing dataset — traces are tagged with it and added to that dataset. The dataset must already exist; invalid UUIDs are rejected up front.
• run (str, optional): Run name — groups traces within a dataset for comparison. Requires dataset_id.

dataset context manager

Tag spans with an existing dataset UUID independently:

with promptic_sdk.ai_component("my-agent"):
    with promptic_sdk.dataset("<dataset-uuid>"):
        agent.run(test_input)

Tracing workflows with custom spans

Most users don't need this. With the right [extras] installed, auto-instrumentation already creates spans for every LLM and tool call. Reach for custom spans only when you have meaningful non-LLM workflow logic (retrieval, normalization, business rules, control flow) you want represented in the trace.

When you do need it, wrap your workflow stages in custom OpenTelemetry spans. Auto-instrumented provider spans automatically nest under whichever custom span is active.

Recommended pattern:

1. Wrap the whole run in one root workflow span inside ai_component(...).
2. Add a child task span for each meaningful stage of the pipeline.
3. Record the stage's input and output as span attributes so the trace reads as a transformation, not just a list of LLM calls.

import json
import promptic_sdk
from opentelemetry import trace

promptic_sdk.init()
tracer = trace.get_tracer(__name__)

with promptic_sdk.ai_component("my-agent"):
    with tracer.start_as_current_span("run_workflow") as root:
        root.set_attribute("traceloop.span.kind", "workflow")
        root.set_attribute("traceloop.entity.input", json.dumps(user_input))

        with tracer.start_as_current_span("retrieve_context") as span:
            span.set_attribute("traceloop.span.kind", "task")
            span.set_attribute("traceloop.entity.input", json.dumps(query))
            context = retrieve(query)
            span.set_attribute("traceloop.entity.output", json.dumps(context))

        with tracer.start_as_current_span("generate_answer") as span:
            span.set_attribute("traceloop.span.kind", "task")
            # Auto-instrumented LLM call nests under this task span
            answer = llm_call(context)

        root.set_attribute("traceloop.entity.output", json.dumps(answer))

Span attribute conventions:

• traceloop.span.kind="workflow" — the top-level run
• traceloop.span.kind="task" — an internal pipeline stage
• traceloop.entity.input / traceloop.entity.output — JSON-serialized stage payloads
• gen_ai.* — reserved for LLM/tool spans; auto-instrumentors emit these

Tips:

• Use semantic span names (retrieve_context, rerank_results) instead of generic function names when several calls would otherwise collide.

• For large payloads, log a small preview plus a count rather than the full object — traces are not meant to store data:

  span.set_attribute(
      "traceloop.entity.output",
      json.dumps({
          "items": items[:5],
          "item_count": len(items),
          "additional_item_count": max(len(items) - 5, 0),
      }),
  )
  

Verify with promptic traces get <trace-id> --json: the root workflow span should carry structured input/output, task spans should appear as its children, and auto-instrumented LLM/tool spans should nest under the task that triggered them.

Custom OpenTelemetry instrumentors

Since Promptic uses standard OpenTelemetry, add any OTel-compatible instrumentor:

import promptic_sdk
from opentelemetry.instrumentation.requests import RequestsInstrumentor

promptic_sdk.init()
RequestsInstrumentor().instrument()  # Spans exported to Promptic

LangGraph / deepagents integration

pip install promptic-sdk[langchain] installs OpenLLMetry's opentelemetry-instrumentation-langchain (≥0.60), which covers LangChain chains, LangGraph (create_agent), and deepagents with subagents. Emits the official OpenTelemetry GenAI semantic conventions (gen_ai.tool.definitions, gen_ai.operation.name, gen_ai.usage.*) for flat agents and multi-agent graphs uniformly.

Users who prefer the LangSmith OTel bridge (e.g. for hybrid dual-export to LangSmith) can opt in by setting LANGSMITH_TRACING=true and LANGSMITH_OTEL_ENABLED=true before calling init(). Note: the LangSmith bridge does not emit tool definitions, so tool metadata may be incomplete on LangSmith-bridged traces.

API Client

Both sync (PrompticClient) and async (AsyncPrompticClient) clients with identical method signatures.

from promptic_sdk import PrompticClient

with PrompticClient() as client:
    traces = client.list_traces(limit=10)

from promptic_sdk import AsyncPrompticClient

async with AsyncPrompticClient() as client:
    traces = await client.list_traces(limit=10)

Constructor args: api_key, access_token, ai_application_id, endpoint, timeout (default 30s). workspace_id is a deprecated alias for ai_application_id.

API reference

For detailed method signatures and parameters, see references/api.md.

Agent Optimization: external submissions

Use AgentGymClient.run_and_submit(...) or promptic agent-gym run when a complete Agent runs outside Promptic. Promptic supplies one immutable benchmark version; the trusted runner executes every case and progressively persists its predictions, generated files, and optional traces before requesting scoring.

Choose the workflow before writing code:

1. Existing benchmark: authenticate → pull the published dataset → inspect the public contract and inputs → implement → run and submit → inspect and compare → iterate.
2. New benchmark: author the contract, cases, and evaluators → review and publish → calibrate with a baseline → begin variant iteration.
3. Changed benchmark: inspect and resolve the draft → publish → re-evaluate existing predictions for evaluator-only changes, or rerun variants when the execution contract changed.
4. Isolated execution: use a trusted coordinator and resumable session → materialize inputs → persist predictions → finalize → wait or recover.

When the task includes creating or revising the benchmark, first read references/agent-benchmark.md. It explains the Agent contract, representative cases, expected behavior, evaluator selection, evaluator configuration, immutable versions, re-evaluation, and when variants must be resubmitted. Then read references/agent-gym.md for the trust boundary, existing-benchmark discovery, callback contract, resumable sessions, durable uploads, scoring submission, result inspection, comparison, and recovery. For exact public signatures, also read the Agent Optimization section of references/api.md. Do not add or describe Auto Engineer or autonomous optimization loops; they are not part of this workflow.

Prompt Optimization Workflow

Optimize prompts via experiments:

from promptic_sdk import PrompticClient

with PrompticClient() as client:
    # Create experiment
    exp = client.create_experiment(
        ai_component_id="comp_...",
        target_model="gpt-4.1-nano",
        task_type="classification",  # or "textGeneration", "structuredOutput"
        initial_prompt="Classify the following text into categories.",
        optimizer="prompticV2",      # or "miproV2", "bootstrapFewShot"
    )

    # Add training data as dataset cases on the experiment's dedicated dataset
    client.create_dataset_cases(exp["aiComponentId"], exp["datasetId"], [
        {"inputPayload": {"message": "Great product!"}, "expectedPayload": "positive"},
        {"inputPayload": {"message": "Terrible service"}, "expectedPayload": "negative"},
    ])

    # Add evaluators
    client.create_evaluators(exp["id"], [
        {"name": "accuracy", "type": "f1", "weight": 1.0},
    ])

    # Start optimization
    client.start_experiment(exp["id"])

    # After completion, deploy the best prompt
    best = client.get_best_iteration(exp["id"])
    client.deploy("comp_...", exp["id"])

    # Fetch deployed prompt at runtime
    prompt = client.get_deployed_prompt("comp_...")
    print(prompt["prompt"])

Tool Optimization

Distinct from prompt optimization, Promptic also optimizes the tool descriptions an LLM chooses between so the model picks the right tool for a query (task type toolSelection). It's a separate optimizer: the input is a set of tool definitions and representative queries, and each iteration returns optimized descriptions in toolDescriptions plus selectionSystemPrompt when system-prompt optimization is enabled.

Tool definitions come either from an MCP server URL (with Bearer-token or OAuth 2.0 auth) that Promptic auto-discovers, or from a JSON array pasted directly into the wizard (Anthropic input_schema, OpenAI {type, function}, or plain {name, description} shapes are all normalized). A tool-optimization experiment can also attach an optional systemPrompt (used as fixed context during evaluation); when the "Also optimize the system prompt" toggle is on, the optimizer rewrites that system prompt alongside the tool descriptions. Read both results from the best iteration as selectionSystemPrompt and toolDescriptions.

Create one programmatically with create_tool_selection_experiment(ai_component_id, *, tools, test_cases, target_model=None, tool_source="manual", system_prompt=None, optimize_system_prompt=False, epochs=None, train_split_ratio=None, name=None, description=None). It atomically creates the experiment, its managed dataset, the tool definitions and canonical cases, the system-prompt settings, and the required toolSelection evaluator as one pending experiment — call start_experiment(...) to run it. tools is a list of {"name", "description", "input_schema"?} and test_cases a list of {"query", "expected_tool"} (use "", or a supported no-tool alias, as expected_tool for queries that should call no tool). tool_source is "manual" (definitions supplied directly) or "mcp". This is a dedicated method, so toolSelection is not a task_type you pass to create_experiment(...) and its evaluator is not created via create_evaluators(...). Auto-discovering tools from an MCP server URL is a dashboard flow; the SDK takes the tool definitions directly.

Existing tool-optimization experiments, canonical dataset cases, evaluators, and iterations come back through the normal SDK methods. Use get_iteration(...) or get_best_iteration(...) to retrieve the optimized toolDescriptions and optional selectionSystemPrompt.

CLI

The promptic CLI mirrors the API client. All commands support --json for JSON output.

# Auth
promptic login                      # Browser auth (device flow)
promptic logout                     # Clear saved credentials
promptic configure                  # Save API key & endpoint (CI/CD)

# Agent Optimization
promptic agent-gym status <benchmark-id>
promptic agent-gym dataset-pull <benchmark-id> -o ./benchmark-inputs
promptic agent-gym apply agent.json
promptic agent-gym run <benchmark-id> my_agent:run \
  --name my-agent --version 1.0.0 --architecture architecture.md
promptic agent-gym results <benchmark-id> <run-id>
promptic agent-gym compare-runs <benchmark-id> <baseline-run-id> <candidate-run-id>
promptic agent-gym reevaluate <benchmark-id> <run-id>

# AI Application
promptic ai-application info         # Show current AI Application details
promptic ai-application list         # List accessible AI Applications
promptic ai-application select <id>  # Select active AI Application

# Traces
promptic traces list                # List recent traces
promptic traces get <trace-id>      # Get trace with spans and events
promptic traces artifacts <trace-id> # List trace artifacts
promptic artifacts get <artifact-id> -o file.bin # Download artifact bytes
promptic traces stats               # Aggregated tracing stats

# Components
promptic components list            # List AI components
promptic components create <name>   # Create a component
promptic components get <id>        # Get component details
promptic components delete <id>     # Delete a component

# Experiments
promptic experiments list           # List experiments
promptic experiments create         # Create experiment (interactive wizard)
promptic experiments create-tool-selection --component-id <id> --tools tools.json --test-cases cases.json [--start]
promptic experiments get <id>       # Get experiment details
promptic experiments update <id>    # Update a pending experiment
promptic experiments delete <id>    # Delete an experiment
promptic experiments start <id>     # Start optimization
promptic experiments duplicate <id> [--start] [-p PROMPT]    # Clone experiment (dataset cases + evaluators)
promptic experiments continue <id> [--start]                 # Clone, seed initial prompt from source's best iteration

# Evaluators
promptic evaluators list <exp-id>                # List evaluators
promptic evaluators add <exp-id> -n <name> -t <type>  # Add evaluator
promptic evaluators delete <exp-id> <eval-id>    # Delete an evaluator

# Iterations
promptic iterations list <exp-id>   # List iterations
promptic iterations get <exp-id> <iter-id>  # Get iteration with scores
promptic iterations best <exp-id>   # Get best iteration, including tool-selection outputs

# Deployments
promptic deployments status <comp-id>            # Show active deployment
promptic deployments deploy <comp-id> <exp-id>   # Deploy experiment
promptic deployments prompt <comp-id>            # Show deployed prompt
promptic deployments undeploy <comp-id>          # Remove deployment

# Datasets
promptic datasets create --component <id> --name <n>  # Create dataset
promptic datasets list --component <id>          # List datasets
promptic datasets get <ds-id> --component <id>   # Get dataset with its cases
promptic datasets delete <ds-id> --component <id>  # Delete dataset
# Individual dataset cases (training/eval data) are managed through the Python
# client (create_dataset_cases / update_dataset_case / delete_dataset_case) or
# the dataset-case REST endpoints, not a dedicated CLI command.

Key Types

Enums (Literal types):

• ExperimentStatus: "pending" | "scheduled" | "running" | "completed" | "failed"
• ModelProvider: "openai" | "openrouter" | "custom" | "google"
• TaskType: "classification" | "textGeneration" | "structuredOutput" | "toolSelection" — "toolSelection" experiments are created with the dedicated create_tool_selection_experiment(...) method, not by passing a task_type to create_experiment(...); the value is also surfaced by get_experiment(...) / list_experiments(...) for existing tool-selection / MCP-optimization experiments.
• EvaluatorType: "f1" | "referenceJudge" | "comparisonJudge" | "generalJudge" | "similarity" | "structuredOutput" | "toolSelection" — the toolSelection evaluator is attached automatically by create_tool_selection_experiment(...); it is not a value to pass into create_evaluators(...), but it is surfaced by list_evaluators(...) on a tool-selection experiment.
• OptimizerType: "promptic" | "prompticV2" | "miproV2" | "bootstrapFewShot" | "gepa" — "promptic" is the legacy v1 value retained for historical experiments; use "prompticV2" for new ones.