Analytics Instrumentation

Analytics Instrumentation

Event Naming

All events MUST be prefixed with opik_. Segment routes opik_* events to PostHog. The tooling enforces this automatically, but event names defined in code should already include the prefix.

Examples: opik_onboarding_agent_name_submitted, opik_eval_suite_created, opik_optimization_created

Frontend Events

Files

• Tracking utility: apps/opik-frontend/src/lib/analytics/tracking.ts (mode-agnostic; safe to import from any project code)
• Segment init: apps/opik-frontend/src/plugins/comet/analytics/index.ts (comet-only)
• Plugin init: apps/opik-frontend/src/plugins/comet/init.tsx (comet-only)

Adding a new event

1. Add the event name to the OpikEvent const in tracking.ts:

export const OpikEvent = {
  ONBOARDING_AGENT_NAME_SUBMITTED: "opik_onboarding_agent_name_submitted",
} as const;

2. Call trackEvent from the component or hook where the action happens:

import { trackEvent, OpikEvent } from "@/lib/analytics/tracking";

trackEvent(OpikEvent.ONBOARDING_AGENT_NAME_SUBMITTED, {
  agent_name: agentName,
});

How it works

• trackEvent() safely no-ops when Segment isn't loaded (OSS mode)
• opik_ prefix is enforced at runtime as a safety net
• OPIK_ANALYTICS_ENVIRONMENT is injected into event properties automatically by trackEvent()
• Frontend custom events flow through Segment (same pipeline as backend): Segment → PostHog
• PostHog still handles automatic pageviews, user identification, and feature flags directly

Backend Events

Files

• Service: apps/opik-backend/src/main/java/com/comet/opik/infrastructure/bi/AnalyticsService.java
• Config: apps/opik-backend/src/main/java/com/comet/opik/infrastructure/AnalyticsConfig.java
• YAML config: apps/opik-backend/config.yml (under analytics:)

API

AnalyticsService exposes two overloads:

void trackEvent(String eventType, Map<String, String> properties);
void trackEvent(String eventType, Map<String, String> properties, String identity);

• 2-arg resolves identity from the current request scope via RequestContext.
• 3-arg takes an explicit identity — use it any time the call executes outside a request scope (reactive schedulers, background threads, event listeners).

How it works

• trackEvent() no-ops when OPIK_ANALYTICS_ENABLED is false (default).
• opik_ prefix is auto-prepended if missing — but keep the prefix in code for grep-ability.
• environment property is auto-injected from OPIK_ANALYTICS_ENVIRONMENT.
• Events flow: Backend → comet-stats → Segment → PostHog.
• AnalyticsService.sendEvent wraps the body in catch (RuntimeException) — callers must not add their own try/catch.

From a synchronous request handler

Inject and call inline. The 2-arg overload resolves identity from RequestContext.

private final @NonNull AnalyticsService analyticsService;

analyticsService.trackEvent("opik_onboarding_first_trace",
        Map.of("trace_id", traceId, "project_id", projectId));

From a reactive chain (doOnSuccess, doOnNext, etc.)

Two things are required: offload with Schedulers.boundedElastic() and pass identity explicitly.

Why offload: when identity is absent AnalyticsService.resolveIdentity() falls back to UsageReportService.getAnonymousId(), which is a synchronous JDBC read. Inside a doOnSuccess lambda that runs on the reactor event loop, that read blocks a scheduler-critical thread.

Why explicit identity: RequestContext is bound to the request thread via a Guice scope — inside the scheduler's lambda it throws ProvisionException, and you silently degrade to the anonymous-ID fallback, losing user attribution.

Capture userName up front from the reactor context alongside workspaceId, then pass both into the scheduled call:

return Mono.deferContextual(ctx -> {
    String workspaceId = ctx.get(RequestContext.WORKSPACE_ID);
    // Use getOrDefault on paths that internal/system callers reach without seeding USER_NAME
    // (e.g. a self-triggered cancellation written only with WORKSPACE_ID in the context).
    String userName = ctx.getOrDefault(RequestContext.USER_NAME, null);

    return someDao.write(...)
            .doOnSuccess(__ -> Schedulers.boundedElastic().schedule(
                    () -> analyticsService.trackEvent("opik_thing_happened",
                            Map.of(
                                    "thing_id", thing.id().toString(),
                                    "workspace_id", workspaceId),
                            userName)));
});

If you already depend on a Schedulers.boundedElastic().schedule(() -> { ... }) block that does other non-reactive work (e.g. a blocking datasetService.getById like ExperimentService.trackEvalSuiteRunIfApplicable), add the trackEvent call inside that existing lambda instead of nesting another.

Don'ts

• Don't add try/catch around trackEvent — sendEvent catches RuntimeException internally. Extra catches are noise and diverge from the codebase pattern.
• Don't add helper methods that only delegate to trackEvent — inline the call at the entry point. Wrap in a helper only when it encapsulates real logic (e.g. applicability check + enrichment + tracking).
• Don't re-fetch ClickHouse rows to get "fresh" values for analytics payloads — a write and a read-after-write can land on different replicas, so you may see a stale snapshot or even a spurious NotFound. Use the pre-write snapshot; some analytics drift is acceptable, a failed user-facing request is not.
• Don't add unit tests that verify(analyticsService)... — the codebase convention is for existing integration tests to exercise these paths organically. Sister analytics PRs (#6326 eval suite, #6333 onboarding, #6338 agent config) ship without emission assertions.
• Don't assume trackEvent is fully non-blocking — the Javadoc contract is aspirational; the identity-fallback path is synchronous JDBC today. Offload from reactive chains as shown above.

Python SDK Events

Files

sdks/python/src/opik/analytics/ — api.py (public surface), rules.py (when reporting is allowed), worker.py (background thread), comet_stats.py (the HTTP call). Config lives in sdks/python/src/opik/config.py, prefixed analytics_.

Identity and environment metadata are shared with Sentry error tracking, not reimplemented. Both live at the top level so neither subsystem depends on the other: opik/environment.py::get_user_identifier() (workspace name, falling back to a hostname/username hash) and opik/environment_details.py (collect_tags_once() / collect_context_once()). Analytics and error_tracking/before_send.py both read them, so an event carries the same user id, the same session_id and the same environment details as any error report from the same run. Add environment metadata there, not in either consumer.

API

One function, called explicitly as the first line of whatever is being reported:

from opik import analytics

analytics.track_event("client", "create_dataset")
analytics.track_event("integration", "openai")
analytics.track_event("evaluation", "metric_created", metric=name)

No decorators, by design: the payload is written out at the call site, so what gets sent is whatever you can read right there.

The positional arguments form a path, broadest first, and go as deep as an event needs:

analytics.track_event("integration", "bedrock")                  # the integration
analytics.track_event("integration", "bedrock", "invoke_agent")  # one part of it

The first element is a closed set (analytics.Component): client, evaluation, integration — extend it there rather than passing a new string. Every level after it is free-form, and the second is normally just the method being reported.

A longer path is a different event, not a repeat of the shorter one, so instrumenting part of a feature never silences the feature itself.

Names are composed by joining the path with a double underscore — opik_python_sdk__integration__bedrock__invoke_agent — in one private helper, so the scheme can be changed for every event at once without touching a call site. The separator is doubled so the name splits back into the path: segments are method names, so they contain single underscores but never a pair. A test enforces that (test_event_names.py); keep it true when adding events. Extra properties are keyword arguments; adding one never changes the API.

Adding an event

1. Pick the path. First element from the closed analytics.Component set — client, evaluation, integration. Second is normally the method being reported. Add further levels only to narrow a feature down ("integration", "bedrock", "invoke_agent"), remembering a longer path is a separate event, not a repeat of the shorter one.

2. Check the segments. No level may contain a double underscore, because that is the separator the name is joined with. Method names never do, so this is normally free — test_event_names.py fails the build if it is ever not.

3. Put the call on the first line of the user-facing function, before it does its work, so a call that goes on to fail still counts as usage. Do not wrap it in try/except and do not guard it with a config check; it already swallows everything and no-ops when reporting is off.

4. Decide the properties, if any. Keyword arguments, scalars only. 97 of the 98 events carry none — reach for one only when the event genuinely has variants worth splitting, as metric_created does. Never a value the user chose: report the Opik-owned name and "custom" otherwise.

5. Check it should be reported at all. Skip it if Opik calls the same entry point internally (litellm's track_completion), or if it is a per-call hot path (Opik.trace(), Opik.span(), an OTel on_start) — instrument the constructor or the user-facing function instead.

6. Verify it locally. Intercepting the HTTP call is the quickest way to see the exact payload without sending anything:

   import json, os, unittest.mock
   import httpx
   
   os.environ["OPIK_ANALYTICS_ENABLE"] = "true"
   
   sent = []
   
   def fake_post(self, url, **kwargs):
       sent.append(kwargs["json"])
       return type("R", (), {"status_code": 201})()
   
   # Patched for the block only. Replacing `post` outright leaves every later
   # request in the process - the SDK's own included - talking to the stub.
   with unittest.mock.patch.object(httpx.Client, "post", fake_post):
       from opik import analytics
   
       ...                              # exercise your new call site
       analytics.flush(timeout=10)      # batched; nothing appears without this
   
   print(json.dumps(sent, indent=2))
   

   Reporting is off under pytest, so this has to be a plain script, not a test. Then run the suite from the SDK directory, where it lives:

   cd sdks/python && pytest tests/unit/analytics    # 67 tests
   

7. Know how it will be read. The event surfaces on the Python SDK Usage dashboard, where every tile counts uniq(distinct_id). A new event needs no dashboard change to appear in the adoption tiles, which group on the name.

How it works

• track_event() never raises, never blocks on I/O, and no-ops when reporting is off.
• Calls Opik makes into its own API are not reported. evaluate_threads calls search_threads, get_or_create_dataset calls get_dataset, the CLI calls both — 35 of the 69 instrumented Opik methods are reachable this way. An event is dropped when either the reporting function was reached from a different Opik module, or some function further up the stack is already reporting. Being called from the reporter's own module is not enough on its own, so a private helper reporting on its caller's behalf (as BaseMetric does) still works. Internal calls record nothing, so the user's own call to the same API still reports.
• Nothing needs decorating for that to hold. Reporting functions are recognised by code object the first time they report, so a new track_event call site joins in automatically.
• Counting is safe across threads and forks. Claiming an event is done under a lock (a check-then-add lets every racing thread report a copy), and the worker is rebuilt after fork() — with _ALREADY_REPORTED deliberately inherited, so a child reports its own events but not the parent's. Separate processes cannot share that state, so a spawn pool reports one copy per worker: count uniq(anonymous_id), never raw event volume.
• Events go to Comet's stats collector from a single background thread (analytics/worker.py), and on through Segment to PostHog — the same route the backend reports through. The collector takes no credentials, so there is no write key to configure; OPIK_ANALYTICS_URL alone points it somewhere else.
• Each event is reported once per process. Analytics answers "how many users use this feature", not "how often", so Opik.span() in a hot loop costs one event and a set lookup. Events differing in their properties count as different events, so one name still covers variants (each metric class, say).
• OPIK_ANALYTICS_ENABLE=false is the only way to switch reporting off. Reporting is also skipped under pytest, which is not a user-facing switch but the thing keeping test suites from making network calls. Add another process-level veto with analytics.register_rule(lambda config: ...) before the first tracked event.
• Config is read once, on the first tracked event - not at import time - so opik.configure(...) is taken into account.

Don'ts

• Don't pass user data as properties. Whatever a call site passes is what gets sent — there is no scrubbing layer. Properties are typed as scalars (str | int | float | bool | None); pick each key deliberately. Counts, flags and library names only.
• Don't report a user-defined class or function name. Report the Opik-owned name and "custom" otherwise (see _track_metric_creation in evaluation/metrics/base_metric.py).
• Don't add try/except around track_event - it already swallows everything.
• Don't instrument an entry point that Opik itself calls internally. The litellm track_completion case is why: LiteLLMChatModel calls it, so the event would measure Opik's own behaviour rather than the user's.
• Don't instrument per-callback hot paths (e.g. an OTel on_start). Instrument the constructor or the user-facing function instead. Opik.trace() and Opik.span() are deliberately uninstrumented for the same reason - the backend already sees them.

Environment Variables

Backend analytics is disabled by default; the Python SDK's is opt-out. OSS installations are unaffected on the backend.

Event Flow

Frontend custom events:  Browser → Segment → PostHog
Backend events:          Java → comet-stats → Segment → PostHog
Python SDK events:       Python → comet-stats → Segment → PostHog (background thread)
PostHog native:          Browser → posthog-js → PostHog (pageviews, feature flags, identification)

Event Property Conventions

• Consistent typing per property: A given property key should always carry the same kind of value. Don't pass a UUID in one code path and a human-readable name in another for the same key.
• Separate ID and name properties: When both a UUID and a display name exist, use distinct keys (e.g. blueprint_id for the UUID, blueprint_name for the display name). If one is unavailable in a code path, omit the key or send an empty string — don't repurpose the other key.
• Include workspace_id: All backend analytics events should include the workspace ID for segmentation.

Deciding Frontend vs Backend vs Python SDK

• Frontend: UI interactions (button clicks, wizard steps, form submissions, page visits)
• Backend: SDK-triggered actions (trace creation, test suite runs), server-side computations, events that happen without the user being on the page
• Python SDK: which SDK APIs, integrations and metrics users reach for, and in which environment (Python version, OS, cloud vs self-hosted vs local). Use it when the backend cannot see the difference - e.g. track_openai vs track_anthropic both produce ordinary spans server-side.