Kora Telemetry Metrics

Kora Telemetry Metrics

Kora sub-skill — obey the kora-v2 meta rules on every task: R0 ground the workspace on Kora 2.0 refs before starting (framework source at tag 2.0.0.RC1 + kora-examples at migration/2.0; kora-docs is 1.x only) · R1 read this sub-skill before writing code · R2 Kora 2.0 APIs only — no Spring/Micronaut/Quarkus, no Kora 1.x APIs, no invented annotations or config keys · R3 journal any incorrect Kora usage. Add comments/Javadoc only if asked.

⚠ In Kora 2.0 component metrics are OFF by default — and the failure is silent

TelemetryConfig (telemetry/telemetry-common) sets:

So http.server.request.duration, http.client.request.duration, db.client.operation.duration and every other component metric do not exist until you switch them on per component:

httpServer            { telemetry.metrics.enabled = true }
httpServer.system     { telemetry.metrics.enabled = true }   # optional: instrument the system server itself
httpClient.petApi     { telemetry.metrics.enabled = true }
jdbc                  { telemetry.metrics.enabled = true }

Nothing warns you. The app starts, GET /metrics answers 200, and the JVM meters (jvm_*, process_*, system_*, kora_up) are all there — so the endpoint looks healthy while every component metric is missing. Any example, template or dashboard that claims to demonstrate metrics must enable them explicitly.

There is a second, independent gate: a MeterRegistry must be in the graph

Every telemetry factory computes the flag as an AND of two conditions. From DefaultHttpServerTelemetryFactory.get(...):

var metricEnabled = this.meterRegistry != null && config.metrics().enabled();

HttpServerModule injects it as @Nullable MeterRegistry, and the only thing that supplies one is MetricsModule. So:

The same meterRegistry != null && config.metrics().enabled() line appears in every Default*TelemetryFactory (HTTP client, database, Kafka, cache, gRPC, scheduling, resilient, S3, SOAP, JMS). Details and the full config-root table: metrics-config-reference.md.

Changed in 2.0 — check this before porting 1.x metrics code

The opentelemetrySpec switch is gone: grep -r opentelemetrySpec over the 2.0 source returns nothing, and no module reads a top-level metrics config path. 2.0 emits one fixed naming scheme — the one 1.x called V123 — and several meter names moved on top of that. Do not carry a 1.x metric list into a 2.0 dashboard; see metrics-reference.md.

Quick start

1. Dependencies

micrometer-module already brings PrometheusMeterRegistry; adding io.micrometer:micrometer-registry-prometheus yourself only risks a version clash. Versions come from the BOM — never pin a io.koraframework:* artifact.

Java (build.gradle, with koraVersion=2.0.0.RC1 in gradle.properties):

repositories { mavenCentral() }

dependencies {
    koraBom platform("io.koraframework:kora-bom:$koraVersion")
    annotationProcessor "io.koraframework:annotation-processors"

    implementation "io.koraframework:micrometer-module"
    implementation "io.koraframework:http-server-undertow"
    implementation "io.koraframework:config-hocon"
    implementation "io.koraframework:logging-logback"
}

Kotlin (build.gradle.kts):

repositories { mavenCentral() }

dependencies {
    implementation(platform("io.koraframework:kora-bom:${property("koraVersion")}"))
    ksp("io.koraframework:symbol-processors:${property("koraVersion")}")

    implementation("io.koraframework:micrometer-module")
    implementation("io.koraframework:http-server-undertow")
    implementation("io.koraframework:config-hocon")
    implementation("io.koraframework:logging-logback")
}

Full build wiring (configurations { koraBom … }, toolchain 25): kora-project-setup-java / kora-project-setup-kotlin.

2. Put MetricsModule on the graph

Java:

package com.example;

import io.koraframework.application.graph.KoraApplication;
import io.koraframework.common.annotation.KoraApp;
import io.koraframework.config.hocon.HoconConfigModule;
import io.koraframework.http.server.undertow.UndertowPublicHttpServerModule;
import io.koraframework.logging.logback.LogbackModule;
import io.koraframework.micrometer.module.MetricsModule;

@KoraApp
public interface Application extends
        HoconConfigModule,
        LogbackModule,
        MetricsModule,
        UndertowPublicHttpServerModule {

    static void main(String[] args) {
        KoraApplication.run(ApplicationGraph::graph);
    }
}

Kotlin:

package com.example

import io.koraframework.application.graph.KoraApplication
import io.koraframework.common.annotation.KoraApp
import io.koraframework.config.hocon.HoconConfigModule
import io.koraframework.http.server.undertow.UndertowPublicHttpServerModule
import io.koraframework.logging.logback.LogbackModule
import io.koraframework.micrometer.module.MetricsModule

@KoraApp
interface Application :
    HoconConfigModule,
    LogbackModule,
    MetricsModule,
    UndertowPublicHttpServerModule

fun main() {
    KoraApplication.run(ApplicationGraph::graph)
}

UndertowPublicHttpServerModule extends UndertowSystemHttpServerModule, so the system server — and with it /metrics, /system/liveness, /system/readiness — comes along automatically.

3. Enable metrics and expose the endpoint

httpServer {
  port = 8080                          # public business traffic
  system.port = 8085                   # system server: metrics + probes
  system.metricsPath = "/metrics"      # default; shown for clarity

  # WITHOUT THIS LINE no http.server.* metric is ever recorded
  telemetry.metrics.enabled = true
}

4. Record a custom metric

Inject MeterRegistry through the constructor and register each meter once.

Java:

package com.example;

import io.koraframework.common.annotation.Component;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;

import java.time.Duration;
import java.util.concurrent.Callable;

@Component
public final class MetricsService {

    private final Timer userCreationTimer;
    private final Counter userCreationCounter;

    public MetricsService(MeterRegistry meterRegistry) {
        this.userCreationTimer = Timer.builder("user.creation.duration")
                .description("Time taken to create users")
                .serviceLevelObjectives(
                        Duration.ofMillis(50),
                        Duration.ofMillis(100),
                        Duration.ofMillis(250),
                        Duration.ofMillis(500))
                .register(meterRegistry);
        this.userCreationCounter = Counter.builder("user.creation.total")
                .description("Total number of users created")
                .register(meterRegistry);
    }

    public <T> T recordUserCreation(Callable<T> action) throws Exception {
        var result = this.userCreationTimer.recordCallable(action);
        this.userCreationCounter.increment();
        return result;
    }
}

Custom meters need only MetricsModule — they are registered directly on the MeterRegistry and are not gated by any telemetry.metrics.enabled flag. Those flags govern Kora's own component instrumentation.

5. Verify

curl -s http://localhost:8085/metrics | head -5          # framework alive + registry bound
curl -s http://localhost:8085/metrics | grep '^kora_up'
curl -s http://localhost:8085/metrics | grep http_server_request_duration

A 200 alone proves nothing — see Diagnosing an empty /metrics.

What's in references/ and assets/

Where /metrics lives

MetricsHandler is registered by SystemHttpServerModule, which UndertowSystemHttpServerModule binds to the config path httpServer.system:

default HttpServerRequestHandler systemMetricsHttpServerRequestHandler(
        @SystemApi SystemHttpServerConfig config, ValueOf<Optional<MetricsScraper>> meterRegistry) {
    return new MetricsHandler(config, meterRegistry);
}

SystemHttpServerConfig defaults: port() = 8085, metricsPath() = "/metrics", readinessPath() = "/system/readiness", livenessPath() = "/system/liveness".

The endpoint is never on the public port, and there is no config key to move it there. Keep it that way — metrics expose internal state.

A stale 1.x key such as privateApiHttpMetricsPath is simply an unrecognised HOCON key: it is ignored without a warning, the system server falls back to 8085 + /metrics, and a service that deliberately used a custom port comes up green on the wrong one.

Diagnosing an empty /metrics

MetricsHandler.handle returns 200 in every case:

var registry = this.meterRegistry.get().orElse(null);
if (registry == null) {
    return HttpServerResponse.of(200, HttpBody.plaintext("# Metric Scraper disabled"));
}
return HttpServerResponse.of(200, HttpBodyOutput.of("text/plain", registry::scrape));

So read the body, not the status code:

MetricsModule supplies the scraper as a @DefaultComponent: prometheusMetricsScraper(MeterRegistry) returns prometheus::scrape for a PrometheusMeterRegistry and a no-op writer for any other registry — so swapping in a non-Prometheus registry without also supplying your own MetricsScraper yields an empty (but still 200) body.

Built-in metric names in 2.0

Read from the *MetricsFactory classes that register them. Dots become underscores in Prometheus, and timers gain _count / _sum / _bucket / _max.

Several of these were renamed relative to 1.x (db.client.request.duration → db.client.operation.duration, cache.duration → cache.operation.duration, messaging.receive/publish.duration → the messaging.process.* / messaging.client.* pair, s3.client.duration → rpc.client.duration), and rpc.*.requests_per_rpc / responses_per_rpc do not exist in 2.0 at all. Full tables with tag keys, plus the JVM binder list: metrics-reference.md.

The HTTP server duration timer carries no status-code tag in 2.0. Its tags are server.name, server.port, http.request.method, http.route, url.scheme, server.address, error.type. A 1.x dashboard filtering on http_response_status_code=~"5.." matches nothing — use error_type!="" instead. (The HTTP client timer does carry http.response.status_code.)

Configuration keys

TelemetryConfig.MetricsConfig declares exactly three settings, and every component inherits them:

Two components add one key each on top: jdbc.telemetry.metrics.driverMetrics (default true, binds Hikari's own pool meters) and the Kafka consumer/publisher driverMetrics (default false).

Bare numbers in slo are milliseconds (DurationConfigValueMapper maps a NumberValue with Duration.ofMillis); strings accept "250ms", "1s", "PT1S".

httpServer.telemetry.metrics {
  enabled = true
  slo = [ 5, 25, 100, 250, 500, 1000, 5000 ]
  tags { "deployment.environment" = "production" }
}

Config roots differ per component (httpServer, httpServer.system, httpClient.<name>, jdbc, cassandra, grpcServer, grpcClient.<Service>, scheduling.telemetry, resilient.telemetry, lettuce, and the annotation-chosen paths for @KafkaListener / @KafkaPublisher / @Cache) — the complete table is in metrics-config-reference.md.

Common tags on every metric

MetricsModule.prometheusMeterRegistry(All<PrometheusMeterRegistryInitializer> initializers) applies every initializer once, in graph-init order, before the JVM binders run. PrometheusMeterRegistryInitializer extends Function<PrometheusMeterRegistry, PrometheusMeterRegistry>, so it must return the registry.

Java:

package com.example;

import io.koraframework.common.annotation.Module;
import io.koraframework.micrometer.module.PrometheusMeterRegistryInitializer;

@Module
public interface CustomMetricsConfig {

    default PrometheusMeterRegistryInitializer commonTagsInit() {
        return registry -> {
            registry.config().commonTags("service", "order-service", "environment", "production");
            return registry;
        };
    }
}

Kotlin:

package com.example

import io.koraframework.common.annotation.Module
import io.koraframework.micrometer.module.PrometheusMeterRegistryInitializer

@Module
interface CustomMetricsConfig {

    fun commonTagsInit(): PrometheusMeterRegistryInitializer =
        PrometheusMeterRegistryInitializer { registry ->
            registry.config().commonTags("service", "order-service", "environment", "production")
            registry
        }
}

This is a global alternative to per-component telemetry.metrics.tags. Keep the values bounded — they multiply onto every series.

Tag cardinality

Each unique tag-value combination is one time series. Tag values must be a bounded, finite set, otherwise the registry grows without limit and the process leaks memory.

For a runtime tag with a bounded value set, cache the meter per value instead of rebuilding it:

private final ConcurrentHashMap<String, Counter> counters = new ConcurrentHashMap<>();

private Counter counter(String provider) {
    return counters.computeIfAbsent(provider, p ->
        Counter.builder("user.creation.total")
            .tag("email.provider", p)
            .register(registry));
}

Kora's own factories do exactly this — see the requestDurationCache / activeRequestsCache maps in DefaultHttpServerMetricsFactory, guarded by the source comment DO NOT ADD DYNAMIC TAGS IN BUILDER, use metric key instead of metric collision will happen.

See metrics-cardinality-reference.md.

Export model — Prometheus pull only

The registry is a PrometheusMeterRegistry scraped over HTTP. Kora exposes no configuration to swap it for a push exporter (StatsD, OTLP, …). To reach another backend, scrape /metrics with an agent (Prometheus, OpenTelemetry Collector, vmagent) and forward from there.

# prometheus.yml
scrape_configs:
  - job_name: 'order-service'
    scrape_interval: 15s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['order-service:8085']   # the SYSTEM port

Trace export is a separate concern handled by opentelemetry-tracing-exporter-{grpc,http} and does not affect the meter registry — see kora-telemetry-tracing. For structured logs see kora-telemetry-logging.

Replacing the registry itself is possible: both prometheusMeterRegistry and prometheusMetricsScraper are @DefaultComponents, so your own @Component wins. See metrics-export-reference.md.

Common pitfalls

Verification

# 1. registry bound + framework version
curl -s http://localhost:8085/metrics | grep '^kora_up'

# 2. component instrumentation actually on (make a request first)
curl -s http://localhost:8080/your/route > /dev/null
curl -s http://localhost:8085/metrics | grep -E '^http_server_(request_duration|active_requests)'

# 3. exact exported series name, before you hard-code it into a dashboard
curl -s http://localhost:8085/metrics | grep '^# TYPE http_server_request_duration'

Step 2 is the one that matters: it is the only check that distinguishes "endpoint answers" from "metrics are being recorded".