Glean Observability

Glean Observability

Overview

Glean aggregates enterprise knowledge across dozens of connectors, making indexing health and search quality the two pillars of observability. Monitor connector sync status to catch stale content before users notice, track search latency to maintain sub-second responses, and measure zero-result rates to identify coverage gaps. Degraded indexing silently erodes search relevance, so proactive alerting is essential.

Key Metrics

Instrumentation

async function trackGleanSearch(query: string, client: GleanClient) {
  const start = Date.now();
  try {
    const results = await client.search({ query });
    const latency = Date.now() - start;
    metrics.histogram('glean.search.latency', latency);
    metrics.increment('glean.search.total');
    if (results.totalCount === 0) metrics.increment('glean.search.zero_results');
    return results;
  } catch (err) {
    metrics.increment('glean.search.errors', { error: err.code });
    throw err;
  }
}

Health Check Dashboard

async function gleanHealth(): Promise<Record<string, string>> {
  const connectors = await gleanAdmin.getConnectorStatus();
  const staleRatio = await gleanAdmin.getStaleContentRatio(30);
  const searchP95 = await metrics.query('glean.search.latency', 'p95', '5m');
  return {
    connectors: connectors.every(c => c.status === 'ok') ? 'healthy' : 'degraded',
    content_freshness: staleRatio < 0.1 ? 'healthy' : 'stale',
    search_latency: searchP95 < 400 ? 'healthy' : 'slow',
  };
}

Alerting Rules

const alerts = [
  { metric: 'glean.search.latency_p95', condition: '> 1000ms', window: '10m', severity: 'warning' },
  { metric: 'glean.search.zero_result_rate', condition: '> 0.10', window: '1h', severity: 'warning' },
  { metric: 'glean.indexing.sync_errors', condition: '> 3', window: '1h', severity: 'critical' },
  { metric: 'glean.indexing.doc_count_delta', condition: 'drop > 20%', window: '1d', severity: 'critical' },
];

Structured Logging

function logGleanEvent(event: string, data: Record<string, any>) {
  console.log(JSON.stringify({
    service: 'glean', event,
    connector: data.connector, doc_count: data.docCount,
    query: data.query ? data.query.substring(0, 100) : undefined,
    latency_ms: data.latency, result_count: data.resultCount,
    timestamp: new Date().toISOString(),
  }));
}

Error Handling

Prerequisites

• A data-minimization policy for logs and traces, metric owners, alert routes, retention limits, and an approved restricted evidence destination.
• Synthetic probes for availability, freshness, and both authorized and denied access outcomes.
• Correlation IDs and configuration revisions that let responders connect signals without logging query text, titles, snippets, identities, or credentials.

Instructions

1. Instrument aggregate request, error, latency, freshness, queue, and ACL-probe metrics at the connector and datasource boundary.
2. Redact event fields before export, bound payload capture, and validate that a failed redaction fails closed.
3. Alert on sustained failures, freshness regression, access-boundary difference, and evidence-pipeline failure with an owner and response runbook.
4. Test alerts with synthetic events, verify the destination/retention policy, and remove test data after the exercise.
5. Review dashboards after every connector or ACL change and retain only the minimum receipt needed for incident reconstruction.

Output

Produce an observability receipt containing dashboard revision, metric/alert IDs, synthetic test results, owner, destination, retention, correlation ID, and remediation/rollback status. Raw source content and credentials are prohibited.

Examples

dashboard=search-r11; alert=freshness-lag; probe=pass; allow=pass; deny=pass; destination=restricted-ops; retention=14d; rollback=alert-r10 is a safe alert-test result.

Resources

• Glean Developer Portal
• Indexing API
• Search API

Next Steps

See glean-incident-runbook.