Osdu Quality

OSDU Quality Skill

Analyze test reliability across OSDU platform CI/CD pipelines using the osdu-quality CLI.

For installation, authentication, output formats, and common troubleshooting, see the shared CLI reference.

Quick Start

Before first use, verify the tool is available:

osdu-quality --version

If the command is not found, stop and use the setup skill to install missing dependencies. Do NOT attempt to install tools yourself — the setup skill handles installation with the correct sources, user approval, and verification.

If installed, skip exploration and go straight to the intent detection table below.

When to Use This Skill

• Test pass rates or reliability metrics
• Flaky or intermittent test failures
• Test status by stage (unit, integration, acceptance)
• Provider-specific test results (Azure, AWS, GCP, IBM, CIMPL)
• Test regressions or quality trends
• Acceptance test parity — what CSP integration tests cover that cimpl doesn't
• Test coverage gaps between providers

Do NOT use for:

• Open MRs, pipeline status, or issues → use osdu-activity
• Contributor rankings or review patterns → use osdu-engagement
• Single-repo GitLab operations → use glab directly
• Running acceptance tests locally against a live environment → use acceptance-test

Intent Detection

Commands

For full CLI options and JSON output structures, see reference/commands.md.

Output Handling

Always pass --output markdown for AI consumption. It produces token-optimized output that's easy to summarize. Use --output json only when you need raw data for field extraction. Never omit the flag — the default is tty which outputs ANSI codes that break parsing.

Interpreting Results

Pass Rate Thresholds:

By Stage:

• Unit tests should be near 100% — failures here indicate real bugs
• Integration tests typically 70-90% — some flakiness expected
• Acceptance tests are most variable — environment-dependent

Flaky Test Signals: Tests that alternate pass/fail across runs, high variance in pass rates, or inconsistent results across providers all indicate flakiness.

Workflow Examples

"How healthy are partition tests?"

osdu-quality analyze --project partition --output markdown

Summarize: lead with the headline ("Partition unit tests are solid at 100%, but acceptance is struggling at 60%"), show the stage breakdown, identify patterns, offer to dig into specific failures.

"What's failing on Azure across the platform?"

osdu-quality analyze --provider azure --output markdown

Group findings by service, highlight the worst offenders, note if failures are provider-specific or cross-provider.

"Show me the actual test failures for storage"

osdu-quality tests --project storage --output markdown

This parses job logs to show individual test class results and specific failure messages.

Acceptance Test Parity Analysis

Compare what CSP integration tests cover vs what cimpl acceptance tests cover. This identifies coverage gaps — test classes that run in Azure/AWS/GCP CI but have no equivalent in cimpl.

When to Use

• "what acceptance test gaps does partition have?"
• "how does cimpl test coverage compare to azure for storage?"
• "what's missing from cimpl acceptance tests?"
• "generate a parity report for partition"
• "what would it take to bring cimpl in line with azure integration tests?"

Method

Step 1: Fetch test data

osdu-quality tests --project <service> --output json

Step 2: Extract test classes by provider and stage

From the JSON output, group test classes from stages.acceptance and stages.integration by cloud_provider. Each job contains a test_classes array with short_name and tests_run.

Step 3: Compute the parity gap

Compare class names:

• cimpl classes: all short_name values from jobs where cloud_provider == "cimpl"
• CSP classes: all short_name values from jobs where cloud_provider is the comparison target (typically azure)
• Gap: classes present in CSP but absent from cimpl
• Overlap: classes present in both

Step 4: Present the report

## Acceptance Test Parity: <service>

### Current Coverage
| Provider | Stage | Tests | Classes |
|----------|-------|-------|---------|
| cimpl | acceptance | 11 | 4 |
| azure | integration | 83 | 8 |

### Overlap (tested by both)
- GetPartitionByIdApiTest (cimpl: 4, azure: 6)
- HealthCheckApiTest (cimpl: 2, azure: 2)
- ...

### Gap (CSP has, cimpl missing)
| Test Class | CSP Tests | What It Covers |
|------------|-----------|----------------|
| TestCreatePartition | 8 | CRUD create operations |
| TestDeletePartition | 8 | CRUD delete operations |
| TestUpdatePartition | 8 | CRUD update operations |
| ...

### Implementation Plan
To close the gap:
1. The missing test classes exist in `testing/<service>-test-azure/`
2. They can be run locally against a cimpl environment using the `acceptance-test` skill
3. To add them to CI, the `cimpl-acceptance-test` pipeline job needs to run the
   `testing/<service>-test-azure/` module with Keycloak OIDC auth instead of Azure SP

Key Context

• cimpl acceptance tests run the standalone <service>-acceptance-test/ module (smoke-level)
• CSP integration tests (especially azure) run testing/<service>-test-azure/ which includes deeper CRUD coverage
• The gap exists because cimpl CI uses the standalone acceptance module, not the provider-specific test module
• The acceptance-test skill can run the deeper tests locally against a cimpl environment right now
• Closing the CI gap requires updating the cimpl pipeline job to use the provider test module with Keycloak auth

Multiple Services

When asked about parity across multiple services, run osdu-quality tests for each and present a summary table:

| Service | cimpl Tests | Azure Tests | Gap Classes | Coverage % |
|---------|-------------|-------------|-------------|------------|
| partition | 11 | 83 | 8 | 13% |
| storage | 154 | 280 | 5 | 55% |
| legal | 120 | 0 | 0 | — |

Coverage % = cimpl tests / max(cimpl, azure) tests.

Cross-Domain Queries

For comprehensive project health, combine with sibling skills:

# Full health picture for a service
osdu-quality analyze --project partition --output markdown   # test reliability
osdu-activity mr --project partition --output markdown       # open MRs
osdu-activity pipeline --project partition --output markdown  # pipeline status

Synthesize into a unified health assessment.

Reference Documentation

• reference/commands.md — Full CLI options, JSON output structures
• reference/troubleshooting.md — Quality-specific issues
• ../../reference/osdu-cli-reference.md — Installation, auth, output formats, common errors