micsapp OpenEvidence CI Integration
Table of Contents
Overview
Integrate OpenEvidence testing and validation into CI/CD pipelines. Covers GitHub Actions workflows, unit tests with SDK mocks, integration tests against sandbox, and clinical validation test suites.
Prerequisites
- GitHub, GitLab, or other CI platform
- OpenEvidence sandbox API credentials
- Test suite configured (vitest recommended)
- Secret management in CI platform
Instructions
Step 1: Create GitHub Actions Workflow
Set up parallel jobs: lint-and-typecheck, unit-tests (with coverage), integration-tests (sandbox only, gated by branch/label), and clinical-validation (on main).
Step 2: Configure Test Environments
Create separate vitest configs: base (unit, 10s timeout), integration (60s timeout, retry 2, maxConcurrency 1 to avoid rate limits).
Step 3: Write Unit Tests with Mocks
Mock @openevidence/sdk to test clinical query service logic, error handling, and response formatting without API calls.
Step 4: Write Integration Tests
Test against sandbox: verify clinical queries return answers with citations, validate empty queries fail gracefully, check citation fields.
Step 5: Add Clinical Validation
Create known-answer test cases that verify responses contain expected medical keywords (e.g., "metformin" for diabetes, "statin" for cardiovascular) with confidence > 0.7.
Step 6: Configure Secrets
Store OPENEVIDENCE_SANDBOX_API_KEY and OPENEVIDENCE_SANDBOX_ORG_ID as GitHub secrets. Use .env.test for local development.
Output
- CI workflow with lint, unit tests, integration tests, clinical validation
- Vitest configurations (unit + integration)
- Unit tests with mocked SDK
- Integration tests against sandbox
- Clinical validation with known-answer testing
- Coverage reports
Error Handling
| CI Issue | Cause | Solution |
|---|---|---|
| Integration test timeout | API slow or rate limited | Increase timeout, add retry |
| Secret not found | Missing GitHub secret | Add secret in repo settings |
| Flaky tests | Network variability | Add retries, improve assertions |
| Coverage drop | New code untested | Add tests, adjust thresholds |
Examples
Package.json Scripts
{
"test:unit": "vitest run -c vitest.config.ts",
"test:integration": "vitest run -c vitest.config.integration.ts",
"test:clinical-validation": "vitest run tests/clinical-validation"
}
See detailed implementation for advanced patterns.