API Docs Quality Report

API Docs Audit Skill

Crawls every API reference endpoint page on a documentation site, scores each against 5 quality checks, detects site-wide patterns, and outputs an interactive HTML report matching the style of previously-generated audits in this session.

STEP 0 — COLLECT INPUT

You need exactly one input: the docs URL.

If the user hasn't provided it, ask:

Which API docs URL should I audit?

If the URL is provided, strip trailing slashes and confirm it before proceeding.

Special case — "do the same for X": If the user says "do the same" or "same audit for X", they want the identical workflow and HTML report format applied to the new URL. Proceed directly to Step 1.

STEP 1 — DISCOVER ALL ENDPOINT PAGES

1a. Try llms.txt first

Fetch <docs_url>/llms.txt.

If found:

• Extract every line matching the pattern - [Page Title](URL): description
• Filter to API reference pages only — look for paths containing any of: api-reference, api-reference/, /api/, /endpoints/, /rest/, /docs/, /reference/ or versioned paths (e.g., /v1/)
• Exclude: overview/index pages (no HTTP method implied), SDK docs, CLI docs, Terraform docs, guides, quickstart pages
• Store as ENDPOINT_PAGES[] — list of {title, url, description}

If not found or returns 404:

• Fetch the docs homepage
• Look for a nav sidebar or sitemap link referencing API/endpoints
• Extract all API reference page URLs from the nav structure
• Note in the report that llms.txt was unavailable

1b. Identify categories

Group discovered pages by their URL path prefix or nav section:

• e.g. /api-reference/agents/ → Agents category
• e.g. /api-reference/executions/ → Executions category
• Pages at the root level of api-reference → Uncategorized or infer from title

Store CATEGORIES[] — list of {name, page_count, pages[]}.

Report discovery summary:

Found N endpoint pages across K categories. Fetching now...

STEP 2 — CHECK OPENAPI SPEC AVAILABILITY

Before fetching individual pages, check if a standalone OpenAPI spec exists.

Try these discovery methods in order (stop at first success):

1. Inspect the Link HTTP header (RFC 8288) on the homepage or docs root for rel="openapi" or rel="describedby" — this is the standard discovery mechanism for API specifications
2. <docs_url>/openapi.json
3. <docs_url>/openapi.yaml
4. <docs_url>/api/openapi.json
5. Any URL referenced in llms.txt under "## OpenAPI Specs"

Result:

• If accessible: SPEC_AVAILABLE = true, store the URL
• If 404 on all attempts and no Link header found: SPEC_AVAILABLE = false

A missing spec is itself a top-priority finding — record it as Issue #1.

STEP 3 — FETCH AND PARSE ALL ENDPOINT PAGES

Fetch each URL in ENDPOINT_PAGES[]. For each page, extract:

3a. Page-level prose description

The text that appears above the OpenAPI block (or between the # title and the first ## OpenAPI heading). This is the human-written description.

3b. OpenAPI block

Look for a fenced code block with language hint yaml or json that contains an OpenAPI spec. Extract:

• HTTP_METHOD — e.g. GET, POST, PUT, DELETE, PATCH
• HTTP_PATH — e.g. /api/v1/agents/{agent_id}/execute
• REQUEST_BODY_SCHEMA — the requestBody schema section (if any)
• PARAMETERS — path + query params (if any)
• RESPONSES — the full responses object
• RESPONSE_200_SCHEMA — the schema under responses.200 or 201

If no OpenAPI block found, mark SPEC_INLINE = false.

STEP 4 — SCORE EACH ENDPOINT (5 CHECKS)

For every endpoint, evaluate all 5 checks. Store results as {check_name: "pass"|"warn"|"fail", current_state: string, fix_guidance: string}.

Read references/scoring-rules.md for the exact scoring criteria per check.

Check 1 — Endpoint Description Check 2 — OpenAPI Spec Check 3 — Body Param Descriptions Check 4 — Response Codes Check 5 — Response Schema

STEP 5 — DETECT SITE-WIDE PATTERNS

After scoring all endpoints, look for patterns that repeat across many pages.

A pattern qualifies if it affects ≥ 60% of endpoints:

For each detected pattern, write a single paragraph summarising:

• What the pattern is
• How many endpoints it affects
• Root cause (e.g. FastAPI auto-generation, missing middleware docs)
• Impact on developer experience

Store as SITE_WIDE_PATTERNS[].

STEP 6 — COMPILE TOP ISSUES

Rank the 10 most impactful issues across all findings. Prioritise by:

1. Issues affecting the most endpoints
2. Issues on the highest-traffic / most important endpoints
3. Security-sensitive gaps (auth endpoints, credential endpoints with empty schemas)
4. Deprecated endpoints with no migration path
5. Spec availability (404 canonical spec blocks all SDK tooling)

For each issue, write:

• A concise <strong>title</strong>
• 1-2 sentence description with specific endpoint names or counts
• Badge: FAIL or WARN

Store as TOP_ISSUES[] (max 10).

STEP 7 — BUILD SCORECARD

For each category, calculate:

• total — endpoint count
• pass_count — endpoints with all 5 checks passing
• warn_count — endpoints with at least one WARN and no FAIL
• fail_count — endpoints with at least one FAIL

Calculate overall totals.

Store as SCORECARD[].

STEP 8 — GENERATE HTML REPORT

Read references/html-template.md for the complete HTML structure, CSS variables, JavaScript, and all section markup.

Substitute all {{VARIABLE}} placeholders with real audit data, ensuring all values are HTML-escaped to prevent XSS.

Key variable map:

For section HTML generation rules, see references/html-template.md → Section: SECTIONS_HTML.

Output the file to:

/mnt/user-data/outputs/<slug>-api-audit-report.html

Where <slug> is the domain name with dots replaced by hyphens (e.g. docs.kubiya.ai → kubiya-ai).

Then call present_files to deliver it.

ERROR HANDLING

WHAT NOT TO DO

• ❌ Never start generating HTML before all pages are fetched and scored
• ❌ Never skip an endpoint page — fetch every one listed in llms.txt
• ❌ Never infer scores from page titles alone — fetch the actual page content
• ❌ Never mark Response Codes as PASS just because 200 is documented
• ❌ Never mark Response Schema as PASS for schema: {} — that is FAIL
• ❌ Never mark Body Params as PASS for additionalProperties: true — that is WARN at minimum
• ❌ Never report a pattern as site-wide unless it affects ≥ 60% of endpoints
• ❌ Never put generic fix text — every fix_guidance must reference the specific endpoint

REFERENCE FILES

• references/scoring-rules.md — Exact PASS/WARN/FAIL criteria for all 5 checks, with worked examples. Read before scoring any endpoint.

• references/html-template.md — Complete HTML/CSS/JS template with all section markup, variable map, and rendering instructions. Read before writing any HTML.