Website Agent Readiness

Website agent readiness

Things that make a site legible to AI agents and crawlers, plus well-known URIs. Curated from the Website Specification — Agent Readiness (CC BY 4.0).

Required

• Stable URLs. Once published, they should keep working — breaking them invalidates citations, bookmarks, and agent caches.

Recommended

• /llms.txt — a curated markdown index of your most important content.
• Per-page Markdown source endpoints (.md suffix or content negotiation) so agents pull source instead of parsing HTML.
• JSON-LD structured data with schema.org types, giving agents typed facts.
• Machine-readable formats (JSON, RSS, markdown) alongside HTML where it makes sense.
• Explicit allow/disallow per named AI crawler user-agent in robots.txt.
• HTTP Link headers advertising llms.txt, sitemap, api-catalog, RSS.
• Agent Skills discovery via a well-known URI.

Well-known URIs

• /.well-known/ is the standardised place for site-level metadata (RFC 8615).
• /.well-known/security.txt for vulnerability disclosure (see the security skill).
• /.well-known/change-password only if the site has user accounts.

Full item list and rationale: https://specification.website/checklist/