How to evaluate a skills registry

Most registries of Agent Skills are a search box over GitHub. That is a legitimate product, and for finding a well-known skill it is enough. It becomes a problem the moment you install something into a repository you are paid to maintain, because at that point you have added instructions your agent will follow, possibly with scripts attached, and you know nothing about where they came from.

Here are the six things worth checking, roughly in order of how much trouble they save.

1. Licence

Start here, because the answer is worse than you expect.

Across the skills we index, 16.6 percent declare a licence. The remaining 83.4 percent say nothing. No permission to use, none to modify, none to redistribute. Under default copyright that is not a permissive silence, it is an absence of a grant.

This has gone almost entirely unremarked. The conversation about skills has been about prompt injection and script execution, which are real, while the boring legal exposure sat in plain sight in the frontmatter. If your company has a policy about pulling unlicensed code into the build, that policy already covers Agent Skills, and it is currently being violated at scale.

What to check: does the registry show the licence on the skill page, and can you filter by it? If it only shows the file, you are doing this yourself.

2. Provenance

A SKILL.md with no origin is an anonymous instruction file. A registry should tell you the source repository, the author, and ideally the commit the content was taken at.

The commit matters more than it sounds. Repositories get force-pushed, files get rewritten, and a skill you reviewed in March can be different content under the same URL in September. Without a pinned reference, “I reviewed this” has a shelf life you cannot see.

What to check: is there a link back to the exact source, and does the registry pin a commit or just a branch?

3. Capabilities

The useful question is not “is this safe” but “what can this do”. A skill that is pure prose cannot do much. A skill that ships a shell script, fetches a URL at runtime, or reads environment variables has a different risk profile, and that difference is mechanically detectable.

What to check: does the registry flag script execution, network access and credential reads, and does it flag them for companion files rather than only the SKILL.md? The second half is where most checking historically stopped, and it is the half that carries the executable code.

4. Review, and whether the method is published

Several registries advertise security review. Very few say what the review does.

“120 detection patterns across 10 threat categories” is a number, not a method. It tells you nothing about false-negative rate, nothing about what is out of scope, and nothing you could reproduce. The useful form is either a published method or published third-party output, because both are checkable by someone who is not the vendor.

We publish per-skill results from NVIDIA SkillSpector and Cisco AI Defense Skill Scanner at /auditors. That is deliberately third-party: our own opinion about our own index is worth less than an outside tool’s. Coverage is partial and still being backfilled across the index, which is a real limitation and the honest thing to say about it.

What to check: can you see what was run and what it found, or only a badge?

5. Duplication

About 2.09 million files named SKILL.md exist on public GitHub, and roughly 1.098 million of them are unique. Close to half the corpus is copies.

That matters to you in two ways. Search results fill up with the same skill fifteen times, and size claims stop meaning anything. A registry that has deduplicated will have a smaller number and better results, which is a bad trade for marketing and a good one for you.

What to check: search for a popular skill. Count how many near-identical results you get before something new appears.

6. Install path

A registry that gives you a copy button has solved discovery and left distribution to you. That is fine for one skill. For a bundle of twelve it produces twelve manual steps, which people either skip or get wrong.

Real install paths look like a CLI that writes to the right directory for your agent, an MCP server so the agent can fetch skills itself, a native plugin format the host already understands, or a CI action that keeps skills current in a repository.

What to check: install a bundle, not a single skill. That is where the difference shows up.

What this adds up to

Discovery is a solved problem and nobody differentiates on it, because every registry indexes the same public repositories. The remaining questions are whether anything was checked, whether you can tell where it came from, whether you are allowed to use it, and whether installing it takes one step or twelve.

Judged on that list, the current field is thin, and we do not exempt ourselves. We have the licence data and the third-party scanner results, and our audit coverage across the full index is incomplete. Another registry has better usage signal than we do. A third has better curation.

If you want the field mapped rather than the criteria, the agent skills directory landscape is the companion piece. For what a well-formed SKILL.md is supposed to contain, including the frontmatter fields that carry licence and provenance, see the SKILL.md format reference. For the capability model in more detail, see Agent Skills security.