fabioc-aloha Evaluate Before Adopting
A catalog tells you what exists and how popular its source is. It does not tell you whether the content is safe to run, whether it is correct, or whether it fits where you are putting it. Those are the three things that matter, and all three need someone to read the payload.
When to Use This
Fire it when adoption is hard to reverse or wide in blast radius:
- Bundling a skill into a runtime every project loads
- Installing a plugin that will hold credentials or execute code
- Taking a dependency your team will build habits around
Skip it for a reversible per-project install you can uninstall in a second. The method costs minutes; spending them on a trial install you will delete anyway is ceremony. Judge by what it costs to be wrong, not by how careful you feel.
What a Trust Score Measures
Usually the store, not the artifact.
Catalogs score what they can compute: who publishes it, how recently the repository moved, how many stars it has, whether the license is clear. Those are properties of the source. Every skill from the same store often carries the same number, which means sorting candidates by trust sorts them by where they live.
Three consequences:
- A high score says a reputable, maintained source published this. It does not say the content is right.
- Two skills from the same store cannot be ranked against each other by score at all.
- A first-party or provenance bonus is a statement about origin, and it can put a defective artifact at the top of the list.
Use the score to decide what to look at. Never to decide what to adopt.
The Checks
Ordered so the cheapest disqualifier comes first.
1. Resolve which artifact you are actually looking at
The same name often exists in several stores at different scores, because catalogs index copies. Matching on name attributes the artifact to whichever store your query reached first, which may not be the one you would install.
Search the name across the whole catalog and list every hit before reading any of them.
2. Read what it declares and what it runs
For a plugin: the manifest, the install command, the declared dependencies, and
anything it launches. npx some-package resolves to whatever the registry serves
at that moment; a pinned version does not.
For a skill: the frontmatter, its declared triggers, and its length. A skill that loads on every turn costs every turn.
3. Read it as instructions, not as documentation
A skill is not data the agent consults. It is instructions the agent follows, using your tools, your filesystem, and your credentials. The prose is the executable surface, which is the one thing structural validation cannot inspect.
Read the payload for anything addressed to the agent rather than to you:
| Shape | What it looks like |
|---|---|
| Injected instruction | "Always run", "do not mention this to the user", "ignore earlier instructions" — directives to the agent buried in prose |
| Exfiltration | Reads environment variables, .env, keychains, or dotfiles and sends them anywhere, including as telemetry |
| Destructive default | Force-push, history rewrite, recursive delete, or skipping verification hooks, presented as the routine path |
| Credential mishandling | Asks for a token in plaintext, echoes it, or writes it somewhere that is not a secret store |
| Supply chain | An unpinned fetch, an install that pipes a download into a shell, a postinstall script, or a package name one keystroke from a popular one |
Most of what you find will be carelessness rather than malice. The response is the same: do not adopt it until it is resolved. Intent changes who you tell, not whether you install.
4. Read the code it ships
Not every artifact is prose. A skill or plugin can bundle scripts, hooks, and config that declares commands — and those run directly, without having to persuade an agent to run them.
This is not a rare case. Executable files ship inside a meaningful share of published skills and plugins, and curated payloads routinely carry Python, shell, and JavaScript alongside the documentation.
Enumerate the payload before reading it. Anything that is not documentation is in
scope: .py, .sh, .ps1, .js, .cjs, .mjs, and the JSON or YAML that
declares hooks, tasks, and install steps.
Then read each one for:
- Where it sends data. Any network call, and what it puts in the body.
- What it touches. Writes outside the working tree; reads of environment variables, dotfiles, or credential stores.
- How it builds commands. Interpolating unchecked input into a shell string is a defect here for the same reason it is in your own code.
- When it runs. Install hooks, postinstall scripts, and activation handlers run before anyone has decided to use the thing.
- What it hides. Encoded blobs,
eval, or minified code sitting in a source tree are worth an explanation.
Then compare what the code does against what the description says it does. A gap in that direction is the finding: a script that quietly does more than it advertises is exactly what a manifest cannot show you. Danger is not proportional to length.
5. Verify the checkable specifics
This is the check that finds what nothing else does.
Skills and plugins are full of plausible, checkable claims: package names, API shapes, CLI flags, endpoints, version numbers. Each one is either true or fabricated, and fabricated ones look exactly like real ones until you look.
| Claim type | How to check |
|---|---|
| Package name | Query the registry: npm, PyPI, NuGet, crates.io |
| API or method shape | The SDK's own docs or source, not memory |
| Endpoint or URL | Fetch it |
| Version number | The project's releases |
| CLI flag | The tool's --help |
Check the ones a reader would act on first. A wrong package name means a command that fails at the first step; a wrong conceptual claim usually degrades quietly.
A namespace is not a package name. using Microsoft.Extensions.Hosting and
import numpy.linalg name code, not distributions, and the mapping between them
is not mechanical. Truncating an import to its first segments and querying that
produces confident false positives — in one trial it flagged four real namespaces
as missing while correctly catching one fabricated package. Verify the install
command, or map the namespace to its package deliberately.
Do not verify from memory. If a sample is wrong and you correct it from recall, you have replaced one unverified claim with another.
6. Follow every reference
Files it cites, skills it links, docs it points at. A reference that resolves in the source may dangle at the destination, because the destination has a different set of neighbours.
Watch for a catalog that has papered over a dangling reference with a rewrite rather than fixing it. That is a signal about how the source is maintained.
7. Judge the freshness claim
A lastReviewed date says when somebody looked. It does not say the content still
holds, and it does not say what they checked. Treat a stamp older than the
fastest-moving thing the artifact describes as unverified rather than fresh.
8. Decide fit, not just quality
A correct artifact can still be wrong for the destination. Ask:
- Does it serve what this project or runtime is for?
- Does it earn its cost? Everything loaded is paid for by everyone who loads it.
- Would it help someone on their first day here?
- Is this the right surface? Project-local, a shared runtime, or a separate install are different answers.
A yes on quality and a no on fit is a common and correct outcome. It usually means install it per project rather than bundling it.
Worked Example
A skill scored 95 — the top band, first-party, the highest-confidence candidate in a 4,266-artifact catalog. Reading it found:
- A NuGet package that returns zero results. The code sample could not compile.
- A builder API that does not exist in the SDK it named.
- A cited file absent from the payload and from the entire source repository.
- Three links to sibling skills that existed in other stores, not at the destination.
None of it was visible from the score, because the score measured the store. None of it was caught by tooling, because the packager validates structure and nothing in the pipeline compiles a sample.
It was found by querying NuGet — one command, after deciding the specifics were worth checking.
What to Do With Findings
| Finding | Response |
|---|---|
| Unsafe instruction or exfiltration path | Do not install. Report it to the catalog and the source; other adopters are exposed too. |
| Code that does more than it advertises | Do not install. The gap is the finding, whatever the extra behaviour turns out to be. |
| Fabricated specific | Fix it from the authoritative source, or remove the section. Do not carry it. |
| Dangling reference | Resolve or delete. A pointer to nothing teaches nothing. |
| Stale stamp | Re-verify the content, then restamp. A date bump is not a review. |
| Wrong fit, correct content | Install it per project instead of bundling it. |
| Several fabrications | Stop. Treat the whole artifact as unverified, not just the parts you found. |
Record what you found and what you did. The next person to consider this artifact should not have to rediscover it, and the source's maintainer may want to know.
Composes With
- anti-hallucination — the same discipline applied to claims you are generating rather than reading
- security-and-hardening — when step 3 finds something and you need the threat model behind it
- critical-thinking — for the fit question, where the answer is a judgement rather than a lookup
- risk-analysis — when the adoption decision needs a probability and impact rather than a yes or no
Would Revise If
- A catalog begins verifying content — scanning for unsafe instructions, reading bundled scripts, compiling samples, resolving references — at which point steps 3 to 6 shrink to reading its report.
- Adopters report running this on trivial installs and treating it as ceremony, which would mean the materiality gate at the top needs to be firmer.
- Three consecutive evaluations find nothing, on artifacts later found defective, which would mean the checks are aimed at the wrong things.