Github Security Audit

GitHub Security Audit — Risk Evaluation for Repositories

Systematic audit of GitHub repositories before integrating them into a production stack or recommending them to clients. Inspired by Trail of Bits' frameworks (supply-chain-risk-auditor, insecure-defaults, sharp-edges) but adapted for an MCP-first agentic workflow.

Philosophy

Don't evaluate repos with intuition. Evaluate with data.
Don't look for reasons to adopt it. Look for reasons NOT to adopt it.
If you can't find reasons to reject it, then it can come in.

Golden rule: A repo that solves a problem your stack already covers adds NO value, regardless of its quality. Evaluate first whether there is a real need before spending time on the full analysis.

Two execution modes

STEP 0 — Need Gate (MANDATORY, before ANY analysis)

Before analyzing security, answer:

1. What problem does this repo solve? (define in one sentence)
2. Is that problem already covered by your current stack? (MCPs, skills, tools)
3. Does it bring genuinely new capability?

If the answer to (2) is YES and to (3) is NO → VERDICT: SKIP — do not continue. Inform the user with the justification and suggest the existing tool.

If there's reasonable doubt or the user insists → continue with Phase 1.

PHASE 1 — Project Health

Sources: web_fetch of the GitHub repo page, GitHub API via web_fetch.

Collect:

Scoring: Each metric 0-10, total weight = 20% of final score.

See references/health-scoring.md for the detailed scoring table.

PHASE 2 — Code Security

Sources: web_fetch of key repo files.

2.1 Static analysis (without executing code)

Review via web_fetch:

• Hardcoded secrets: search source code for API key patterns, tokens, passwords (password =, api_key =, secret, token, committed .env files)
• Known vulnerable dependencies: review package.json, requirements.txt, Cargo.toml, go.mod — cross-reference versions with known CVEs via web_search
• Excessive permissions: in GitHub Actions workflows (.github/workflows/*.yml), look for permissions: write-all, pull_request_target, secrets in fork PRs
• Dangerous dynamic execution: eval(), exec(), subprocess.call(shell=True), dangerouslySetInnerHTML, innerHTML, unsafe deserialization
• Insecure default configuration: following Trail of Bits' insecure-defaults pattern — does the repo work insecurely if you don't configure anything?

2.2 Transitive dependencies

• How many direct dependencies does it have?
• Are there dependencies with a single maintainer?
• Are there dependencies that have been compromised previously?

Scoring: 0-10, weight = 25% of final score.

See references/security-patterns.md for the complete checklist.

PHASE 3 — Supply Chain

Sources: web_search of maintainer profile, repo history.

Scoring: 0-10, weight = 20% of final score.

See references/supply-chain-signals.md for detailed analysis.

PHASE 4 — Code Quality

Sources: web_fetch of representative files.

• Test coverage: Mentioned in README/CI? What percentage?
• Type hints / static types: Does it use typing?
• API documentation: Docstrings/JSDoc/rustdoc?
• Architecture: Modular or monolithic? Separation of concerns?
• Error handling: Generic try/catch or specific handling?
• Logging: Structured logging?

Scoring: 0-10, weight = 10% of final score.

PHASE 5 — License and Legal Compatibility

Sources: web_fetch of LICENSE file.

Scoring: 0-10, weight = 10% of final score.

PHASE 6 — MCP/Plugin/Skill Attack Surface

Applies when: the repo is an MCP server, Claude Code plugin, skill, integration tool, or any software that runs with access to client data.

Inspired by Trail of Bits' agentic-actions-auditor and sharp-edges.

MCP/Plugin Checklist

• What permissions does it request? (filesystem, network, env vars, secrets)
• Does it send data to third parties? (telemetry, analytics, external APIs)
• Does it store data locally? Where? Encrypted?
• Does it have hooks that run automatically? (pre/post tool use)
• Is the MCP server's code auditable? Or is it a binary/remote service?
• Does it use eval(), exec(), or dynamic execution of user-provided code?
• Does it validate LLM inputs before executing? (prompt injection surface)
• Does it have rate limiting?
• What happens if it fails? (fail-open vs fail-secure)

Rationalizations to Reject

(Inspired by Trail of Bits' "Rationalizations to Reject")

• "It's open source, so it's secure" → NO. Open source ≠ audited.
• "It has many stars" → NO. Stars measure popularity, not security.
• "Company X uses it" → NO. Unless company X has done a public audit.
• "It's read-only" → VERIFY. Really? Doesn't it write logs, cache, temp files?
• "I'll only use it internally" → Doesn't reduce risk if it touches client data.
• "The maintainer seems trustworthy" → VERIFY with data, not gut feeling.

Scoring: 0-10, weight = 15% of final score (0% if not applicable, redistributed).

PHASE 7 — External Trust Signals

Sources: web_search.

• Has it been mentioned in security publications? (positive or negative)
• Does it have third-party audits?
• Does it appear on curated quality lists? (awesome-X, Trail of Bits skills-curated)
• Has it been reported as malicious at any point?
• Does it have a bug bounty program or SECURITY.md?

Scoring: 0-10, weight = bonus (doesn't subtract, only adds up to +5 points to total).

Final Score Calculation

SCORE = (P1 × 0.20) + (P2 × 0.25) + (P3 × 0.20) + (P4 × 0.10) + (P5 × 0.10) + (P6 × 0.15) + bonus_P7

If P6 doesn't apply:
SCORE = (P1 × 0.24) + (P2 × 0.29) + (P3 × 0.23) + (P4 × 0.12) + (P5 × 0.12) + bonus_P7

Verdict

Override rules (override numeric scoring)

• No license → automatic AVOID
• AGPL/SSPL → automatic AVOID (except internal use without SaaS)
• Hardcoded secrets in repo → automatic AVOID
• Single maintainer + last commit >1 year → maximum CAUTION
• Unsanitized dynamic execution in MCP → automatic AVOID

Deliverable

Generate an interactive HTML report with:

1. Header: Repo name, URL, date, verdict with color badge
2. Executive summary: 3-5 sentences with key findings
3. Radar chart: The 7 dimensions in a radial chart (or 6 if P6 doesn't apply)
4. Detail per phase: Table with metrics, values found, scoring
5. Critical findings: List of issues requiring attention (if any)
6. Rejected rationalizations: If detected during analysis
7. Final verdict: Score + verdict + recommended action
8. Stack comparison: What current tool covers similar functionality?

Save in the working/output directory.

Limitations

State these limitations when delivering the report:

• Does not execute code or discover unknown vulnerabilities. Not static analysis, not a pentest. Analyses what is published: source, manifests, workflows, license, history. A backdoor written to look like ordinary code will not be caught this way.
• Does not replace human review when the repo will touch sensitive data. The score prioritises where to look; it does not authorise not looking.
• The score reflects the repo on the day it was measured. Recommend re-running before important decisions.

When to Use

• Before integrating any repo/library/MCP/plugin/skill into your stack
• When a client asks about a tool's security
• When evaluating a new dependency for a project
• When the user shares a GitHub link asking whether to use it
• To audit third-party MCPs or skills before installing them

When NOT to Use

• For SEO audits of websites
• For analyzing your own code (use your dev/test workflow)
• For creating new skills
• To evaluate tools that are not open source / have no public repo

References

Load on demand:

• references/health-scoring.md — Detailed scoring table per health metric
• references/security-patterns.md — Complete security pattern checklist
• references/supply-chain-signals.md — Supply chain evaluation framework