# Github Project Audit

> Deep evaluation of a GitHub repository — structure, security, licensing, maintenance signals, and hype-vs-substance analysis.

- Skill: `jahfaliabdulrahman-dev/github-project-audit` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add jahfaliabdulrahman-dev/github-project-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jahfaliabdulrahman-dev/github-project-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: jahfaliabdulrahman-dev (https://skillmd.com/u/jahfaliabdulrahman-dev)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jahfaliabdulrahman-dev/github-project-audit

---


# GitHub Project Audit

Use when the user asks to evaluate, review, or analyze a GitHub repository in depth.

## Methodology

Conduct a multi-layered investigation in parallel where possible:

### Layer 1 — Verified Stats (GitHub API)
```python
# Use execute_code to fetch and parse repo metadata
from hermes_tools import web_extract
result = web_extract(["https://api.github.com/repos/OWNER/REPO"])
```
Report: Stars, Forks, Language, Description, Default branch, Created, Pushed, Size, Open issues, License.

### Layer 2 — Contributors & Bus Factor
Fetch contributors via web_extract on `https://api.github.com/repos/OWNER/REPO/contributors?per_page=10`. Report top contributors, concentration risk, and bus factor.

### Layer 3 — Commit Activity & Velocity
Fetch commits via web_extract on `https://api.github.com/repos/OWNER/REPO/commits?per_page=15`. Check: Is it still active? How many commits/day? Single maintainer or team?

### Layer 4 — Documentation Analysis
Read these files if they exist:
- `README.md` — what they claim to be
- `LICENSE` — verify actual license file (not just Cargo.toml/package.json metadata)
- `SECURITY.md` — responsible disclosure process
- `ROADMAP.md` — future direction
- `CONTRIBUTING.md` — community governance

**Critical:** Compare declared license in config files vs actual LICENSE file. GitHub API returns `license: null` when no LICENSE file exists even if package metadata claims MIT/Apache.

### Layer 5 — Source Code Security Audit

#### 5a. Cross-Language Red Flag Patterns (check all repos)
Use GitHub code search via web_search or web_extract to scan for dangerous patterns:
- **Shell execution:** `exec.Command`, `Runtime.exec`, `subprocess`, `child_process`, `Process()`, `NSTask`
- **Network calls:** `http.Get`, `http.Post`, `URLSession`, `fetch`, `axios`, `net.Dial`
- **Obfuscation:** `base64`, `eval`, `exec`, `atob`, dynamic code generation

#### 5b. Dependency Audit
- **Go:** Review `go.mod` — look for unknown modules, excessive deps, or suspicious packages
- **Swift:** Review `Package.swift` dependencies — verify all git URLs are legitimate repos
- **Node:** Check `package.json` + `package-lock.json` for supply chain risks
- **Python:** Check `requirements.txt`/`pyproject.toml` for pinned versions and known vulns

#### 5c. Data Flow Analysis
Trace where data goes:
- **Inbound:** What permissions/API keys does the tool require?
- **Storage:** Where does it persist data? (local DB, files, cloud?)
- **Outbound:** What servers does it contact? Are they user-initiated or background?
- **Exfiltration risk:** Does any code send user data to unexpected endpoints?

#### 5d. Language-Specific Checks
- **Rust:** `unsafe_code` policy in Cargo.toml, TLS choice (rustls vs openssl), input sanitization in exec/shell paths
- **Go:** `exec.Command` usage, HTTP server exposure, CGO usage (`go-sqlite3` = native code)
- **Python:** dependency pinning, sandboxing, subprocess usage
- **Node:** eval usage, dependency audit, env variable handling
- **Swift:** `Process()` (NSTask) shell execution, `URLSession` network calls, keychain access, Accessibility API usage

### Layer 6 — Issues & PRs Reality Check
Fetch issues via web_extract on `https://api.github.com/repos/OWNER/REPO/issues?state=open&per_page=50`. Compare "open issues" count from repo metadata vs actual issues returned. Issues-disabled repos still show a count that includes PRs.

### Layer 7 — Releases & Maturity
Fetch releases via web_extract on `https://api.github.com/repos/OWNER/REPO/releases`. No releases = proof-of-concept, not production-ready.

### Layer 8 — ML/AI Model Legitimacy Verification
When the repo claims to release an ML model (especially foundation models, fine-tuned LLMs, or financial/timeseries models), verify the weights are REAL — not just config dummies or LFS pointer stubs.

#### 8a. Hugging Face Model Audit
- Hit `https://huggingface.co/api/models/ORG/MODEL` — check: `siblings` list, `downloads`, `likes`
- Models with zero downloads + zero likes + recent creation = red flag
- **Critical pitfall:** Hugging Face API returns `size: 0` for all LFS-tracked files (like `model.safetensors`). This is normal — LFS pointers are tiny text files pointing to the real binary. Do NOT flag "0 byte model" as a red flag — download the actual file to verify.
- Download the safetensors file via `https://huggingface.co/ORG/MODEL/resolve/main/model.safetensors` and check actual byte size with `wc -c`

#### 8b. Parameter Count Verification
For safetensors models, read the raw header WITHOUT torch dependency (see `references/safetensors-inspection.md`):
```bash
curl -sL "https://huggingface.co/ORG/MODEL/resolve/main/model.safetensors" -o /tmp/model.safetensors
python3 references/safetensors-inspection.md  # or inline the script
```
- Expected: `total_params ≈ claimed_params` (±1% tolerance for embedding/bias mismatches)
- For fp32 models: `file_size_bytes ≈ claimed_params × 4` (e.g., 24.7M params ≈ 94-99 MB)
- For fp16 models: `file_size_bytes ≈ claimed_params × 2`

#### 8c. Weight Distribution Sanity Check
Trained weights should show:
- Mean near 0 (not 0.5, not 1.0 — those suggest random init or bugs)
- Std dev in 0.02-0.30 range (not 0.001 — dead weights; not 50+ — exploding)
- No tensors that are all-zeros or all-identical
- No extreme outliers (>50σ from mean)
- The output head/projection layers often have wider spread than internal layers — this is normal for trained models

Red flags for fake/dummy weights:
- All weights exactly 0 or exactly 1 — not trained
- Perfect uniform distribution — random init, not trained
- File is only config JSON with no actual tensor data (1-2 KB total)

#### 8d. Academic Paper Verification
For repos claiming a paper:
- Search arXiv: `https://arxiv.org/abs/XXXX.XXXXX` — does paper exist?
- Verify venue claims via DBLP: `https://dblp.org/search/publ/api?q=TITLE`
- Check Semantic Scholar: `https://api.semanticscholar.org/graph/v1/paper/ArXiv:XXXX.XXXXX?fields=citationCount,publicationVenue`
- Verify authors are real researchers (Google Scholar profiles, university affiliations)
- Cross-reference: does the GitHub author name match the paper's first author?

#### 8e. Training Code Reality Check
- Is there actual training/pre-training code, or only inference wrappers?
- Many academic repos release weights + inference but keep training pipeline closed — note this honestly
- Complete absence of any training code while claiming "open-source model" = partial openness

## Output Format

Structure the report as:
1. **Verified Stats** — what the numbers actually say
2. **Architecture & Structure** — code organization, tech stack
3. **Protocol/Design** — how it works internally
4. **Issues Analysis** — what kind of problems exist
5. **Security Assessment** — positives and concerns (red/yellow/green flags)
6. **License & Open Source** — is it legally usable?
7. **Maintenance Outlook** — will it be updated?
8. **Hype vs Substance** — honest verdict
9. **Risk Summary Table** — tool × risk level × key concern matrix

### Layer 9 — Business Model & Marketing Funnel Detection

When the repo is an "awesome list" or curated collection, trace the money:

- Check FUNDING.yml — does the repo accept sponsorships?
- Scan README for product banners (LaunchKit, SaaS, paid services)
- Map the org ecosystem — do they own multiple "awesome-*" repos funneling to a paid product?
- Watch for "Request private X" CTAs — paid service disguised as community feature
- Read CONTRIBUTING.md — does it actually accept external PRs, or are contributions blocked?
- AI-speak detection — files with "theatre-like darkness", "radical subtraction", "chiaroscuro" are AI-generated, not hand-curated
- Preview gap — do claimed files (preview.html) actually exist in the repo, or only on the monetized website?
- Auth routes on companion site — sign-in flows indicate a SaaS product behind the curtain

Flag pattern: "awesome list → product funnel" is a legitimate growth strategy but label it as marketing, not community altruism.

### Layer 10 — Comparison Audit

When the user asks "compare against what we have," benchmark the third-party repo against the user's equivalent toolkit:

1. Identify the user's equivalent capability (Stitch MCP, internal pipeline, etc.)
2. Load the relevant skill for the user's tool to get current capabilities
3. Build a comparison table: Nature, Creator, Output, Customization, Integration, Ownership, Business Model, Pricing, Risks
4. Render a verdict: complementary (use both), substitutive (pick one), or inferior (skip)?

**⚠️ PITFALL — Category Error in Comparison.** The most common failure: comparing form without understanding substance. Impeccable inspects rendered DOM (visual output); building a regex source-code scanner and calling it "the Flutter equivalent" is a category error. When the third-party tool operates at a different abstraction layer than your equivalent, state that honestly. Don't force-fit the comparison.

**⚠️ PITFALL — Adversarial validation before building.** After a comparison audit concludes with a "gap," the natural reflex is to build the equivalent immediately. Run a Light 2-Round adversarial consultation (see `multi-model-consultation` skill) before building. The consultation will catch whether you're building the right tool or just copying the form. Battle-tested: Flutter Design Anti-Patterns audit (2026-07-11) — consultation redirected from 31 regex patterns to 3 custom_lint rules.

Table template and real example in `references/comparison-audit-template.md`.

### Layer 11 — Dev-Tool Claims Verification (Oracle Test)

When a dev tool claims accuracy ("retrieves any function", "detects dead code"), do not take README numbers at face value — and do not rely on the tool's own eval corpus alone. Run a mini-oracle on the user's OWN code before adoption:

1. Build ground truth cheaply with the language's own tooling: Python `ast` module (~20 lines), Dart `dart analyze` + a quick AST pass, TS `ts-morph`. A few hundred symbols is enough.
2. Run the tool on the same code; compute precision/recall/F1 (tp/fp/fn against your oracle).
3. Compare against the dumb baseline the tool claims to beat (usually grep). Graph-vs-grep is a fair headline — but check the comparison isn't rigged: exact-name matching vs substring search inflates precision trivially.
4. Judge the tool's own evals for honesty: do they publish oracles, baselines, and raw tp/fp/fn? (Battle-tested 2026-08-22: code-graph-rag publishes oracle evals — 0.91 F1 graph retrieval vs 0.38 grep — that transparency was the deciding trust signal.)
5. Adopt only if it beats grep/language tooling on YOUR code by a margin that matters for the workflow. Otherwise the verdict comes free from the oracle, no live install needed.

## Pitfalls
- **Skills.sh ecosystem:** See `references/skills-ecosystem-audit.md` for the full methodology on auditing skills published on skills.sh. Key trap: short URLs like `/s/<name>` can return HTTP 200 for non-existent skills because Next.js renders a shell page. Always cross-reference the sitemap and verify GitHub source.
- High star count ≠ quality (viral drops can hit 100K+ stars in days)
- "Source-available" ≠ "open source" (no license = all rights reserved)
- Disabled issues = controlled narrative, not community-driven
- 2-3 contributors = bus factor of 1, high abandonment risk
- Compare age (days since creation) to stars for hype ratio
- GitHub API rate limits (60/hr unauthenticated) — fall back to web_search when hit
- Hardcoded API keys in source may be public demo keys — not always a real secret
- Third-party protocol libraries may violate platform ToS even if code is clean
- **"Is it real?" is the most common deep-dive intent** — prioritize Layers 1 (stats), 3 (commits), 6 (issues), and 8 (model verification). Skip security-heavy Layer 5 unless the repo is a dev tool the user might install.
- **Hugging Face API returns 0-byte sizes for LFS files** — this is normal. Download to verify real size.
- When API tools are unavailable, use terminal with curl, saving to temp files before processing
- **AI skills repos: see `references/ai-skills-ecosystem-audit.md`** — Claude/Cursor/Codex marketplace repos are predominantly markdown, not software. File-type composition is the #1 signal.
- **Single-dominant-contributor (>70% commits) = solo project** regardless of listed contributor count. Check distribution, not headline numbers.
- **Confounding variable trap (self-auditing):** When auditing your OWN skills/repos before publishing, the #1 failure is attributing success to the wrong variable. Example: a governance skill claimed constitutional scaffolding improved EPIC completion — but zero profiles ever loaded it, zero cron jobs ran, and SOUL rewrites (2→581 lines) happened simultaneously. Always verify: did the thing we CLAIM fixed it actually get USED? Timeline match? Simultaneous changes that better explain the outcome?
- **Skills ecosystem repos are predominantly markdown:** Claude/Cursor/Codex marketplace repos can have 9,500+ entries that are just SKILL.md files. File-type composition is the #1 signal. Check for actual code, scripts, and references — not just markdown wrappers. Skills.sh short URLs like `/s/<name>` can return HTTP 200 for non-existent skills (Next.js renders a shell page) — always cross-reference sitemaps.