Vibecoded Design Tells Analysis

vibecoded-design-tells-analysis

Skill by ara.so — Design Skills collection.

This project provides Reddit-mined data ranking the visual, textual, and code tells that make something look AI-generated ("vibe-coded" or "slop"). It scanned 3.2M posts across 47 subreddits, tabulated what people flag as AI tells, verified the findings against real quotes, and packaged three Claude skills with standalone scanners: unslop-ui (websites/design), unslop-text (prose), and unslop-code (source code).

What it does

• Data-driven tell rankings: What design/writing/code patterns people actually complain about, ranked by frequency in on-topic discussions
• Three unslop skills: Remove AI tells from UI, text, or code while preserving functionality
• Standalone scanners: Python scripts that grep a codebase, score vibe-coded patterns, and gate CI on exit code
• Reproducible pipeline: Scripts to re-run the Reddit mining, analysis, and chart generation yourself

Installation

Clone the repository:

git clone https://github.com/JCarterJohnson/vibecoded-design-tells.git
cd vibecoded-design-tells
pip install -r requirements.txt

Install the unslop skills

Each skill can be installed into Claude Desktop or uploaded to claude.ai:

# UI skill (removes AI design tells)
unzip skill/unslop-ui.skill -d ~/.claude/skills/

# Text skill (removes AI writing tells)
unzip unslop-ai-text/skill/unslop-text.skill -d ~/.claude/skills/

# Code skill (removes AI code tells)
unzip unslop-ai-code/skill/unslop-code.skill -d ~/.claude/skills/

Or upload the .skill files directly in the claude.ai skills UI.

Using the standalone scanners

Each skill includes a Python scanner that checks your project and exits non-zero if tells are found.

UI scanner

cd skill/scripts
python3 devibe_scan.py /path/to/your/website

Example output:

Scanning: /Users/dev/my-site
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🎨 Vibe score: 42/100

Found 5 tells:
  • shadcn/Tailwind defaults (3 files)
    - src/components/Button.tsx: className="rounded-lg"
    - src/App.tsx: className="container mx-auto"
  
  • AI purple gradient (2 files)
    - styles.css: background: linear-gradient(to right, #667eea, #764ba2)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
❌ CI gate: FAIL (score < 80)

Text scanner

cd unslop-ai-text/skill/scripts
python3 scan_text.py /path/to/docs

Flags tells like "it's not just X, it's Y", em dashes, "delve", "leverage", etc.

Code scanner

cd unslop-ai-code/skill/scripts
python3 scan_code.py /path/to/src

Flags leftover chat artifacts, placeholder comments, emoji in code, swallowed errors, etc.

Reproducing the study

Run the pipeline in order. Each script is resumable and writes outputs to the current folder.

cd unslop-ai-ui

# Phase 1-2: Aggregate stats
python3 collect.py

# Phase 3: Harvest on-topic posts (pass target count)
python3 harvest.py 3000

# Phase 4: Harvest comments from canonical threads
python3 harvest_comments.py

# Phase 5: Analyze posts for tells
python3 analyze.py

# Phase 6: Analyze comments for tells (cleaner signal)
python3 analyze_comments.py

# Phase 7-8: Generate charts
python3 make_charts.py
python3 make_charts2.py

Key scripts

collect.py: Queries Arctic Shift for per-subreddit totals and matched-by-year aggregates. No auth required.

import urllib.request
import json

url = "https://arctic-shift.photon-reddit.com/api/stats"
params = {"subreddit": "webdev", "term": "AI slop"}
req = urllib.request.Request(f"{url}?{urllib.parse.urlencode(params)}")
with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
    print(data["total_posts"])

harvest.py: Pulls full post text for on-topic submissions. Resumes from checkpoint.

python3 harvest.py 5000  # Target 5000 on-topic posts

analyze.py: Detects tells via synonym lexicon, counts occurrences, writes tell_counts.csv and tell_examples.md.

TELLS = {
    "shadcn_tailwind": ["shadcn", "default tailwind", "tw-"],
    "ai_purple": ["purple gradient", "#667eea", "purple-blue"],
    # ...
}

analyze_comments.py: Same as analyze.py but for comments (cleaner signal). Produces comment_tell_counts.csv.

Key outputs

All data lives in unslop-ai-ui/, unslop-ai-text/, and unslop-ai-code/:

• corpus.jsonl.gz: 46,971 on-topic posts (UI study)
• comments.jsonl: 3,033 comments from 125 canonical threads
• tell_counts.csv / comment_tell_counts.csv: Ranked tells with counts and percentages
• tell_examples.md: Verbatim quotes with permalinks
• Charts: PNG files for growth, ranking, scale, co-occurrence, sentiment

Example comment_tell_counts.csv:

tell,count,total_comments,share
shadcn_tailwind,421,3033,0.1388
ai_purple_gradient,389,3033,0.1282
gradient_hero_text,312,3033,0.1029
neon_glow,287,3033,0.0946
emoji_as_icons,245,3033,0.0808

Common patterns

Add the UI scanner to CI

# .github/workflows/vibe-check.yml
name: Vibe Check
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
      - run: |
          pip install -r requirements.txt
          python3 skill/scripts/devibe_scan.py . --threshold 80

Programmatically check a file

import re

TELLS = {
    "shadcn": r"(shadcn|cn\(|class.*rounded-lg)",
    "ai_purple": r"(#667eea|#764ba2|purple.*gradient)",
}

def scan_file(path):
    with open(path) as f:
        content = f.read()
    found = {}
    for tell, pattern in TELLS.items():
        if re.search(pattern, content, re.I):
            found[tell] = re.findall(pattern, content, re.I)
    return found

results = scan_file("src/App.tsx")
if results:
    print(f"Found tells: {list(results.keys())}")

Query Arctic Shift for custom terms

import urllib.request
import json

def query_arctic(subreddit, term, year=None):
    url = "https://arctic-shift.photon-reddit.com/api/stats"
    params = {"subreddit": subreddit, "term": term}
    if year:
        params["year"] = year
    req = urllib.request.Request(f"{url}?{urllib.parse.urlencode(params)}")
    with urllib.request.urlopen(req) as resp:
        return json.loads(resp.read())

data = query_arctic("webdev", "bento grid", year=2024)
print(f"Posts mentioning 'bento grid' in r/webdev (2024): {data['total_posts']}")

Configuration

The studies use hardcoded subreddit lists and keyword lexicons. To customize:

Edit the subreddit list in collect.py:

SUBREDDITS = [
    "webdev", "web_design", "SaaS", "Entrepreneur",
    # Add your own
]

Edit the tell lexicon in analyze.py or analyze_comments.py:

TELLS = {
    "your_tell": [
        "keyword1",
        "keyword2",
        r"regex.*pattern",  # prefix with r for regex
    ],
}

Adjust scanner thresholds in devibe_scan.py:

# Line ~15
THRESHOLD = 80  # Fail CI if vibe score < 80
WEIGHTS = {
    "shadcn_tailwind": 15,
    "ai_purple_gradient": 12,
    # Adjust weights per tell
}

Troubleshooting

"No posts found" when harvesting: The term or subreddit may have no matches. Check scanned_totals_by_sub.csv to verify the subreddit was indexed by Arctic Shift.

Scanner false positives: The lexicon uses keywords and regex. Refine patterns in TELLS dict. Example: exclude "purple" alone, require "purple gradient".

Resuming harvest after interrupt: The harvest scripts checkpoint by post ID. Just re-run; they skip already-fetched posts.

Charts not generating: Ensure matplotlib is installed and tell_counts.csv exists. Run analyze.py before make_charts.py.

Memory issues with large corpus: The text corpus can be large. Use corpus.jsonl.gz (committed snapshot) or stream-process with:

import gzip
import json

with gzip.open("corpus.jsonl.gz", "rt") as f:
    for line in f:
        post = json.loads(line)
        # Process one at a time

Arctic Shift API rate limits: The free endpoint has no auth but may throttle. The scripts sleep 0.5s between requests. If you hit limits, increase the sleep in harvest.py.

Real-world usage

Before deploying a new landing page, run the UI scanner:

python3 skill/scripts/devibe_scan.py ./public

If score < 80, review flagged tells and replace with deliberate choices.

In a pre-commit hook:

#!/bin/sh
python3 skill/scripts/devibe_scan.py . --threshold 80 || exit 1

To audit AI-written docs:

python3 unslop-ai-text/skill/scripts/scan_text.py ./docs

To clean AI-generated code:

python3 unslop-ai-code/skill/scripts/scan_code.py ./src

Review output, manually fix or prompt an AI to rewrite flagged sections without the tells.

Advanced: Adding a new tell

1. Add to the lexicon in analyze_comments.py:

TELLS = {
    # ...
    "your_new_tell": [
        "keyword phrase",
        r"regex.*pattern",
    ],
}

2. Re-run analysis:

python3 analyze_comments.py
python3 make_charts.py

3. Verify against quote bank in comment_tell_examples.md.

4. Add to scanner in skill/scripts/devibe_scan.py:

PATTERNS = {
    # ...
    "your_new_tell": {
        "pattern": r"keyword.*pattern",
        "weight": 8,
        "label": "Your Tell Label",
    },
}

5. Test:

python3 skill/scripts/devibe_scan.py /path/to/test/project