vibecoded-design-tells-analysis

# vibecoded-design-tells-analysis > Skill by [ara.so](https://ara.so) — Design Skills collection. This project provides Reddit-mined data ranking the visual, textual, and code tells that make something look AI-generated ("vibe-coded" or "slop"). It scanned 3.2M posts across 47 subreddits, tabulated what people flag as AI tells, verified the findings against real quotes, and packaged three Claude skills with standalone scanners: **unslop-ui** (websites/design), **unslop-text** (prose), and **unslop-code** (source code). ## What it does - **Data-driven tell rankings**: What design/writing/code patterns people actually complain about, ranked by frequency in on-topic discussions - **Three unslop skills**: Remove AI tells from UI, text, or code while preserving functionality - **Standalone scanners**: Python scripts that grep a codebase, score vibe-coded patterns, and gate CI on exit code - **Reproducible pipeline**: Scripts to re-run the Reddit mining, analysis, and chart generation yourself ## Installation Clone the repository: ```bash git clone https://github.com/JCarterJohnson/vibecoded-design-tells.git cd vibecoded-design-tells pip install -r requirements.txt ``` ### Install the unslop skills Each skill can be installed into Claude Desktop or uploaded to claude.ai: ```bash # UI skill (removes AI design tells) unzip skill/unslop-ui.skill -d ~/.claude/skills/ # Text skill (removes AI writing tells) unzip unslop-ai-text/skill/unslop-text.skill -d ~/.claude/skills/ # Code skill (removes AI code tells) unzip unslop-ai-code/skill/unslop-code.skill -d ~/.claude/skills/ ``` Or upload the `.skill` files directly in the claude.ai skills UI. ## Using the standalone scanners Each skill includes a Python scanner that checks your project and exits non-zero if tells are found. ### UI scanner ```bash cd skill/scripts python3 devibe_scan.py /path/to/your/website ``` Example output: ``` Scanning: /Users/dev/my-site ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 🎨 Vibe score: 42/100 Found 5 tells: • shadcn/Tailwind defaults (3 files) - src/components/Button.tsx: className="rounded-lg" - src/App.tsx: className="container mx-auto" • AI purple gradient (2 files) - styles.css: background: linear-gradient(to right, #667eea, #764ba2) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ❌ CI gate: FAIL (score < 80) ``` ### Text scanner ```bash cd unslop-ai-text/skill/scripts python3 scan_text.py /path/to/docs ``` Flags tells like "it's not just X, it's Y", em dashes, "delve", "leverage", etc. ### Code scanner ```bash cd unslop-ai-code/skill/scripts python3 scan_code.py /path/to/src ``` Flags leftover chat artifacts, placeholder comments, emoji in code, swallowed errors, etc. ## Reproducing the study Run the pipeline in order. Each script is resumable and writes outputs to the current folder. ```bash cd unslop-ai-ui # Phase 1-2: Aggregate stats python3 collect.py # Phase 3: Harvest on-topic posts (pass target count) python3 harvest.py 3000 # Phase 4: Harvest comments from canonical threads python3 harvest_comments.py # Phase 5: Analyze posts for tells python3 analyze.py # Phase 6: Analyze comments for tells (cleaner signal) python3 analyze_comments.py # Phase 7-8: Generate charts python3 make_charts.py python3 make_charts2.py ``` ### Key scripts **collect.py**: Queries Arctic Shift for per-subreddit totals and matched-by-year aggregates. No auth required. ```python import urllib.request import json url = "https://arctic-shift.photon-reddit.com/api/stats" params = {"subreddit": "webdev", "term": "AI slop"} req = urllib.request.Request(f"{url}?{urllib.parse.urlencode(params)}") with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) print(data["total_posts"]) ``` **harvest.py**: Pulls full post text for on-topic submissions. Resumes from checkpoint. ```python python3 harvest.py 5000 # Target 5000 on-topic posts ``` **analyze.py**: Detects tells via synonym lexicon, counts occurrences, writes `tell_counts.csv` and `tell_examples.md`. ```python TELLS = { "shadcn_tailwind": ["shadcn", "default tailwind", "tw-"], "ai_purple": ["purple gradient", "#667eea", "purple-blue"], # ... } ``` **analyze_comments.py**: Same as `analyze.py` but for comments (cleaner signal). Produces `comment_tell_counts.csv`. ## Key outputs All data lives in `unslop-ai-ui/`, `unslop-ai-text/`, and `unslop-ai-code/`: - **corpus.jsonl.gz**: 46,971 on-topic posts (UI study) - **comments.jsonl**: 3,033 comments from 125 canonical threads - **tell_counts.csv** / **comment_tell_counts.csv**: Ranked tells with counts and percentages - **tell_examples.md**: Verbatim quotes with permalinks - **Charts**: PNG files for growth, ranking, scale, co-occurrence, sentiment Example `comment_tell_counts.csv`: ```csv tell,count,total_comments,share shadcn_tailwind,421,3033,0.1388 ai_purple_gradient,389,3033,0.1282 gradient_hero_text,312,3033,0.1029 neon_glow,287,3033,0.0946 emoji_as_icons,245,3033,0.0808 ``` ## Common patterns ### Add the UI scanner to CI ```yaml # .github/workflows/vibe-check.yml name: Vibe Check on: [push, pull_request] jobs: scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-python@v4 - run: | pip install -r requirements.txt python3 skill/scripts/devibe_scan.py . --threshold 80 ``` ### Programmatically check a file ```python import re TELLS = { "shadcn": r"(shadcn|cn\(|class.*rounded-lg)", "ai_purple": r"(#667eea|#764ba2|purple.*gradient)", } def scan_file(path): with open(path) as f: content = f.read() found = {} for tell, pattern in TELLS.items(): if re.search(pattern, content, re.I): found[tell] = re.findall(pattern, content, re.I) return found results = scan_file("src/App.tsx") if results: print(f"Found tells: {list(results.keys())}") ``` ### Query Arctic Shift for custom terms ```python import urllib.request import json def query_arctic(subreddit, term, year=None): url = "https://arctic-shift.photon-reddit.com/api/stats" params = {"subreddit": subreddit, "term": term} if year: params["year"] = year req = urllib.request.Request(f"{url}?{urllib.parse.urlencode(params)}") with urllib.request.urlopen(req) as resp: return json.loads(resp.read()) data = query_arctic("webdev", "bento grid", year=2024) print(f"Posts mentioning 'bento grid' in r/webdev (2024): {data['total_posts']}") ``` ## Configuration The studies use hardcoded subreddit lists and keyword lexicons. To customize: **Edit the subreddit list** in `collect.py`: ```python SUBREDDITS = [ "webdev", "web_design", "SaaS", "Entrepreneur", # Add your own ] ``` **Edit the tell lexicon** in `analyze.py` or `analyze_comments.py`: ```python TELLS = { "your_tell": [ "keyword1", "keyword2", r"regex.*pattern", # prefix with r for regex ], } ``` **Adjust scanner thresholds** in `devibe_scan.py`: ```python # Line ~15 THRESHOLD = 80 # Fail CI if vibe score < 80 WEIGHTS = { "shadcn_tailwind": 15, "ai_purple_gradient": 12, # Adjust weights per tell } ``` ## Troubleshooting **"No posts found" when harvesting**: The term or subreddit may have no matches. Check `scanned_totals_by_sub.csv` to verify the subreddit was indexed by Arctic Shift. **Scanner false positives**: The lexicon uses keywords and regex. Refine patterns in `TELLS` dict. Example: exclude "purple" alone, require "purple gradient". **Resuming harvest after interrupt**: The harvest scripts checkpoint by post ID. Just re-run; they skip already-fetched posts. **Charts not generating**: Ensure `matplotlib` is installed and `tell_counts.csv` exists. Run `analyze.py` before `make_charts.py`. **Memory issues with large corpus**: The text corpus can be large. Use `corpus.jsonl.gz` (committed snapshot) or stream-process with: ```python import gzip import json with gzip.open("corpus.jsonl.gz", "rt") as f: for line in f: post = json.loads(line) # Process one at a time ``` **Arctic Shift API rate limits**: The free endpoint has no auth but may throttle. The scripts sleep 0.5s between requests. If you hit limits, increase the sleep in `harvest.py`. ## Real-world usage **Before deploying a new landing page**, run the UI scanner: ```bash python3 skill/scripts/devibe_scan.py ./public ``` If score < 80, review flagged tells and replace with deliberate choices. **In a pre-commit hook**: ```bash #!/bin/sh python3 skill/scripts/devibe_scan.py . --threshold 80 || exit 1 ``` **To audit AI-written docs**: ```bash python3 unslop-ai-text/skill/scripts/scan_text.py ./docs ``` **To clean AI-generated code**: ```bash python3 unslop-ai-code/skill/scripts/scan_code.py ./src ``` Review output, manually fix or prompt an AI to rewrite flagged sections without the tells. ## Advanced: Adding a new tell 1. Add to the lexicon in `analyze_comments.py`: ```python TELLS = { # ... "your_new_tell": [ "keyword phrase", r"regex.*pattern", ], } ``` 2. Re-run analysis: ```bash python3 analyze_comments.py python3 make_charts.py ``` 3. Verify against quote bank in `comment_tell_examples.md`. 4. Add to scanner in `skill/scripts/devibe_scan.py`: ```python PATTERNS = { # ... "your_new_tell": { "pattern": r"keyword.*pattern", "weight": 8, "label": "Your Tell Label", }, } ``` 5. Test: ```bash python3 skill/scripts/devibe_scan.py /path/to/test/project ```