research-archival

name	research-archival
description	Scrape AI research URLs, archive with frontmatter, create GitHub Issues with identity verification.
allowed-tools	Read, Bash, Grep, Glob, Edit, Write

Research Archival

Scrape AI research conversations (ChatGPT, Gemini, Claude) and web pages, archive them as markdown files with YAML frontmatter, and create cross-referenced GitHub Issues — with mandatory identity verification at every step.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

FIRST - TodoWrite Task Templates

MANDATORY: Select and load the appropriate template before any archival work.

Template A - Full Archival (scrape + save + issue)

1. Identity preflight — verify GH_ACCOUNT or resolve via curl /user
2. Scrape URL — route to Firecrawl or Jina per url-routing.md
3. Save to file — YYYY-MM-DD-{slug}-{source_type}.md with frontmatter
4. Survey labels — gh label list, reuse existing, max 3-6
5. Create GitHub Issue — use --body with heredoc or --body-file
6. Update frontmatter — add github_issue_url and github_issue_number
7. Post canonical backlink comment on Issue

Template B - Save Only (no issue)

1. Identity preflight (still required for consistency)
2. Scrape URL — route to Firecrawl or Jina per url-routing.md
3. Save to file — YYYY-MM-DD-{slug}-{source_type}.md with frontmatter

Template C - Issue Only (file already exists)

1. Identity preflight
2. Read existing file frontmatter
3. Survey labels — gh label list, reuse existing, max 3-6
4. Create GitHub Issue — use --body with heredoc or --body-file
5. Update file frontmatter with issue cross-reference
6. Post canonical backlink comment on Issue

---

Identity Preflight (MANDATORY — Step 0)

MUST execute before any gh write command. Non-negotiable.

The gh-repo-identity-guard.mjs PreToolUse hook provides a safety net, but this skill performs its own check as defense-in-depth.

Resolution Order

1. Fast-path — GH_ACCOUNT env var (set by mise per-directory)
2. Token filename — scan ~/.claude/.secrets/gh-token-* for single base match
3. API call — curl -sH "Authorization: token $GH_TOKEN" https://api.github.com/user

Verification

/usr/bin/env bash << 'IDENTITY_EOF'
# Resolve authenticated user
if [ -n "$GH_ACCOUNT" ]; then
  AUTH_USER="$GH_ACCOUNT"
  AUTH_SOURCE="GH_ACCOUNT"
else
  AUTH_USER=$(curl -sf --max-time 5 -H "Authorization: token $GH_TOKEN" \
    https://api.github.com/user 2>/dev/null | grep -o '"login":"[^"]*"' | cut -d'"' -f4)
  AUTH_SOURCE="API /user"
fi

# Resolve target repo owner
REPO_OWNER=$(git remote get-url origin 2>/dev/null | sed -n 's|.*github\.com[:/]\([^/]*\)/.*|\1|p')

echo "Authenticated as: $AUTH_USER (via $AUTH_SOURCE)"
echo "Target repo owner: $REPO_OWNER"

if [ "$AUTH_USER" != "$REPO_OWNER" ]; then
  echo ""
  echo "MISMATCH — do NOT proceed with gh write commands"
  echo "Fix: export GH_TOKEN=\$(~/.claude/tools/bin/gh-token-for-repo)"
  exit 1
fi
echo "Identity verified — safe to proceed"
IDENTITY_EOF

BLOCK if mismatch — display diagnostic and do NOT continue to any gh write operation.

---

Scraping Workflow

Route scrape requests based on URL pattern. See url-routing.md for full details.

Decision Tree

Any JS-rendered share link (chatgpt.com/share/, gemini.google.com/share/, claude.ai/artifacts/)
  → Firecrawl public API (POST https://api.firecrawl.dev/v2/scrape)
  → Use curl (not WebFetch — it summarizes instead of returning raw)

Simple static page
  → Either works. Jina Reader (https://r.jina.ai/{URL}) is one GET and fine here.

Default to Firecrawl. Measured 2026-08-13 on two chatgpt.com/share/* links, Jina returned 17% and 12% of Firecrawl's content and truncated mid-sentence; Firecrawl reached the true page footer both times. Jina also needs -H "x-timeout: 30" or it returns ~321 bytes of login chrome.

Firecrawl Scrape

No health check, no preflight, no revival. The public API needs no key and has no host to be down — handle a failed request per request rather than gating the run on a liveness probe.

/usr/bin/env bash << 'SCRAPE_EOF'
set -euo pipefail

# waitFor gives the SPA time to render; without it a share link returns the shell.
RESPONSE=$(curl -sS --max-time 180 -X POST https://api.firecrawl.dev/v2/scrape \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg u "$URL" \
        '{url: $u, formats: ["markdown"], waitFor: 8000, timeout: 60000}')")

CONTENT=$(printf '%s' "$RESPONSE" | jq -r '.data.markdown // empty')

if [ -z "$CONTENT" ]; then
  echo "ERROR: Firecrawl returned no markdown:" >&2
  printf '%s\n' "$RESPONSE" | head -c 400 >&2
  echo "Retry once, then fall back to: curl -H 'x-timeout: 30' https://r.jina.ai/${URL}" >&2
  exit 1
fi

printf '%s\n' "$CONTENT"
SCRAPE_EOF

Do not reintroduce a self-hosted Firecrawl. The littleblack deployment (ports 3002/3003, five containers) was retired 2026-08-13 and reclaimed ~18 GB. It required health checks, container restarts, and WORKER-STALLED triage that the public API makes unnecessary at this volume.

---

File Saving

Naming Convention

YYYY-MM-DD-{slug}-{source_type}.md

• slug — kebab-case summary (max 50 chars)
• source_type — from enum: chatgpt, gemini, claude, web

Default location: docs/research/ in the current project.

YAML Frontmatter

See frontmatter-schema.md for the full field contract.

---
source_url: https://chatgpt.com/share/...
source_type: chatgpt-share
scraped_at: "2026-02-09T18:30:00Z"
model_name: gpt-4o
custom_gpt_name: Cosmo
claude_code_uuid: SESSION_UUID
github_issue_url: ""
github_issue_number: ""
---

Leave github_issue_url and github_issue_number empty — update after Issue creation.

---

GitHub Issue Creation

Label Survey

Survey existing labels first — reuse preferred, create only when concept is genuinely novel.

gh label list --repo owner/repo --limit 100

Policy: Max 3-6 labels per issue. Common labels: research, ai-output, chatgpt, gemini, archival.

Create Issue

Use --body with heredoc for inline composition, or --body-file for very large content.

/usr/bin/env bash << 'ISSUE_EOF'
# Write body to temp file
cat > "/tmp/issue-body-${SLUG}.md" << 'BODY_EOF'
## Summary

Brief description of the archived research content.

## Source

- **URL**: SOURCE_URL
- **Type**: source_type
- **Model**: model_name
- **Scraped**: scraped_at

## Key Findings

- Finding 1
- Finding 2

## Archived File

`docs/research/FILENAME.md`
BODY_EOF

# Create issue
gh issue create \
  --repo owner/repo \
  --title "Research: descriptive title here" \
  --body-file "/tmp/issue-body-${SLUG}.md" \
  --label "research,ai-output"

# Clean up
rm -f "/tmp/issue-body-${SLUG}.md"
ISSUE_EOF

Update Frontmatter

After issue creation, update the archived file's frontmatter with the issue URL and number.

---

Canonical Backlink Comment

Post a comment on the Issue linking back to the archived file:

**Archived**: `docs/research/YYYY-MM-DD-slug-source_type.md`
Scraped: 2026-02-09T18:30:00Z
Source: [chatgpt-share](https://chatgpt.com/share/...)
Session: SESSION_UUID

---

Post-Change Checklist

After modifying THIS skill:

1. YAML frontmatter valid (no colons in description)
2. Trigger keywords current in description
3. All ./references/ links resolve
4. Identity preflight section remains FIRST in workflow
5. Append changes to evolution-log.md
6. Validate: uv run plugins/plugin-dev/scripts/skill-creator/quick_validate.py plugins/gh-tools/skills/research-archival
7. Validate links: bun run plugins/plugin-dev/scripts/validate-links.ts plugins/gh-tools/skills/research-archival

---

Troubleshooting

Issue	Cause	Fix
Wrong account posting	GH_TOKEN mismatch	Check mise env | grep GH_TOKEN, verify GH_ACCOUNT
Body exceeds 65536 chars	GitHub API limit	Split across issue body + first comment
Firecrawl returns no markdown	Transient API failure	Retry once, then fall back to Jina with -H "x-timeout: 30"
Scrape returns the page shell	SPA had not rendered yet	Raise waitFor (8000 → 15000) and timeout in the request body
Jina returns ~321 bytes	Missing timeout header	Add -H "x-timeout: 30" — without it Jina returns login chrome
Jina output truncated	Jina under-covers JS-heavy pages	Expected — use Firecrawl; Jina got 17%/12% coverage in the 2026-08-13 test
mise parse error	Stale .mise.toml syntax	Run mise doctor, check [hooks.enter] syntax
Identity guard blocks	Non-owner account	export GH_TOKEN=$(~/.claude/tools/bin/gh-token-for-repo)

References

• Frontmatter Schema — YAML field contract
• URL Routing — Scraper routing table
• Evolution Log — Change history

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.