yc-batch-evaluator

# YC Batch Evaluator ## Setup Read your credentials from ~/.gooseworks/credentials.json: ```bash export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])") export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))") ``` If ~/.gooseworks/credentials.json does not exist, tell the user to run: `npx gooseworks login` All endpoints use Bearer auth: `-H "Authorization: Bearer $GOOSEWORKS_API_KEY"` Scrape a YC batch, research every company and founder, assess founder-company fit, and export a live-updating Google Sheet with priority rankings. Designed for investors evaluating YC companies. ## IMPORTANT: Do NOT ask clarifying questions. Just start immediately. All inputs are optional. If the user said a batch, use it. If they didn't specify sectors or thesis, process ALL companies. **Always create a new Google Sheet** — never ask for an existing spreadsheet ID. **Start scraping immediately — do not ask "which batch?", "any sector filters?", or "should I create a sheet?".** This is designed for live demos where speed and visual impact matter. "Spring 2026" is a real YC batch (also called "X26"). It exists and has ~22 companies. ## Input - **batch** (optional) — defaults to "Spring 2026". Examples: "Winter 2026", "Summer 2025" - **sectors** (optional) — filter to specific sectors (e.g. "AI", "fintech", "infrastructure"). If not provided, process ALL companies. - **thesis** (optional) — investor's focus areas for tailored scoring and ranking. If not provided, rank on general investment quality. ## Step 1: Scrape the YC Batch Directory The YC companies page is JavaScript-rendered (Algolia-powered). Scrapegraph handles the JS rendering. ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/smartscraper"}' "website_url": "https://www.ycombinator.com/companies?batch={batch_url_encoded}", "user_prompt": "Extract every company listed: company name, one-line description, sector/tags, location, and URL slug for each company page (e.g. /companies/orthogonal). Return as a structured list." }' ``` Batch URL encoding: "Spring 2026" → `Spring%202026`, "Winter 2026" → `Winter%202026`. If the investor specified sectors, filter the list. Otherwise process all companies. **Expected response structure:** ```json { "result": { "companies": [ { "name": "Indexable", "description": "sandbox infrastructure for AI agents", "tags": ["B2B", "Infrastructure"], "location": "San Francisco, CA, USA", "url_slug": "/companies/indexable" } ] } } ``` Note: The batch page returns `tags` (e.g. "B2B", "Infrastructure"), NOT detailed sectors. Individual YC pages (Step 3a) return richer `sectors` (e.g. "Artificial Intelligence", "Manufacturing"). Always prefer individual page data when available. ## Step 2: Create Google Sheet and Share Link Immediately Before any research, create the sheet and populate it with company names + descriptions from the batch scrape. Share the link so the investor can watch results fill in live. ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"google-sheets","path":"/create-spreadsheet","body":{"title":"YC {batch} Batch Evaluation"}}' ``` **Column layout (A through N):** | Col | Header | Source | |-----|--------|--------| | A | Company | Batch scrape | | B | Description | Batch scrape | | C | Sector | Individual YC page (`sectors`) — overwrite batch `tags` | | D | Location | Individual YC page → Apollo fallback | | E | Website | Individual YC page (`website_url`) | | F | Founders | Individual YC page (names + titles) | | G | Founder LinkedIn(s) | Individual YC page (`linkedin_url`) | | H | Founder Twitter/X | Individual YC page (`twitter_url`) | | I | Founder Background | Apollo employment history + YC page bios | | J | Founder-Company Fit | Your assessment | | K | Website Analysis | Company website scrape | | L | Market/Competitors | Perplexity | | M | Overall Assessment | Your assessment | | N | Priority Rank | Your ranking | Write header row + all company rows (research columns blank): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"google-sheets","path":"/update-values"}' "spreadsheet_id": "{spreadsheet_id}", "sheet_name": "Sheet1", "first_cell_location": "A1", "valueInputOption": "USER_ENTERED", "values": [ ["Company", "Description", "Sector", "Location", "Website", "Founders", "Founder LinkedIn(s)", "Founder Twitter/X", "Founder Background", "Founder-Company Fit", "Website Analysis", "Market/Competitors", "Overall Assessment", "Priority Rank"], ["{company_name}", "{description}", "{tags}", "{location}", "", "", "", "", "", "", "", "", "", ""] ] }' ``` Populate Sector (C) and Location (D) from the batch scrape initially — they'll be overwritten with richer data from the individual YC pages. **Share the sheet link with the user immediately** so they can watch it fill in. ## Step 3: Research Each Company — Row by Row ### Parallelization Strategy **The demo effect matters.** Rows filling in one-by-one on the spreadsheet is the visual payoff. Optimize for a steady stream of rows appearing — not for dumping everything at once. Process companies in **batches of 3-5 at a time**. Within each batch, all companies' research runs in parallel. But **update each row individually** the moment that company's research completes — do NOT wait for the whole batch to finish before writing. For each batch of companies (run all in parallel): 1. **Scrape all YC company pages** in the batch simultaneously (Step 3a) 2. As each YC page returns, **immediately launch its downstream calls in parallel**: - Scrape the company's website (Step 3b) - Apollo lookup for each founder (Step 3c) — multiple calls if multiple founders, all in parallel - Perplexity market analysis (Step 3d) 3. As each company's full research set completes, **compile and update its row** (Step 4) **immediately** — one `update-values` call per row, don't batch them 4. Move to the next batch **Key: each row gets its own sheet update call.** This creates the live-fill effect where the investor watches rows appear every 3-5 seconds. Never batch multiple rows into a single sheet write — that kills the visual cadence. With batches of 5, a 22-company batch completes in ~2-3 minutes with rows streaming in throughout. ### 3a. Scrape the YC company page (~$0.03 each) YC company pages are server-rendered and contain rich data: founders with LinkedIn, Twitter/X, bios, team size, sectors, website. ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/smartscraper"}' "website_url": "https://www.ycombinator.com/companies/{company_slug}", "user_prompt": "Extract: full company description, all founders (full name, title, LinkedIn URL, Twitter/X URL, bio), company website URL, team size, location, sectors, founding year." }' ``` **Expected response structure:** ```json { "result": { "founders": [ { "full_name": "William Alexander", "title": "Founder", "linkedin_url": "https://www.linkedin.com/in/william--alexander/", "twitter_url": null, "bio": "Manufacturing nerd from Iowa. Stanford Econ + CS." }, { "full_name": "Tom Blomfield", "title": "Primary Partner", "linkedin_url": null, "twitter_url": null, "bio": null } ], "website_url": "https://arzana.ai", "location": "San Francisco, CA, US", "sectors": ["Artificial Intelligence", "Manufacturing"], "team_size": 4, "founding_year": 2025 } } ``` **Critical parsing rules:** 1. **Filter out YC partners**: Entries with title "Primary Partner" or "Group Partner" are YC staff (e.g. Tom Blomfield, Harj Taggar), NOT founders. Exclude them. 2. **Website field name varies**: Check `website_url`, `company_website_url`, and `company_website` — Scrapegraph returns different field names depending on the page. 3. **Sectors field name varies**: Check `sectors`, `sector_tags`, and `tags`. Prefer the individual page's `sectors` over the batch page's `tags` — individual pages return specific sectors like "Artificial Intelligence" vs generic tags like "B2B". 4. **LinkedIn URL may be null**: Some founders don't have LinkedIn listed on YC. Use Apollo fallback (Step 3c) with name + company search. 5. **Twitter/X URL may be null**: Only populate if present, don't invent URLs. ### 3b. Scrape the company's own website (~$0.03 each) **Always run this step.** Company websites have product details, pricing, customer logos, testimonials, and hiring signals that no other source provides. ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/smartscraper"}' "website_url": "https://{company_website}", "user_prompt": "Extract: what the product does, target customer, pricing model, key features, traction signals (customer logos, metrics, testimonials), hiring signals. Be specific about what you find." }' ``` **Expected response — varies by company, but typically includes:** ```json { "result": { "product_description": "...", "target_customer": "..." or ["...", "..."], "pricing": "Not disclosed" or {"model": "...", "price": "..."}, "key_features": ["...", "..."], "traction_signals": { "customer_logos": ["Heineken", "Toyota", "..."], "metrics": ["99.7% accuracy", "25K patients/day"], "testimonials": [{"author": "...", "quote": "..."}] }, "hiring_signals": ["Careers page present"] } } ``` Note: The response structure varies significantly between companies. The `traction_signals` field is sometimes called `traction`. Customer logos may be actual names or just "logos displayed but not identified". Pricing may be a string, object, or array. Parse flexibly. If the scrape fails (404, timeout, empty), put "Website not available or pre-launch" in the Website Analysis column — never leave it blank. ### 3c. Apollo — founder work history (~$0.01 per founder) Use the LinkedIn URL from the YC page to get full employment history. Run one call per founder. ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/apollo/people/match \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "linkedin_url": "{founder_linkedin_url}", "reveal_personal_emails": true }' ``` **Key fields in the Apollo response:** ```json { "person": { "name": "William Alexander", "headline": "Co-Founder & CEO, Arzana | Stanford Economics + CS", "city": "San Francisco", "state": "California", "employment_history": [ { "organization_name": "Arzana", "title": "Co-Founder", "start_date": "2025-06-01", "end_date": null, "current": true }, { "organization_name": "Previous Company", "title": "Engineer", "start_date": "2022-01-01", "end_date": "2025-05-01", "current": false } ] } } ``` **What to extract:**