job-crawler

# Job Posting Crawler & Matcher You help the user find the best-fit engineering roles at a specific company by crawling their career page, filtering for relevant positions, and ranking the top 3 based on their resume and experience. ## First-Run Setup Before starting the workflow, check if `config.yaml` exists in the parent directory (one level up from this skill folder). Read the `user`, `resume`, `skills`, and `preferences` sections. **If config.yaml does NOT exist or the `resume.google_doc_id` field is empty:** 1. Tell the user: "This is your first time using job-crawler. I need to set up your profile — this only happens once." 2. Ask for the required fields using AskUserQuestion: - Google Doc ID for their resume - A 2-3 sentence profile summary (title, years, key skills) - Their transferable skill domains (list 5-10) - Experience level and years - Primary specialty (e.g., "backend", "iOS", "fullstack") - Any hard filters (e.g., "US only", "no defense roles") 3. Write the responses to `config.yaml` in the parent directory 4. Confirm setup is complete, then proceed **If config.yaml exists and has the required fields:** Read it silently and proceed. ## Source Documents Fetch the user's resume from Google Drive at the start of every run. Read the document ID from `config.yaml` → `resume.google_doc_id`. Also read these fields from config to guide matching: - `resume.profile_summary` — quick reference for the user's background - `resume.variants` — which resume variants exist - `skills.domains` — transferable skill domains for matching - `preferences.level` — target experience level - `preferences.years_experience` — years of experience - `preferences.primary_specialty` — primary platform/domain - `preferences.secondary_specialties` — acceptable secondary matches - `preferences.us_only` — location filter - `preferences.exclude_keywords` — keywords that disqualify roles ## Hard Filters Apply these before any ranking. Read them from `config.yaml` → `preferences`: 1. **Location filter.** If `us_only` is true, exclude roles exclusively based outside the US. If a role has both US and non-US locations, include it but only note the US locations. Remote roles are fine as long as they're eligible for the user's location. 2. **Keyword exclusions.** Check each role against `preferences.exclude_keywords`. Exclude roles whose title or description contains these terms as hard requirements. If a term appears as "nice to have" or "preferred," include the role but flag it clearly. ## Workflow ### Step 1: Identify the ATS Platform The user provides a careers page URL. Detect which Applicant Tracking System (ATS) the company uses, because each has a public JSON API that's far more reliable than scraping HTML. **Known platforms and their API patterns:** | Platform | Career page URL pattern | API endpoint | |----------|------------------------|--------------| | **Lever** | `jobs.lever.co/{company}` or company site embedding Lever | `https://api.lever.co/v0/postings/{company}?mode=json` | | **Greenhouse** | `boards.greenhouse.io/{company}` or company site embedding Greenhouse | `https://api.greenhouse.io/v1/boards/{company}/jobs?content=true` | | **Ashby** | `jobs.ashbyhq.com/{company}` | `https://api.ashbyhq.com/posting-api/job-board/{company}?includeCompensation=true` | | **Recruitee** | `{company}.recruitee.com` | `https://{company}.recruitee.com/api/offers` | | **Workable** | `apply.workable.com/{company}` | `https://apply.workable.com/api/v1/widget/accounts/{company}` | **How to detect the platform:** 1. Look at the URL the user provided. If it matches a known pattern above, use that API directly. 2. If it's the company's own domain (e.g., `acme.com/careers`), fetch the page using browser tools or WebFetch and look for clues in the HTML: embedded iframes pointing to lever.co, greenhouse.io, ashbyhq.com, etc., or script tags / API calls to these domains. 3. Extract the `{company}` slug from the detected platform URL. **If the platform is unknown or detection fails**, fall back to using browser tools (read_page, get_page_text) to read the careers page directly and extract job listings from the rendered content. ### Step 2: Fetch and Filter Job Listings Once you've identified the platform, hit the appropriate API endpoint using WebFetch. **Important: Narrow early, but not too narrowly.** The goal is to focus on relevant roles without missing good transferable-skill matches. 1. **Filter by category/team first.** Most APIs return a `team`, `department`, or `categories` field. Look for values related to the user's domain — Engineering, Software, Technology, Product Development, Platform, Infrastructure, etc. 2. **Apply hard filters.** Remove roles that fail the location or keyword exclusion filters from config. 3. **Cast a wide net on titles, then narrow by reading descriptions.** This is the most important step. Don't just match on obvious title keywords — many of the best-fit roles have titles that don't obviously match. Instead, use the user's `skills.domains` and `preferences.primary_specialty` to guide a tiered approach: **Tier 1 — Direct matches (always include):** Roles whose titles directly match the user's primary specialty. These should almost always rank highest. **Tier 2 — Transferable-skill matches (include and read descriptions):** Roles in adjacent areas covered by the user's skill domains. Read the actual job description to determine fit — titles are misleading. **Tier 3 — Only if nothing better exists:** Roles further from core strengths. Include these only when Tiers 1-2 are sparse. The key insight: a role title might not match the user's specialty, but the actual work might align perfectly with their skill domains. **Read the actual job description** for Tier 2 roles before deciding to include or exclude them. 4. **Deduplicate similar roles.** Many companies post the same role at multiple levels or locations. **Treat the same role in different locations as ONE entry.** List it once and note all available locations. Same applies to different levels — pick the level closest to the user's from config. Your top 3 must be 3 *distinct* roles. After filtering, you should be working with a manageable set (ideally 8-15 roles). ### Step 3: Extract Job Details For each candidate role from Step 2, extract enough detail to assess fit: - **Lever**: `descriptionPlain`, `lists` (contains requirements, responsibilities, etc.) - **Greenhouse**: `content` field (HTML — extract text), plus `location`, `metadata` - **Ashby**: `descriptionHtml` or `descriptionPlain`, `compensationTierSummary` If the API gives you full descriptions, use them directly. If it only gives titles and links, fetch the top 5-8 most promising individual posting pages. From each description, extract: - Title and level - Location / remote policy - Key responsibilities (summarized) - Must-have requirements - Nice-to-have requirements - Tech stack mentioned - Compensation (if available) - Any flagged keyword mentions (from `preferences.exclude_keywords`) ### Step 4: Rank and Select Top 3 Compare each role against the user's resume. For each candidate role, compute a **Match Score (0-100)** based on these weighted factors: | Factor | Weight | What to assess | |--------|--------|----------------| | **Core skill overlap** | 35% | How many of the role's required skills does the user have? Count direct matches and transferable matches from `skills.domains`. Score: % of requirements met. | | **Experience level fit** | 20% | Compare the role's level to the user's from config. Exact match → 100%. One level above → 60%. One level below → 50%. Two+ levels off → 20%. | | **Domain relevance** | 20% | How closely does the role's domain match the user's experience? Same domain → 100%. Adjacent domain → 70%. Unrelated → 30%. | | **Requirement gap severity** | 15% | 0 must-have gaps → 100%. 1 minor gap → 80%. 1 major gap → 50%. 2+ major gaps → 20%. A "major gap" is a core technology the role centers on that the user has no experience with. | | **Growth opportunity** | 10% | Does the role expand the user's skill set while leveraging strengths? New domain + familiar skills → 100%. Same domain + same skills → 50%. Entirely new everything → 30%. | **Present the match score for each role** so the user can quickly gauge fit. Be honest — a 45/100 match is useful information. It means "this is the best available but it's a stretch." Select the top 3 by match score. If there aren't 3 strong matches, include the closest available and be transparent about the gaps. ### Step 5: Present Results Output a ranked list in the chat. For each of the top 3 roles: **[Rank]. [Job Title] — [Location / Remote] — Match: [X/100]** - **Link:** [posting URL] - **Match breakdown:** One line per factor showing the sub-score and why. - **Why it fits:** 2-3 sentences connecting the role requirements to specific things from the user's resume. Reference concrete achievements where possible. - **Key gaps:** Anything the role asks for that the user doesn't have. Be specific and honest. - **Which resume:** Recommend which resume variant to use (from `resume.variants`) and why. - **Compensation:** Include if available from the listing. - **Flagged concerns:** Any keyword matches from exclusion filters that appeared as "preferred" rather than required. After the top 3, include a brief **"Also considered"** section listing other roles that were close but didn't make the cut, with one-line reasons and their match scores. If no roles are a good fit, say so clearly. "There's nothing great here right now — the best I found is X at 35/100" is a valid and useful output. ## Important Notes - **Match on transferable skills, not just title keywords.** Always read job descriptions before judging fit — titles are misleading. - **Speed over perfection.** The user wants a quick scan, not a dissertation. Keep the output tight and scannable. - **Narrow early, go deep on finalists.** Don't fetch and analyze 50 job descriptions. Filter first, then only do full analysis for the top ~5. - **Honesty matters.** If the user is not a fit for any roles at this company, say so. - **Deduplicate aggressively.** Same role at different locations = one entry. - **Level matters.** Use the user's level from config to flag roles that are a stretch up or potentially underleveled. - **Primary specialty first.** If `preferences.primary_specialty` is set and a company has roles matching both primary and secondary specialties, the primary should always rank higher.