apify-jobs-data

# Jobs Data Extract clean, structured job-posting data from 20+ boards in one Apify run, then put it to use. The pipeline is the same regardless of purpose: **acquire → de-noise → normalize → { analyze | export | rank by résumé }** The reusable value is the cleaned dataset: postings deduplicated across boards, cross-board reposts merged, likely ghost jobs flagged, and fields normalized to a consistent schema. Ghost-job and repost pollution corrupts demand counts, salary statistics, and dashboards just as much as it wastes a job seeker's time — so de-noise is the shared core, and every output mode runs on top of it. **Never invent a posting, a salary, a count, or a keyword.** Everything is scraped live from a named Apify Actor and carries its source; missing fields stay blank, and every statistic reports the share of postings it is computed from — see [Quality rules](#quality-rules). Ghost-job detection is heuristic and single-run: it compares the same (company, title, location) across boards for conflicting posted dates and reads the JD for pipeline language. The data carries no first-seen or edit history, so a flag is a suspicion, never a verdict. The Actors do the data work; the agent routes, de-noises, aggregates, and grounds: | Step | Apify Actor | Agent | |---|---|---| | Acquire across 20+ boards | `agentx/all-jobs-scraper` | routes the query | | De-noise ghost jobs / reposts | cross-board scraped fields — conflicting `posted_date` across boards, JD-body match across "company" names (no first-seen / edit-history field exists) | applies the rule | | Salary benchmark (optional) | `memo23/glassdoor-scraper-ppr` | aggregates, labels coverage | ## Note on overlap with analysis-first job skills A more analysis-first skill may also answer "hiring demand / in-demand skills / salary" questions. This skill's center of gravity is the **cleaned dataset** — cross-board duplicates and reposts merged, likely ghost jobs flagged, and fields normalized — which every mode then runs on. If you only need a quick market read, an analysis-only skill is lighter. Reach for this one when **data quality matters** (deduped demand counts, coverage-labeled salary stats), when you need the **raw rows exported** to CSV / an Apify dataset for a dashboard, or when you need a **résumé ranked** against live postings — none of which an analysis-only skill produces. ## Output modes — pick one or more The shared core (Steps 1–5) is identical; Step 6 produces whichever mode(s) the user wants. Default to whatever the request implies; if unclear, ask. | Mode | Produces | Reference | |---|---|---| | **Market & salary analysis** | hiring demand on **deduped** counts (by company / location / seniority), in-demand skills, remote-hybrid mix, and **coverage-labeled** salary distribution | [reference/analysis.md](reference/analysis.md) | | **Structured export** | the normalized rows as CSV / JSON, or the raw Apify dataset ID for direct BI / dashboard ingestion | [reference/output-formats.md](reference/output-formats.md) | | **Résumé-fit** | postings ranked against a résumé, with ATS-keyword gaps and apply-ready briefs | [reference/fit-scoring.md](reference/fit-scoring.md) | ## Cost discipline (best quality for the lowest cost) This is a design goal, not an afterthought — keep every run as cheap as it can be while still answering the question: - **One cheap actor, no subscriptions.** Default to the pay-per-result aggregator (≈ $0.0035/job + $0.01 start on the free tier, as of 2026-09-16 — check the live rate in `reference/gotchas.md` / the Apify console before quoting a number). A typical run is **cents to a few dollars**. - **Smallest sample that answers it.** `max_results` is *per platform* (× every platform the Actor supports for the country unless `platforms` is pinned), so pin `platforms` and start small — a quick scan needs ~10–15/platform; a market analysis ~25–50/platform — and scale only if the result is too thin. Always estimate, and confirm before a big run. - **One run feeds every mode.** Analysis, export, and résumé-fit all read the *same* scrape — never re-scrape to add a second mode. - **De-noise so you never pay for junk.** Removing ghost jobs / reposts / duplicates keeps the billed sample honest and the counts real. - **For salary, prefer the Glassdoor benchmark over a mega-scrape.** Salary disclosure is low (~2–6% in many markets), so scraping thousands of postings to harvest a few disclosed figures is wasteful — one cheap `memo23/glassdoor-scraper-ppr` call per company **with `maxItems` ≤ 50** (its default is 20,000 rows per URL) gives a better signal for less. ## Prerequisites (No need to check this upfront.) Two execution paths — pick the one that matches your environment. **MCP path (default in an agent session with MCP, recommended).** If the Apify MCP server is connected, no setup is needed — auth runs through the user's Apify account. Use the `call-actor`, `get-actor-run`, and `get-dataset-items` MCP tools. **CLI path (scripted / non-interactive execution).** Requires the [Apify CLI](https://docs.apify.com/cli) v1.5.0+ and auth via `apify login` or an `APIFY_TOKEN` env var ([get a token](https://console.apify.com/settings/integrations)). Every CLI call in this skill carries three flags — `--json`, `--user-agent apify-awesome-skills/apify-jobs-data`, and `2>/dev/null` (`apify api` prints JSON by itself and rejects `--json` — give it the other two). **Responsible use.** These Actors scrape third-party boards (LinkedIn, Glassdoor, Indeed, ZipRecruiter and others) against those sites' Terms of Service — the user's call to make. All routes run on Apify's infrastructure (no user login), so they never put the user's own board accounts at risk. ## Workflow Copy this checklist and track progress. ``` Task Progress: - [ ] Step 0: Pick the output mode(s) - [ ] Step 1: Collect the query anchors (one block) - [ ] Step 2: Route to the right board Actor(s) (primary + fallback) - [ ] Step 3: Build the input, estimate cost, confirm if over threshold - [ ] Step 4: Run, wait, pull the dataset - [ ] Step 5: De-noise + normalize into clean rows - [ ] Step 6: Produce the selected mode(s) — analysis / export / résumé-fit - [ ] Step 7: Deliver, with provenance and coverage ``` ### Step 0: Pick the output mode(s) Decide what the user wants *done with the data* — it changes only Step 6, not the shared core. Infer from the request; confirm if ambiguous: - "what skills/salary/demand for X", "job-market for Y" → **Market & salary analysis**. - "scrape / export / give me a dataset / CSV for a dashboard" → **Structured export**. - "which of these fit my résumé", "rank these for me" → **Résumé-fit** (needs a résumé; anchor #7). - Multiple are fine — e.g. analysis + export of the same run. ### Step 1: Collect the query anchors Ask these as **one block** before any Actor call. Defaults shown — always surface them so the user can override. (Anchor-block pattern from `apify-verified-email-finder`.) 1. **Role / keywords** — e.g. `senior backend engineer`. Required. 2. **Location** — city, country, or `remote`. Required. `remote` flips the remote-only filter on. 3. **Boards** — `auto` (default → aggregator, all boards the Actor supports for the country) or a named subset passed as the aggregator's `platforms` array (exact enum values from the live schema — e.g. `LinkedIn`, `Indeed`, `Glassdoor`; Google Jobs is **not** in the enum). Drives Step 2 and cost (Step 3). 4. **Result cap** — for the aggregator this is **per platform** (`max_results: 25` ≈ 25 × the platforms hit — every platform the Actor supports for the country unless `platforms` is pinned), **minimum 1**. Start small (`10–15` for a quick scan, `25–50` for analysis) and scale only if the sample is too thin — together with `platforms` it's the main cost lever, so budget `max_results × platforms` ([Cost discipline](#cost-discipline-best-quality-for-the-lowest-cost), Step 3). Maps to each Actor's own field (`max_results` for the aggregator / `maxItemsPerSearch` for Indeed / `maxItems` for Glassdoor — actor-index.md). 5. **Recency window** — default `2 weeks`. The aggregator takes this as a natural-language string (`"2 weeks"`, `"1 month"`), not a day count. 6. **Filters** — optional constraints that *drop* a posting: `salary_floor`, `remote_only`, `job_type` (`fulltime` / `contract` / `internship` — no hyphen). Collect what the user volunteers. 7. **Résumé / profile** — *only for résumé-fit mode*: résumé text or a must-have + nice-to-have skill list. Without it, résumé-fit falls back to a mechanical score labeled `Fit (partial)`. **Ambiguity rule:** if role or location is missing or vague, ask **one** clarifying question before running. Never burn Actor compute on a guessed query. ### Step 2: Route to the right Actor(s) Default to the **aggregator** — one run, 20+ boards (LinkedIn, Indeed, Glassdoor, ZipRecruiter, and more; **not** Google Jobs), country-aware routing, cheapest per-result. **Every Actor here is pay-per-result — no subscriptions.** The aggregator already covers LinkedIn cheaply, so there is no need for a per-board LinkedIn subscription Actor. If the user asks for Google Jobs specifically, say this skill does not cover it. | User wants | Actor | Notes | |---|---|---| | All boards (default) | `agentx/all-jobs-scraper` | One query → LinkedIn, Indeed, Glassdoor, ZipRecruiter + 38 more in the live `platforms` enum (no Google Jobs). **Pay-per-result** (≈ $0.0035/job + $0.01 start, free tier, 2026-09-16). | | Indeed only (cheap, focused) | `misceres/indeed-scraper` | Community, pay-per-result (≈ $0.006/job, free tier, 2026-09-16). | | Glassdoor salary benchmark | `memo23/glassdoor-scraper-ppr` | Pay-per-result; analysis mode only, to cross-check posted salaries (analysis.md). | Start with the aggregator unless the user explicitly wants a single board. **Named boards stay in one run:** if the user names one or more boards, pass them in the aggregator's `platforms` array (anchor #3) — a single run whatever the number of boards, never one Actor per board. The only standalone route chosen up front is `misceres/indeed-scraper`, and only when the user wants Indeed *exclusively*. **Per-board fallback rule:** after the run, count rows per requested board *that match anchor #2 (location)*. A board the user named that comes back with **zero (or near-zero) in-area rows** — blocked, or the Actor ignored the location, as Indeed's path does — gets **one** fallback run with its standalone Actor from the table (`misceres/indeed-scraper` for Indeed, capped with `maxItemsPerSearch`), if budget and run count allow; its rows carry their own `Source` board and are deduped against the aggregator's in Step 5. Only if budget or the run count is gone do you report the board as a coverage gap instead — a board the user asked for is worth that one capped run. The fallback is **conditional on a measured zero**: outside it, never run two Actors against the same query. Never a second aggregator retry for the same board — whatever the cause (blocked, location ignored, or TIMED-OUT before the board was reached); do the per-board in-area count first, and if the aggregator timed out, narrow `platforms`/`max_results` only in the *first* run's design, never in a rerun. Full schemas and field mappings: [reference/actor-index.md](reference/actor-index.md). ### Step 3: Build the input, estimate cost, confirm Map anchors → the chosen Actor's field names (per-Actor table in [reference/actor-index.md](reference/actor-index.md)). Aggregator example: ```json { "keyword": "senior backend engineer", "location": "Berlin", "country": "Germany", "max_results": 50, "posted_since": "2 weeks", "remote_only": false } ``` Verified-from-live-schema notes for the aggregator: `country` is a **full country name** (`Germany`, `United States`), not an ISO-2 code; `posted_since` is a **natural-language string** (`"2 weeks"`), not a number of days; `job_type` is `fulltime` / `parttime` / `contract` / `internship` / `all` (no hyphen); and **`max_results` is per platform** — with `platforms` empty the actor fans out to every platform it supports for the country, so `max_results: 50` can return several hundred rows; pin `platforms` to bound it. Other Actors use their own field names (actor-index.md). **Estimate before running.** Formula and live rates in [reference/gotchas.md](reference/gotchas.md). Guardrails: - Estimated cost **> $5** → warn with a rough number ("around $X"). - Estimated cost **> $20** → require explicit confirmation before running. - All routes are pay-per-result — cost scales with `max_results × boards`. Keep `max_results` modest to keep runs in the cents. ### Step 4: Run, wait, pull the dataset **MCP path:** `call-actor` with `actor`, `input`, `callOptions` `{"timeout": 900, "memory": 2048}`. It returns `runId` + `datasetId`. If `status` is `RUNNING`, poll `get-actor-run` (waitSecs ≤ 45) until `SUCCEEDED`. Capture both IDs for the `run_metadata.json` sidecar (and surface `datasetId` in export mode). **CLI path:** ```bash # Run options travel in --params: timeout 900 s (this skill's recommendation for the # multi-board aggregator — same as the MCP callOptions above) paired with # maxTotalChargeUsd = your Step 3 estimate (here the $5 warn threshold), the platform's # cap on spend. `apify actors call` has no flag for maxTotalChargeUsd — use the API mode. apify api POST "actors/agentx~all-jobs-scraper/runs" \ --params '{"timeout":900,"maxTotalChargeUsd":5}' \ --body '{"keyword":"senior backend engineer","location":"Berlin","country":"Germany","max_results":50,"posted_since":"2 weeks"}' \ --user-agent apify-awesome-skills/apify-jobs-data 2>/dev/null # the response returns immediately (data.id, data.defaultDatasetId); wait for the run: apify api GET "actor-runs/RUN_ID" --params '{"waitForFinish":60}' \ --user-agent apify-awesome-skills/apify-jobs-data 2>/dev/null # repeat until data.status is SUCCEEDED # then, using data.defaultDatasetId: apify datasets get-items DATASET_ID --format json \ --user-agent apify-awesome-skills/apify-jobs-data 2>/dev/null > /tmp/jobs.json ``` On the MCP path, pull with `get-dataset-items` (`clean: true` + a `fields` list). If the dataset exceeds the response cap, fetch directly: `curl -H "Authorization: Bearer $APIFY_TOKEN" 'https://api.apify.com/v2/datasets/<id>/items?clean=true&fields=...' | jq` — auth goes in the header, **never** as `?token=` in the URL (it lands in access logs). On the CLI path prefer `apify api GET "datasets/<id>/items" --params '{"clean":"true"}' --user-agent apify-awesome-skills/apify-jobs-data 2>/dev/null`, which authenticates itself. **Report every failure explicitly** (Actor, input, error) — never silently drop a board. If a board returns 0 in-area rows, apply the per-board fallback rule (Step 2) before concluding "no data". A board returning 0 while others return results is usually a block, not an empty market. ### Step 5: De-noise + normalize into clean rows The shared core. Two parts: 1. **De-noise** — walk every row and remove the noise: location mismatches (the Actor's location filter is not trusted), duplicates/reposts across boards, hard-filter violations, off-target roles; flag (don't drop) ghost jobs, staffing-agency reposts, and bare-`Remote` rows with no stated region (`remote: unverified` — kept, but not counted as in-area). Skipped rows are **kept** in a separate `Skipped` section with a one-line reason — never silently dropped. Apply the location check first, then hard filters, then dedupe, so the funnel reads `raw → after location check → after hard filters → after dedupe → clean`. Full detection logic: [reference/skip-pass.md](reference/skip-pass.md). 2. **Normalize** — map each surviving row onto the consistent schema in [reference/output-formats.md](reference/output-formats.md) (title, company, location, remote flag, salary min/max/currency, seniority, skills, posted date, source board, apply URL). Field coverage varies by board — leave missing fields blank, never inferred.