job-signal-collector

name

job-signal-collector

description

Use this skill when the user wants to collect public job-source signals, validate recruitment source profiles, crawl or scrape public JD/list data, normalize public job records, build source run packages, or prepare source evidence handoff from ATS boards, company career pages, Greenhouse, Lever, Ashby, Workday, BOSS直聘, 智联招聘, 前程无忧, 猎聘, 丁香人才, PCB人才网, 半导体人才网, or says Job Signal Collector, Public Job Source Collector, 职位信号采集, source源头, JD抓取, 职位爬取, 招聘源采集. It routes sources through Source Registry, Adapter Factory, public-data acquisition, and Evidence Lineage rules, then outputs normalized public job records plus a traceable source run package. It blocks login, captcha, resume, private-contact collection, DB writes, staging promotion, and downstream signal interpretation.

Job Signal Collector

Overview

This skill is the source layer for public job signals. It turns public recruitment sources into normalized, traceable job-source records that downstream skills can validate, interpret, compare, store, or report.

The MVP boundary is intentionally narrow: source profile -> source classification -> acquisition strategy -> public list/detail acquisition -> normalized job records -> evidence lineage -> source run package. It does not own signal-card interpretation, cross-signal judgment, long-term storage, DB writes, staging promotion, or product-level reports.

The crawler technology stack stays fixed: Firecrawl crawl/search/scrape, public ATS APIs, direct HTTP, and optional Crawlee public fallback. Use scripts/jd_harvest.js as the MVP collector CLI. Legacy downstream artifacts are disabled by default and require compatibilityOutputs: true; MVP success is judged only by source records, evidence records, lineage, run log, run manifest, and report.

Resource Guide

• Read references/acquisition-ladder.md before any task that needs a target count such as 10 JD records.
• Read references/job-signal-mvp.md before tasks that require MVP scope, source run packages, source profiles, run logs, reports, company dimension, or position dimension.
• Read references/source-registry.md before validating or designing source profiles, company source pools, sector source lists, or source admission rules.
• Read references/adapter-factory.md before adding, repairing, or promoting a site/platform adapter or deciding whether a source needs manual adapter discovery.
• Read references/evidence-lineage.md before creating evidence records, lineage manifests, source run manifests, or handoff packages.
• Read references/source-strategy.md before choosing a command or judging a site family.
• Read references/public-platform-adapters.md for ATS API, Firecrawl search/scrape, detail URL filter, and Crawlee fallback patterns.
• Read references/field-schema.md before normalizing JD output or defining extraction fields.
• Read references/foxnex-handoff-contract.md before handing source output to FOXNEX Evidence Validation, Signal Card Builder, Cross-Signal Judgment, Runtime Governance, storage, or reporting skills.
• Read references/validation-matrix-2026-06-19.md when the user asks about the sites already tested in this project, or asks whether a source can reach 10 public JD records.
• Use scripts/summarize_firecrawl_crawl.js after every crawl batch that writes Firecrawl JSON.
• Use scripts/jd_harvest.js when the request is to collect records across one or more public sources. Treat .records.jsonl, .records.csv, .evidence-records.jsonl, .run-manifest.json, .run-log.json, and .report.md as the MVP outputs. .job-signal-cards.jsonl and .job-diff.json are disabled unless the config explicitly sets compatibilityOutputs: true.
• Use examples/retest-prompts.md when regression testing this skill or checking a new site category.
• Use examples/job-signal-company-source-profiles.example.json for company-dimension collection.
• Use examples/job-signal-position-targets.example.json for position/keyword-dimension collection.

Routing

Classify each target URL before running Firecrawl:

Source type	Examples	Default route
Enterprise careers page	Figure, Harvey, Cursor, Perplexity, Figma	Crawl only the careers/jobs section, then inspect discovered ATS or detail links.
ATS board	Greenhouse, Lever, Ashby, Workday	Prefer public platform APIs where available, then scrape/crawl detail pages; do not treat all ATS systems the same.
Vertical recruiting site	PCB人才网, 医药人才网, 半导体人才网, 丁香人才	Use public detail paths, Firecrawl search/scrape, link expansion, and direct HTTP where lawful.
Comprehensive recruiting site	BOSS直聘, 智联招聘, 前程无忧, 猎聘	Use public pages only; prefer indexed detail URLs, batch scrape, and link expansion over broad crawl.
Social or private source	脉脉, LinkedIn, 微信群, 会员区, 内推系统	Do not force automation. Use only public pages or user-exported authorized files.

If the user gives only a company name, search or inspect the official careers page first, then choose the source type. If the user gives many URLs, group them by source type and run one pilot per group before scaling.

Before collecting from a recurring company or sector source, create or validate a Source Registry entry. Do not let the collector repeatedly rediscover the same source when a stable source profile can own the URL, ATS type, guard status, parser strategy, and evidence boundary.

Collection Dimensions

Use company dimension when the user gives companies, company career URLs, ATS boards, or asks for company hiring signals. Preserve company_id, company_name, sector, source_url, source_type, collection_strategy, risk_level, and parser_strategy in the source profile.

Use position dimension when the user gives a role, keyword, city, function, or industry-wide signal question. Use platform search/scrape targets with collectionDimension: "position", target_role, query, sector, and URL filters. If company fields are missing from public snippets, keep them null and record the missing fields.

Workflow

1. Define the crawl target.

   • Capture the user goal, source URLs, max page budget, required JD fields, and whether the task is discovery-only or detail extraction.
   • If the page requires login, membership, captcha, or private group access, stop and ask for a public URL or user-exported data.

2. Validate the source profile.

   • Use references/source-registry.md to classify the source as confirmed, candidate, needs_review, blocked, or unknown.
   • Preserve source identity: company, sector, source URL, source type, ATS/platform, source priority, validation status, guard status, parser strategy, and evidence boundary.
   • Candidate or unknown sources may run only small diagnostics. Blocked sources must not be forced.
   • Apply the Source Admission Gate before any API, HTTP, Firecrawl, or browser call. Normal collection requires validation_status=confirmed, guard_status=pass, and collector_allowed=true. diagnostic_only forces limit=1; false or blocked states make no network request.

3. Choose the acquisition strategy.

   • Use references/acquisition-ladder.md to pick the next allowed level.
   • Use references/source-strategy.md to pick the smallest viable crawl or scrape route.
   • Prefer source-specific paths such as /careers, /jobs, /job, /position, /zhaopin, or ATS board paths.
   • Never start with a whole-domain crawl.
   • If the source requires a new parser or site-specific route, use references/adapter-factory.md to keep the adapter as a candidate until a fixture, smoke run, and failure boundary exist.

4. Run a pilot or harvester.

   • If the task is to collect source-profile-backed job signals, prefer node job-signal-collector/scripts/jd_harvest.js --config <profiles.json> --out <records.json> --limit 10.
   • If the task is source diagnosis, use npx -y firecrawl-cli@latest crawl.
   • Every crawl pilot command must include --limit, --wait, --timeout, --pretty, and -o.
   • Use Firecrawl scrape for known public detail URLs and Firecrawl search for indexed URL discovery.
   • Use --progress when running crawl interactively.
   • Use --max-depth 0 for single-page diagnostics and --max-depth 1 for list-to-detail pilots.

Example command shape:

npx -y firecrawl-cli@latest crawl "https://job-boards.greenhouse.io/example" \
  --limit 3 \
  --max-depth 1 \
  --wait \
  --progress \
  --timeout 120 \
  --pretty \
  -o ".firecrawl/example-greenhouse.json"

5. Validate the output.

   • Run node job-signal-collector/scripts/summarize_firecrawl_crawl.js <crawl-json...>.
   • Count list pages, JD detail pages, loading shells, status-only outputs, blocked pages, and unknown pages.
   • A successful JD pilot needs at least one detail page with title plus location/company plus responsibilities, requirements, or their Chinese equivalents.

6. Decide the next action.

   • If the pilot has JD detail pages, scale slowly with a larger limit and the same path constraints.
   • If it has only list pages, extract detail links and run a smaller detail crawl or scrape batch.
   • If crawl has loading shells or status-only output, mark plain crawl as failed and continue through the acquisition ladder instead of retrying a broader crawl.
   • If it has login walls, captcha, abnormal traffic, or private pages, stop that branch and ask for public URLs or authorized exports.
   • If Firecrawl returns per-minute rate limits, wait and retry sequentially; do not confuse rate limits with credit exhaustion.
   • If Firecrawl credits are exhausted, use the public API/direct HTTP/Crawlee fallback patterns in references/public-platform-adapters.md.
   • If the user asks for a large crawl, create a checkpoint before increasing page budget.

7. Normalize public JD fields only.

   • Use references/field-schema.md.
   • Output records must follow the SRS Normalized Job Record schema.
   • Every record must include run_id and schema_version.
   • Every run must write a run log and Markdown report.
   • Keep public job information. Drop application-form fields, candidate questions, phone codes, resumes, personal contact data, cookies, tokens, and session material.

8. Build a Source Run Package.

   • Use references/evidence-lineage.md to link source profile, acquisition route, raw page/API reference, normalized records, evidence records, run log, and run manifest.
   • Build Evidence Record objects with source URLs, evidence level, confidence, content hash, review status, and missing fields.
   • Build Run Manifest with source-layer file contracts, schema versions, counts, adapter/parser route, compliance boundary, and downstream handoff hints.
   • Do not create trend claims, organization-signal claims, DB write plans, staging promotion decisions, or product-level reports inside this skill.

Output Protocols

For a pilot result, report:

## JD Crawl Pilot

- source_type:
- strategy:
- commands_run:
- output_files:
- page_count:
- jd_detail_pages:
- list_pages:
- failed_pages:
- strategy_attempts:
- records_collected:
- remaining_gap:
- extracted_fields:
- verdict: pass | partial | fail
- next_action:
- compliance_notes:

For a Source Run Package, report:

## Source Run Package

- run_id:
- collection_dimension: company | position
- source_registry_status: confirmed | candidate | needs_review | blocked | unknown
- admission_status: allowed | diagnostic_only | blocked
- config_file:
- output_json:
- output_jsonl:
- output_csv:
- evidence_records:
- run_log:
- run_manifest:
- report:
- schema_version:
- schema_validation_status: pass | warn | fail
- target_count:
- final_record_count:
- evidence_record_count:
- human_review_required_records:
- partial_or_failed_sources:
- adapter_status: reused | candidate | needs_review | blocked
- lineage_status: pass | warn | fail
- compliance_notes:
- downstream_handoff:

For a site strategy decision, report:

## Strategy

- source:
- classification:
- first crawl:
- pass signal:
- fail signal:
- fallback:
- scale condition:

Boundaries

Do not automate login, captcha solving, account creation, membership access, session-cookie reuse, or private group scraping. Do not collect resumes, applicant data, personal contact data, or application-form answers. Do not bypass robots, rate limits, paywalls, or platform controls.

Do not claim a source is crawlable just because Firecrawl returns HTTP 200 or a completed job. Workday loading shells, status-only JSON, empty ATS boards, and login/captcha pages are failures for JD extraction unless they contain public JD detail content.

Do not run broad crawls over entire company domains or comprehensive recruiting sites without a CHECKPOINT / STOP approval. The checkpoint must name the exact page budget, path filters, output location, failure condition, and rollback plan.

Do not own downstream product functions in this skill. Signal cards, organization capability tags, cross-signal judgment, product reports, storage, DB writes, staging promotion, and runtime governance belong to downstream skills. Compatibility files require explicit opt-in and must not be embedded in the default source-layer payload.

Quality Standard

A good run satisfies all of these checks:

• The source type is named before crawling.
• Recurring sources have a source profile or a clear reason why no profile exists.
• Source Registry fields preserve URL, platform, guard status, validation status, and evidence boundary.
• Source Admission is enforced before network access; recorded registry states and runtime behavior must agree.
• Every Firecrawl crawl command is scoped with limit, timeout, output path, and depth or path control.
• The result distinguishes discovery pages from JD detail pages.
• Loading shells, status-only outputs, login walls, captcha, and empty boards are labeled honestly.
• Public JD fields are normalized using references/field-schema.md.
• Job signal exports follow the SRS fields in references/field-schema.md, not ad hoc crawler fields.
• Run log, report, JSONL, CSV, Evidence Record, and Run Manifest files are generated for collection runs.
• Evidence Lineage links source profile, acquisition route, raw reference, normalized records, evidence records, and manifest.
• Low-confidence snippets stay low confidence and require review before downstream interpretation.
• The final answer includes commands, files, verdict, and next action.
• The workflow composes with $firecrawl-crawl, but uses search, scrape, APIs, direct HTTP, or Crawlee fallback when the evidence justifies them.
• A 10-record request is judged by records collected and strategies exhausted, not by whether one crawl command completed.
• Known-source claims are backed by references/validation-matrix-2026-06-19.md or by a fresh validation run.
• Downstream interpretations are handed off, not invented in this source-layer skill.

Anti-Patterns

• "One command crawls all recruitment websites."
• Treating a careers landing page as JD data.
• Treating a Workday Loading page as success.
• Retrying blocked comprehensive sites with larger limits.
• Keeping application-form or candidate personal fields in the JD dataset.
• Crawling social, private, or member-only sources without explicit public authorization.
• Treating a platform search page as one full JD record. If only public snippets are available, use snippet content conservatively and mark missing detail fields.
• Treating DB write, staging promotion, signal-card interpretation, or cross-signal judgment as this skill's MVP.
• Creating a new adapter directly from one successful page without fixture, failure boundary, and promotion gate.
• Recording collector_allowed=false or a blocked guard state while still executing the source.
• Emitting capability tags or organization signals in the default source-layer payload.