| name | job-search-validation |
| description | How to validate a job posting URL before adding it to the pipeline or recommending it to the user. Built from real failures during search sessions — HTTP 200 ≠ active posting, and most ATS systems serve empty pages to scrapers. |
Job Search Validation
The single biggest source of frustration during a search session
is recommending dead links and stale postings. This skill encodes what works.
The core rule
HTTP 200 does NOT mean the posting is active. Many ATS systems serve a
"this job is no longer active" page with a 200 status. Examples confirmed
in the wild:
- The Hershey Company (
careers.thehersheycompany.com) — returns 200 + "This job posting is no longer active"
- Lincoln Financial (
jobs.lincolnfinancial.com) — returns 200 + "This position is no longer posted"
- UPMC (
careers.upmc.com) — returns 200 + "Job not found - UPMC" as the page title
- Vanguard (
vanguardjobs.com) — returns 200 + "Page not found - Vanguard Careers" for stale IDs
You must read the page content (the actual job title, description, or
the absence of "no longer active" / "not found" language) to validate.
Validation hierarchy (fast → slow)
Tier 1: curl + title-and-body grep
Fast batch validation. Use this for screening ~20 URLs at once.
UA="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
URL="..."
body=$(curl -A "$UA" -L -s --max-time 12 "$URL")
title=$(echo "$body" | grep -oE '<title>[^<]+' | head -1 | sed 's|<title>||')
if echo "$body" | grep -qiE 'no longer (active|posted|available|accepting)|position is closed|job has been (closed|filled)|posting (has expired|is closed)|requisition is no longer|job not found|page not found|expired'; then
echo "❌ CLOSED | $title"
elif echo "$title" | grep -qiE 'not found|page not found|expired'; then
echo "❌ NOT FOUND | $title"
elif [ -n "$title" ]; then
echo "✅ LIKELY OPEN | $title"
else
echo "❓ NEED MANUAL CHECK | (empty title — JS-rendered ATS)"
fi
Tier 2: WebFetch with explicit content prompt
Use when curl returns empty title (JS-rendered) OR to confirm Tier 1 hits
before recommending.
WebFetch(
url: "...",
prompt: "Is this an active, currently-open job posting? Or does the page say
'no longer active', 'this job is closed', 'this position has been
filled', or redirect to a generic page? Quote the relevant text."
)
Tier 3: Browser only
Workday-based ATS (*.wd5.myworkdayjobs.com) and dejobs.org are heavily
JS-rendered. Neither curl nor WebFetch can read the job content. Either:
- Hand the link to the user to verify in browser, OR
- Pull the same role from a Greenhouse/LinkedIn mirror if one exists
ATS reliability ranking
| ATS / host | Reliability | Validation method |
|---|
Greenhouse (job-boards.greenhouse.io, boards.greenhouse.io) | ⭐⭐⭐⭐⭐ | WebFetch reads content reliably |
| Direct ATS (Lockheed, Erie Insurance, Hershey, Lincoln Financial) | ⭐⭐⭐⭐ | Tier 1 works — read body for "no longer" patterns |
LinkedIn (linkedin.com/jobs/view/{id}) | ⭐⭐⭐ | Rate-limited; same URL returns different content per call. Newer job IDs (43xxxxxxxx+) more likely fresh; older (38xxxxxxxx) usually stale. Use WebFetch with retry. |
Ashby (jobs.ashbyhq.com) | ⭐⭐⭐ | WebFetch sometimes returns minimal content |
Workday (*.wd5.myworkdayjobs.com) | ⭐ | JS-rendered, scraper-blind. Browser only. |
| dejobs.org | ⭐ | JS-rendered aggregator. Browser only. |
| JS-rendered company sites (Northrop, UPMC newer pages) | ⭐ | Browser only |
LinkedIn job ID heuristic
LinkedIn job IDs are auto-incrementing integers. Rough freshness signal:
43xxxxxxxx and 44xxxxxxxx → posted 2026, likely fresh
42xxxxxxxx → late 2025, possibly fresh
40xxxxxxxx–41xxxxxxxx → mid 2025, mixed
38xxxxxxxx and below → 2024 or earlier, usually closed and redirects to generic search
When LinkedIn redirects a closed job, the page title becomes something like:
"7,000+ Senior Product Design Engineer jobs in United States" or
"813 Klaviyo jobs in United States" (a generic search result). That title
pattern is the dead-job signal.
A live LinkedIn job has a title like:
"Stanley Black & Decker, Inc. hiring Lead Engineer, Embedded Software in Towson, MD | LinkedIn"
The presence of " hiring " in the title is the alive marker.
Recommendation rules
Before handing a list of links to the user:
- Every link must be content-validated, not HTTP-status-validated
- State the validation method per link in the response (e.g., "WebFetch confirmed", "browser-only — couldn't validate")
- Never pad a list with unverified URLs — fewer real links beats more padded ones
- If you can't validate ≥20 links, give the user fewer and link to live search pages instead
- Re-check the day of application — postings expire fast in this market
Sanity checks before adding to the pipeline
All criteria come from profile.yaml — see profile.yaml.example at the repo root.