원클릭으로 Manus에서 모든 스킬 실행

download-gated-pdfs

스타1

포크0

업데이트2026년 7월 22일 09:34

Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

kennethkhoocy

kennethkhoocy/applied-micro-skills

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Download bot-gated PDFs via Wayback id_

Problem

Many think-tank and publisher sites (taxpolicycenter.org, urban.org, SSRN delivery) serve an HTML bot-challenge page instead of the PDF to non-browser clients. A browser User-Agent header does not help. The downloaded "PDF" is actually HTML.

Context / Trigger Conditions

curl -o file.pdf <url> succeeds but the file starts with <!DOC
pypdf raises invalid pdf header: b'<!DOC' or PdfStreamError: Stream has ended unexpectedly
Firecrawl scrape returns clean markdown for the same URL (its proxies get through), but Firecrawl does not return the binary — only parsed content

Solution

Request the file through the Wayback Machine's raw-content (id_) endpoint, which serves the original archived binary without rewriting:
```
curl -sL -A "Mozilla/5.0 ... Chrome/126.0 Safari/537.36" \
  "https://web.archive.org/web/<YYYY>id_/<original-pdf-url>" -o out.pdf
```
<YYYY> is any year likely to have a snapshot (e.g. publication year); Wayback redirects to the nearest capture. The id_ suffix after the timestamp is what requests the untouched original.

Verify the download with pypdf — a bot page fails immediately:

from pypdf import PdfReader
r = PdfReader("out.pdf"); print(len(r.pages), "pages")

If Wayback has no capture, fall back to: another mirror found via search (Exa/Firecrawl), or Firecrawl scrape for the parsed text when the binary is not strictly needed.

Verification

PdfReader opens the file and reports a plausible page count; first-page text matches the expected title.

Example

Verified 2026-07-15: taxpolicycenter.org/sites/default/files/publication/165884/ssrn-id4797771.pdf and urban.org/sites/default/files/publication/80621/2000790-...pdf both bot-gated to direct curl (with UA), both downloaded intact via https://web.archive.org/web/2024id_/<url> and .../web/2023id_/<url> (18 and 12 pages).

Notes

Government data hosts (e.g. ticdata.treasury.gov) are usually NOT gated — try direct curl first; Wayback is the fallback, not the default.
Wayback captures can be stale for frequently-revised documents; check the snapshot date if currency matters.
See also: the pdf skill (parsing/extraction after download) and firecrawl:firecrawl-scrape (parsed markdown when the binary is unnecessary).

이 저장소의 다른 Skills

같은 저장소

adjudication-sheets

kennethkhoocy/applied-micro-skills

Build human adjudication / hand-labeling sheets from LLM-pipeline data without evidence truncation. Use when: (1) preparing a CSV/Excel sheet for a human to rule on cases an LLM classifier or rater panel judged, (2) a labeler reports "there is no information to label from" or cells look empty in Excel, (3) excerpt columns cluster at one exact length (e.g. all 1,500 chars — a hard truncation cap). Covers: full rating-basis recovery, Excel 32,767-char cell cap, multi-line CSV mangling, ruling dropdowns, companion text files.

2026-07-221

adversarial-empirical-review

kennethkhoocy/applied-micro-skills

N-round adversarial review pipeline for empirical research output — the chain from data to LaTeX tables to a manuscript that cites them. A Claude drafter proposes minimal diffs, a deterministic mechanical battery gates every diff from a clean state with a regression gate, a Codex reviewer files check-backed critiques, and a blind judge panel decides residual disputes. Manual-invoke ONLY: trigger when the user explicitly runs /adversarial-empirical-review or names 'adversarial-empirical-review' / 'adversarial empirical review'. Do NOT auto-trigger on generic 'review my results', 'check my tables', or manuscript-editing requests. For prose-style refinement use style-emulation instead; this skill AUDITS WHETHER THE TABLES ARE CORRECT — that each number in the tables is what the analysis code computes, reproduces from the data, and is internally consistent. It is an empirical + code review: the manuscript is read only to resolve table numbering, and prose is not examined.

2026-07-221

annotator-input-parity-check

kennethkhoocy/applied-micro-skills

Before designing, training, or auditing ANY model that replicates human-annotated labels, audit the annotation protocol's INPUT — the exact document/evidence the human labelers consulted — and give the model that same input. Use when: (1) designing a classifier/LLM extractor whose target is a hand-coded label set, (2) a label-replication model shows low recall concentrated in a label subset and the diagnosis on offer is "the label's information is not in the features", (3) reviewers propose construct splits (e.g. "designation vs record-evident"), adjudication sittings, or per-domain stop rules to explain residual disagreement with gold, (4) validating an extraction pipeline against labels transcribed from a source document. Symptom of the underlying failure: elaborate theory accumulates to explain why gold is "partially unpredictable" when the model was simply never shown the document the annotators read.

2026-07-221

asyncopenai-concurrency-httpx-pool

kennethkhoocy/applied-micro-skills

Raise real concurrency in asyncio LLM batch scorers built on the OpenAI SDK (AsyncOpenAI, including OpenAI-compatible providers like DeepSeek). Use when: (1) raising an asyncio.Semaphore above ~100 produces no throughput gain, (2) a batch pipeline saturates near 100 in-flight requests despite a larger semaphore, (3) planning a high-concurrency campaign against a provider with no hard rate limit (DeepSeek v4-flash tolerates 2000+ in flight). Root cause: AsyncOpenAI's default httpx pool caps max_connections at 100, silently bottlenecking any larger semaphore — you must pass a custom http_client with httpx.Limits sized to the semaphore.

2026-07-221

cite-placement

kennethkhoocy/applied-micro-skills

Place pre-screened literature citations into a LaTeX or Word manuscript, or restyle the citations already in one. Three modes: (1) inline placement — inline \cite{}/\citet{}/\citep{} with a compiled references.bib, for author-date journals (APA, MLA, Harvard, Chicago author-date, IEEE, Vancouver); (2) footnote placement — full formatted \footnote{} or OOXML footnotes for legal and notes styles (Bluebook, OSCOLA, Chicago, APA, McGill) with Id./supra short forms; (3) restyle — convert existing footnote citations from one style to another. This skill is manual-invoke ONLY — trigger ONLY when the user explicitly runs /cite-placement or explicitly names the "cite-placement" skill. Do NOT auto-trigger on general citation, footnote, or reference requests.

2026-07-221

event-study-cars

kennethkhoocy/applied-micro-skills

Complete methodology for computing publication-quality cumulative abnormal returns with proper event-study test statistics, matching the robustness of Kaspereit's eventstudy2 for Stata. Covers dateline construction, event-date mapping, estimation and event windows, thin-trading adjustment, OLS with Theil prediction error correction, abnormal return computation, CAR/CAAR/AAR accumulation, boundary contamination guards, and common tests such as Patell, BMP, Kolari-Pynnonen, generalized sign, Wilcoxon, and GRANK-T. Use when the user mentions abnormal returns, event windows, market-model regressions, CARs, CAAR, AAR, eventstudy2, thin trading, trade-to-trade returns, or event-study test statistics.

2026-07-221

name	download-gated-pdfs
description	Download the actual PDF binary from bot-gated sites (taxpolicycenter.org, urban.org, SSRN-hosted mirrors, think-tank/publisher sites) via the Wayback Machine id_ URL form. Use when: (1) curl/WebFetch of a .pdf URL returns HTML instead of a PDF even with a browser User-Agent, (2) pypdf fails with "invalid pdf header: b'<!DOC'" or "EOF marker not found" on a freshly downloaded file, (3) Firecrawl can parse the PDF to markdown but you need the original file on disk (e.g., filing a reference copy).
author	Claude Code
version	1.0.0
date	"2026-07-15T00:00:00.000Z"

Download bot-gated PDFs via Wayback id_

Problem

Context / Trigger Conditions

curl -o file.pdf <url> succeeds but the file starts with <!DOC
pypdf raises invalid pdf header: b'<!DOC' or PdfStreamError: Stream has ended unexpectedly
Firecrawl scrape returns clean markdown for the same URL (its proxies get through), but Firecrawl does not return the binary — only parsed content

Solution

Request the file through the Wayback Machine's raw-content (id_) endpoint, which serves the original archived binary without rewriting:
```
curl -sL -A "Mozilla/5.0 ... Chrome/126.0 Safari/537.36" \
  "https://web.archive.org/web/<YYYY>id_/<original-pdf-url>" -o out.pdf
```
<YYYY> is any year likely to have a snapshot (e.g. publication year); Wayback redirects to the nearest capture. The id_ suffix after the timestamp is what requests the untouched original.

Verify the download with pypdf — a bot page fails immediately:

from pypdf import PdfReader
r = PdfReader("out.pdf"); print(len(r.pages), "pages")

If Wayback has no capture, fall back to: another mirror found via search (Exa/Firecrawl), or Firecrawl scrape for the parsed text when the binary is not strictly needed.

Verification

PdfReader opens the file and reports a plausible page count; first-page text matches the expected title.

Example

Notes

Government data hosts (e.g. ticdata.treasury.gov) are usually NOT gated — try direct curl first; Wayback is the fallback, not the default.
Wayback captures can be stale for frequently-revised documents; check the snapshot date if currency matters.
See also: the pdf skill (parsing/extraction after download) and firecrawl:firecrawl-scrape (parsed markdown when the binary is unnecessary).