webseclist-archive-references

## Source security Before handling third-party material, read and follow [the shared source-security policy](../../source-security.md). It governs sandboxed reading/conversion, trusted-policy reviews and validation before any source-derived action. These requirements apply to this entire workflow, including retries, imports and bulk work; no unsafe host fallback is permitted. # Archive the cited resources as Markdown and PDF ## Related sources Apply [research stories and related sources](references/related-sources.md) whenever researching, adding, updating, archiving or reviewing a reference. It defines the maintained relationship registry, source roles, per-source preservation, linked media and corpus migration. ## What this is This repository is a reading list: finalized Markdown files (`2006.md` through `2025.md`, plus `2016-17.md`) and explicitly marked preliminary files such as `2026-ai.md`, each containing links to web-hacking research. This skill turns those links into a durable local archive so a technique survives the article that described it going offline. For every cited resource that is not a video, the archive keeps: - a **Markdown copy** at `archived-references/md/<year>/<slug>.md`, with an attribution block naming the source, and - a **PDF copy** at `archived-references/pdf/<year>/<slug>.pdf` - our Markdown printed offline, or, when the source was already a PDF, the original PDF itself. The two format trees are parallel on purpose: the binary PDFs stay out of the Markdown tree, so diffs, sparse checkouts and exclusion rules can treat prose and print separately. `refslib/collections.py` owns the layout - never move a file between folders by hand. A recording is not archived - it is a third-party stream this repository cannot hold - but where the same research was also GIVEN AS A TALK, the reference records a link to it and how sure the archive is that it is the right one. See *The talk behind the research*. Every Markdown file's frontmatter follows **Google's Open Knowledge Format (OKF) v0.2** - `type`, `title`, `resource`, `tags`, `generated`, `verified`, `status`, `stale_after` and `sources` - with the archive's own fields kept as the custom keys the specification permits. `verified` is deliberately absent until the validation gate has run: under OKF, absence IS the honest statement. The local spec note is `references/okf-v0.2.md`; refresh it only if the specification moves. ```text year lists -> archive inventory -> acquisition -> Markdown -> PDF ``` ## Never download a program **No executable is ever fetched, on any operating system.** A program is not a document: the technique lives in the write-up, so downloading the binary gains the archive nothing and puts an executable on the maintainer's disk and into the content store. `refslib/kinds.py` gives these the kind `executable` before any other rule runs, and acquisition records them link-only. Two parts of the list are the ones people argue with, so they are the ones to remember: - **A format that looks harmless is still a program.** `.chm` reads as a help file and is a compiled, scriptable Windows binary. Judge a format by what it can execute, never by how mild its name sounds. - **An archive is refused too** - `.zip`, `.tgz`, `.7z`, `.iso` - because what is inside is unknown until it is unpacked, and unpacking is how an executable arrives without ever being named. Source TEXT is not an executable and stays archivable: `.py`, `.ps1`, `.cs` are read as text and never run. If a citation points ONLY at a download, archive the write-up that describes it and leave the download as a link. ## Antivirus deletes archived exploit text The corpus is exploit research, and a scanner reads it as what it describes. One deleted **233 objects out of the content store** - leaving 116 references with a perfectly good Markdown file on disk and nothing behind it, because the store holds the evidence while `archived-references/` holds the rendering. `verify` catches this as `missing store object`, and `index` lists every one of them on `store-gaps.md`; no other rule notices, because they all ask whether a document was PRODUCED. Recovery is a re-fetch, not a re-render: `check --missing-store` first, then `acquire --missing-store --force --refetch` for the sources that still answer and `wayback --missing-store` for those that do not. **A store gap is not a needs-work row.** Those references ARE archived - their Markdown and PDF are published and readable - and only the evidence behind the files is gone. Listing them as unfetched work put 1,011 fully archived references on `document-gaps.md` and buried the two that genuinely had no document. Keep the two reports apart: `document-gaps.md` answers *could we not get the document*, `store-gaps.md` answers *can we still show what the document was made from*. Exclude the store directory AND the repository from the scanner before a long run, or the next sweep quietly undoes it. ## The one rule **Never edit a year list.** Nothing here writes `20xx.md`. This workflow READS them and writes the archive under `archived-references/`. `verify` fingerprints every year file before a run and FAILS if one changed during it. If you want to add, remove or fix a link in a list, that is a hand edit to the list, done separately - archiving is never a required follow-up to it. A reference is filed under the source collection that cites it. That decision lives in one place, `refslib/collections.py`: a file cited by `2019.md` lands in `archived-references/md/2019/` (its PDF in `archived-references/pdf/2019/`), while `2026-ai.md` lands in parallel `2026-ai/` folders. A reference cited by more than one collection is filed under the first. Never move a file between year folders by hand. ### Preliminary collections Tracked files matching `YYYY-ai.md` are discovered automatically. They must contain exactly one ordered marker pair: ```text <!-- archived-references:start --> ... preliminary research bullets ... <!-- archived-references:end --> ``` Only links inside that boundary are archive input. Scoring notes, watchlists and dropped candidates outside it are intentionally ignored. Use `--collection YYYY-ai` with `check`, `acquire` and `pdf` to process only that provisional tree. ## The pipeline Every command is `python tools/references/refs.py <command>`. Only `check`, `check-browser`, `acquire`, `papers`, `images`, `wayback`, `insecure` and `transcripts` touch the network; `pdf` needs a browser but never a network. ```text python tools/references/refs.py harvest # cited URLs in the year lists python tools/references/refs.py sync # citations/folders, offline python tools/references/refs.py check # health of each URL [NETWORK] python tools/references/refs.py check-browser # only the walled rows [NETWORK] python tools/references/refs.py acquire # preserve, convert, render [NETWORK] python tools/references/refs.py papers # the article's own PDF [NETWORK] python tools/references/refs.py images # its figures, re-encoded [NETWORK] python tools/references/refs.py pdf # a PDF for each md [BROWSER] python tools/references/refs.py translate # anything not in English python tools/references/refs.py digest # a summary and tags per document python tools/references/refs.py index # regenerate the folder index python tools/references/refs.py verify # the offline gate ``` `digest` is the one step the tool cannot do for you, and the one a run most easily forgets: the archive can tell you a document is 40KB of prose, never that it is about a parser differential. Skip it and the reference is preserved but unfindable - the website has nothing to show and nothing to search. `papers` and `images` run BEFORE `pdf` and decide what it publishes. A reference that is already a PDF is copied; otherwise the publisher's own PDF of the article wins if `papers` found one; otherwise our Markdown is printed, carrying whatever figures `images` preserved. `images` never stores what it fetched - every image is decoded and re-encoded, which is what strips metadata, appended payloads and anything hidden in the low bits, and SVG is refused rather than rasterised. ### Ordinary sync, after a year list changed ```text python tools/references/refs.py check --prune python tools/references/refs.py check-browser python tools/references/refs.py acquire python tools/references/refs.py papers python tools/references/refs.py images python tools/references/refs.py pdf python tools/references/refs.py translate --prepare # then translate, then --apply python tools/references/refs.py index python tools/references/refs.py verify ``` For a new preliminary collection, keep the same sequence but scope the network and PDF stages with `--collection YYYY-ai`. ### Promote a preliminary collection when the real list arrives Keep both source files long enough to review their normalized URL sets: ```text python tools/references/refs.py compare 2026-ai 2026 ``` After a maintainer has accepted the finalized `2026.md` and deliberately removed `2026-ai.md` from the repository and app registry, reconcile the archive: ```text python tools/references/refs.py sync --prune --refile --prune-files python tools/references/refs.py pdf --force --collection 2026 python tools/references/refs.py index python tools/references/refs.py verify ``` `sync` is offline. Shared URLs retain their manifest record and stored bytes; their Markdown files move from `2026-ai/` to `2026/` with refreshed `cited_by` frontmatter. PDFs are regenerated from the refiled Markdown or copied again from their preserved original bytes, so none can retain a preliminary citation. URLs found only in the removed preliminary file leave the manifest, and `--prune-files` removes their published files. The content-addressed store is intentionally not garbage-collected: it is recovery evidence, not the public archive tree. `acquire` without `--force` only processes references that have no file yet, and `pdf` without `--force` only makes PDFs that do not exist yet, so this is cheap to repeat. **One `refs.py` command at a time.** Every command loads the whole manifest at start and saves it whole at exit, so the last writer wins: a long `pdf` run finishing after a batch of `import`s silently erased every import record while the files stayed on disk. Never run two commands concurrently - background a long run only when nothing else will touch the archive until it finishes. **Set `WEBSEC_REFS_STORE`** to a durable directory first (see below). **Do not stop at `acquire`.** A newly fetched foreign-language page renders into a complete-looking file its reader cannot read, so translation belongs in the same run that fetched it, and `verify` warns while any document is untranslated. ### The store: set it before every run ```text export WEBSEC_REFS_STORE=/some/durable/path # bash $env:WEBSEC_REFS_STORE = "D:\some\durable\path" # PowerShell ``` Every fetched byte is content-addressed into this store, and the Markdown/PDF in `archived-references/` reference it by hash. Without the variable the tool falls back to a git-ignored workspace cache, which `git clean -xfd` would destroy - so a page that is already gone online would be lost. `verify` warns whenever the store is that fallback. ## Making PDFs `pdf` gives every archived reference a self-contained PDF in the `pdf/<year>/` tree, parallel to its Markdown. - It prints **our archived Markdown**, converted offline to HTML by `refslib/makepdf.py`, using headless Chromium only inside the locked-down toolbox container. No host browser or host browser profile is used. The container has no network while printing, no third-party page script runs, and images become labelled links rather than remote `<img>`. - A source that was **already a PDF** (a whitepaper, a conference paper) is copied verbatim from the content store instead of re-rendered, so its own typesetting survives. - **A video is skipped** - a talk is not a page. `config.json -> pdf.skip_kinds` controls this, and `config.json -> layout.pdf_tree` names the `pdf/` tree. - It needs Docker for Markdown-origin PDFs. With no container runtime, PDF-origin sources are still copied and everything that must be rendered is skipped with a clear message. Do not work around that by launching Chrome or Edge on the host. `refs.py pdf --only <substring>` does one reference; `--force` remakes existing PDFs (after the Markdown changed). **`pdf --stale` UNSCOPED IS A CORPUS-WIDE REPRINT, and its selector is file mtimes.** One of the four staleness tests is `md newer than pdf`, so anything that touches Markdown timestamps without changing a byte - a branch switch, a fresh clone, a checkout - makes the whole archive look stale. A run meant to refresh the two documents `attribution --rewrite` had just re-published reprinted 536 PDFs before it was stopped, every one of them differing from its committed version only in the timestamps a PDF embeds. Scope it: `pdf --stale --only <substring>`, or `pdf --only <substring> --force` for a reference you just changed. If an unscoped run does get away, the manifest is the thing to check - `pdf` saves it in batches, so a run killed early leaves the entries untouched and `git checkout -- archived-references/pdf` restores the tree without disturbing new files, which are untracked. ## When a plain fetch is not enough The hard-won recovery routes from the upstream tool all still apply - the corpus is the open web either way. The route-by-route playbook, with its commands and traps, is in [references/recovery-routes.md](references/recovery-routes.md). Read it before working any reference the ordinary HTTP route does not deliver cleanly: - an app shell, sign-in redirect or waiting page -> the Docker browser ladder (`check-browser`); never a host browser - a video or PoC citation -> find, verify and archive its WRITTEN counterpart, then record the clip as `derivative` - a dead URL, or a citation pinned to a wall or shell capture -> a better Wayback snapshot (`wayback`), nearest its date; use `--replay-url` when known - a live landing page with a paper, code and slides -> preserve the full paper with `acquire --linked-document-url`; keep distinct sibling sources in the related-source registry. Use `--also-at` only for another location of the SAME document. The cited landing URL remains the identity. An ABSTRACT page needs this too and nothing will ask for it: it extracts well over the content floor and grades as research, so acquisition succeeds and the archive publishes the abstract - an expired certificate -> `insecure`, then `acquire --force`, then `images --insecure` for the figures on the same host - no useful CDX path -> bounded historical-path discovery with the pinned Docker waymore route (`historical-urls`), then verify each candidate - a Wayback-wrapped URL -> its kind is the CAPTURED page's kind, never the wrapper host's - a GitHub advisory, blob or issue -> the public API, never the rendered shell - captions, expired certificates, unreadable source PDFs -> the container sandbox (`transcripts`, `insecure`, `pdf-text`, `pdf-pages`) - a genuine scan or rendered deck -> page-image transcription with subagents Three rules from that playbook bind every session that touches the archive, so they stay here: - **`stored` MEANS A FILE WAS WRITTEN, NOT THAT IT IS THE DOCUMENT.** Every outcome count this tool prints is about the fetch, and the three ways an acquisition goes quietly wrong all report `stored`: | Symptom | What happened | Route | |---|---|---| | 3-8KB where a paper would be 100KB+ | the abstract page, not the paper | `acquire --linked-document-url` | | `title:` is `Preprint`, `Code`, `Paper` | the citation's link text, because the document declared no title | `decisions[url].title` | | prose reading `signicant`, `congur` | the font's ligatures were deleted | `acquire --force` (routes to poppler) | Read the character count `acquire` prints against what the source should hold, and open the first page of what it wrote. None of these reaches any gap report, because all three ARE archived - a document exists, and it is the wrong one or a damaged one. `verify` will not tell you either. - **A faulty capture goes on `document-gaps.md` the moment it is found.** Whenever a capture is discovered to be faulty - the manifest advertises a Markdown or PDF that is not in the tree, the file holds the wrong page (a parked or taken-over domain, a consent wall, a homepage or the site's chrome instead of the article), or the bytes behind a file are gone - record it before the session ends so it is listed for recapture. `document-gaps.md` is GENERATED state: never edit it by hand, and never leave the finding as a session note or an app-side workaround. The record is the manifest: set the entry's `content_gap` to `faulty capture: <what is wrong>; <remedy> (reported <date>)` - or, inside this workflow, let `verify`/`check` record it - then regenerate with `python tools/references/refs.py index` (`WEBSEC_REFS_STORE` set, as always) so the entry appears with its reason and remedy. This rule binds every session that touches the archive, not only this workflow: work on the year lists or on the browsing apps that stumbles over a bad capture files it here too. - **One research story can have several separately credited sources.** Existing same-line companions establish membership. Additional papers, parts, recordings and analyses belong in the maintained related-source registry described above; they inherit the story’s citation without rewriting historical year lists. Each written source has its own archive record and available copies. ## Importing documents obtained by hand Read [the manual import procedure](references/manual-imports.md) before importing hand-obtained documents, repairing an import, or changing its title. ## Grades, and what gets no file Every archived reference is graded in the manifest (this does NOT change its folder - the year does): - **research** - a document that carries technique. What the archive is for. - **records** - a record ABOUT a product rather than research: a CVE row, a vendor advisory, release notes, a package page, a talk with no transcript, a stub. Kept, and marked. - **excluded** - no file at all, with the reason recorded so the next run skips it: a broken capture (a bot wall, a consent gate, a 404 that answered), a URL that was never a research citation, or a maintainer decision. Two categories no rule can safely decide - a page that restates a source already archived, and a tool's usage page with no technique - are found with `refs.py report --candidates` and written by hand into `decisions` in `tools/references/overrides.json`. No rule overwrites a decision, and a skip is not fetched again. ## The generated folder ```text archived-references/ README.md the index, grouped by year (the only discovery route) document-gaps.md ONLY what could not be archived, with reasons and remedies store-gaps.md archived references whose stored bytes went missing excluded.md everything kept with NO document, and why manifest.json the record of record history.jsonl the append-only journal md/2006/ ... 2025/ finalized archived references as Markdown, per year md/2026-ai/ a bounded preliminary collection, when present pdf/<collection>/ PDF copies in a tree parallel to md/ ``` ### Always rebuild the work queue after archiving **No archive run is finished until `index` has run after the final archive mutation.** `document-gaps.md`, `README.md`, `excluded.md` and `store-gaps.md` are generated views of the final manifest *and the active content store*. Running `index` without `WEBSEC_REFS_STORE` set reads the empty fallback cache and reports the whole archive as store gaps. An index made halfway through a run is stale as soon as a later `acquire`, `wayback`, `import`, `translate`, `pdf`, pointer repair, fault report or manifest correction changes that state. Run this once nothing else will write the archive: ```text WEBSEC_REFS_STORE=<durable-store> python tools/references/refs.py index python tools/references/refs.py verify ``` `document-gaps.md` is the fetch queue and nothing else: a reference belongs there only while the archive could not get its document - a failed acquisition, a stub or metadata-only capture, or a filed faulty capture. Anything with a good published document stays off it, whatever else is wrong with the record. Use `index --prune-files` when a recovery changed a slug, replaced a faulty capture, refiled a collection or otherwise left generated files that the final manifest no longer claims. Never edit `document-gaps.md` by hand. If `verify` or a final review discovers a fault and you correct `content_gap` or any other manifest field, the correction is another archive mutation: run `index` again so `document-gaps.md` contains the actual remaining work before handing off. ## Translation is a stage of the pipeline, not an afterthought Check for foreign-language documents after every acquisition. Read [the translation procedure](references/translation.md) before preparing, applying or recovering translations; only genuinely foreign prose gets a translation. ## Naming the researcher Read [the attribution procedure](references/attribution.md) when extracting or publishing document-derived bylines. Credit only names supported by the source.