scrape-polls

# scrape-polls Pull newly-published Gallup and Maskína polls into the kosningaspá data set, with browser-mediated verification of the parts the article-body regex scraper gets wrong. ## When this skill applies The user wants the post-election polling data refreshed. Cadence is monthly — Gallup publishes Þjóðarpúls around the start of the month (reporting the *previous* month's field period), Maskína publishes irregularly through the month. Run this skill whenever a new poll might be available — discovering "no new polls" in 30 seconds is fine; missing one for a month is not. ## Working directory assumption All file paths are relative to the **kosningaspá repo root** (the directory containing `R/`, `Stan/`, `data/`). Run shell commands from there. If you're invoked from somewhere else, `cd` in first; if you're already inside a worktree of it, stay put. ## What this skill does — and doesn't **Does:** - Discovers new poll articles since the last tribble entry, primarily via RÚV's canonical poll list (a JSON object embedded in `ruv.is/kosningar/kannanir-a-landsvisu`). - Verifies each candidate by reading the source article in Chrome — type-of-poll, field-period midpoint, sample size. - Edits the `tribble()` in `R/scrape_polls.R::get_hardcoded_polls()` — the durable record. - Regenerates `data/post_election_polls.csv` from the updated tribble. **Does not:** - Refit the polling-watch Stan model (`R/fit_polling_watch.R` — minutes-long; user decides when). - Regenerate the metill-platform JSON (separate human step in `~/metill-platform`). - Commit anything to git (user reviews the diff first). Hand the next steps off to the user with a clean summary at the end. ## Source of truth (read this before editing anything) The durable record of post-election polls is the `tribble()` literal inside `get_hardcoded_polls()` in `R/scrape_polls.R`. The CSV at `data/post_election_polls.csv` is **regenerated from that tribble each time the script runs**. If you write only to the CSV, your changes vanish on the next run. Always edit the tribble. Treat the CSV as build output, not source. ## Where the data actually lives Four reliable sources. Use whichever is freshest: 1. **RÚV's canonical poll list at `https://www.ruv.is/kosningar/kannanir-a-landsvisu`.** The page embeds a `window.pollsArray` JavaScript variable containing every poll RÚV has on file, with per-party `ratio` to 6 decimal places, the firm name, the canonical Icelandic title (e.g. "Þjóðarpúls Gallup mars 2026", "Maskína 23. mars 2026"), and a publication date. **The cleanest source when it's up to date.** Read it via `javascript_tool` running `JSON.stringify(window.pollsArray)`. 2. **Gallup's own site at `https://www.gallup.is/frettir/` and the Þjóðarpúls PDF.** Gallup typically publishes the Þjóðarpúls on gallup.is several days before RÚV picks it up — in May 2026 the lag was ≥ 8 days. The news listing at `gallup.is/frettir/` shows the latest articles; each Þjóðarpúls article links through to a PDF viewer page (`gallup.is/<slug>/`) whose iframe `src` points at the actual PDF. **The PDF is authoritative for methodology** — it states the field period (e.g. "1. - 29. apríl 2026"), heildarúrtak, and response rate that the RÚV blob doesn't carry. Fetch it with `curl` and read it directly. **The PDF drops sub-3% parties from its table** (Píratar, every reading since June 2026) — recover those from source #2b below, not by guessing from the prose. 2b. **Gallup's Looker back-end — the best source, and fully scriptable.** Verified working 2026-09-02. It carries *every* party to 2 dp, including the ones the PDF and RÚV drop, for every month back to 2021. Four steps, all `curl`: ```bash # 1. the page embeds a SIGNED Looker SSO url in its iframe src curl -sL 'https://www.gallup.is/nidurstodur/thjodarpuls/fylgi-flokka-til-althingis/' -o page.html # 2. follow that url with a cookie jar -> authenticates an anonymous embed session curl -sL -c cj.txt -b cj.txt "<iframe src>" -o looker.html # 3. read the csrf-token meta from looker.html, then get the dashboard definition curl -s -b cj.txt -H "X-CSRF-Token: $CSRF" \ 'https://gogn.gallup.is/api/internal/core/4.0/dashboards/gallup_dw::fylgi_til_althingis' # 4. run the query id from the "Þróun á fylgi flokka til Alþingis" element curl -s -b cj.txt -H "X-CSRF-Token: $CSRF" \ 'https://gogn.gallup.is/api/internal/core/4.0/queries/<query_id>/run/json' ``` Rows are keyed by `gallup_althingisfylgi.response_date` (the reading's **month-end**, e.g. `2026-08-31` for the 4.–31. ágúst fielding — *not* the tribble's midpoint date) with `…hlutfall` giving per-party percentages. Only the `/api/internal/` prefix works; `/api/4.0/` returns 401. Use this to recover dropped parties **and** to reconcile the whole Gallup block in one pass. Note the old claim that this dashboard was "gated and cross-origin" was wrong. **PDF hosting paths (important):** - **May 2025 onward:** `cdnx.gallup.is/media/documents/Puls_MMYY_Fylgi_flokka.pdf` — deterministic and URL-guessable (`MMYY` = publication month, e.g. `0526` = May 2026 PDF reporting April 2026 field period). - **Pre-May 2025:** `gallup.is/documents/<opaque_id>/Puls_MMYY_Fylgi_flokka.pdf` — `<opaque_id>` is not enumerable; find it via the iframe `src` on the PDF viewer page, don't try to guess. For pre-May-2025 polls where this PDF can't be located, the next-month PDF's "comparison" column provides secondary confirmation. 3. **Maskína's own site at `https://maskina.is/frettir/` and the Fylgi-Althingi PDF report.** Maskína publishes a PDF report for every **regular monthly Alþingi reading** at `maskina.is/wp-content/uploads/YYYY/MM/YYYY-MM-Fylgi-Althingi_Maskinuskyrsla.pdf` (sometimes with `-DD` date suffix when multiple). This is the analogue of Gallup's PDF and the authoritative source for `svarendur` and exact field period. Discover the URL via the news listing at `maskina.is/frettir/` (each monthly poll gets its own news article whose body links to the PDF — paginate via `/frettir/page/N/`) or via `WebSearch site:maskina.is "<month> <year>"`. **Important: regular monthly vs commissioned side-publications.** Not every poll that appears under the "Maskína" brand goes through this channel. Maskína occasionally publishes commissioned polls for partner outlets — Vísir, Sýn (Stöð 2), Sýn's Kryddsíldarþáttur, etc. — that **don't** get their own Maskína-authored article or PDF, only the partner outlet's write-up. Examples: the 2025-11-10 reading was for Vísir; the 2026-02-10 one (published 24 Feb) was for Sýn. Both appear on the dashboard as monthly entries but Maskína never published a self-authored article or PDF for them. If `WebSearch site:maskina.is` returns nothing for a known poll date, assume it's a commissioned side-publication. Take the field period and values from the downstream article (source #4), but take `svarendur` from the dashboard sum below when the article doesn't print it, and never estimate n. **The definitive PDF list is WordPress's media API**: `curl 'https://maskina.is/wp-json/wp/v2/media?search=Fylgi-Althingi&per_page=100&_fields=source_url,date'`. It lists every Maskínuskýrsla ever uploaded (upload folder ≠ reading month: `2026/07/2026-06-…`). If a reading is not listed there, it is media-only and has no PDF, so stop probing URLs. The PDFs are PowerPoint exports whose `pdftotext` output is fragmented glyph by glyph, so `-layout` and `-raw` are both garbled. Strip **all** whitespace from the plain `pdftotext` output, then regex the joined string: `Könnuninfórframdagana…` / `Súfyrrivarframkvæmddagana…` for the field period, and `Gildsvör([\d.]+)100,0%` for `svarendur`. Table rows read `<Party><count><pct>%±<ci><pct>%`; the repeated pct pins down where the count ends. There's also a public dashboard at [`maskina.is/fylgi-flokka-a-althingi/`](https://maskina.is/fylgi-flokka-a-althingi/): a Tableau Public viz (`public.tableau.com/views/FylgiFlokka-heimasa/Njastamling?:embed=y&:showVizHome=no`) covering every Maskína reading back to Nov 2021, including commissioned ones. There is no `.csv` export (`allowDataAccess: false`), **but it is the one source of exact `svarendur` for media-only readings**. Pick the month in `Veldu mælingu (Bakgrunnur)` (the options are DOM elements), then hover each segment of the `Allir` bar with **real** mouse hovers, since Tableau ignores synthetic JS events. Read `document.querySelector('.tab-tooltipContent').innerText` (`Hlutfall` + `Fjöldi`) after each hover. The segments are sorted by share, so each centre is at `x0 + width·(cum% + pct/2)/100`. **Σ `Fjöldi` = `Gild svör` exactly** (exact on the 3 readings with an independently published n, 2026-09-23). Hover the tail for Lýðræðisflokkurinn/Ábyrg framtíð slivers in early 2025. The dashboard's free-text methodology line is **not** reliable: for 2026-08 it says 3.460, while the counts and press both give 3.172. **Vísir's Infogram embeds are the fastest discovery and reconciliation channel.** Vísir edits the same two charts in place every month. `e.infogram.com/120df4ad-46bb-41ce-be41-e7f86616dbde` (bar chart, every reading since Apr 2023, labelled by *publication* date) and `e.infogram.com/68399b2f-2841-4338-91fe-7c17c3b3317c` (line chart, one row per month) both carry 1-dp values in `window.infographicData`, which you can pull with `curl`. Diff the bar chart against the tribble on every ingest to catch readings that were never ingested; that diff found 11 on 2026-09-23. It has had one-off slips (2025-08 B 6,3 against the PDF's 6,5), so a PDF or the dashboard wins on conflict. 4. **The downstream article on RÚV (`ruv.is/frettir/innlent/...`), Vísir (`visir.is/g/...`), or mbl.is (`mbl.is/frettir/innlent/...`).** Useful when the firm's own publication is delayed or you only have an indirect lead. Article bodies sometimes disclose `svarendur` and field period directly; when paywalled, per-party percentages can still be cross-checked via search snippets. The R scraper (`scrape_polls.R::scrape_gallup` / `scrape_maskina`) historically did article-body regex against the tag pages on RÚV and Vísir. **Both tag pages now lazy-load via JavaScript and `rvest` sees nothing**, so the scraper silently returns empty results. Don't rely on it for discovery; treat it as a deprecated fallback. ## Procedure ### 1. Find the gap Read `R/scrape_polls.R` and note the latest dated entry per firm in `get_hardcoded_polls()`. Quick way: ```bash grep -E '^\s+"[0-9]{4}-' R/scrape_polls.R | awk -F'"' '{printf "%s %s\n",$2,$4}' | sort ``` That's the lower bound for what to look for. ### 2. Pull the canonical poll list from RÚV In Chrome, navigate to `https://www.ruv.is/kosningar/kannanir-a-landsvisu`. The page hydrates `window.pollsArray` after a brief delay (~2 seconds), so wait before reading or you'll get an empty array. Note that top-level `await` is not allowed in the eval context — use a promise: ```javascript new Promise(r => setTimeout(() => r(JSON.stringify(window.pollsArray)), 2500)) ``` Use `javascript_tool`. This returns a clean JSON list of every poll RÚV has on file (~50 entries), each with `identifier`, `text`, `shorttext`, `type`, `date`, `ispoll`, `calculator`, plus per-party `ratio` to 6 decimal places. Filter to entries dated *after* the last tribble entry per firm, and where the firm is **Gallup** or **Maskína** (skip Prósent and Félagsvísindastofnun). If nothing's newer than the existing tribble in `pollsArray`, **don't immediately report "no new polls"** — RÚV can lag Gallup's own publication. Also check `gallup.is/frettir/` for a Þjóðarpúls article newer than the last tribble entry (the news listing is server-rendered, so `get_page_text` works); if one exists, follow its "Skoða PDF" link to find the PDF URL embedded in an iframe, then fetch and read the PDF for the values and methodology. Skip the gallup.is check if the user has confirmed nothing's been published. ### 3. Verify each candidate against its source article The JSON entries don't include the article URL. To find each article: - **Gallup:** browse `https://www.ruv.is/frettir/tag/thjodarpuls-gallup` in Chrome (this also lazy-loads — wait ~2 seconds before reading the page text). The article whose headline matches the JSON entry's `text` field is the one you want. - **Maskína:** browse `https://www.visir.is/t/2296` similarly. Headlines match the JSON entry's `shorttext`. Once you've found the article, navigate to it and confirm four things from the body: | What | Where to look | Why | |---|---|---| | Type of poll | Headline + lede | Skip Reykjavík/Akureyri/Hafnarfjörður municipal polls, single-issue polls, leader-favourability polls — only national party support feeds the model. National polls always say "á landsvísu" or list multiple national parties; municipal ones name the city or a mayoral candidate | | Collection dates | "Könnunin var framkvæmd dagana X til Y" / "fór fram dagana X.–Y." / "kannað dagana X til Y" | Take the **midpoint** as the date. Publication date is wrong; field-period midpoint is what the model expects | | Sample size | See "Sample-size convention" below | Different convention per firm | | Per-party percentages | Confirm article body matches the JSON blob | Cross-check is cheap and catches the rare cases where RÚV's canonical list is mid-update or wrong | ### 4. Sample-size convention (matches the existing tribble — important) The existing rows in `get_hardcoded_polls()` use **different conventions per firm**, and you must match them so the column stays meaningful: - **Gallup:** use `heildarúrtak` (gross sample, e.g. `9958`, `10000`, `10887`). The methodology line in the article reports both gross sample and response rate; take the gross sample. If the article omits methodology entirely (recap articles, mid-month follow-ups), default to **`10000`** — it's the median of the recent rows and the model isn't sensitive to ±5% on the contacted count. - **Maskína:** use `svarendur` (response count, e.g. `2617`, `1786`, `886`). The Maskína article body reports the number of people who took a position on a party — that's the right column. Do not switch conventions row-by-row, even if you think one is "more correct" — the entire prior data set sets the convention and the polling-watch model was fit against it. ### 5. Field-period midpoint rules | Situation | Use this date | |---|---| | Single field period like "1.–14. apríl 2026" | Linear midpoint (April 7) | | Field period spans two months ("28 March – 11 April") | Linear midpoint (April 4) — not the start, not the publication date | | Composite Maskína poll pooling 2–3 sub-surveys | Linear midpoint of [first sub-period start, last sub-period end]. Don't average the per-sub-period midpoints — the existing rows use the span midpoint | ### 6. Update the tribble Open `R/scrape_polls.R` and find `get_hardcoded_polls()`. Insert verified rows. The columns are: ``` ~date, ~fyrirtaeki, ~n_total, ~S, ~D, ~B, ~C, ~M, ~F, ~P, ~V, ~J, ``` Match the existing column alignment when adding rows — R doesn't care about whitespace, but humans do, and `git diff` reads much cleaner when columns line up. Look at the surrounding rows for the spacing pattern. Two structural notes: - The existing tribble groups Gallup and Maskína into separate blocks. Add to the appropriate block; don't interleave. - If a party isn't reported in the article (Maskína sometimes omits parties below 2%), use the value the JSON blob gives — `Other` is computed automatically as `100 − sum(named)`, so a literal `J = 0.0` for "didn't report" is fine. - Don't enter "Annað" / "Other" explicitly; the script appends a remainder row. ### 7. Update `data_sources.md` The provenance audit trail at `data_sources.md` (repo root) is the durable record of *where each tribble row came from*. It must stay in sync with the tribble — every row you added or changed in step 6 needs a matching row appended to the relevant pollster's table in `data_sources.md`. Row format: ```markdown | Date | n_total | Source | Notes | |---|---|---|---| | 2026-04-15 | 10,484 | [PDF](https://www.gallup.is/documents/2237/Puls_0526_Fylgi_flokka.pdf) ([RÚV](https://www.ruv.is/frettir/innlent/2026-05-08-thjodarpuls-...)) | Fielded 1.–29. apríl 2026; úrtak 10.484, svh 43% | ``` Conventions for the row: - **Date**: same as the tribble (midpoint of fielding, YYYY-MM-DD). - **n_total**: same as the tribble, with comma thousands separator (e.g. `10,484`, not `10484`). - **Source**: primary URL first, secondary in parentheses if available. - Gallup → `[PDF](gallup.is/documents/...)` first, `[RÚV](ruv.is/frettir/innlent/...)` second - Maskína → `[PDF](maskina.is/wp-content/uploads/...)` when a Maskínuskýrsla exists, else `[Vísir](visir.is/g/...)`; add `[maskina.is dashboard](https://maskina.is/fylgi-flokka-a-althingi/)` in parentheses when n came from the dashboard sum - Use `_not found_` only as a last resort - **Notes**: terse, Icelandic field-period phrasing.