| name | scraper-runner |
| description | Run a scraper (_Helpers/scripts/scrapers/scrape_*.py) with the right flags. Covers scrape_skool (Skool programvideogames), scrape_official (odin-lang.org), scrape_zylinski (zylinski.se), scrape_gingerbill.py, scrape_newsletters.py, scrape_jakubtomsu.py, scrape_showcase.py, download_gists.py, download_odin_examples.py. All re-entrant. |
Run a scraper
Skill for re-running one of the project's scrapers.
All scrapers are re-entrant: an already exported file is skipped unless --force is passed.
Overview
| Scraper | Source | Output | Prerequisites |
|---|
scrape_skool.py | Skool "programvideogames" | odin-knowledge-base/courses/programvideogames/ | skool-cli + Playwright |
scrape-official.py | odin-lang.org/docs/ + awesome-odin | odin-knowledge-base/docs/official/ | requests + BeautifulSoup |
scrape-zylinski.py | zylinski.se blog | odin-knowledge-base/docs/karl_zylinski/ | requests + BeautifulSoup |
scrape-gingerbill.py | gingerbill.org RSS | odin-knowledge-base/docs/gingerbill/ | requests + BeautifulSoup |
scrape-newsletters.py | odin-lang.org/news/ | odin-knowledge-base/docs/newsletters/ | requests + BeautifulSoup |
scrape-jakubtomsu.py | jakubtomsu.github.io RSS | odin-knowledge-base/docs/jakubtomsu/ | requests + BeautifulSoup |
scrape-showcase.py | odin-lang.org/showcase/ | odin-knowledge-base/docs/showcase/ | requests + BeautifulSoup |
scrape_odin_changelog.py | github.com/odin-lang/Odin releases | odin-knowledge-base/docs/official/changelog/ | requests |
scrape_raylib_changelog.py | github.com/raysan5/raylib releases | odin-knowledge-base/docs/raylib/changelog/ | requests |
build_gitingest.py | local clones + gitingest CLI | odin-knowledge-base/gitIngest/ | gitingest |
download_gists.py | awesome-odin gist URLs | code/gists/ | None (stdlib urllib) |
download_odin_examples.py | Odin repo examples/ | code/examples/ | None (stdlib urllib) |
How
Re-scrape a source (idempotent skip)
python _Helpers/scripts/scrapers/scrape_official.py
python _Helpers/scripts/scrapers/scrape_zylinski.py
python _Helpers/scripts/scrapers/scrape_skool.py
-> Skip already-present files, only write new ones.
Force a full rewrite
python _Helpers/scripts/scrapers/scrape_official.py --force
python _Helpers/scripts/scrapers/scrape_zylinski.py --force
python _Helpers/scripts/scrapers/scrape_skool.py --overwrite-existing-lessons
-> Rewrite ALL files (useful after a KB refactor or an HTML format change on the source side).
scrape_skool - useful options
python _Helpers/scripts/scrapers/scrape_skool.py --lesson "entities state physics"
python _Helpers/scripts/scrapers/scrape_skool.py --skip-until 50
python _Helpers/scripts/scrapers/scrape_skool.py --download-video
python _Helpers/scripts/scrapers/scrape_skool.py --download-support-files
export SKOOL_EMAIL="your@email.com"
export SKOOL_PASSWORD="yourpassword"
python _Helpers/scripts/scrapers/scrape_skool.py
Exit codes:
0: success1: missing prerequisites (skool-cli not found, credentials missing)2: (reserved)
Prerequisites per scraper
scrape_skool
- Node.js: https://nodejs.org/
- skool-cli:
npm install -g skool-cli
- Playwright Chromium:
npx playwright install chromium
- skool-cli patches (anti-bot AWS WAF + retry goto):
- On Windows:
%AppData%\npm\node_modules\skool-cli\dist\core\browser-manager.js
- On Windows:
%AppData%\npm\node_modules\skool-cli\dist\core\page-ops.js
- yt-dlp (if
--download-video): any directory on your PATH is fine
- Skool credentials:
- Env var
SKOOL_EMAIL / SKOOL_PASSWORD, OR
- File
_Private/.config/skool_credentials.txt (gitignored)
- Windows: disable Defender firewall (Playwright Chromium otherwise fails with
ERR_NETWORK_ACCESS_DENIED)
Changelog scrapers (Odin / Raylib)
python _Helpers/scripts/scrapers/scrape_odin_changelog.py
python _Helpers/scripts/scrapers/scrape_odin_changelog.py --limit 50
python _Helpers/scripts/scrapers/scrape_raylib_changelog.py
python _Helpers/scripts/scrapers/scrape_raylib_changelog.py --check
Both are idempotent (release file already present + matching tag: in frontmatter = skip). Optional env var GITHUB_TOKEN raises the GitHub rate limit from 60 to 5000 requests/hour.
gitingest snapshot generator (Perplexity Space)
python _Helpers/scripts/scrapers/build_gitingest.py
python _Helpers/scripts/scrapers/build_gitingest.py --force
python _Helpers/scripts/scrapers/build_gitingest.py --check
python _Helpers/scripts/scrapers/build_gitingest.py --only metroidvaniaish odin-raylib-hot-reload-game-template
python _Helpers/scripts/scrapers/build_gitingest.py --include-skipped
The script reads _Helpers/config/gitIngest_repos.jsonc and invokes gitingest for each entry. Each snapshot is paired with a <id>.json sidecar (provenance metadata + max mtime of the source tree) so a re-run skips anything unchanged.
scrape-official / scrape-zylinski
pip install requests beautifulsoup4 markdownify lxml
Diagnostics
Typical successful output
=== With -config odinfmt.json ===
odinfmt : <path-to-odinfmt>/odinfmt.exe
Mode : WRITE
Target : odin-knowledge-base
Scanned : TBD (depends on your KB)
Modified : 0 <- already clean
Errors : 0
Common errors
| Error | Cause | Fix |
|---|
[ERR] skool-cli not found | npm install -g skool-cli not done | Run the command |
ERR_NETWORK_ACCESS_DENIED | Windows firewall blocks Playwright | Disable Defender or add an exception |
Sign in to confirm you're not a bot | YouTube rate-limit | Wait + add YOUTUBE_COOKIES_FILE |
HTTP Error 429/403 | YouTube anti-bot | Automatic exponential backoff (90/180/360s) |
[ERR] odinfmt exit=1 | Odin syntax error | Fix the source code |
Missing path to format | odinfmt called without argument | Add -stdin or a file |
Anti-patterns
- Do NOT delete
_Helpers/scripts/fixes/fix_mojibake.py even if the KB is clean: it is a safety net if a new scrape introduces mojibake.
- Do NOT disable re-entrancy by adding an automatic
force. Idempotence is valuable for large KBs (Skool courses can run into the hundreds of lessons).
- Do NOT scrape in
--force mode without running --check first to estimate the scope of the changes.
