learn-sync

name	learn-sync
description	Sync, ingest, download, or refresh University of Waterloo Learn (Brightspace/D2L) course materials into the local vault. Use when the user wants to pull/update/download course content, ingest a semester or specific course, refresh Learn, set up a new term, or back up everything from their courses.

learn-sync — pull & refresh UW Learn into the vault

This repo ingests everything from UW Learn via the raw D2L JSON API by reusing a logged-in browser session (agent-browser). It is term- and structure-agnostic — it discovers courses from enrolment and mirrors whatever module structure each course uses. Never hardcode course/term/layout.

Always use the venv interpreter: PY=.learn/.venv/bin/python.

Workflow

1. Harvest the session — auth opens an agent-browser window; the user signs in to Learn once (WatIAM + Duo) there. If a step reports "not logged in" / HTML instead of JSON, the session expired — tell the user to re-run auth and sign in again in that window.

   $PY .learn/learn.py auth
   

   Success = whoami: OK as <name> and wrote .learn/session.json (never print its contents).

2. Calibrate once (only if .learn/calibration.json is missing): $PY .learn/learn.py calibrate.

3. Sync (incremental — safe to re-run weekly; the manifest.json ledger skips existing files):

   $PY .learn/learn.py sync                 # the latest enrolled term (current courses) — DEFAULT
   $PY .learn/learn.py sync --label 2B      # name the folder '2B_<Term>' (else just '<Term>')
   $PY .learn/learn.py sync --only MSE232,MSE253   # a subset by course-code prefix
   $PY .learn/learn.py sync --term 1265     # a specific enrolled term by code
   

   sync runs the full pipeline: ingest → render course outlines → convert to markdown → build INDEX maps → audit.

4. Verify completeness:

   $PY .learn/learn.py verify        # writes .learn/AUDIT.md
   

   Then read .learn/AUDIT.md and each course's _meta/content_map.json (path set on every row = every D2L topic resolved to a real file/link). Report any flags honestly.

What gets pulled

Content files (PDF/PPTX/DOCX/XLSX/IPYNB/Rmd/HTML packages) + their markdown twins, external links, assignments (dropbox + materials-in-content + news-attached), announcement attachments, discussions, quiz metadata, grades. Each file lands under <course>/content/<module-or-week>/….

Discover what's available

$PY .learn/learn.py terms        # all enrolled terms + courses (term codes auto-detected)

Edge cases

• Past (completed) terms: UW's student API only lists the active term (403s on others). To pull a past course, get its org-unit id from the course's /d2l/home/<id> URL and run $PY .learn/learn.py sync --courses <id>.
• Quiz question text is not in the API (metadata only). Embedded streaming video and external resources (GitHub Classroom, eTextbooks) are surfaced as links, not files. These are platform limits, not bugs — handle gracefully and tell the user.
• Never print or commit .learn/session.json (live cookies).

The browser-driven steps here (session harvest via auth, outline rendering) run on agent-browser (npm install -g agent-browser && agent-browser install). Load the agent-browser skill for the mechanics or any browser automation beyond the API.