openclaw/notcrawl

Latest recorded source activity: September 13, 2026 at 04:51

SkillsMP catalog refreshed: November 5, 2026 at 19:57

skills collected: 1

GitHub stars: 139
GitHub forks: 20

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/openclaw/notcrawl.

Repository README

Repository README source: README.md

Read the source README

Expand full README

# notcrawl 🗞️ — Your Notion memory, on disk [![CI](https://img.shields.io/github/actions/workflow/status/openclaw/notcrawl/ci.yml?branch=main&style=flat-square&label=ci)](https://github.com/openclaw/notcrawl/actions/workflows/ci.yml) [![GitHub release](https://img.shields.io/github/v/release/openclaw/notcrawl?style=flat-square)](https://github.com/openclaw/notcrawl/releases/latest) [![Platforms](https://img.shields.io/badge/platforms-macOS%20%7C%20Linux-blue?style=flat-square)](https://github.com/openclaw/notcrawl/releases/latest) [![License](https://img.shields.io/github/license/openclaw/notcrawl?style=flat-square)](LICENSE) [![Homebrew](https://img.shields.io/badge/Homebrew-openclaw%2Ftap-orange?style=flat-square&logo=homebrew)](https://github.com/openclaw/homebrew-tap/blob/main/Formula/notcrawl.rb) <img src="docs/notcrawl_banner.jpg" alt="notcrawl banner"/> `notcrawl` mirrors a Notion workspace into local SQLite and normalized Markdown. It is for people and agents that need to search, query, diff, or share workspace history without depending on the Notion UI. SQLite is the canonical archive. Markdown is the durable human-readable export. ## Install Homebrew is the smallest install path on macOS and Linux: ```sh brew install openclaw/tap/notcrawl ``` GitHub Releases also provide signed and notarized macOS archives (macOS 13.0 or newer), Linux archives, and `.deb` and `.rpm` packages. Download the appropriate file from the [latest release](https://github.com/openclaw/notcrawl/releases/latest). ## Quick start With Notion Desktop installed and opened at least once: ```sh notcrawl sync --source desktop notcrawl search "launch plan" notcrawl export-md ``` The sync reads a snapshot of Notion's local cache. Search uses the SQLite FTS5 index, and `export-md` writes the normalized archive under `~/.notcrawl/pages`. Run `notcrawl doctor` if the desktop cache is not found. ## Choose a source | Source | Use it for | Setup | |---|---|---| | `desktop` | Fast, local ingestion of pages Notion Desktop has cached | None beyond Notion Desktop | | `api` | Pages, blocks, users, comments, databases, and rows shared with an integration | Set `NOTION_TOKEN` | | `notion-mcp` | Targeted repair of incomplete pages through the Notion app connected in Codex | Configure the experimental connector in `config.toml` | The official API is the stable remote integration: ```sh export NOTION_TOKEN="secret_..." notcrawl sync --source api ``` For a slow or stalled sync, add `--verbose` to that invocation: ```sh notcrawl sync --source api --verbose ``` Verbose diagnostics go to stderr and report source phases, elapsed time, and counts. Official API sync also reports request attempts, endpoint classes, numeric HTTP statuses, and retry delays. No credentials, headers, request/response bodies, raw URLs, cursors, page identifiers, or upstream error text are logged. Warnings become counts and failures use a fixed message in verbose mode. Normal stdout is unchanged and may still contain local paths; without `--verbose`, the existing progress, warnings, and errors are unchanged. Desktop and MCP sync report source phases and counts, not per-request API traces. Official API requests have a 60-second timeout. Read-only requests retry transient failures, including request timeouts, up to four total attempts while the sync context remains active. Canceling the sync or reaching its caller deadline stops retries. Notion MCP can repair known incomplete pages, fetch a page by ID or URL, or run a bounded workspace search: ```sh notcrawl sync --source notion-mcp --page PAGE_ID notcrawl sync --source notion-mcp --query "launch plan" --limit 25 ``` Without `--page` or `--query`, the MCP source retries known Desktop pages with missing cached content and API pages with incomplete block sync. It does not enumerate the entire workspace. Empty connector responses leave archived content unchanged and remain eligible for retry. The transport uses Codex authentication through the experimental ChatGPT apps gateway. Targeted repair retries gateway HTTP 429/503 responses up to five times, honoring `Retry-After` with a 60-second maximum delay and stopping when the sync context is canceled. MCP sync selects the connector's named `search` and `fetch` actions, preferring the shortest qualified name when auxiliary variants are also advertised. Descriptions mentioning search or fetch do not make a tool a page action. ## Work with the archive `notcrawl tui` opens a three-pane terminal browser for workspaces, teamspaces, pages, and databases. It supports keyboard and mouse navigation, filtering, sorting, local refresh, opening or copying the selected Notion URL, and local/remote state in the footer. Markdown exports preserve archived HTTP(S) destinations for bookmarks, embeds, link previews, tweets, and videos, alongside available text or captions. Existing archives can be re-exported without resyncing. These blocks remain links; export does not fetch third-party page or media content. API simple tables render as Markdown tables from their archived cell values. Tables without column headers receive an empty Markdown header; cell whitespace is normalized. Existing archives can export these tables without resyncing. Run `notcrawl maintain` to rebuild search entries for previously archived table cells. API text preserves word and punctuation boundaries across inline formatting changes. Run `notcrawl sync --source api` and then `notcrawl export-md` to repair text previously archived with extra spaces between formatted spans. API code blocks retain indentation and line breaks, with Markdown fences that safely contain embedded backticks. The same resync and export commands repair previously flattened code blocks. Database rows can be exported separately from the Markdown archive: ```sh notcrawl databases notcrawl export-db --database DATABASE_ID --format csv --output roadmap.csv notcrawl export-db --all --dir exports/csv ``` The main commands are: | Command | Purpose | |---|---| | `init` | Write a starter config | | `doctor` | Check config, SQLite, the desktop cache, and token presence | | `status` | Show archive counts, last sync time, and database/WAL size | | `report` | Summarize recent page, database, space, and comment activity | | `maintain` | Rebuild FTS, optimize indexes, and optionally run `VACUUM` | | `sync` | Ingest `desktop`, `api`, `notion-mcp`, or all enabled sources | | `tap` | Alias for `sync --source desktop` | | `export-md` | Render normalized Markdown from SQLite | | `databases` / `export-db` | List and export crawled Notion databases | | `search` | Search page and comment text through FTS5 | | `tui` | Browse archived pages and databases | | `sql` | Run read-only SQL against the archive | | `publish` | Export SQLite tables and Markdown into a git share repository | | `subscribe` / `update` | Merge current or historical git share snapshots | | `metadata` / `status --json` / `doctor --json` | Emit crawlkit control data for automation | Run `notcrawl --help` for the full command summary. `sql` accepts one SELECT, WITH, or PRAGMA statement and opens the archive read-only. Leading comments, multiline queries, and quoted punctuation are supported. It never creates or migrates the archive; run `sync` first to prepare a new archive or upgrade an older schema. ## Share an archive Git share mode publishes compressed JSONL table snapshots and normalized Markdown. Another machine can subscribe and search the archive without Notion credentials. `publish --tag NAME` creates an immutable checkpoint. `subscribe` and `update` merge snapshots without deleting local-only rows by default; `--restore` requests exact replacement, and `--retain-revisions` saves replaced local payloads. Snapshot imports preserve signed 64-bit integer literals exactly, including ordering values above 2^53; out-of-range integer literals fail the import without changing the archive. Decimal and exponent notation retain floating-point behavior. An archive with no Markdown pages can still be published and subscribed to. Secrets are not included in Markdown or git share snapshots. Structured signed-file URL credentials are removed from these outgoing projections; ordinary link parameters and private local raw recovery payloads are retained. ### Troubleshooting export failures `export-md` and `publish` stop with an error if protected provider JSON is malformed or a signed file URL cannot be safely sanitized. They do not skip the record or remove its raw payload from the local archive. Before investigating, stop archive writers and make a consistent, private SQLite backup, including any uncheckpointed WAL data. Keep the original archive and backup unchanged. Use read-only queries on the backup to inspect JSON validity and record counts; do not print or share raw payloads, signed URLs, or credentials in diagnostics or issue reports. If the source is still available, use a separate configuration with new `db_path`, `cache_dir`, `markdown_dir`, and `[share].repo_path` locations to reimport it. Select the intended source explicitly. Desktop cache coverage is limited, and a normal resync is not guaranteed to repair historical records. Preserve the original archive for records the source can no longer supply. A failed export can leave partial or stale files in the Markdown or share directory; output changes are not rolled back atomically. Do not manually commit, push, or distribute that output. Retry into separate output locations after resolving the source problem, and require a successful export before using the result. ## Configuration `notcrawl init` writes `~/.notcrawl/config.toml`. The default data paths are: | Data | Path | |---|---| | SQLite archive | `~/.notcrawl/notcrawl.db` | | Desktop snapshots | `~/.notcrawl/cache` | | Markdown archive | `~/.notcrawl/pages` | | Git share checkout | `~/.notcrawl/share` | See [`config.example.toml`](config.example.toml) for every setting. To limit future Desktop ingestion to selected workspaces, set `space_ids` under `[notion.desktop]` in your config: ```toml [notion.desktop] enabled = true space_ids = ["01234567-89ab-cdef-0123-456789abcdef"] ``` An empty list (the default) includes all cached workspaces. IDs match regardless of case or UUID hyphens. Find known IDs with `notcrawl sql "SELECT id, name FROM spaces"` in an existing archive. The filter applies to `sync --source desktop`, `tap`, and the Desktop phase of `sync`. Rows without a workspace ID are skipped when a list is set. The list filters workspace content, not the complete archive: existing rows and their Markdown remain, shared user metadata is still imported, and cache snapshots still contain the complete local Notion database. API and MCP sync are unaffected. For a new Desktop-only archive, use separate `db_path`, `cache_dir`, and `markdown_dir` settings and disable the API and MCP sources. Interactive terminal runs check for a newer release once per day. `notcrawl check-update` checks immediately; set `NOTCRAWL_NO_UPDATE_CHECK=1` or `CRAWLKIT_NO_UPDATE_CHECK=1` to disable the passive check. ## Safety model Desktop mode is read-only. It snapshots Notion's local SQLite database before reading it and never writes to Notion application storage. Cache coverage is opportunistic, so missing rows are not treated as deletions; explicit Notion tombstones still retire records. Markdown marks pages whose bodies were not cached so the API or MCP source can fill them later. API mode uses the official Notion API and stores raw payloads alongside normalized rows so exports can improve without another crawl. A successful, nonempty full API discovery retires API pages omitted from both page search and database/data-source queries, along with their API blocks and comments. Retired pages leave normal queries, search, and the next Markdown export; live Desktop or MCP copies remain available. Rediscovery restores API content. Discovery with no pages or rows (even if databases are found), failed or cancelled syncs, incomplete block walks, and targeted syncs do not retire unseen pages. This is a mirror-retention policy: Notion search can temporarily omit accessible pages, so retirement does not prove permanent deletion. Tombstones and local recovery rows remain in SQLite. Database containers themselves are not retired by omission. Notion MCP mode is read-only and targeted. It reads the Codex bearer credential at request time, never stores it, resolves only the connected Notion search and fetch tools, and strips signed URL credentials before persisting connector Markdown. Credentials are sent only to the configured HTTPS ChatGPT apps gateway. The gateway and Codex auth-file format are experimental contracts. ## Architecture `notcrawl` uses [`crawlkit`](https://github.com/openclaw/crawlkit) for config paths, SQLite helpers, snapshot packing and import, git-backed sharing, output formatting, status payloads, and the terminal explorer. Notion API and Desktop parsing, schemas, Markdown rendering, and FTS content remain in this repository. See [`SPEC.md`](SPEC.md) for the data model and archive contracts. Maintainers can find release packaging and verification in [`docs/distribution.md`](docs/distribution.md). ## Development Go 1.27.1 or newer is required. Builds and CI select the patched Go 1.27.3 toolchain through `go.mod`. ```sh make build make test make check ``` `make check` runs the dependency, formatting, vet, dead-code, test, smoke, release-config, and snapshot gates used by CI. See [`CONTRIBUTING.md`](CONTRIBUTING.md) before sending a change. ## License MIT. See [`LICENSE`](LICENSE).

Loading the repository README…

Skills in this repository

Office Clerks, General

1 occupation category · 100% classified

skill

occupation

description

updated

notcrawl

Office Clerks, General

Use for local Notion archive search, sync freshness, Markdown/database exports, git-share snapshots, and Notcrawl repo/release work.

2026-09-13

Showing 1 of 1 collected skill.