# gitcrawl 🕷️ — Pull GitHub down to earth
[](https://github.com/openclaw/gitcrawl/actions/workflows/ci.yml)
[](https://github.com/openclaw/gitcrawl/releases/latest)
[](https://github.com/openclaw/gitcrawl/releases/latest)
[](LICENSE)
[](https://gitcrawl.sh)
`gitcrawl` mirrors GitHub issues and pull requests into local SQLite for maintainers and agents. It provides local search, related-thread clustering, JSON output, and a terminal interface without running a local HTTP service.
<img width="1797" height="1096" alt="Gitcrawl terminal interface showing issue and pull request clusters" src="https://github.com/user-attachments/assets/54a0a6cf-5862-451d-9552-5d18656976ff" />
## Install
Homebrew is the smallest install on macOS and Linux:
```sh
brew install openclaw/tap/gitcrawl
```
Prebuilt archives for macOS, Linux, and Windows are available from [GitHub Releases](https://github.com/openclaw/gitcrawl/releases/latest). The [installation guide](docs/installation.md) also covers source builds and update checks; source builds require Go 1.27.2 or newer for CrawlKit v0.17.0 and select the patched Go 1.27.3 toolchain automatically. Binaries built with this toolchain require macOS 13 Ventura or newer. A [Docker source build](docs/docker.md) keeps its runtime state under one mounted directory.
Sync needs a GitHub token from `GITHUB_TOKEN` or `gh auth token`. Generating summaries and embeddings also needs `OPENAI_API_KEY`; semantic search and clustering use those stored embeddings, while ordinary sync and keyword search do not.
## Quick start
Initialize the local archive, sync a repository, and query it:
```sh
gitcrawl init
gitcrawl sync openclaw/gitcrawl
gitcrawl search issues "SQLite" -R openclaw/gitcrawl \
--state open --json number,title,url --limit 5
```
The default sync fetches open issues and pull requests and sweeps recently closed threads. Use `--state all` for an initial historical backfill, `--include-comments` for discussion, or `--with pr-details` for pull request files, commits, checks, and review state.
The [quickstart](docs/quickstart.md) continues through embeddings, clusters, and the terminal UI. For a quota-conscious maintainer workflow, see [Maintainer archive workflow](docs/maintainer-archive.md).
## Search and automation
Direct search supports keyword, semantic, and hybrid modes over one local repository. The `gh search`-shaped form used above lets existing scripts query the SQLite mirror without spending GitHub search quota. Add `--sync-if-stale 5m` when an agent should refresh an old mirror before searching.
`gitcrawl embed owner/repo --include-closed` includes historical threads. Candidate collection uses an ordering index added automatically when the archive opens for writing, so large archives do not need a full sort for every page.
For `analytics --once` and `analytics --watch`, keep the historical discovery receipt (`status.json`) beside the selected config file, including when selecting it through `GITCRAWL_CONFIG`. Analytics uses that location regardless of the working directory.
Commands support structured output with `--json`. Gitcrawl reserves stdout for results and sends diagnostics to stderr, so its output can feed `jq` or another process directly. See [Search](docs/search.md) and [Automation](docs/automation.md) for the supported shapes.
## Cluster and review
`gitcrawl refresh owner/repo` runs sync, embedding, and clustering in order. Clusters combine vector similarity with direct GitHub references, then preserve maintainer decisions such as local closes, member exclusions, and canonical threads across later runs.
Open `gitcrawl tui [owner/repo]` for the keyboard- and mouse-driven cluster browser. The TUI reads SQLite and refreshes its view every 15 seconds; it does not call GitHub itself. See [Clustering](docs/clustering.md), [Governance](docs/governance.md), and [TUI](docs/tui.md).
## Archive modes
The default local archive keeps configuration, SQLite data, vectors, caches, and logs in platform-native user directories. Exact paths and environment overrides are in [Configuration](docs/configuration.md).
[Portable stores](docs/portable-stores.md) publish a compact SQLite snapshot through Git so multiple machines can share a read-mostly archive. [Cloud archives](docs/cloud-archives.md) configure authenticated reads and snapshot publication through a separately deployed Worker service. Local, portable, and cloud modes keep distinct storage and credential boundaries.
Octopool owns pooled live `gh` reads. Gitcrawl keeps local mirror, search, clustering, and TUI workflows; use `gh` or Octopool for final live verification and GitHub write actions. See the [migration note](docs/gh-shim.md).
## Command map
| Job | Command | Guide |
| --- | --- | --- |
| Check archive health | `gitcrawl status` / `gitcrawl doctor` | [Configuration](docs/configuration.md) |
| Refresh a portable subscriber | `gitcrawl portable refresh --expected-remote URL` | [Portable stores](docs/portable-stores.md#routine-subscriber-refresh) |
| Mirror GitHub threads | `gitcrawl sync owner/repo` | [Sync](docs/sync.md) |
| Collect repository counters and releases | `gitcrawl metrics collect\|import\|status --config metrics.json` | [Repository metrics](docs/metrics.md) |
| Search threads or indexed code | `gitcrawl search ...` | [Search](docs/search.md) |
| Build and inspect clusters | `gitcrawl refresh`, `clusters`, `tui` | [Clustering](docs/clustering.md) |
| Export a code-free conversation snapshot | `gitcrawl capture owner/repo` | [Capture](docs/capture.md) |
The [command reference](docs/commands.md) lists the complete CLI surface and flags.
## Development
Use one canonical development checkout and branches by default. Additional
worktrees require an explicit user request, a named owner and end-of-task cleanup.
Keep deployment artifacts and evidence outside Projects; preserve live dependencies
and held work before retiring a checkout. See [workspace rules](AGENTS.md).
```sh
make build
make test
make check
```
`make check` runs the formatting, vet, vulnerability, dead-code, coverage, smoke, release-script, documentation, and snapshot gates used by CI. The documentation build uses Node.js 26.12.1 (pinned in `.node-version`); `make docs` runs that gate alone. CI pins GoReleaser v2.18.2 for snapshot builds.
On Linux or macOS, use Python 3 and the Go toolchain declared in `go.mod` to
compare durable clustering output across scalar, SIMD-enabled and forced-fallback
builds:
```sh
python3 -m unittest discover -s scripts -p test_cluster_simd_parity.py
python3 scripts/cluster_simd_parity.py
```
The runner builds two test binaries and reuses the SIMD-enabled binary with
`GODEBUG=simd=0` for fallback. It exercises a synthetic 2000-node SQLite fixture
through full, limited, model-partial and strict-rejection commands, without
providers or real archives. Temporary binaries and output are removed on exit.
Durable identities, vectors, coverage and lineage must match exactly; only
enumerated generated clocks are normalized, and membership scores allow an
absolute difference of `1e-12`. Stored JSON text stays lossless, including SQL
NULL versus JSON null, and clock ordering retains nanosecond precision.
This checks correctness, not performance or
hardware SIMD availability. Ubuntu CI runs it explicitly; ordinary `go test`
skips the output-producing test unless its output path is supplied.
macOS CI runs on the Apple Silicon `xcode-27` image with an explicit macOS 13.0 deployment target. Release verification executes Intel binaries under Rosetta, installing it if needed, and rejects binaries that raise the macOS 13 minimum.
Maintainers can validate the signed release pipeline on `main` without publishing:
```sh
gh workflow run release-unified.yml --ref main -f version=0.16.0 -f dry_run=true
```
Use the current version for this dry run. It builds, signs, and verifies the artifacts without creating tags or releases or updating Homebrew.
## License
MIT — see [LICENSE](LICENSE).