<div align="center">
<img src="assets/logo.svg" style="width: 128px; height: 128px;"/>
<a href="https://mqlang.org">Visit the site 🌐</a>
—
<a href="https://mqlang.org/book">Read the book 📖</a>
—
<a href="https://mqlang.org/playground">Playground 🎮</a>
<h1>mq</h1>
**Query. Filter. Transform Markdown.**
[](https://github.com/harehare/mq/actions/workflows/ci.yml)
[](https://github.com/harehare/mq/actions/workflows/audit.yml)
[](https://crates.io/crates/mq-markdown)
[](https://codecov.io/gh/harehare/mq)
[](https://codspeed.io/harehare/mq)
[](LICENSE)
mq is a command-line tool that processes Markdown using a syntax similar to jq.
It's written in Rust, allowing you to easily slice, filter, map, and transform structured data.
</div>
> [!IMPORTANT]
> This project is under active development.
## Why mq?
mq makes working with Markdown files as easy as jq makes working with JSON. It's especially useful for:
- **LLM Workflows**: Efficiently manipulate and process Markdown used in LLM prompts and outputs
- **LLM Input Generation**: Generate structured Markdown content optimized for LLM consumption, since Markdown serves as the primary input format for most language models
- **Documentation Management**: Extract, transform, and organize content across multiple documentation files
- **Content Analysis**: Quickly extract specific sections or patterns from Markdown documents
- **Batch Processing**: Apply consistent transformations across multiple Markdown files
Since LLM inputs are primarily in Markdown format, mq provides efficient tools for generating and processing the structured Markdown content that LLMs require.
## Features
- **Slice and Filter**: Extract specific parts of your Markdown documents with ease.
- **Map and Transform**: Apply transformations to your Markdown content.
- **Command-line Interface**: Simple and intuitive CLI for quick operations.
- **Extensibility**: Easily extendable with custom functions.
- **Built-in support**: Filter and transform content with many built-in functions and selectors.
- **REPL Support**: Interactive command-line REPL for testing and experimenting.
- **IDE Support**: VSCode Extension and Language Server **Protocol** (LSP) support for custom function development.
- **Debugger**: Includes an experimental debugger (`mq-dbg`) for inspecting and stepping through mq queries interactively.
- **External Subcommands**: Extend mq with custom subcommands by placing executable files starting with `mq-` in `~/.local/bin/`.
## Installation
### Quick Install
```bash
curl -sSL https://mqlang.org/install.sh | bash
```
Downloads the latest mq binary for your platform, installs it to `~/.local/bin/`, and updates your shell profile to add mq to your PATH.
On Windows (PowerShell):
```powershell
powershell -c "irm https://mqlang.org/install.ps1 | iex"
```
### Package Managers
| Method | Command |
| ---------------------- | --------------------------------------------------------------------------------- |
| Homebrew (macOS/Linux) | `brew install mq` |
| MacPorts (macOS) | `sudo port install mq` |
| Scoop (Windows) | `scoop bucket add mq https://github.com/harehare/scoop-mq && scoop install mq/mq` |
| Arch (yay) | `yay -S mq-bin` |
| Cargo (crates.io) | `cargo install mq-run` |
| Docker | `docker run --rm ghcr.io/harehare/mq:latest` |
<details>
<summary>More install options: cargo variants, binstall, pre-built binaries</summary>
```sh
# Install from Github
cargo install --git https://github.com/harehare/mq.git mq-run --tag v0.10.1
# Latest Development Version
cargo install --git https://github.com/harehare/mq.git mq-run --bin mq
# Install the debugger
cargo install --git https://github.com/harehare/mq.git mq-run --bin mq-dbg --features="debugger"
# Install using binstall
cargo binstall mq-run@0.10.1
```
Pre-built binaries for macOS, Linux, and Windows are also available on the [GitHub releases page](https://github.com/harehare/mq/releases).
</details>
### Editor & CI Integrations
| Integration | Link |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| VS Code | [](https://marketplace.visualstudio.com/items?itemName=harehare.vscode-mq) [](https://open-vsx.org/extension/harehare/vscode-mq) |
| Chrome | [](https://chromewebstore.google.com/detail/mq-for-markdown/pcgdpjopokfnfeeclnjdikecnjpoholf) |
| Neovim | [](https://github.com/harehare/mq/blob/main/editors/neovim/README.md) |
| Zed | [](https://github.com/harehare/mq/blob/main/editors/zed/README.md) |
| JetBrains IDEs | [](https://plugins.jetbrains.com/plugin/33547-mq--jq-like-tool-for-markdown-processing) |
| Obsidian | [](https://community.obsidian.md/plugins/mq) |
| Helix | [](https://mqlang.org/book/start/syntax-highlighting.html#helix) |
| GitHub Actions | [](https://github.com/marketplace/actions/setup-mq) |
```yaml
steps:
- uses: actions/checkout@v7
- uses: harehare/setup-mq@v1
- run: mq '.code' README.md
```
## Packages
mq is a Rust + TypeScript monorepo.
<details>
<summary>Rust crates and npm packages (click to show)</summary>
| Name | Description | Version |
| ------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`mq-run`](crates/mq-run) | Command-line interface for mq | [](https://crates.io/crates/mq-run) |
| [`mq-lang`](crates/mq-lang) | Core language implementation | [](https://crates.io/crates/mq-lang) |
| [`mq-markdown`](crates/mq-markdown) | Markdown parsing and manipulation utilities | [](https://crates.io/crates/mq-markdown) |
| [`mq-lsp`](crates/mq-lsp) | Language Server Protocol implementation | [](https://crates.io/crates/mq-lsp) |
| [`mq-repl`](crates/mq-repl) | Interactive REPL | [](https://crates.io/crates/mq-repl) |
| [`mq-formatter`](crates/mq-formatter) | Code formatter for the mq query language | [](https://crates.io/crates/mq-formatter) |
| [`mq-lint`](crates/mq-lint) | Static analysis linter | [](https://crates.io/crates/mq-lint) |
| [`mq-check`](crates/mq-check) | Type checker | [](https://crates.io/crates/mq-check) |
| [`mq-hir`](crates/mq-hir) | High-level Internal Representation (HIR) | [](https://crates.io/crates/mq-hir) |
| [`mq-dap`](crates/mq-dap) | Debug Adapter Protocol implementation | [](https://crates.io/crates/mq-dap) |
| [`mq-crawler`](crates/mq-crawler) | Directory crawler for batch Markdown processing | [](https://crates.io/crates/mq-crawler) |
| [`mq-macros`](crates/mq-macros) | Procedural macros for builtin function registration | [](https://crates.io/crates/mq-macros) |
| [`mq-test`](crates/mq-test) | Test runner for mq | [](https://crates.io/crates/mq-test) |
| [`mq-ffi`](crates/mq-ffi) | C API for integrating mq into C applications | \- |
| [`mq-wasm`](crates/mq-wasm) | WebAssembly bindings | \- |
| [`mq-web`](packages/mq-web) | Official WebAssembly build for the browser | [](https://www.npmjs.com/package/mq-web) |
| [`mq-nodejs`](packages/mq-nodejs) | Node.js bindings | [](https://www.npmjs.com/package/mq-nodejs) |
</details>
mq is also available as a hosted REST API (see [`mq-web-api`](https://github.com/harehare/mq-web-api)). See the [REST API docs](https://mqlang.org/book/start/web_api) or try it in the [Playground](https://mqlang.org/playground).
```bash
curl --data-binary @doc.md https://api.mqlang.org/.h1
```
## Language Bindings
Language bindings are available for Elixir, Python, Ruby, Java, and Go. See the [Language Bindings documentation](https://mqlang.org/book/start/language_bindings.html) for details.
## Usage
For more detailed usage and examples, refer to the [documentation](https://mqlang.org/book/).
Looking for a solution to a specific problem? See the [Cookbook](https://mqlang.org/book/cookbook/) for task-first recipes, or the [Example Guide](https://mqlang.org/book/start/example/) for a full tour of mq's selectors and built-in functions.
### Basic usage
<details>
<summary>Complete list of options (click to show)</summary>
```sh
Usage: mq [OPTIONS] [QUERY OR FILE] [FILES]... [COMMAND]
Commands:
repl Start a REPL session for interactive query execution. Optional FILES are combined as the initial input (same file/format handling as `mq QUERY FILES...`)
commands List built-in and external subcommands
completion Generate a shell completion script and print it to stdout
help Show documentation for a builtin function, selector, standard module, standard-module function, or the `examples` topic
compile Compile a query to experimental `.mqc` bytecode
Arguments:
[QUERY OR FILE]
[FILES]...
Options:
-q, --quiet
Suppress normal query output while preserving diagnostics, exit status, and query side effects
-A, --aggregate
Collect each input's nodes into a single array before the query runs. Combine with `--eval-all` to get one array across all files
-I, --input-format <INPUT_FORMAT>
Set input format [possible values: markdown, mdx, html, text, null, raw, bytes, cbor, csv, gron, json, jsonl, psv, toml, toon, tsv, xml, yaml]
--csv-delimiter <CHAR>
Custom delimiter for `-I csv` input (a single ASCII character). Has no effect on `-I tsv`/`-I psv`, which use a fixed tab/pipe delimiter by design; pass `-I csv` with this flag instead if you need a different delimiter (e.g. `;`)
--no-header
Treat csv/tsv/psv input as headerless: each row becomes an array of values instead of a dict keyed by header names. Applies to `-I csv`, `-I tsv`, and `-I psv`
--no-frontmatter
Do not read frontmatter at the start of Markdown or MDX input: the lines between `---` or `+++` are then text, a heading or a rule. Applies to `-I markdown` and `-I mdx`, and to `.md` and `.mdx` files
-L, --directory <MODULE_DIRECTORIES>
Search modules from the directory
-M, --module-names <MODULE_NAMES>
Load additional modules by name
-m, --import-module-names <IMPORT_MODULE_NAMES>
Import modules by name, making them available as `name::fn()` in queries
--allow-http-import[=<DOMAIN>...]
Allow `import`/`include` to fetch modules over HTTP(S). Disabled by default. `--allow-http-import=DOMAIN` (repeat the flag, or comma-separate) also allows those domains beyond the default. Use `github.com/{user}/{repo}` for one repository (expanded automatically), or a plain domain like `example.com` for any path under that host. The `=` is required so a bare domain isn't taken as a query/file positional
--refresh-modules
Force re-fetch of mutable-ref (HEAD/branch) HTTP-imported modules, ignoring the local cache. Versioned (tagged) modules are never re-fetched regardless of this flag
--clear-cache
Remove all HTTP module cache including versioned (tagged) modules and lock files. Use this to fully reset the cache when something goes wrong
--no-lockfile
Disable the mq.lock integrity check for HTTP imports. By default a fetched URL's content is checked against mq.lock, and a mismatch is rejected unless --refresh-modules is also passed
--frozen
Fail instead of recording a new mq.lock entry. Use in CI so a new module's content is only trusted during a reviewable local run whose mq.lock diff gets committed
--lockfile <PATH>
Path to the mq.lock file used for HTTP import integrity checks. Defaults to ./mq.lock (relative to the current directory)
-N, --allow-net[=<DOMAIN>...]
Allow the `http` function to make outbound HTTPS requests. Disabled by default; requests are HTTPS-only and blocked from reaching loopback/private/link-local addresses regardless of this flag. Pass with no value to allow any domain, or `--allow-net=DOMAIN` (repeat the flag, or comma-separate, to add more) to restrict requests to just those domains (and any path under them). The `=` is required so a bare domain after the flag isn't swallowed as a query/file positional instead
-R, --allow-read[=<PATH>...]
Allow the `read_file`/`read_file_bytes`/`open_file`/`collection`/`walk_files`/`file_exists`/`file_info`/`embed_images` functions to read from the filesystem. Disabled by default. Pass with no value to allow reading anywhere, or `--allow-read=PATH` (files or directories; repeat the flag, or comma-separate, to add more) to restrict reads to just those paths and their descendants. The `=` is required so a bare path after the flag isn't swallowed as a query/file positional instead
-W, --allow-write[=<PATH>...]
Allow the `write_file`/`mkdir`/`extract_images` functions to write to the filesystem. Under `--sandbox`, this also gates `-o`/`--output` (and `--append`, `--no-clobber`, `--atomic-output`), which are rejected unless the target is allowed. Without `--sandbox`, `-o` is not restricted. Disabled by default. Pass with no value to allow writing anywhere, or `--allow-write=PATH` (files or directories; repeat the flag, or comma-separate, to add more) to restrict writes to just those paths and their descendants. The `=` is required so a bare path after the flag isn't swallowed as a query/file positional instead
--allow-run[=<COMMAND>...]
Allow the `system` function to execute external commands. Disabled by default. Commands run directly (never through a shell), so shell metacharacters in arguments are never interpreted. Pass with no value to allow any command, or `--allow-run=COMMAND` (repeat the flag, or comma-separate, to add more) to restrict execution to just those commands. The `=` is required so a bare command after the flag isn't swallowed as a query/file positional instead
-E, --allow-env[=<NAME>...]
Allow `$VAR`/`${$VAR}` interpolation and debugger logpoints to read environment variables. Disabled by default. Pass with no value to allow reading any variable, or `--allow-env=NAME` (repeat the flag, or comma-separate, to add more) to restrict access to just those names. The `=` is required so a bare name after the flag isn't swallowed as a query/file positional instead
-a, --allow-all
Grant every sandboxed permission at once (read/write/net/run/env), and also enable HTTP module imports as if --allow-http-import were passed. Disabled by default. Cannot be combined with the individual --allow-* flags above
--sandbox <SANDBOX>
Named preset of sandboxed capabilities. Cannot be combined with --allow-* flags, except --allow-write, which adds write access (including `-o`/`--output`) on top of the preset. Under a preset, `-o`/`--output` is denied unless write access is granted [possible values: strict, read-only, networked, unsafe]
-f, --from-file
load filter from the file
--arg <NAME> <VALUE>
Sets a named string argument. NAME is accessible directly in queries, and also via ARGS."named" when --arg or --args is given
--argjson <NAME> <JSON_VALUE>
Sets a named JSON argument. NAME is accessible directly in queries
--rawfile <NAME> <FILE>
Sets file contents that can be referenced at runtime
--slurpfile <NAME> <FILE>
Sets a named argument from a JSON file. NAME is bound to an array of every JSON value found in FILE (jq --slurpfile compatible), so a file containing a single JSON value becomes a one-element array
--stream
Enable streaming mode for processing large files line by line
--watch
Watch the input file(s) for changes and automatically re-run the query whenever they change. Requires at least one input file (stdin cannot be watched). With --from-file, the query file is watched too. Runs until interrupted (Ctrl-C); a query error is printed to stderr and watching continues rather than exiting
--eval-all
Merge all input files into one input and run the query once (like yq's `eval-all`), instead of once per file
-F, --output-format <OUTPUT_FORMAT>
Set output format. When omitted, inferred from the `-o`/`--output` file extension if given (e.g. `.json` -> json, `.csv` -> csv), else defaults to markdown [possible values: markdown, html, text, json, table, grep, gron, raw, csv, jsonl, toml, toon, xml, yaml, shell, none]
--shell-prefix <PREFIX>
Prefix for variable names in `-F shell` output (must be non-empty) [default: MQ_]
-U, --update
Update matching Markdown nodes and write the result to stdout
--diff
With --update, print a unified diff instead of the transformed content; nothing is written. Multiple files are diffed one at a time with their path in the headers; stdin is labeled `<stdin>`. Exits 1 if anything would change
-i, --in-place
With --update, rewrite each input file with the result instead of printing it. Writes atomically and keeps the file mode; symlinks are followed and files whose content would not change are left untouched. Combined with --diff, only previews
--backup <SUFFIX>
With --in-place, copy each changed file to `<file>SUFFIX` before rewriting it
--unbuffered
Unbuffered output
--list-style <LIST_STYLE>
Set the list style for markdown output (default: keep the original) [possible values: dash, plus, star]
--link-title-style <LINK_TITLE_STYLE>
Set the link title surround style for markdown output [default: double] [possible values: double, single, paren]
--link-url-style <LINK_URL_STYLE>
Set the link URL surround style for markdown links [default: none] [possible values: none, angle]
--no-table-padding
Do not pad table cells to a common column width
-S, --separator <QUERY>
Specify a query to insert between files as a separator
-o, --output <FILE>
Output to the specified file. With several inputs, results are written one after another
--atomic-output <ATOMIC_OUTPUT>
Write `-o`/`--output` atomically via a same-directory temp file + fsync + rename, so a crash or full disk mid-write can't truncate the target [default: auto] [possible values: auto, always, never]
--no-clobber
Fail instead of overwriting `-o`/`--output` if the target already exists
--append
Add to `-o`/`--output` instead of replacing it, creating it if missing. Use `--atomic-output never` for concurrent-safe appends across processes
-C, --color[=<WHEN>]
When to colorize output. `-C` alone means `always` [default: auto] [alias: --color-output] [possible values: auto, always, never]
-B, --before-context <NUM>
Show NUM nodes before each match. Only effective with -F grep
--after-context <NUM>
Show NUM nodes after each match. Only effective with -F grep
--context <NUM>
Show NUM nodes before and after each match. Only effective with -F grep
-e, --exit-status
Exit with 4 if every result is empty (None, "", [] or an unmatched select), otherwise 0
--count
Output only the count of matching (non-None) results. With multiple files, prints "filename: N" per file and "total: N" at the end
--skip <N>
Skip the first N matching results before outputting
--limit <N>
Limit output to at most N results
--no-position
Omit Markdown node position information from structured output (json, table, gron, csv, toml, toon, xml, yaml). Reduces output size when source line/column spans aren't needed
-c, --compact
Print JSON on a single line, without pretty-printing. Only valid with -F json
--indent <N>
Number of spaces per indent level in pretty-printed output (0-7), jq's `--indent`. Only valid with -F json or -F xml; default is 2
--tab
Indent pretty-printed output with tabs instead of spaces, jq's `--tab`. Only valid with -F json or -F xml
-T, --format <FORMAT>
Set both input and output format at once (shorthand for `-I FORMAT -F FORMAT`). An explicit `-I`/`-F` overrides this for that side. Only accepts formats valid on both sides; e.g. `-I mdx` or `-F table` still require the dedicated flag [possible values: markdown, html, text, json, jsonl, gron, raw, csv, toml, toon, xml, yaml]
-P, --parallel-threshold <PARALLEL_THRESHOLD>
Number of files to process before switching to parallel processing [default: 10]
--args [<ARGS>...]
Positional string arguments, available as ARGS."positional" in queries
--timeout <SECONDS>
Maximum time in seconds allowed for query evaluation before aborting (e.g. 0.5, 5). Must be a finite positive number. No timeout by default
--stack-trace
Include VM frames in uncaught runtime errors
--error-format <ERROR_FORMAT>
Format for the final uncaught error. Combines with `--stack-trace` [default: human] [possible values: human, json]
-h, --help
Print help (see more with '--help')
-V, --version
Print version
# Examples
mq 'query' file.md
mq compile -f query.mq
mq query.mqc file.md
Compiled .mqc programs are experimental. Recompile them after upgrading mq.
Run `mq help examples` for more usage examples, or `mq help <name>` for
function/selector/module docs.
```
</details>
Here's a basic example of how to use `mq`:
```sh
# Extract all headings from a document
mq '.h' README.md
# Extract only h1 headings
mq '.h(1)' README.md
# Extract h1 and h2 headings
mq '.h(1, 2)' README.md
# Extract headings from level 1 to 3 using a range
mq '.h(1..3)' README.md
# Extract only Rust code blocks
mq '.code("rust")' example.md
# Extract code blocks containing "name"
mq '.code | select(contains("name"))' example.md
# Extract code values from code blocks
mq -A 'pluck(.code.value)' example.md
# Extract language names from code blocks
mq '.code.lang' documentation.md
# Extract URLs from all links
mq '.link.url' README.md
# Filter table cells containing "name"
mq '.[][] | select(contains("name"))' data.md
# Select lists or headers containing "name"
mq 'select(.[] || .h) | select(contains("name"))' docs.md
# Exclude JavaScript code blocks
mq '.code | select(.code.lang != "js")' examples.md
# Convert CSV to markdown table
mq 'csv::csv_to_markdown_table' example.csv
# Extract a section by title
mq -A 'section::section("Installation")' README.md
# Filter sections by heading level (scalar or range)
mq -A 'section::sections() | section::by_level(2)' README.md
mq -A 'section::sections() | section::by_level(1..2)' README.md
```
### Composing Workflows with Subcommands
`mq` subcommands are designed to work together via Unix pipes.
```sh
# Convert Excel report to Markdown, then extract all headings
mq conv report.xlsx | mq '.h'
# Convert a Word document and extract a specific section
mq conv document.docx | mq -A 'section::section("Summary")'
# Convert and view Markdown directly in the terminal
mq conv slides.pdf | mq view
```
Run `mq commands` to see all available subcommands (built-in and external).
## External Subcommands
You can extend `mq` with custom subcommands by placing executable files starting with `mq-` in `~/.local/bin/` or anywhere in your `PATH`.
This makes it easy to add your own tools and workflows to `mq` without modifying the core binary.
See the [External Subcommands documentation](https://mqlang.org/book/start/external_subcommands) for the full list and details.
## Support
- 🐛 [Report bugs](https://github.com/harehare/mq/issues/new?template=bug_report.yml)
- 💡 [Request features](https://github.com/harehare/mq/issues/new?template=feature_request.yml)
- ⭐ [Star the project](https://github.com/harehare/mq) if you find it useful!
## Contributing
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, coding conventions, and how to submit changes. Please also review our [Code of Conduct](.github/CODE_OF_CONDUCT.md).
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.