ClickHouse/silk

Latest recorded source activity: September 21, 2026 at 20:18

SkillsMP catalog refreshed: October 29, 2026 at 18:30

skills collected: 3

GitHub stars: 332
GitHub forks: 14

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/ClickHouse/silk.

Repository README

Repository README source: README.md

Read the source README

Expand full README

<picture> <img align="left" width="200" style="margin-right: 20px;" src="https://github.com/user-attachments/assets/0c64f89e-55c7-42f8-8a82-0da220f1f9f5" alt="The Silk logo."> </picture> ### Silk A cooperative fiber scheduler for Linux with per-CPU scheduler threads, io_uring integration, and topology-aware work-stealing. Fibers are lightweight stackful coroutines that suspend rather than block their OS thread, enabling high concurrency with low overhead. ## Documentation - [`docs/coroutines.md`](docs/coroutines.md) — stackless coroutines vs stackful fibers: design differences and performance data - [`docs/perf.md`](docs/perf.md) — `net-perf` and `file-perf` benchmark results and fio comparison - [`docs/scheduler.md`](docs/scheduler.md) — scheduler loop, context switching, suspension pattern, async IO, sleep cancellation, work-stealing design, and performance benchmarks - [`docs/simulator.md`](docs/simulator.md) — fiber workload simulator: pipeline config format, step types, parameterized scenarios, and modeling limits - [`docs/sync.md`](docs/sync.md) — synchronization primitives: `FiberFuture`, `FiberFutex`, `FiberMutex`, `FiberSequencer`, `FiberEvent`, `FairFiberMutex` - [`docs/tls.md`](docs/tls.md) - thread-local storage with migrating fibers: why native `thread_local` is unsafe across a suspension, the rules for fiber code, and how silk hardens its own context accessors - [`docs/util.md`](docs/util.md) — utility library: lock-free data structures, TSC timing, memory pool, CPU topology, logging, assertions - [`src/fibers/tests/`](src/fibers/tests/) — usage examples: fiber lifecycle, futures, synchronization primitives, async IO - [`src/gdb/fiber.py`](src/gdb/fiber.py) — GDB extension; load with `source src/gdb/fiber.py`, then use `fiber-list`, `fiber-savecontext`, `fiber-restorecontext`, `fiber-switchcontext` ## Requirements - CMake >= 3.28 - Ninja - Clang 21 - ccache (optional) - Boost headers (`libboost-dev`) - libelf (`libelf-dev`) — optional, required only for `src/profiler`; the profiler is silently skipped if absent. GTest, Google Benchmark, libbacktrace, liburing, librseq, libbpf, bpftool, and cxxopts are bundled as submodules under `contrib/` and do not need to be installed separately. The two Boost.Context asm sources used for fiber switching are vendored under `contrib/fcontext/`, and systemtap's USDT headers (`<sys/sdt.h>` + `<sys/sdt-config.h>`) under `contrib/systemtap-sdt/` (both vendored directly, no submodule). Poco, the AWS SDK, and jemalloc are built on demand via `--build-poco`, `--build-aws`, and `--build-jemalloc` passed to `configure`. Runtime dependencies for optional benchmarks: nginx (only for `http-perf --nginx`; the default uses an internal Poco-based server built into the `http-perf` binary), fio (for `fio-perf`), and MinIO (for `s3-perf`). MinIO is downloaded automatically to `.tools/` if not in PATH; the others must be installed separately. ## Build ``` ./bb [options] [command] ``` ### Global options | Option | Values | Default | Description | |---|---|---|---| | `-b`, `--build` | `debug`, `release` | `debug` | Build type | | `-s`, `--sanitizer` | `thread`, `address`, `memory`, `undefined` | | Enable sanitizer | | `-v`, `--verbose` | | | Print every command before running it; also passes `--verbose` to perf binaries to enable their debug logging | ### Commands #### `configure [--build-poco] [--build-aws] [--build-jemalloc]` Configure (or reconfigure) the CMake build directory. Optional flags enable components that are off by default: `--build-poco` enables `http-perf` (requires Poco), `--build-aws` enables `s3-perf` (requires the AWS SDK), `--build-jemalloc` enables jemalloc (used by `http-perf` and `s3-perf` to improve allocator performance). ``` ./bb configure ./bb configure --build-poco --build-aws ./bb -b release configure ``` #### `fmt [--check]` Format the C++ sources with clang-format-21 and the Python sources (`bb`, `ci/`) with black. Pass `--check` to verify formatting without modifying files (exits non-zero if any file would be changed). ``` ./bb fmt ./bb fmt --check ``` #### `lint` Type-check the Python sources (`bb`, `ci/commands/`) with mypy in strict mode. ``` ./bb lint ``` #### `clean` Remove the entire `build/` directory. ``` ./bb clean ``` #### `build [targets]` Build the project. Configures automatically if the build directory does not exist. `build` is the default command when none is specified. ``` ./bb # debug build ./bb -b release # release build ./bb -s thread # debug build with TSan ./bb -b release -s address # release build with ASan ./bb build fibers-test # build a specific target ``` #### `test [-R pattern] [-N] [ctest flags...]` Build and run tests. Runs in parallel using all available CPUs. Any extra flags are forwarded directly to `ctest`. | Flag | Description | |---|---| | `-R <pattern>` | Run only tests matching the regex pattern | | `-N` | List tests without running them | | `--timeout SECONDS` | Per-test timeout in seconds (default: 180, 0=none) | | `--coverage` | Instrument with coverage, run tests, and generate an HTML report, an lcov file, and a Cobertura XML report under `build/debug-coverage/` | | `--rerun-failed` | Rerun only tests that failed in the last run | | `--repeat until-fail:<n>` | Repeat each test up to `n` times, stopping on first failure (useful for flaky test hunting) | | `--output-on-failure` | Print test output when a test fails | ``` ./bb test ./bb test -R FiberMutex ./bb -s thread test ./bb test --rerun-failed ./bb test --coverage ``` #### `bench [-R pattern] [-N] [gbench flags...]` Build and run benchmarks. | Flag | Description | |---|---| | `-R <pattern>` | Run only benchmarks matching the pattern | | `-N` | List benchmarks without running them | | `--timeout SECONDS` | Per-benchmark timeout in seconds (default: 180, 0=none) | ``` ./bb -b release bench ./bb -b release bench -R LockFreeQueue ``` --- ## Performance commands Each perf command builds the relevant binary and runs the benchmark, printing results as a Markdown table. Duration, warmup, and delay options accept a unit suffix (`ns`, `us`, `ms`, `s`, `m`); a bare number is interpreted as seconds. All perf commands accept `--timeout SECONDS` (per-run timeout; default: 180, 0=none), and `--help` on any of them lists every option with its default. #### `file-perf` Async file I/O benchmark using io_uring. | Option | Default | Description | |---|---|---| | `--file PATH` | `/dev/shm/file-perf.bin` | Test file path | | `--bs SIZE` | `4k` | Block size | | `--size SIZE` | `1g` | File size | | `--duration DURATION` | `10` | Measurement duration | | `--warmup DURATION` | `2` | Warmup duration | | `--numjobs N [N ...]` | `1` | Number of parallel jobs | | `--iodepth N [N ...]` | `16` | IO queue depth per job | | `--rw MODE [MODE ...]` | `randread` | Access mode(s): `randread`, `randwrite`, `seqread` | | `--flamegraph` | | Profile and generate flamegraph SVG | | `--print-counters` | | Print perf counters after each run | | `--fixed-buffers` | | Use registered buffers (`IORING_OP_READ_FIXED` / `WRITE_FIXED`) | ``` ./bb -b release file-perf ./bb -b release file-perf --bs 64k --size 4g ./bb -b release file-perf --numjobs 1 16 --iodepth 1 16 ./bb -b release file-perf --rw randread randwrite ./bb -b release file-perf --fixed-buffers ./bb -b release file-perf --flamegraph ``` #### `fio-perf` fio comparison using io_uring engine. Same options as `file-perf` (except `--flamegraph` and `--print-counters`). Does not build anything. ``` ./bb fio-perf ./bb fio-perf --bs 64k ./bb fio-perf --numjobs 1 16 --iodepth 1 16 ``` #### `net-perf` TCP echo benchmark. Starts a local server and runs the client against it. When `--host` points to a remote host, the server is not started locally. | Option | Default | Description | |---|---|---| | `--host` | `127.0.0.1` | Server host | | `--port` | `17777` | Server port | | `--msg-size BYTES` | `64` | Echo message size | | `--duration DURATION` | `10` | Measurement duration | | `--warmup DURATION` | `2` | Warmup duration | | `--connections N [N ...]` | `1000` | Connection counts to sweep | | `--delay DURATION` | `0` | Server-side delay per message (e.g. `1ms`, `100us`) | | `--flamegraph` | | Profile client and generate flamegraph SVG | | `--print-counters` | | Print perf counters after each run | ``` ./bb -b release net-perf ./bb -b release net-perf --connections 1 64 256 1024 ./bb -b release net-perf --delay 1ms ./bb -b release net-perf --host 10.0.0.2 ./bb -b release net-perf --flamegraph ``` #### `net-perf-asio` TCP echo benchmark using Boost.Asio C++20 coroutines. Same options as `net-perf`. ``` ./bb -b release net-perf-asio ./bb -b release net-perf-asio --connections 1 64 256 1024 ./bb -b release net-perf-asio --delay 1ms ./bb -b release net-perf-asio --flamegraph ``` #### `simulator` Runs `fibers-simulator`: a synthetic fiber workload described by a pipeline config file, for exercising the scheduler (chains of dependent wakes, micro-bursts, fan-out/fan-in, injected stalls, load steps) without a network stack. `CONFIG` is a file path or the name of a bundled config from `src/perf/simulator/configs/` (`net-baseline`, `net-stall`, `chain`, `micro-burst`, `fan-out`, `phased`, `mixed`, `pool`, `amber`). A config has a `params` section and a `pipeline` section. `params` declares the run settings (`duration`, `warmup`, `seed`) plus any `$name` placeholders with their defaults; every param can be overridden with `--param name=value`. `pipeline` is a tree of named steps, each with a `type` key: `cpu` (busy-loop), `wait` (fiber sleep, optionally exponential), `yield`, `sequence`, `loop` (`count` / `duration` / `period` pacing), `parallel` (fan out N fibers over the child steps and join), `concurrent` (one fiber per child step), `chain` (a token ring of dependent wakes), `maybe` (probabilistic body, e.g. stalls), `pool` (worker fibers serving submitted requests, optionally combining a `batch` per execution, with `affinity` binding each submitter to one worker's queue), and `submit` (call a pool and wait the completion); any value may reference `$name`. `measure = 1` on a step reports its wall-time percentiles; loops measure per iteration and chains per round. Each parameterized config's header documents the measured cases as ready-made `--param` lines: the `net-stall` cases from `docs/work-stealing.md` and the `amber` transfer-fiber counts (1, 32, 200, 400, 800). | Option | Default | Description | |---|---|---| | `--duration DURATION` | config | Override the run duration | | `--warmup DURATION` | config | Override the warmup | | `--param NAME=VALUE` | config | Override a config param (repeatable; comma-separated pairs allowed) | | `--cpus CPUS` | | taskset CPU list for the run | | `--flamegraph` | | Profile the run and generate flamegraph SVG | | `--print-counters` | | Print perf counters after the run | | `--disable-cpu-adjust` | | Pin the scheduler CPU width at full (static-width baselines) | | `--timeout SECONDS` | 180 | Per-run timeout (0=none) | ``` ./bb -b release simulator chain ./bb -b release simulator net-stall --duration 60s --print-counters ./bb -b release simulator net-stall --cpus 0-15 --param connections=256,stall_rate=100,workers=256 ./bb -b release simulator amber --param fibers=800 ./bb -b release simulator src/perf/simulator/configs/mixed.cfg --cpus 0-15 ./bb -b release simulator micro-burst --flamegraph ``` #### `http-perf` HTTP/1.1 GET benchmark. Defaults to silk's internal HTTP server (Poco's `HTTPServerConnection` over `FiberSocketImpl`, one fiber per connection); pass `--nginx` to run against nginx instead. | Option | Default | Description | |---|---|---| | `--host` | `127.0.0.1` | Server host | | `--port` | `18080` | Server port | | `--duration DURATION` | `10` | Measurement duration | | `--warmup DURATION` | `2` | Warmup duration | | `--connections N [N ...]` | `1000` | Connection counts to sweep | | `--delay DURATION` | `0` | Server-side per-request delay (e.g. `1ms`, `100us`); fiber server uses `silk::FiberScheduler::sleep`, nginx uses lua sleep | | `--threads` | | Use thread-per-connection client mode instead of fibers | | `--nginx` | | Run client against nginx instead of the internal server | | `--flamegraph` | | Profile client and generate flamegraph SVG | | `--print-counters` | | Print perf counters after each run | ``` ./bb -b release http-perf ./bb -b release http-perf --threads ./bb -b release http-perf --nginx ./bb -b release http-perf --delay 5ms ./bb -b release http-perf --connections 1 512 1024 2048 ./bb -b release http-perf --flamegraph ``` #### `s3-perf` S3 object storage benchmark. Starts a local MinIO server (downloaded automatically to `.tools/` if not in PATH) and runs the client against it. | Option | Default | Description | |---|---|---| | `--endpoint URL` | `http://127.0.0.1:9000` | S3 endpoint | | `--bucket NAME` | `test-bucket` | S3 bucket | | `--key NAME` | `test-object` | S3 object key | | `--region NAME` | `us-east-1` | S3 region | | `--access-key KEY` | `minioadmin` | S3 access key | | `--secret-key KEY` | `minioadmin` | S3 secret key | | `--size SIZE` | `4096` | Object size; no units = bytes (e.g. `4096`, `64k`, `1g`) | | `--duration DURATION` | `10` | Measurement duration | | `--warmup DURATION` | `2` | Warmup duration | | `--numjobs N [N ...]` | `1` | Number of parallel jobs | | `--iodepth N [N ...]` | `16` | IO queue depth per job | | `--rw MODE [MODE ...]` | `read` | Access mode(s): `read`, `write`, `readwrite` | | `--threads` | | Also run with thread executor | | `--flamegraph` | | Profile first config and generate flamegraph SVG | | `--print-counters` | | Print perf counters after each run | | `--data-dir PATH` | `/dev/shm/minio-data` | MinIO data directory | ``` ./bb -b release s3-perf ./bb -b release s3-perf --rw read write ./bb -b release s3-perf --numjobs 1 16 --iodepth 1 64 ./bb -b release s3-perf --threads ./bb -b release s3-perf --flamegraph ``` #### `perf` Run multiple perf benchmarks in one shot. Targets are positional values, listed after the options. | Target | Description | |---|---| | `file` | file-perf | | `fio` | fio comparison | | `net` | net-perf | | `net-asio` | net-perf-asio | | `net-epoll` | net-perf-epoll | | `http` | http-perf (internal server, fiber client) | | `http-threads` | http-perf (internal server, thread client) | | `http-nginx` | http-perf against nginx (fiber client) | | `s3` | s3-perf (fibers) | | `s3-threads` | s3-perf (threads) | | `all` | run every target above | | Option | Description | |---|---| | `--duration DURATION` | Override per-binary measurement duration (e.g. `60s`) | | `--warmup DURATION` | Override per-binary warmup duration (e.g. `10s`) | | `--timeout SECONDS` | Per-run timeout (default 180, 0 = none) | ``` ./bb -b release perf file net ./bb -b release perf all ./bb -b release perf --duration 60s --warmup 10s file net net-asio ./bb -b release perf --duration 60s --warmup 10s all ```

Loading the repository README…

Skills in this repository

Software Quality Assurance Analysts & Testers

1 occupation category · 33% classified

skill

occupation

description

updated

debug

unclassified

Reproduce and diagnose a failing, flaky or hanging silk test or run - CI failures, timeouts, sanitizer reports, crash dumps - and measure test coverage. Use when a test fails, hangs or flakes, when a run wedges, or when the user asks for coverage.

2026-09-21

profile

unclassified

Profile a silk perf target and read the result - flamegraph, folded stacks, scheduler-latency histograms, counters, paired runs against a baseline. Use when the user asks why a run is slow, wants a flamegraph or counters, or needs a throughput or latency…

2026-09-21

check-ci

Software Quality Assurance Analysts & Testers

Fetch and report CI results for a silk PR. Use when the user asks to investigate, address, or fix CI failures, or refers to a PR without specifying what's broken.

2026-06-24

Showing 3 of 3 collected skills.