Run any Skill in Manus with one click

perf-ab-testing

Stars0

Forks0

UpdatedJune 28, 2026 at 11:37

Use when measuring or comparing PlainQ's gRPC performance — running the perf/ AB harness, comparing a branch or PR against a stable ref, reasoning about a latency/throughput regression, reading the "PlainQ AB Performance" Grafana dashboard or the markdown report, editing the k6 gRPC load script, the VictoriaMetrics scrape config, the Performance AB GitHub workflow, or the perf Dockerfile. Covers the candidate-vs-baseline AB model, the report verdict, and the per-PR CI integration.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

marsolab

marsolab/plainq

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	perf-ab-testing
description	Use when measuring or comparing PlainQ's gRPC performance — running the perf/ AB harness, comparing a branch or PR against a stable ref, reasoning about a latency/throughput regression, reading the "PlainQ AB Performance" Grafana dashboard or the markdown report, editing the k6 gRPC load script, the VictoriaMetrics scrape config, the Performance AB GitHub workflow, or the perf Dockerfile. Covers the candidate-vs-baseline AB model, the report verdict, and the per-PR CI integration.
version	1.0.0
tags	["performance","benchmarking","k6","grpc","victoriametrics","grafana","ci","plainq"]

PlainQ performance AB testing

Overview

PlainQ ships a self-contained AB harness under perf/. It builds two servers — candidate (current checkout) and baseline (a stable git ref) — runs an identical k6 gRPC workload (Send → Receive → Delete) at both at once, stores everything in VictoriaMetrics, and reports the comparison via a Grafana dashboard and a markdown report with a pass/fail verdict. Every metric is tagged variant=baseline|candidate and op=send|receive|delete|total.

Core idea: never read a single absolute number — always compare candidate against the baseline ref under the same load.

When to use

"Did this change make PlainQ slower/faster?" / suspected perf regression.
Comparing a branch or PR against main (or a tag) for latency/throughput.
"Just spin a load test" against a running server.
Reading the AB Grafana dashboard or perf/results/report-*.md.
Editing the harness: cmd/perfctl, k6/ab_test.js, k6/load_test.js, victoriametrics/scrape.yml, grafana/, Dockerfile.plainq, scripts/, or .github/workflows/perf.yml.

Quick reference

perfctl (cmd/perfctl) is the single-entry CLI; make targets wrap it.

Action	Command
Build the CLI	`make -C perf cli` → `./perf/perfctl`
AB run vs `origin/main` (leaves stack up)	`perfctl ab` or `make -C perf ab`
AB vs a specific ref, heavier run	`perfctl ab -baseline v0.1.0 -vus 50 -duration 5m`
Load one running server (no baseline)	`perfctl load -target localhost:8080`
Stack up / URLs / down / cleanup	`perfctl up` / `dashboard` / `down` / `clean`
Re-run only AB k6 on the live stack	`make -C perf k6`

Each flag has an env fallback (-vus/VUS, -baseline/BASELINE_REF, -target/TARGET_ADDR, …). Use -target host.docker.internal:PORT to load a server on the host. Knobs: BASELINE_REF, VUS, DURATION, BATCH_SIZE, MSG_BYTES, RUN_ID, KEEP_UP. See perf/README.md for the full table.

Reading results

Markdown report — perf/results/report-<run-id>.md (also printed by run.sh). Verdict: ✅ NO REGRESSION, 🚀 IMPROVEMENT, or ⚠️ REGRESSION (candidate end-to-end p95 >10% above baseline, or worse error rate). report.py exits non-zero on regression so run.sh / CI can gate.
Grafana — http://localhost:3000 → PlainQ AB Performance: latency (p50/p95/p99), throughput, errors, per-op latency, server resources, all baseline-vs-candidate.
Metrics in VictoriaMetrics (:8428, prefix k6_): k6_plainq_latency_p95{variant,op}, k6_plainq_reqs_total{variant,op}, k6_plainq_errs_total{variant,op}.

On every PR (CI)

.github/workflows/perf.yml runs the harness automatically:

Pull requests → candidate = the PR, baseline = the target branch; a light run (10 VUs, 45s). The report is posted as a sticky PR comment.
Push to main → candidate = new main, baseline = previous commit; a thorough run (30 VUs, 2m) for the baseline trend.

It is informational — the verdict never fails the check. Full data is in the run's perf-results artifact. To make it a hard gate, remove continue-on-error from the Run AB test step.

Common mistakes

Comparing absolute numbers across machines. Only candidate-vs-baseline on the same run is meaningful; CI runners are noisy.
Forgetting KEEP_UP=0 in automation — run.sh leaves the stack up by default for interactive use.
Expecting an xk6 plugin build. k6's gRPC support is native (k6/net/grpc); no plugin compile is needed.
Server-resource panels show n/a. They depend on the server exposing process/Go metrics; the k6 client-side latency/throughput is the source of truth regardless.
Editing the dashboard JSON by hand and breaking it. Validate with jq empty perf/grafana/dashboards/plainq-ab.json.

Agent/platform notes

This skill uses Claude Code tool names. On Codex/others, map them per .agents/skills/using-superpowers/references/codex-tools.md (e.g. Bash → your shell). The harness itself is plain make/docker/git — identical on every platform.

More from this repository

same repository

go-cli

marsolab/plainq

Building command-line tools in Go with the stdlib `flag` package. ALWAYS use this skill when writing or reviewing Go CLI code — top-level flags with `flag.Parse`, subcommands with `flag.NewFlagSet`, exit codes, writing errors to `os.Stderr`, signal handling, environment-variable fallbacks, `-h`/`-help` output, and `cmd/<tool>` project layout. Avoid reaching for cobra/urfave-cli unless the tool clearly needs it (~5+ subcommands with rich help). Pair with go-style for naming and config patterns, go-errors for exit-code mapping, and go-logging for diagnostic output.

2026-05-190

go-concurrency

marsolab/plainq

Disciplined Go concurrency. ALWAYS use this skill when writing or reviewing Go code that touches goroutines, channels, `context.Context`, `sync.WaitGroup`, `sync.Mutex` / `sync.RWMutex`, `sync.Once`, `errgroup.Group`, worker pools, fan-out/fan-in pipelines, request cancellation, timeouts, deadlines, or any "run this in the background" pattern. Also use when debugging deadlocks, goroutine leaks, race conditions, or channel-related bugs. Pair with go-errors for error propagation across goroutines.

2026-05-190

go-dev

marsolab/plainq

Umbrella skill for Go development. ALWAYS use this skill when the user's task involves Go or Golang in any way — writing new Go code, reviewing existing Go code, building Go services, creating Go CLI tools, working with Go tests, setting up Go linting, using sqlc for database access, or anything else Go-related. This includes mentions of: Go, Golang, .go files, go.mod, go test, golangci-lint, Chi router, sqlc, goose migrations, generics, goroutines, channels, interfaces, error handling, concurrency, table-driven tests, slog, iterators, or any Go package/tool. This skill is a router — it loads fast and points to the focused sibling skill(s) that own the deep reference material for the task at hand.

2026-05-190

go-errors

marsolab/plainq

Idiomatic Go error handling. ALWAYS use this skill when writing or reviewing Go code that raises, returns, wraps, matches, or aggregates errors — questions about `fmt.Errorf` with `%w` vs `%v`, `errors.Is` and `errors.As`, sentinel errors, typed error structs, `errors.Join` for combining failures, the "handle exactly once" rule, panic vs error, guard clauses and early return, error message formatting, or any `if err != nil` pattern. Use alongside go-style for general idioms.

2026-05-190

go-http

marsolab/plainq

Building HTTP services in Go with the Chi router and stdlib `net/http`. ALWAYS use this skill when writing or reviewing Go HTTP code — defining routes with `chi.Router`, middleware (`chi.Use`, `middleware.Logger`, `middleware.Recoverer`, `middleware.RequestID`), handler signatures, request decoding, JSON responses, status codes from typed errors, graceful shutdown with `http.Server.Shutdown`, `httptest` patterns, service project layout (`cmd/server`, `internal/handler`, `internal/service`), and timeouts (`ReadTimeout`, `WriteTimeout`, `IdleTimeout`). Pair with go-sql for the data layer, go-logging for request-scoped logging, go-errors for status code mapping, and go-testing for handler tests.

2026-05-190

go-lint

marsolab/plainq

Linting and formatting Go code with `gofmt`, `goimports`, `golangci-lint`, and a curated `.golangci.yml`. ALWAYS use this skill when setting up or troubleshooting Go linting — installing golangci-lint, dropping the bundled `.golangci.yml` into a project, configuring pre-commit hooks, picking which linters to enable (errcheck, govet, staticcheck, unused, misspell, prealloc, gosec, revive, gocritic), suppressing findings with `//nolint` comments, reading lint output, or wiring lint into CI. Pair with go-style for the underlying idioms most lint rules enforce.

2026-05-190

name	perf-ab-testing
description	Use when measuring or comparing PlainQ's gRPC performance — running the perf/ AB harness, comparing a branch or PR against a stable ref, reasoning about a latency/throughput regression, reading the "PlainQ AB Performance" Grafana dashboard or the markdown report, editing the k6 gRPC load script, the VictoriaMetrics scrape config, the Performance AB GitHub workflow, or the perf Dockerfile. Covers the candidate-vs-baseline AB model, the report verdict, and the per-PR CI integration.
version	1.0.0
tags	["performance","benchmarking","k6","grpc","victoriametrics","grafana","ci","plainq"]

PlainQ performance AB testing

Overview

Core idea: never read a single absolute number — always compare candidate against the baseline ref under the same load.

When to use

"Did this change make PlainQ slower/faster?" / suspected perf regression.
Comparing a branch or PR against main (or a tag) for latency/throughput.
"Just spin a load test" against a running server.
Reading the AB Grafana dashboard or perf/results/report-*.md.
Editing the harness: cmd/perfctl, k6/ab_test.js, k6/load_test.js, victoriametrics/scrape.yml, grafana/, Dockerfile.plainq, scripts/, or .github/workflows/perf.yml.

Quick reference

perfctl (cmd/perfctl) is the single-entry CLI; make targets wrap it.

Action	Command
Build the CLI	`make -C perf cli` → `./perf/perfctl`
AB run vs `origin/main` (leaves stack up)	`perfctl ab` or `make -C perf ab`
AB vs a specific ref, heavier run	`perfctl ab -baseline v0.1.0 -vus 50 -duration 5m`
Load one running server (no baseline)	`perfctl load -target localhost:8080`
Stack up / URLs / down / cleanup	`perfctl up` / `dashboard` / `down` / `clean`
Re-run only AB k6 on the live stack	`make -C perf k6`

Reading results

Markdown report — perf/results/report-<run-id>.md (also printed by run.sh). Verdict: ✅ NO REGRESSION, 🚀 IMPROVEMENT, or ⚠️ REGRESSION (candidate end-to-end p95 >10% above baseline, or worse error rate). report.py exits non-zero on regression so run.sh / CI can gate.
Grafana — http://localhost:3000 → PlainQ AB Performance: latency (p50/p95/p99), throughput, errors, per-op latency, server resources, all baseline-vs-candidate.
Metrics in VictoriaMetrics (:8428, prefix k6_): k6_plainq_latency_p95{variant,op}, k6_plainq_reqs_total{variant,op}, k6_plainq_errs_total{variant,op}.

On every PR (CI)

.github/workflows/perf.yml runs the harness automatically:

Pull requests → candidate = the PR, baseline = the target branch; a light run (10 VUs, 45s). The report is posted as a sticky PR comment.
Push to main → candidate = new main, baseline = previous commit; a thorough run (30 VUs, 2m) for the baseline trend.

It is informational — the verdict never fails the check. Full data is in the run's perf-results artifact. To make it a hard gate, remove continue-on-error from the Run AB test step.

Common mistakes

Comparing absolute numbers across machines. Only candidate-vs-baseline on the same run is meaningful; CI runners are noisy.
Forgetting KEEP_UP=0 in automation — run.sh leaves the stack up by default for interactive use.
Expecting an xk6 plugin build. k6's gRPC support is native (k6/net/grpc); no plugin compile is needed.
Server-resource panels show n/a. They depend on the server exposing process/Go metrics; the k6 client-side latency/throughput is the source of truth regardless.
Editing the dashboard JSON by hand and breaking it. Validate with jq empty perf/grafana/dashboards/plainq-ab.json.