원클릭으로 Manus에서 모든 스킬 실행

perf-ab-testing

스타0

포크0

업데이트2026년 6월 28일 11:37

Use when measuring or comparing PlainQ's gRPC performance — running the perf/ AB harness, comparing a branch or PR against a stable ref, reasoning about a latency/throughput regression, reading the "PlainQ AB Performance" Grafana dashboard or the markdown report, editing the k6 gRPC load script, the VictoriaMetrics scrape config, the Performance AB GitHub workflow, or the perf Dockerfile. Covers the candidate-vs-baseline AB model, the report verdict, and the per-PR CI integration.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

marsolab

marsolab/plainq

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

PlainQ performance AB testing

Overview

PlainQ ships a self-contained AB harness under perf/. It builds two servers — candidate (current checkout) and baseline (a stable git ref) — runs an identical k6 gRPC workload (Send → Receive → Delete) at both at once, stores everything in VictoriaMetrics, and reports the comparison via a Grafana dashboard and a markdown report with a pass/fail verdict. Every metric is tagged variant=baseline|candidate and op=send|receive|delete|total.

Core idea: never read a single absolute number — always compare candidate against the baseline ref under the same load.

When to use

"Did this change make PlainQ slower/faster?" / suspected perf regression.
Comparing a branch or PR against main (or a tag) for latency/throughput.
"Just spin a load test" against a running server.
Reading the AB Grafana dashboard or perf/results/report-*.md.
Editing the harness: cmd/perfctl, k6/ab_test.js, k6/load_test.js, victoriametrics/scrape.yml, grafana/, Dockerfile.plainq, scripts/, or .github/workflows/perf.yml.

Quick reference

perfctl (cmd/perfctl) is the single-entry CLI; make targets wrap it.

Action	Command
Build the CLI	`make -C perf cli` → `./perf/perfctl`
AB run vs `origin/main` (leaves stack up)	`perfctl ab` or `make -C perf ab`
AB vs a specific ref, heavier run	`perfctl ab -baseline v0.1.0 -vus 50 -duration 5m`
Load one running server (no baseline)	`perfctl load -target localhost:8080`
Stack up / URLs / down / cleanup	`perfctl up` / `dashboard` / `down` / `clean`
Re-run only AB k6 on the live stack	`make -C perf k6`

Each flag has an env fallback (-vus/VUS, -baseline/BASELINE_REF, -target/TARGET_ADDR, …). Use -target host.docker.internal:PORT to load a server on the host. Knobs: BASELINE_REF, VUS, DURATION, BATCH_SIZE, MSG_BYTES, RUN_ID, KEEP_UP. See perf/README.md for the full table.

Reading results

Markdown report — perf/results/report-<run-id>.md (also printed by run.sh). Verdict: ✅ NO REGRESSION, 🚀 IMPROVEMENT, or ⚠️ REGRESSION (candidate end-to-end p95 >10% above baseline, or worse error rate). report.py exits non-zero on regression so run.sh / CI can gate.
Grafana — http://localhost:3000 → PlainQ AB Performance: latency (p50/p95/p99), throughput, errors, per-op latency, server resources, all baseline-vs-candidate.
Metrics in VictoriaMetrics (:8428, prefix k6_): k6_plainq_latency_p95{variant,op}, k6_plainq_reqs_total{variant,op}, k6_plainq_errs_total{variant,op}.

On every PR (CI)

.github/workflows/perf.yml runs the harness automatically:

Pull requests → candidate = the PR, baseline = the target branch; a light run (10 VUs, 45s). The report is posted as a sticky PR comment.
Push to main → candidate = new main, baseline = previous commit; a thorough run (30 VUs, 2m) for the baseline trend.

It is informational — the verdict never fails the check. Full data is in the run's perf-results artifact. To make it a hard gate, remove continue-on-error from the Run AB test step.

Common mistakes

Comparing absolute numbers across machines. Only candidate-vs-baseline on the same run is meaningful; CI runners are noisy.
Forgetting KEEP_UP=0 in automation — run.sh leaves the stack up by default for interactive use.
Expecting an xk6 plugin build. k6's gRPC support is native (k6/net/grpc); no plugin compile is needed.
Server-resource panels show n/a. They depend on the server exposing process/Go metrics; the k6 client-side latency/throughput is the source of truth regardless.
Editing the dashboard JSON by hand and breaking it. Validate with jq empty perf/grafana/dashboards/plainq-ab.json.

Agent/platform notes

This skill uses Claude Code tool names. On Codex/others, map them per .agents/skills/using-superpowers/references/codex-tools.md (e.g. Bash → your shell). The harness itself is plain make/docker/git — identical on every platform.

이 저장소의 다른 Skills

같은 저장소

go-cli

marsolab/plainq

Building command-line tools in Go with the stdlib `flag` package. ALWAYS use this skill when writing or reviewing Go CLI code — top-level flags with `flag.Parse`, subcommands with `flag.NewFlagSet`, exit codes, writing errors to `os.Stderr`, signal handling, environment-variable fallbacks, `-h`/`-help` output, and `cmd/<tool>` project layout. Avoid reaching for cobra/urfave-cli unless the tool clearly needs it (~5+ subcommands with rich help). Pair with go-style for naming and config patterns, go-errors for exit-code mapping, and go-logging for diagnostic output.

2026-05-190

go-concurrency

marsolab/plainq

Disciplined Go concurrency. ALWAYS use this skill when writing or reviewing Go code that touches goroutines, channels, `context.Context`, `sync.WaitGroup`, `sync.Mutex` / `sync.RWMutex`, `sync.Once`, `errgroup.Group`, worker pools, fan-out/fan-in pipelines, request cancellation, timeouts, deadlines, or any "run this in the background" pattern. Also use when debugging deadlocks, goroutine leaks, race conditions, or channel-related bugs. Pair with go-errors for error propagation across goroutines.

2026-05-190

go-dev

marsolab/plainq

Umbrella skill for Go development. ALWAYS use this skill when the user's task involves Go or Golang in any way — writing new Go code, reviewing existing Go code, building Go services, creating Go CLI tools, working with Go tests, setting up Go linting, using sqlc for database access, or anything else Go-related. This includes mentions of: Go, Golang, .go files, go.mod, go test, golangci-lint, Chi router, sqlc, goose migrations, generics, goroutines, channels, interfaces, error handling, concurrency, table-driven tests, slog, iterators, or any Go package/tool. This skill is a router — it loads fast and points to the focused sibling skill(s) that own the deep reference material for the task at hand.

2026-05-190

go-errors

marsolab/plainq

Idiomatic Go error handling. ALWAYS use this skill when writing or reviewing Go code that raises, returns, wraps, matches, or aggregates errors — questions about `fmt.Errorf` with `%w` vs `%v`, `errors.Is` and `errors.As`, sentinel errors, typed error structs, `errors.Join` for combining failures, the "handle exactly once" rule, panic vs error, guard clauses and early return, error message formatting, or any `if err != nil` pattern. Use alongside go-style for general idioms.

2026-05-190

go-http

marsolab/plainq

Building HTTP services in Go with the Chi router and stdlib `net/http`. ALWAYS use this skill when writing or reviewing Go HTTP code — defining routes with `chi.Router`, middleware (`chi.Use`, `middleware.Logger`, `middleware.Recoverer`, `middleware.RequestID`), handler signatures, request decoding, JSON responses, status codes from typed errors, graceful shutdown with `http.Server.Shutdown`, `httptest` patterns, service project layout (`cmd/server`, `internal/handler`, `internal/service`), and timeouts (`ReadTimeout`, `WriteTimeout`, `IdleTimeout`). Pair with go-sql for the data layer, go-logging for request-scoped logging, go-errors for status code mapping, and go-testing for handler tests.

2026-05-190

go-lint

marsolab/plainq

Linting and formatting Go code with `gofmt`, `goimports`, `golangci-lint`, and a curated `.golangci.yml`. ALWAYS use this skill when setting up or troubleshooting Go linting — installing golangci-lint, dropping the bundled `.golangci.yml` into a project, configuring pre-commit hooks, picking which linters to enable (errcheck, govet, staticcheck, unused, misspell, prealloc, gosec, revive, gocritic), suppressing findings with `//nolint` comments, reading lint output, or wiring lint into CI. Pair with go-style for the underlying idioms most lint rules enforce.

2026-05-190

name	perf-ab-testing
description	Use when measuring or comparing PlainQ's gRPC performance — running the perf/ AB harness, comparing a branch or PR against a stable ref, reasoning about a latency/throughput regression, reading the "PlainQ AB Performance" Grafana dashboard or the markdown report, editing the k6 gRPC load script, the VictoriaMetrics scrape config, the Performance AB GitHub workflow, or the perf Dockerfile. Covers the candidate-vs-baseline AB model, the report verdict, and the per-PR CI integration.
version	1.0.0
tags	["performance","benchmarking","k6","grpc","victoriametrics","grafana","ci","plainq"]

PlainQ performance AB testing

Overview

Core idea: never read a single absolute number — always compare candidate against the baseline ref under the same load.

When to use

"Did this change make PlainQ slower/faster?" / suspected perf regression.
Comparing a branch or PR against main (or a tag) for latency/throughput.
"Just spin a load test" against a running server.
Reading the AB Grafana dashboard or perf/results/report-*.md.
Editing the harness: cmd/perfctl, k6/ab_test.js, k6/load_test.js, victoriametrics/scrape.yml, grafana/, Dockerfile.plainq, scripts/, or .github/workflows/perf.yml.

Quick reference

perfctl (cmd/perfctl) is the single-entry CLI; make targets wrap it.

Action	Command
Build the CLI	`make -C perf cli` → `./perf/perfctl`
AB run vs `origin/main` (leaves stack up)	`perfctl ab` or `make -C perf ab`
AB vs a specific ref, heavier run	`perfctl ab -baseline v0.1.0 -vus 50 -duration 5m`
Load one running server (no baseline)	`perfctl load -target localhost:8080`
Stack up / URLs / down / cleanup	`perfctl up` / `dashboard` / `down` / `clean`
Re-run only AB k6 on the live stack	`make -C perf k6`

Reading results

Markdown report — perf/results/report-<run-id>.md (also printed by run.sh). Verdict: ✅ NO REGRESSION, 🚀 IMPROVEMENT, or ⚠️ REGRESSION (candidate end-to-end p95 >10% above baseline, or worse error rate). report.py exits non-zero on regression so run.sh / CI can gate.
Grafana — http://localhost:3000 → PlainQ AB Performance: latency (p50/p95/p99), throughput, errors, per-op latency, server resources, all baseline-vs-candidate.
Metrics in VictoriaMetrics (:8428, prefix k6_): k6_plainq_latency_p95{variant,op}, k6_plainq_reqs_total{variant,op}, k6_plainq_errs_total{variant,op}.

On every PR (CI)

.github/workflows/perf.yml runs the harness automatically:

Pull requests → candidate = the PR, baseline = the target branch; a light run (10 VUs, 45s). The report is posted as a sticky PR comment.
Push to main → candidate = new main, baseline = previous commit; a thorough run (30 VUs, 2m) for the baseline trend.

It is informational — the verdict never fails the check. Full data is in the run's perf-results artifact. To make it a hard gate, remove continue-on-error from the Run AB test step.

Common mistakes

Comparing absolute numbers across machines. Only candidate-vs-baseline on the same run is meaningful; CI runners are noisy.
Forgetting KEEP_UP=0 in automation — run.sh leaves the stack up by default for interactive use.
Expecting an xk6 plugin build. k6's gRPC support is native (k6/net/grpc); no plugin compile is needed.
Server-resource panels show n/a. They depend on the server exposing process/Go metrics; the k6 client-side latency/throughput is the source of truth regardless.
Editing the dashboard JSON by hand and breaking it. Validate with jq empty perf/grafana/dashboards/plainq-ab.json.