perf-echo

name	perf-echo
description	Use when updating, validating, regenerating, or documenting py-netty perf_echo benchmarks, charts, README benchmark data, or local echo performance comparisons against asyncio and threaded sockets.

perf_echo Benchmark Workflow Skill

Use this workflow when updating, validating, or regenerating the local echo performance benchmark for py-netty.

Goal

Produce reproducible local benchmark data for py-netty echo traffic, optionally compare it with Python standard-library implementations, generate charts, and update README benchmark documentation.

Benchmark Runner

Primary command:

python integration_tests/perf_echo.py --case all --engine all --timeout 20 --json

High connection-count scaling command:

python integration_tests/perf_echo.py --case high_connection_scaling --engine all --timeout 30 --json

Useful smoke command:

python integration_tests/perf_echo.py --case single_connection_latency --messages 3 --payload-size 32 --timeout 5 --engine all

Run py-netty clients with multiple event loops:

python integration_tests/perf_echo.py --case large_payload_throughput --engine py-netty --connections 32 --messages 64 --client-eventloops 2

Override the traffic pattern when needed:

python integration_tests/perf_echo.py --case large_payload_throughput --sequential
python integration_tests/perf_echo.py --case single_connection_latency --no-sequential

Run against a separately deployed echo server:

python integration_tests/perf_echo.py --case single_connection_latency --messages 20 --payload-size 64 --timeout 5 --engine py-netty --port 19080
python integration_tests/perf_echo.py --case single_connection_latency --messages 20 --payload-size 64 --timeout 5 --engine py-netty --host 10.0.0.5 --port 19080

Supported engines:

• py-netty
• asyncio
• threaded
• all

Supported cases:

• single_connection_latency
• backpressure_smoke
• large_payload_throughput
• small_payload_concurrency
• connection_ramp_up
• high_connection_128
• high_connection_256
• high_connection_512
• high_connection_scaling
• all

all runs the default benchmark suite only. high_connection_scaling is a separate group for heavier local runs that stress 128, 256, and 512 concurrent connections to identify where the one-thread-per-connection threaded engine starts losing ground.

Test Principle

By default, each case starts a local in-process echo server for the selected engine and creates matching clients. When --port is provided, the runner skips local server startup and connects clients to the separately deployed echo server at --host and --port; --host defaults to 127.0.0.1. Clients send fixed-size framed payloads. The first 8 bytes encode:

• connection_id
• sequence

The remaining bytes are deterministic filler. The client validates every echoed frame to handle TCP split/coalesced reads correctly.

The run fails only for functional errors:

• connection failures
• missing echoes
• payload mismatches
• sequence mismatches
• timeouts

Throughput and latency values are informational and environment-dependent.

Data Collection

Save benchmark JSON outside the repo unless the user explicitly asks to commit raw data:

python integration_tests/perf_echo.py --case all --engine all --timeout 20 --json > /tmp/py-netty-perf-compare.json

For high connection-count comparisons:

python integration_tests/perf_echo.py --case high_connection_scaling --engine all --timeout 30 --json > /tmp/py-netty-high-connections.json

Before using the data, confirm every result has:

• empty errors
• timed_out set to false
• received_messages == sent_messages
• received_bytes == sent_bytes

Chart Generation

Generate charts from /tmp/py-netty-perf-compare.json into img/.

Required chart files:

• img/perf_echo_throughput.png
• img/perf_echo_message_rate.png
• img/perf_echo_latency.png
• img/perf_echo_ramp_up.png

Chart requirements:

• Sort cases by connection count ascending.
• Use grouped bars for py-netty, asyncio, and threaded when comparison data is available.
• Label axes with units.
• Keep filenames stable so README links do not churn.

Recommended metric mapping:

• Throughput: bytes_per_second / 1024 / 1024, unit MiB/s.
• Message rate: messages_per_second, unit msg/s.
• Latency: latency_p95_ms, unit ms.
• Ramp-up: ramp_up_seconds * 1000, unit ms.

README Update

Update the ## Benchmark section in README.md.

The benchmark section should include:

• the exact command used to collect data
• local environment summary
• a table with case, engine, connections, payload, messages, throughput, message rate, p50 latency, p95 latency, and ramp-up
• chart links to the four img/perf_echo_*.png files
• a note that metrics are informational and environment-dependent
• a note that benchmark failures are functional failures, not performance threshold failures

Keep the table ordered by connection count:

1. single_connection_latency
2. backpressure_smoke
3. large_payload_throughput
4. small_payload_concurrency
5. connection_ramp_up

When documenting high connection-count scaling, keep those rows separate from the default benchmark table unless the user asks to replace the main benchmark data. Order them by connection count:

1. high_connection_128
2. high_connection_256
3. high_connection_512

For each case, list engines in this order:

1. py-netty
2. asyncio
3. threaded

Validation

Run these checks after modifying the benchmark runner, charts, or README:

python integration_tests/perf_echo.py --help
python integration_tests/perf_echo.py --case single_connection_latency --messages 3 --payload-size 32 --timeout 5 --engine all
python integration_tests/perf_echo.py --case high_connection_scaling --engine threaded --connections 2 --messages 1 --payload-size 64 --timeout 5 --json
pytest -q

When changing external echo server support, also validate --port against a running echo server. --host can be omitted for 127.0.0.1:

python integration_tests/perf_echo.py --case single_connection_latency --messages 3 --payload-size 32 --timeout 5 --engine py-netty --port <external-echo-port>

Validate chart files are readable and non-empty:

python - <<'PY'
from pathlib import Path
from PIL import Image

for path in sorted(Path("img").glob("perf_echo_*.png")):
    with Image.open(path) as image:
        print(path, image.size, image.mode)
PY

Cautions

• Do not add these benchmark cases to default pytest; they are local integration/performance checks.
• Do not commit /tmp raw output files.
• Do not use hard performance thresholds unless the user explicitly asks.
• Do not compare results from different machines as absolute performance claims.
• If a high-connection case fails with Too many open files, lower --connections or raise the OS file descriptor limit before rerunning.
• Do not run the full high_connection_scaling group as a casual smoke check; use overrides such as --connections 2 --messages 1 when validating CLI behavior only.