| name | profile-performance-certus-api-bench |
| description | Measure system performance of this platform using |
| argument-hint | [output-dir] |
The purpose of this skill is to capture a snapshot of performance by running certus-server (release build) and certus-api-bench.py.
Create the output directory from the output-dir argument, for example:
mkdir -p
Write PROFILE.md to /PROFILE.md.
Collect data for combinations of the following parameters:
- SSD device count D (1..N), where N is the number of detected SSD PCI devices.
Detect available SSDs via
lspci -d <vendor>:<device> or equivalent, sort PCI addresses lexicographically,
and use the first D addresses.
Pass each selected device as repeated flags:
--device-pci --device-pci ... --device-pci
If D > N, skip that combination and record SKIPPED: requested D > available N in PROFILE.md.
- Memory tier size: 4GB, 8GB, 16GB with
--memory-tier-size <bytes> (use bytes, e.g. 4294967296).
- Number of clients: 1, 2, 4, 8, 16 with
certus-api-bench --clients <count>.
- Cache block size: 1MB, 2MB, 4MB, 5MB with
certus-api-bench --block-size <bytes> (use bytes, e.g. 1048576).
- Batch size: 8, 16, 32, 64 with
certus-api-bench --batch-size <count>.
- Objects per lookup batch: 16, 32 with
certus-api-bench --num-objects <count>.
- Iterations: 10, 20, 50 with
certus-api-bench --iterations <count>.
Other requirements:
- Run certus-server with
--format.
- Run a full factorial Cartesian product across the listed values. If the total number of combinations exceeds 200, sample uniformly at random to 200 combinations and document the strategy in PROFILE.md.
- Execute experiments sequentially by default, restarting certus-server between combinations and avoiding parallel runs.
- For each combination:
a. Stop any running certus-server.
b. Start certus-server with the selected
--device-pci flags, --memory-tier-size, and --format.
c. Retry startup up to 3 times with a 5s delay if it fails. If startup still fails, record the combination as FAILED in PROFILE.md with stderr and continue.
d. After a healthy start, run a 1-iteration warm-up of certus-api-bench before measurement.
e. Run the measured iterations.
f. If certus-api-bench exits non-zero, capture stdout/stderr to <output-dir>/logs/<run-name>.log, mark the run as ERROR in PROFILE.md, retry once, and if still failing continue to the next combination.
g. Stop certus-server and collect logs.
- For each run write raw JSON to:
/results/devicecount-_mem-_clients-_bs-_batch-_objs-_iter-.json
and a human-readable log to:
/logs/devicecount-_mem-_clients-_bs-_batch-_objs-_iter-.log
Run combinatorial experiments and build a result table in /PROFILE.md.
PROFILE.md must include a table with columns:
device_count | pci_list | memory_tier_size_bytes | clients | block_size_bytes | batch_size | num_objects | iterations | throughput_GB_per_sec (populate) | throughput_GB_per_sec (hot lookup) | throughput_GB_per_sec (cold lookup) | notes
Include raw output file links and log links in PROFILE.md.
Summarize the data and identify performance anomalies.
Create a plot with Y-axis throughput, X-axis device_count for:
- Populate
- Lookup (hot)
- Lookup (cold)
Save plot in output directory.
