菜单
Use when triaging a misbehaving ovstorage-broker - covers startup failures, authz denies, listener authn mismatches, streaming write aborts, reload misfires, and drain hangs.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | ovstorage-operator-debug-broker |
| description | Use when triaging a misbehaving ovstorage-broker - covers startup failures, authz denies, listener authn mismatches, streaming write aborts, reload misfires, and drain hangs. |
| license | CC-BY-4.0 |
| version | 0.1.0 |
| author | NVIDIA Omniverse |
| tags | ["ovstorage","broker","debugging"] |
| tools | ["Read","Bash"] |
| compatibility | Requires access to broker logs, metrics, configuration, and operator-approved diagnostic commands. Credentials are user-supplied. |
Triage a misbehaving broker using the typed errors, metric labels, and tracing fields the daemon emits. The broker fails closed by design; most live-incident classes surface as a typed error before any backend dispatch.
docs/public/broker-operator/README.md
§ Debug runbook for the structured walkthrough.cargo run -p xtask -- list-routes --config /etc/ovstorage/broker.toml
against the broker's config. The same TOML validator the daemon
runs at startup will report bad routes, unresolvable secret
refs, and invalid listener authn config without binding any
sockets.broker_authz_decisions_total{outcome="deny"}. The deny path
threads the plugin's reason and explanation into the gRPC
PermissionDenied message; the explanation is typically a
stable rule id from the configured authz plugin.PolicyEpochStale errors, the request was stamped with a
pre-reload epoch. Have the SDK refresh-and-retry (it's wired
in broker-client's with_route_retry).Unauthenticated), confirm the
mode matches what your callers send:
jwt_verify mode: bearer JWT against the configured
issuer / audience / jwks_url. JWKS is cached
5 minutes; an unknown kid triggers one refetch before
Unauthenticated.peer_cred mode: UDS / npipe only. Invalid on TCP
listeners.trusted_* modes: only on trusted_proxy = true listeners
with trusted_peers CIDR allow-list.mtls: reserved; startup fails with Unsupported if
configured.[[connections]] blocks: backends register at startup; a
missing or misnamed plugin manifest fails startup. Confirm
OVSTORAGE_PLUGIN_DIR is set and readable.Ok(None) from the inbound stream) —
broker commits with the chunks received.Err(status) with
status.code() == Cancelled) — broker aborts with
Status::cancelled. Look for broker.write spans
ending in Cancelled to identify these.WRITE_BODY_BYTE_CAP (64 MiB)
after authz. Larger writes need the redirect branch (S3
multipart, GCS resumable, etc.); the plugin must mint
redirects via WriteStep::Redirects.broker_lifecycle_events_total{event="reload_failed"}
increments on validation failure; old broker stays live.broker_policy_epoch_advances_total increments on
successful reload.drain_timeout (default
DEFAULT_DRAIN_TIMEOUT). A wedged plugin / upstream
pushes the broker past the deadline and forces truncated
in-flight RPCs.Unauthenticated for every jwt_verify
request. Check the IdP's reachability from the broker host.AuthzPlugin cdylib FFI does not propagate cancellation
today; plugin authors are expected to bound their own work).
Watch
histogram_quantile(0.99, rate(broker_rpc_seconds_bucket[5m]))
trending up — though note the RPC-latency histogram is
dormant today (registered but not observed), so this query
may return empty.broker_lifecycle_events_total{event="reload_failed"}
increments and the broker keeps the previous epoch.ResourceExhausted (the cap is the hard-coded
DEFAULT_WATCH_DIRECTORY_FANOUT_LIMIT = 256).Always run
cargo run -p xtask -- list-routes --config <broker-config>
against a config change before SIGHUP or restart. The same
validator the daemon runs at startup will surface typed errors
without binding any socket.
explain-decision <audit-id> tool. Trace the audit_id
through your log aggregator.tracing output.route.id and backend.id are not stamped on per-RPC spans
yet — closing that gap is a tracked work item.docs/public/broker-operator/README.md
— full debug runbook + observability + implementation gaps.ovstorage-operator-monitor-broker
— what to scrape and which spans carry which fields.ovstorage-operator-configure-broker-policy
— interpreting authz denies and policy-epoch advances.ovstorage-operator-deploy-broker —
baseline deployment checklist.Deploy and develop against NVIDIA Omniverse Storage APIs on any Kubernetes cluster (MicroK8s, EKS, AKS, GKE, bare metal). Two storage adapters: (1) Example Storage Adapter — Python filesystem reference implementation + Discovery for learning and custom adapter development; (2) S3/Azure Production Storage Adapter — NVIDIA pre-built S3/Azure adapter + Discovery as minimum, expandable with Event Aggregation, Event Consumer, RabbitMQ, Envoy Auth Extension, Storage Navigator, and Contour ingress. Can deploy on the developer's behalf (generating scripts + .env) or guide manual deployment. Custom adapter development covers three APIs: Storage, Notifications, and Permissions. Use when a developer asks about deploying, configuring, validating, troubleshooting, or building any component of the Omniverse Storage APIs stack.
Use when adding or reviewing an authz plugin for ovstorage-broker or ovstorage-rest - covers the AuthzPlugin trait, the 21 operations, the policy-epoch model, and the cdylib cancellation contract.
Use when adding or reviewing a storage backend plugin for ovstorage.
Use when Rust changes affect ovstorage C or C++ headers and checked-in generated headers must be refreshed.
Use when starting an ovstorage-broker locally for development or integration testing - covers a minimal TOML, the cdylib plugin path, and how to hit the listener with broker-client or curl.
Use when checking the Storage API wire contract that ovstorage-plugin-services-client depends on.