j6-ucp-perfetto-trace-catcher

name

j6-ucp-perfetto-trace-catcher

description

Capture UCP (Horizon Robotics inference SDK) Perfetto traces from a J6 development board. Use this skill whenever the user wants to grab UCP traces, capture Perfetto traces on a Horizon dev board, debug UCP scheduling issues with trace data, or pull .pftrace files from a remote board. Also trigger when the user mentions UCP performance analysis, UCP trace, hrt_model_exec tracing, or BPU trace capture on J6/S1000 platforms. Do NOT use for DSP trace or Chrome trace.

UCP Trace Catcher

Capture Perfetto-based UCP traces from a remote J6 development board and bring them back to the X86 host for analysis.

What this skill does

Automates the full lifecycle of UCP Perfetto trace capture:

1. Transfer config files to the dev board
2. Set up the trace environment and run the application
3. Pull the captured trace back to the host
4. Tell the user what to do next

Only Perfetto trace is in scope — ignore DSP trace and Chrome trace.

Never modify original files in this skill's directory. Copy config files to a temp location under the local current workspace (pwd), edit the copy there, then transfer.

Before starting

You need the following information from the user. Ask for anything not provided:

Parameter	Description	Example
board_ip	IP address or hostname of the dev board	192.168.1.100
board_user	SSH username for the board (default: root if not specified, do not use local username)	root
work_dir	Directory on the board where the UCP app runs	/userdata/ucp_test
mode	in_process or system	system
enable_bpu	Whether to capture BPU trace (system mode only, default: false)	false
output_name	Desired trace filename (default: ucp.pftrace)	my_trace.pftrace
app_command	The UCP application command to run	./hrt_model_exec perf --model_file model.hbm --frame_count 1000 --thread_num 8

If mode is not specified, ask the user:

• in_process: captures only UCP process-internal trace (track_event). Does NOT support thread scheduling states, ftrace, memory stats, process stats, or BPU trace. No daemon needed. Good for UCP scheduling logic analysis.
• system: captures UCP trace + thread scheduling states (ftrace), memory stats, and process stats by default (uses ucp_system.cfg). When BPU trace is enabled, the same ucp_system.cfg is used but BPU trace is additionally turned on via sysfs commands; BPU trace data will appear alongside ftrace data. Needs tracebox daemons. Use system mode when you need system-level visibility beyond UCP internals.

Important: The pre-built config files in reference/ cover all modes. Select the right one by mode — do NOT modify config files unless you have a specific reason (e.g., adjusting buffer size or duration).

Step 1: Transfer config files to the board

Config files are in this skill's reference/ directory. Select the right cfg file based on mode:

Mode	cfg file	json file
in_process	ucp_in_process.cfg	ucp_in_process.json
system	ucp_system.cfg	ucp_system.json

All system mode configs share the same ucp_system.json (it just sets "backend": "system"). System mode always uses ucp_system.cfg regardless of BPU trace — BPU trace is controlled by sysfs commands, not the cfg file.

Transfer via scp:

For in_process mode

scp reference/ucp_in_process.cfg reference/ucp_in_process.json <board_user>@<board_ip>:<work_dir>/

For system mode

scp reference/ucp_system.cfg reference/ucp_system.json <board_user>@<board_ip>:<work_dir>/

If you edited a config copy (e.g., adjusting duration_ms or output_path), scp the copy instead of the original reference/ file.

Step 2: Set up environment and run the application

Environment variables and the app must run in the same SSH session, otherwise the variables are lost. Always combine setup and app execution into a single SSH heredoc.

For in_process mode

ssh <board_user>@<board_ip> << 'EOF'
cd <work_dir>

# Remove old trace file if it exists (Perfetto cannot overwrite)
rm -f <output_name>

# Set environment variables
export HB_UCP_PERFETTO_CONFIG_PATH=ucp_in_process.json
export HB_UCP_ENABLE_PERFETTO=true

# Run the UCP application
<App command>
EOF

For system mode

ssh <board_user>@<board_ip> << 'EOF'
cd <work_dir>

# Remove old trace file if it exists
rm -f <output_name>

# Kill existing tracebox processes to avoid interference
pkill -f "tracebox" 2>/dev/null || true
sleep 1

# Start Perfetto daemons
tracebox traced --background
tracebox traced_probes --background --reset-ftrace

# If BPU trace is enabled, query BPU core count and turn on BPU trace on ALL cores
if <enable_bpu>; then
    BPU_CORE_NUM=$(cat /sys/devices/system/bpu/core_num)
    for i in $(seq 0 $(expr $BPU_CORE_NUM - 1)); do
        echo 0 > /sys/devices/system/bpu/bpu${i}/power_enable 2>/dev/null || true
        echo 1 > /sys/devices/system/bpu/bpu${i}/trace
    done
fi

# Start trace capture in background and record its PID
tracebox perfetto --txt -c <cfg_file_from_step1> -o <output_name> &
PERFETTO_PID=$!

# Set UCP environment variables
export HB_UCP_PERFETTO_CONFIG_PATH=ucp_system.json
export HB_UCP_ENABLE_PERFETTO=true

# Run the UCP application
<App command>

# Wait for the background perfetto process to finish writing trace data
wait $PERFETTO_PID

# If BPU trace was enabled, disable it on ALL cores
if <enable_bpu>; then
    BPU_CORE_NUM=$(cat /sys/devices/system/bpu/core_num)
     i  $( 0 $(  - 1)); 
         0 > /sys/devices/system/bpu/bpu/trace
    

EOF

Important: in system mode, the tracebox perfetto process must remain running for the entire duration of the UCP application. If it exits before the app, the trace will be incomplete. After the app finishes, wait $PERFETTO_PID ensures the perfetto process completes writing all trace data before cleanup.

Step 3: Pull the trace file back to the host

After the application finishes:

scp <board_user>@<board_ip>:<work_dir>/<output_name> ./

Verify the file was transferred and has reasonable size (should be > 0 bytes).

Step 4: Tell the user what to do next

Inform the user:

• The trace file is saved at <local_path>/<output_name>
• They can open it in Perfetto UI for visualization
• If BPU trace was captured, they need the hbperfetto tool (contact Horizon support) to view BPU-specific data
• Key Perfetto UI shortcuts: w/s to zoom in/out, a/d to pan left/right, ? for help

Handling common issues

Problem	Solution
Trace file already exists	Delete it with rm -f ucp.pftrace before capture. Perfetto cannot overwrite.
tracebox: command not found	The tracebox binary may not be in PATH. Ask the user where it's located.
Trace is empty or 0 bytes	Likely env vars weren't set in the same shell as the app, or perfetto process crashed. Check with wait $PERFETTO_PID — if it returns a non-zero exit code, perfetto exited abnormally.
BPU trace not showing	Verify BPU trace was enabled on ALL cores: query cat /sys/devices/system/bpu/core_num for the core count, then check echo 1 > /sys/devices/system/bpu/bpu${i}/trace was executed for each core i from 0 to core_num - 1 before starting the capture. Also verify power_enable was 0 on each core.
Old perfetto processes interfere	Kill existing tracebox perfetto and tracebox traced processes before starting a new capture.
BPU trace not cleaned up	After capture, disable BPU trace on ALL cores: loop through echo 0 > /sys/devices/system/bpu/bpu${i}/trace for each core i from 0 to core_num - 1. Leaving it on may affect subsequent runs.
Config file not found on board	The config files use relative paths — they must be in the same directory as the running program.

Config file reference

All config files are in this skill's reference/ directory. Read them when you need to inspect or modify the configurations before transfer.

File	Mode	Trace modules	Buffers
ucp_in_process.json + ucp_in_process.cfg	in_process	UCP	1
ucp_system.json + ucp_system.cfg	system	UCP + ftrace + BPU trace (BPU data requires sysfs enable)	2

Key parameters you might want to adjust:

Parameter	File	Default	Purpose
duration_ms	all .cfg	10000 (10s)	How long Perfetto captures. For in_process mode, trace data after this duration is discarded — increase this value if the app runs longer than 10s. For system mode, the perfetto process exits after this duration regardless of whether the app has finished.
size_kb	ucp_in_process.cfg	65535	Buffer size for in_process mode.
size_kb	ucp_system.cfg	131072	Buffer size per buffer for system mode (2 buffers).
file_write_period_ms	ucp_in_process.cfg	2500	How often buffer is flushed to file.
write_into_file	ucp_in_process.cfg	true	Periodic flush to file. Keep true for reliability.
bputrace_period_ms	ucp_system.cfg	500	BPU trace sampling period. Decrease under heavy BPU load. Data appears only when BPU trace is enabled via sysfs.
output_path	ucp_in_process.cfg	ucp.pftrace	Trace output path on the board.