bfd-data-acquisition

name	bfd-data-acquisition
description	Use when acquiring STM32 runtime data from global symbols, memory regions, stack-published local slots, or RTT channels and converting captures into structured artifacts for analysis.

BFD Data Acquisition

Self-Improvement Loop (Required)

• When this skill encounters a build, flash, debug, script-usage, or other workflow problem, do not stop at the local workaround after the issue is solved.
• Record the resolved issue and lesson in BFD-Kit/.learnings/ERRORS.md and/or BFD-Kit/.learnings/LEARNINGS.md; unresolved capability gaps go to BFD-Kit/.learnings/FEATURE_REQUESTS.md.
• Promote reusable fixes into the affected BFD-Kit asset in the same task when feasible: update the relevant SKILL.md, script, wrapper, or resource so the next run benefits by default.
• When a learning is promoted into a skill or script, append a short entry to BFD-Kit/.learnings/CHANGELOG.md and mention the improvement in the task close-out.

Purpose

Use this skill when the task is "show the contents of this global/static object" and the object can be resolved from an ELF.

This skill is the default path for:

• global/static object inspection
• pointer-hub decoding
• enum-bearing state objects
• stack-published local-slot capture through pointer symbols
• raw RAM capture only when no stable symbol exists

This skill is not tied to any motor model or business object. The primary workflow is generic ELF + symbol + DWARF.

Required Precheck

Run bootstrap before any acquisition command:

python3 ./.codex/skills/bfd-project-init/scripts/bootstrap.py --project-root . --mode check

Fail fast if ${STM32_ELF} or the active bootstrap profile is missing.

Before the first symbol-auto run, also verify that the exact interpreter behind python3 can import elftools:

python3 -c "import elftools, sys; print(sys.executable)"

If this fails:

• on Ubuntu system Python, install python3-pyelftools
• if an activated conda environment shadows system python3, either install pyelftools into that environment or invoke /usr/bin/python3 explicitly for the acquisition command

Primary Command Contract

Primary script:

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py ...

Primary mode:

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py \
  --elf "${STM32_ELF}" \
  --mode symbol-auto \
  --symbol <global_symbol> \
  --follow-depth 1 \
  --format summary \
  --output logs/data_acq/<symbol>.summary

Meaning of the primary arguments:

• --mode symbol-auto: resolve the symbol from ELF, reflect its DWARF type, reuse or rebuild schema cache, sample RAM, and decode named fields.
• --symbol <global_symbol>: required global or static object symbol.
• --follow-depth <N>: pointer follow depth.
• --format summary|json|csv: output renderer.

symbol-auto rules:

• requires --symbol
• supports only --capture-mode snapshot
• rejects manual decode options such as --decode-profile, --layout, --pointer-array, and --follow-pointer
• writes or reuses cache under .codex/bfd/dwarf_cache/
• when the script is invoked from a central BFD-Kit checkout, it also searches the current working directory upward for .codex/bfd/active_profile.env and .codex/bfd/dwarf_cache, so run it from the target project directory when you want project-local profile/cache resolution
• root object size and typedef-backed array element sizes come from the reflected DWARF schema; if a snapshot is shorter than expected, the script now reports an explicit capture-size mismatch or truncated nested field metadata instead of a raw Python struct.error

Follow-Depth Rules

• --follow-depth 0: decode the root object only; pointers remain address metadata.
• --follow-depth 1: follow one pointer layer. Use this first for pointer hubs and object tables.
• --follow-depth >1: recursively follow nested pointers. Increase only when the object layout requires it.

If --follow-depth >1 on a pointer array or pointer-rich object crashes with errors such as struct.error: unpack requires a buffer of 1 bytes, stop deep recursion on that object. Fall back to --follow-depth 1 to recover the first-layer structs plus child pointer addresses, then inspect the nested child objects separately with another acquisition or raw J-Link mem32 reads.

Standard Workflow Order

1. Use --mode symbol-auto first when the target object has DWARF-supported type information.
2. Fall back to --mode symbol only when:
   • the target type contains unsupported DWARF features
   • a manual decode profile is intentionally narrower than the full object
   • a typed scalar layout is sufficient
3. Use --pointer-symbol only for stack-published local data.
4. Use raw --address capture only when no stable symbol exists.

Native HSS Escalation

When the requirement is high-rate, non-halting sampling of a fixed-address scalar global/static variable on a J-Link target, prefer the native HSS CLI path instead of GUI J-Scope automation or halt/read/go loops:

bash BFD-Kit/scripts/bfd_jlink_hss.sh --json hss sample \
  --symbol chassis_parameter.IMU.yaw \
  --symbol chassis_parameter.IMU.pitch \
  --duration 0.3 \
  --period-us 1000 \
  --output logs/data_acq/imu_yaw_pitch_hss.csv

Use this path only for fixed-address scalar globals/statics. Repeat --symbol for multi-symbol sampling. On J-Link PLUS, SEGGER's model limits and local HSS verification both indicate a 10-symbol ceiling; do not treat hss inspect raw capability word 2 as the symbol-count limit. The wrapper uses BFD-Kit/.runtime/venv automatically after bash BFD-Kit/init_project.sh --project-root .. If the target is a pointer-driven object graph or needs DWARF field decoding, stay on symbol-auto. The command writes a wide CSV to --output and a metadata sidecar JSON to --output.meta.json.

Do not assume requested --period-us equals the achieved sample period on every probe model. Local RSCF_h7 validation with J-Link CE showed a practical single-float floor near 1000 us, even when --period-us was reduced from 1000 down to 1 and SWD was raised from 4000 to 8000 kHz. When a new J-Link model, target, or board wiring is introduced, run an explicit sweep first and report the effective period from captured sample counts instead of trusting the requested value or a capability word name.

ST-Link does not provide an HSS-equivalent path in this revision. If STM32_PROBE=stlink, treat high-rate capture as a separate polling/snapshot design problem rather than a drop-in replacement for native J-Link HSS.

If the requirement is textual RTT evidence over ST-Link, switch to the independent bfd-strtt-rtt skill instead of forcing that flow through J-Link RTT guidance.

For FanX/Tek DAPLink High or another CMSIS-DAP probe, use the PyOCD HSS-compatible sampler when native J-Link HSS is unavailable but a fixed-address scalar CSV is still needed:

RSCF_h7/.tools/pyocd-venv/bin/python BFD-Kit/scripts/bfd_pyocd_hss.py --json sample \
  --elf RSCF_h7/builds/gcc/debug/RSCF_H7_BOARD_ONE.elf \
  --target stm32h723xx \
  --uid 6d1395736d13957301 \
  --frequency 10000000 \
  --symbol g_system_status.status_byte \
  --duration 0.5 \
  --period-us 1000 \
  --output RSCF_h7/logs/data_acq/g_system_status_pyocd_hss.csv

This backend intentionally writes the same wide CSV + .meta.json shape as native HSS, but it is host-polled and uses host monotonic timestamps. Do not label it as SEGGER probe-side HSS. For deterministic high-rate system identification, prefer firmware telemetry mirror/ring or a custom DAPLink vendor-command backend once the probe firmware source is available.

When the question is specifically "how many float values can DAPLink / PyOCD hold at a stable 1000 Hz update?", use the built-in benchmark instead of guessing from one-off captures:

RSCF_h7/.tools/pyocd-venv/bin/python BFD-Kit/scripts/bfd_pyocd_hss.py --json benchmark-float \
  --address 0x200096E8 \
  --min-floats 1 \
  --max-floats 64 \
  --target stm32h723xx \
  --uid 6d1395736d13957301 \
  --frequency 10000000 \
  --duration 0.2 \
  --period-us 1000 \
  --output RSCF_h7/logs/data_acq/pyocd_float_benchmark.json

This report tells you:

• the largest repeated float_count that still meets the configured 1 kHz stability rule
• the effective bytes per second at each float count
• where the host-polled path starts to fall behind

If the needed float count is above the stable 1 kHz boundary, stop trying to stretch host polling and move the fast path into the bundled MCU telemetry ring:

python3 BFD-Kit/scripts/bfd_telemetry_ring.py --json layout \
  --field pos_rad:f32 \
  --field vel_rads:f32 \
  --field torque_nm:f32 \
  --field state:u32 \
  --capacity 128

Then capture the ring through PyOCD once the firmware has published the ring image symbol/address:

RSCF_h7/.tools/pyocd-venv/bin/python BFD-Kit/scripts/bfd_telemetry_ring.py --json capture-pyocd \
  --address 0x24020000 \
  --field pos_rad:f32 \
  --field vel_rads:f32 \
  --field torque_nm:f32 \
  --field state:u32 \
  --target stm32h723xx \
  --uid 6d1395736d13957301 \
  --frequency 10000000 \
  --duration 1.0 \
  --poll-period-us 1000 \
  --output RSCF_h7/logs/data_acq/app_ring_capture.csv

The current bfd_pyocd_hss.py also supports two generic paths that align better with UNI_DataVisualizer style direct-read workflows:

• raw SRAM/MMIO scalar capture via repeated --address-spec name@0xADDR:type
• importing enabled variables from a Windows HSSDVProj file via --project-file

The script now coalesces contiguous addresses into one region read and automatically selects block32 when the merged region is 4-byte aligned and 4-byte sized. On RSCF_h7 with FanX/Tek DAPLink High at 10 MHz SWD, local validation reached about 51.25 kB/s for a packed 41B struct field group and about 128 kB/s for a merged 64B block32 raw-memory capture. This clears the practical 50 kB/s threshold for MCU runtime data collection, but it is still not probe-side HSS and should not be positioned as a 1 MB/s equivalent.

Example: raw address/MMIO capture with automatic block32 merge:

RSCF_h7/.tools/pyocd-venv/bin/python BFD-Kit/scripts/bfd_pyocd_hss.py --json sample \
  --target stm32h723xx \
  --uid 6d1395736d13957301 \
  --frequency 10000000 \
  --duration 0.2 \
  --period-us 500 \
  --address-spec w00@0x200096E8:u32 \
  --address-spec w01@0x200096EC:u32 \
  --address-spec w02@0x200096F0:u32 \
  --address-spec w03@0x200096F4:u32 \
  --output RSCF_h7/logs/data_acq/raw_block32_pyocd_hss.csv

Example: import a Windows DataVisualizer project directly:

RSCF_h7/.tools/pyocd-venv/bin/python BFD-Kit/scripts/bfd_pyocd_hss.py --json sample \
  --project-file .tmp_datavis/DataVisualizer_0.0.4.0/STM32F103VET6.HSSDVProj \
  --target stm32h723xx \
  --uid 6d1395736d13957301 \
  --frequency 10000000 \
  --duration 0.2 \
  --period-us 1000 \
  --output RSCF_h7/logs/data_acq/hssdv_project_capture.csv

Before using an imported HSSDVProj as a formal benchmark source, inspect it for duplicate addresses. The bundled DataVisualizer_0.0.4.0/STM32F103VET6.HSSDVProj demo contains repeated address entries, so importing it without review can make a capture look richer than the underlying unique RAM locations actually are.

Generic Command Templates

1. Decode a Global or Static Object

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py \
  --elf "${STM32_ELF}" \
  --mode symbol-auto \
  --symbol g_object_state \
  --follow-depth 0 \
  --format summary \
  --output logs/data_acq/g_object_state.summary

2. Decode a Pointer Hub or Pointer Array

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py \
  --elf "${STM32_ELF}" \
  --mode symbol-auto \
  --symbol g_object_hub \
  --follow-depth 1 \
  --format json \
  --output logs/data_acq/g_object_hub.json

3. Manual Symbol Fallback When Auto Reflection Is Not Suitable

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py \
  --elf "${STM32_ELF}" \
  --mode symbol \
  --symbol g_object_array \
  --count <N> \
  --decode-profile <profile_name> \
  --format csv \
  --output logs/data_acq/g_object_array.csv

4. Stack-Published Local Variable Capture

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py \
  --elf "${STM32_ELF}" \
  --pointer-symbol g_local_probe_addr \
  --seq-symbol g_local_probe_seq \
  --layout f32x1 \
  --count 20 \
  --interval-ms 20 \
  --mode nonstop \
  --output logs/data_acq/local_probe.csv

5. Raw Address Capture

python3 BFD-Kit/skills/codex/bfd-data-acquisition/scripts/data_acq.py \
  --address 0x20000000 \
  --layout u32x4 \
  --count 20 \
  --mode snapshot \
  --output logs/data_acq/raw_u32.csv

Output Policy

• Save final artifacts under logs/data_acq/.
• Use summary for fast operator judgment.
• Use json for downstream AI or tool consumption.
• Use csv for manual table review.
• Treat .codex/bfd/dwarf_cache/ as a reusable cache, not as the final evidence artifact.
• Do not run multiple J-Link RAM sampling, RTT, or GDB attach flows against the same target in parallel.

Decision Rules

Use this skill instead of hand-written GDB or J-Link memory expressions when:

• an ELF symbol exists
• the task is "inspect this global/static object"
• the task needs named fields rather than raw words
• RTT produced no payload and RAM is the next validation path

Switch to bfd-debug-interface only when:

• the task is register-centric
• the symbol is not available
• lower-level breakpoint, watchpoint, or fault control is required