docker-bench-expert

name

docker-bench-expert

description

Specialised skill for the AMT Docker Bench — a multi-container simulation of a T264 autonomous truck. Use when asked about docker_bench containers (rpk1, rpk2, rpk3, platform-sim, avi-radio, test-manager, vas-aht), Docker Compose configuration, network topology (onboard/offboard/CAN/local-fms-network), NAT setup, CIC/RPK version selection, T264 simulator, VAS:AHT integration, DDS QoS profiles, IP address remapping, overlay files, run_tests.sh, Docker Bench version compatibility, compatible FMS/AMT version pairs, or any issue involving the Docker Bench environment. Also use for debugging container startup failures, networking problems, build errors, image pull failures, version mismatch issues, or service readiness issues in the bench. Can fetch the Docker Bench Versions Compatibility Matrix from Confluence to determine compatible version combinations.

Docker Bench Expert

A specialised skill for understanding, maintaining, and debugging the AMT Docker Bench environment — a collection of Docker containers that emulate the separate physical computers on a T264 autonomous haul truck for testing and validation purposes.

This skill extends the docker-expert skill with domain-specific knowledge of the Docker Bench architecture, its containers, networking, and configuration.

When to Use This Skill

• Understanding how the Docker Bench containers fit together
• Debugging container build failures, startup issues, or service readiness problems
• Analysing or modifying the docker-compose.yml, Dockerfiles, or overlay configuration
• Diagnosing networking issues between containers (onboard, offboard, CAN, FMS)
• Understanding or modifying the NAT setup (avi-radio)
• Selecting or changing CIC/RPK/T264 Sim versions
• Determining compatible FMS and AMT Docker Bench version combinations
• Resolving image pull failures or version mismatch errors
• Understanding the IP address remapping system
• Troubleshooting DDS communication or QoS profile issues
• Working with run_tests.sh or the test-manager container
• Understanding the relationship between Docker Bench and the real truck hardware

FX-8520 Refuel Dispatch Playbook (Validated)

Use this workflow when a Docker Bench refuel scenario reaches Dispatched but the asset does not move, or the assignment is quickly cancelled.

High-signal checks

Run these checks together so geometry, mode, and dispatch state are evaluated in the same time window:

• TM area state: GET /areas/{areaId}/state (Traffic Manager)
• Robot mode/restrictions: GET /RobotControl/AssetIdentifier/{asset}
• Assignment list: GET /Assignments (Assignment Aggregator)
• TM logs: search for DispatchCancelledEvent, No MoveToArea instruction found, LoadAreaId, DumpAreaId

Copy/paste diagnostic command (local Imperium + Docker Bench)

Use this one-liner to capture the key state in one snapshot:

TOKEN=$(curl -s -X POST http://localhost:4020/Auth/locallogin -H "Content-Type: application/json" -d '{"username":"641693","password":"Autonomy@1"}' | python3 -c "import json,sys; print(json.load(sys.stdin)['tokenResponse']['token'])") && \
echo '--- TM area state ---' && \
curl -s "http://localhost:7100/areas/e387c3ae-8bc4-4278-8fdf-83943f6be9ea/state" -H "Authorization: Bearer $TOKEN" | python3 -m json.tool && \
echo '--- RobotControl AHT002 ---' && \
curl -sk "https://robot-control-service.dev.localhost/RobotControl/AssetIdentifier/AHT002" -H "Authorization: Bearer $TOKEN" | python3 -m json.tool && \
echo '--- Assignment Aggregator ---' && \
curl -s -o /tmp/aa_root.out -w "%{http_code}\n" "http://localhost:5660/Assignments" -H "Authorization: Bearer $TOKEN" && \
head -c 1200 /tmp/aa_root.out && echo && \
echo '--- traffic-manager relevant logs ---' && \
docker logs traffic-manager 2>&1 | grep -Ei "AHT002|ELI_REFUEL|no route|route|assignment|Dispatched|DispatchCancelled|MoveToArea|LoadAreaId|DumpAreaId" | tail -120

Interpret this snapshot before making geometry changes.

Known failure signature

If TM logs show both of the following, treat it as a dispatch/assignment interpretation failure (not only a mode problem):

• No MoveToArea instruction found for "LoadAreaId" (and/or DumpAreaId)
• Handling "DispatchCancelledEvent" ... immediately after dispatch is added

Lane-aligned spawn guidance

For scripted spawn positions, derive pose from the lane geometry instead of map eyeballing:

1. Get the lane segment id from TM entryPathPoint.laneSegmentId.
2. Query Mine Model GET /PathSegment and select the matching segment.
3. Use the midpoint PathSegmentPoint (index count//2) for X and Y.
4. Derive heading from midpoint neighbors (mid-1 -> mid+1) with atan2.

Using the midpoint centerpoint avoids truck overhang at lane ends and gives more stable initial routing.

Important note for debugging priorities

If midpoint lane-aligned spawn and autonomy mode are both correct but cancellation persists, prioritise assignment payload/dispatch contract checks over further spawn angle tuning.

Communication Style

• Default: Provide concise, plain-language explanations. Focus on what is happening, why it matters, and how to fix it. Do not assume the reader has deep Docker or networking knowledge.
• When asked for more detail: Give in-depth explanations covering the underlying Docker and networking mechanisms.
• Always provide actionable solutions — not just descriptions of problems.

Docker Bench Architecture Overview

The Docker Bench simulates a T264 autonomous haul truck using seven containers, each representing a physical computer or network device on the real truck:

Container	Role	What It Emulates
rpk1	Autonomy Management Toolkit (AMT)	The main onboard management computer running the RTK launcher, diagnostics watchdog, mine model adapter, blob sync, and FMS bridge
rpk2	Compute IC (CIC)	The control computer running the Control IC software with a web interface on port 8009
rpk3	Robot Perception Kernel (RPK)	The perception computer running obstacle detection and point cloud processing
vas-aht	Vehicle Awareness System (VAS:AHT)	The Motium unit running VAS for position/heading reporting, notifications, and sound management via DDS
platform-sim	T264 Simulator	Simulates the vehicle dynamics and CAN bus of the physical truck, with a web interface on port 8123
avi-radio	NAT Gateway	Emulates the AVI radio that bridges the truck's internal network to the external FMS/OT network using iptables NAT rules
test-manager	Test Automation (optional)	Runs ARCU automation tests; enabled via --profile test-manager

Multi-Asset Support

Docker Bench supports running multiple instances simultaneously, each representing a different truck. The ASSET_ID variable in .env (default: AHT000) controls:

• Compose project name: amt-docker-bench-${ASSET_ID_LOWER} (e.g., amt-docker-bench-aht000)
• Container names: <service>-${ASSET_ID} (e.g., rpk1-AHT000)
• Network names: <network>-${ASSET_ID} (e.g., onboard-AHT000, offboard-AHT000)

Two conventions exist for running commands inside containers:

• Primary: docker exec <service>-AHT000 <command> (e.g., docker exec rpk1-AHT000 bash)
• Alternative: docker compose exec <service> <command> (e.g., docker compose exec rpk1 bash)

The primary convention uses the full container name (service + asset ID). The alternative uses Compose service names and is useful when you do not know or want to type the asset ID. Both are valid; the documentation uses the primary convention.

Key Files

File	Purpose
docker_bench/docker-compose.yml	Orchestrates all containers, networks, volumes, and build arguments
docker_bench/.env	Environment variables: versions, IPs, asset ID, patterns
docker_bench/run_tests.sh	Automated test orchestration script
docker_bench/utils/helpers.sh	Shell utility functions for logging, Docker version checks, container readiness
docker_bench/dockerfile-rpk1	AMT container Dockerfile
docker_bench/dockerfile-rpk2	CIC container Dockerfile
docker_bench/dockerfile-rpk3	RPK container Dockerfile
docker_bench/dockerfile-vas-aht	VAS:AHT container Dockerfile
docker_bench/dockerfile-platform-sim	T264 Simulator Dockerfile
docker_bench/dockerfile-test-manager	Test manager Dockerfile
docker_bench/dockerfile-avi-radio	NAT gateway Dockerfile
docs/docker_bench/index.md	User-facing documentation for Docker Bench

Network Architecture

The Docker Bench uses four networks to mirror the real truck's network topology:

Networks

Network	Subnet	Purpose	Real-World Equivalent
onboard (onboard-${ASSET_ID})	10.10.10.0/24	Internal communication between onboard computers	The truck's internal Ethernet connecting all RPKs
offboard (offboard-${ASSET_ID})	10.10.1.0/24	Connects onboard computers to the AVI radio	The link from onboard computes to the external-facing radio
local-fms-network	172.18.0.0/16	External FMS/cloud connectivity	The OT (Operational Technology) mine network
can (can-${ASSET_ID})	Docker CAN plugin	CAN bus between RPK2 and platform-sim	Physical CAN bus on the truck

IP Address Assignments

Container	Onboard (eno1)	Offboard (eno2)	Other
rpk1	10.10.10.110	10.10.1.110	—
rpk2	10.10.10.111	10.10.1.111	CAN network
rpk3	10.10.10.112	10.10.1.112	—
vas-aht	—	10.10.1.115	—
platform-sim	10.10.10.55	10.10.1.55	CAN network
test-manager	10.10.10.56	10.10.1.56	local-fms-network
avi-radio	—	10.10.1.2	172.18.50.1 (primary), plus dedicated IPs per container

NAT and Port Forwarding (avi-radio)

The avi-radio container performs Network Address Translation using per-container dedicated IPs on the local-fms-network. Each onboard container has its own external IP, so traffic is routed per-container rather than requiring port-specific forwarding:

Container	Offboard IP	Dedicated FMS IP
rpk1	10.10.1.110	172.18.50.110
rpk2	10.10.1.111	172.18.50.111
rpk3	10.10.1.112	172.18.50.112
vas-aht (motium)	10.10.1.115	172.18.50.115
platform-sim	10.10.1.55	172.18.50.55
test-manager	10.10.1.56	172.18.50.56

For each container, SNAT rewrites the source address of outgoing traffic to its dedicated FMS IP, and DNAT rewrites the destination of incoming traffic to its offboard IP.

Explicit port forwarding is additionally set up for web interfaces and SSH access (especially for Windows users who cannot directly access local-fms-network):

Port	Protocol	Destination	Service
8009	TCP	rpk2	CIC web interface
8123	TCP	platform-sim	T264 Sim web interface
2020	TCP	rpk1:22	SSH
2021	TCP	rpk2:22	SSH
2022	TCP	rpk3:22	SSH
2025	TCP	motium:22	SSH

Interface Naming System

Docker assigns default interface names like eth0, eth1 to containers. The RTK service apply-systemd-networkd-config.service (part of the base images) runs at container boot to rename these interfaces to match the systemd-networkd configuration files in /etc/systemd/network/.

The script (/usr/bin/apply_systemd_networkd_config.py) works by:

1. Reading all .network files in /etc/systemd/network/
2. Extracting [Match] Name= as the target interface name
3. Finding the existing Docker interface (eth0, etc.) by matching the [Network] Address= IP
4. Renaming the Docker interface to the target name using ip link set

Interface Name Convention

Container	Onboard Interface	Offboard Interface
rpk1	eno1	eno2
rpk2	eno1	eno2
rpk3	eno1	eno2
vas-aht	—	enp0s29f1
platform-sim	eno1	eno2

Important: Glob Patterns in Name= Are Not Supported

The apply-systemd-networkd-config.service script reads Name= values literally. On real hardware, systemd-networkd treats Name=enp* as a glob pattern. In Docker Bench, the script would rename the interface to the literal string enp* (with asterisk). This was the root cause of FX-8560.

All Docker Bench network configs must use exact interface names (e.g. Name=eno2, Name=enp0s29f1), not glob patterns. The RPK containers handle this by removing the base image's 20-wired.network (which contains Name=enp*) and replacing it with exact-name configs (10-eno1.network, 20-eno2.network). The VAS AHT Dockerfile patches 20-wired.network in place with sed to set Name=enp0s29f1 (matching site configuration).

IP Address Remapping System

Each container runs an overwrite-ip-address-config.service at boot that rewrites IP addresses in configuration files. This allows the bench to work with different subnet configurations without manually editing every config file.

• Controlled by overlay/common/etc/overwrite_ip_address_config/overwrite_ip_address_config.xml
• Executed by overlay/common/usr/bin/overwrite_ip_address_config.py
• Searches through /etc/systemd/network/, /etc/systemd/system/, /opt/amt/resources/, /opt/vas/config/, /opt/vas/share/, /opt/lme/cic/, /opt/lme/rpk/, /usr/share/t264_simulator/, and /persistent/
• Explicitly excludes /opt/field-commissioning/, /opt/field-sync/, /opt/field-signal/, and /opt/field-telemetry/ (field services use DDS on localhost only; no IP rewriting needed)
• Replaces default IPs with actual container IPs using environment variables

Version Selection

CIC Version (rpk2)

• CIC tarballs are stored in docker_bench/overlay/rpk2/persistent/
• The .env variable CIC_FIND_PATTERN controls which version is installed (e.g. cic*0.65*.tgz)
• Set to cic*.tgz to always install the latest available version
• The upgradeCIC.sh script in the overlay handles installation

RPK Version (rpk3)

• RPK tarballs are stored in docker_bench/overlay/rpk3/persistent/
• The .env variable RPK_FIND_PATTERN controls which version is installed (e.g. rpk*FS65*.tgz)
• Set to rpk*.tgz to always install the latest available version

VAS:AHT Version (vas-aht)

• The .env variable VAS_AHT_DOCKER_BENCH_VERSION controls the VAS:AHT base image version
• VAS uses a separate base image (rtk_os_docker_bench_vas_aht) built by the VAS repository, independent of the AMT base images
• Default: pinned to a specific version (e.g. 4.2.1-master.6-0.x1a79a05d)

T264 Sim Version (platform-sim)

• The .env variable T264_SIM_VERSION forces a specific version (in DPKG format)
• If unset, the version bundled with the base image is used

Docker Bench Compatibility Matrix

The authoritative source for compatible version combinations is the Confluence page: Docker Bench Versions Compatibility Matrix

Important ordering note: The newest compatible combinations are at the bottom of the "Master" table, not the top. When scanning manually, verify this by checking that version numbers increase as you go down.

How to Fetch and Use the Compatibility Matrix

When a user asks about version compatibility, version selection, or encounters image pull/startup errors related to version mismatches, fetch the matrix from Confluence using the Atlassian MCP tool:

mcp_com_atlassian_getConfluencePage(
    cloudId: "fmgl-autonomy.atlassian.net",
    pageId: "1226080308",
    contentFormat: "markdown"
)

The response contains a markdown table under the "## Master:" heading with these columns:

Column	Description
FMS Version	Imperium Docker image tag (e.g. 4.1.0-master.776-0.xe4bdb8fd)
Imperium Repo Commit Hash	Git commit to check out the Imperium repo at
AMT Version	AMT Docker Bench image tag (e.g. 0.33.0-master.47-0.x531ee13b)
AMT Repo Commit Hash	Git commit to check out the AMT repo at
T264 Sim Version	T264 Simulator version (DPKG format); "(default)" means use the version bundled in the base image
CIC Version	CIC tarball filename pattern; "(default)" means use the latest in overlay/rpk2/persistent/
RPK Version	RPK tarball filename pattern; "(default)" means use the latest in overlay/rpk3/persistent/
Workarounds/Comments	Known issues, required workarounds, or empty if clean

There is also a "## Release Candidates:" table with the same columns for RC builds.

Version Compatibility Queries

When the user asks a version-related question, fetch the page and answer using these patterns:

"What is the latest compatible version pair?"

• Look at the last row of the Master table.
• Prefer rows where the Workarounds/Comments column is empty or contains only minor notes (not "can't get the truck to move" or similar blockers).
• Report the FMS Version, AMT Version, and any default component versions (T264 Sim, CIC, RPK).

"Is FMS version X compatible with AMT version Y?"

• Search the table for a row matching both the FMS and AMT versions.
• If found, report the row and any workarounds.
• If not found, report that the combination is not in the matrix and suggest the nearest compatible pair(s).

"What AMT version works with FMS version X?"

• Search the table for rows matching the FMS version.
• There may be multiple rows (same FMS version paired with different AMT versions).
• Report all matches, noting which has the fewest workarounds.

"What FMS version works with AMT version X?"

• Search the table for rows matching the AMT version.
• Report all matches with their workarounds.

"Give me a clean version pair with no workarounds"

• Scan from the bottom of the table upward.
• Return the first (newest) row where the Workarounds/Comments column is empty.

Version String Anatomy

Understanding version strings helps identify compatibility:

• FMS: 4.1.0-master.776-0.xe4bdb8fd → major.minor.patch-branch.buildNumber-0.xCommitHash
• AMT: 0.33.0-master.47-0.x531ee13b → same pattern
• T264 Sim: 0.0.0~master.58~0.x83d258a2 → uses ~ instead of - (Debian package format)
• CIC: cic_0.62.0.tgz → tarball filename
• RPK: rpk-533_20260301-175657-release-63152_FS62.1.tgz → tarball with date and FS (Functional Safety) version

The commit hash suffix (after 0.x) can be used to check out the corresponding Git repository at the exact commit that was tested.

Applying Versions

Once a compatible pair is identified, apply them as follows:

FMS version (Imperium side):

# In imperium/Tools/local-environment/
./setVersion.ps1 <FMS_VERSION>
# e.g. ./setVersion.ps1 4.2.1-master.207-0.xc5d2f714

AMT Docker Bench version (AMT side):

• Set AMT_DOCKER_BENCH_VERSION in docker_bench/.env, OR
• Pass -dockerBenchVersion <AMT_VERSION> to Imperium's startAsset.ps1

T264 Sim version (if not "(default)"):

• Set T264_SIM_VERSION in docker_bench/.env

CIC/RPK versions (if not "(default)"):

• Ensure the correct tarball exists in overlay/rpk2/persistent/ (CIC) or overlay/rpk3/persistent/ (RPK)
• Set CIC_FIND_PATTERN / RPK_FIND_PATTERN in docker_bench/.env to match

Imperium repo commit (if the matrix specifies a different commit from the FMS version tag):

• Some rows list an Imperium Repo Commit Hash that differs from the hash in the FMS version string. This means the Docker images are from one commit but the local tooling scripts (PowerShell scripts, compose files) must be from a different commit for compatibility.
• Check out the specified commit: cd ~/workspace/imperium && git checkout <commit_hash>

Common Version Mismatch Symptoms

Symptom	Likely Cause
Image pull fails ("manifest not found")	FMS version tag doesn't exist in Artifactory for one or more services. Check the matrix for a version that has all images published.
Truck won't move / map sync fails	FMS↔AMT version incompatibility. The matrix notes often describe this (e.g. "can't get the truck to move").
ALL_POSE_TIMEOUT in FMS Bridge	Known issue with several version pairs. Restart amt.service or asset-dds-gateway. Check the matrix Workarounds column.
RPK MAP LANE BREACH fault on startup	Known RPK issue with some versions. Clear via CIC Test Page or RCS Health Reset.
Path permission / hash faults (797, 849)	Do a Health Reset in RCS and delete remaining diagnostic IDs with controlled stop latched.
DEADLINE_MISSED_PATH_SEGMENT_PERMISSIONS	Restart mine-model-distributor and mine-model-service.

Container Capabilities and Permissions

The containers require elevated Linux capabilities to function:

Capability	Used By	Why
CAP_NET_ADMIN	All RPKs, vas-aht, platform-sim, test-manager	Network configuration, IP remapping, interface management
CAP_SYS_MODULE	All RPKs, vas-aht, platform-sim, test-manager	Kernel module loading (CAN drivers)
CAP_SYS_PTRACE	rpk1	Process tracing for diagnostics
CAP_IPC_LOCK	rpk2, rpk3, vas-aht, platform-sim	Lock memory for real-time performance
CAP_SYS_NICE	rpk2, rpk3, vas-aht, platform-sim	Set real-time scheduling priorities
CAP_SYS_TIME	vas-aht	System time modification (required by chrony)
NET_ADMIN	avi-radio	NAT and iptables rules
NET_RAW	avi-radio	Raw socket access for network translation

DDS Communication

The bench uses RTI Connext DDS for inter-container communication.

DDS Domain IDs

Domain	Name	Used By
20	CIC	FMS Bridge (rpk1), CIC (rpk2), RPK (rpk3), ARCU tests
30	VAS	VAS Bridge (vas-aht), field-sync, field-signal
50	IAI	Asset Shadow (Imperium), VAS Bridge telemetry route
51	Field	field-commissioning, field-sync, field-signal, field-telemetry

DDS Configuration

• QoS profiles are defined in overlay/common/persistent/USER_QOS_PROFILES.xml
• FMS Bridge communicates externally via unicast through the avi-radio NAT
• The ASSET_SHADOW_DDS_PARTICIPANT_ID environment variable configures the FMS Bridge participant
• VAS:AHT communicates via DDS over the offboard network interface (UDP, no TCP ports required)
• Field services bind DDS to 127.* (localhost) only, routing through VAS Bridge

Spoof Scripts

Located in docker_bench/overlay/common/persistent/, these scripts simulate data that would come from real truck sensors or systems:

Script	What It Simulates
spoof_aci_pose.sh	Vehicle position and attitude (published every 100ms)
spoof_aci_robot_mode.sh	Robot operational mode
spoof_health_event_*.sh	Various health/diagnostic events
spoof_actioned_diagnostics_perception_stop.sh	Perception system diagnostic actions
spoof_health_reset_perception_stop.sh	Perception health status reset

VAS:AHT Container

The vas-aht container runs the Vehicle Awareness System for Autonomous Haul Trucks. On the real truck, VAS:AHT runs on the Motium unit (10.10.1.115). It acts primarily as a network bridge — receiving commands and publishing status via DDS, with the centralised OCS handling all positioning and decision-making.

VAS:AHT Key Details

• Base image: rtk_os_docker_bench_vas_aht (built by the VAS repository, not AMT)
• Version variable: VAS_AHT_DOCKER_BENCH_VERSION (independent of AMT_DOCKER_BENCH_VERSION)
• Network: Offboard only (10.10.1.115) — not connected to onboard or CAN
• No exposed ports: VAS:AHT communicates exclusively via DDS over UDP
• IP remapping: overwrite-ip-address-config.service scans /opt/vas/config/ and /opt/vas/share/ for hardcoded IPs

Disabled Imperium Field Services

One Imperium Field .NET service is pre-installed in the VAS base OS image but is disabled in dockerfile-vas-aht because it is not required in Docker Bench:

Service	Why Disabled
field-commissioning-ui.service	Web UI for commissioning; confirmed not required in Docker Bench context (Craig Brogle)

Enabled Imperium Field Services

field-commissioning.service is enabled in Docker Bench to support manned mode commissioning workflows. It is a .NET service providing an HTTP API and DDS participant for calibration, survey import, and operator login.

Communication path:

• DDS domain 51 bound to 127.* (localhost) - routes through VAS Bridge in the same container
• HTTP (Kestrel) bound to localhost:4050 - not externally reachable
• The 172.18.* entry in AllowInterfacesList is a no-op (no matching interface in Docker Bench)
• /opt/field-commissioning/ is explicitly excluded from IP address remapping

Verification:

docker exec vas-aht-AHT000 systemctl status field-commissioning.service
docker exec vas-aht-AHT000 curl -s -w "\n%{http_code}" http://localhost:4050/Health/liveness
docker exec vas-aht-AHT000 curl -s http://localhost:4050/Health/version
docker exec vas-aht-AHT000 journalctl -u field-commissioning --no-pager -n 30

field-sync.service is enabled in Docker Bench to support manned mode mine model and asset data synchronisation. It provides a DDS participant for distributing mine model, area, zone, and asset state data to the in-cab display (Centurion).

Communication path:

• DDS domain 51 bound to 127.* (localhost) - routes through VAS Bridge in the same container
• DDS domain 30 (additional VAS domain bridge participant)
• HTTP (Kestrel) bound to localhost:5510 - not externally reachable
• Redis connection to localhost:6379 (port corrected from base image default of 6380)
• /opt/field-sync/ is explicitly excluded from IP address remapping

Expected warnings (non-fatal in standalone Docker Bench):

• gRPC Unavailable errors for Asset Shadow and Mine Model Distributor (FMS Core not reachable)
• RabbitMQ connection warnings (no AMQP broker in container)

Verification:

docker exec vas-aht-AHT000 systemctl status field-sync.service
docker exec vas-aht-AHT000 curl -s -o /dev/null -w "%{http_code}" http://localhost:5510/
docker exec vas-aht-AHT000 journalctl -u field-sync --no-pager -n 30

field-signal.service is enabled in Docker Bench to support manned mode real-time data delivery to Centurion (the in-cab display). It hosts a SignalR hub serving operational data (assignments, materials, notifications, asset state, etc.) from Redis to connected SignalR clients.

Communication path:

• DDS domain 51 bound to 127.* (localhost) - routes through VAS Bridge in the same container
• DDS domain 30 (additional VAS domain bridge participant)
• HTTP (Kestrel) bound to localhost:5000 (default, not externally reachable)
• Redis connection to localhost:6379 (default port, no fix needed)
• SignalR hub hosted on the Kestrel port (Centurion connects here)
• /opt/field-signal/ is explicitly excluded from IP address remapping

Dependencies: redis-server.service (hard), field-sync.service (ordering only)

Expected behaviour (normal in standalone Docker Bench):

• No connected SignalR clients (Centurion is not running)
• Redis reads return empty data (until field-sync populates it)
• No gRPC or RabbitMQ errors (field-signal does not connect to external services)

Verification:

docker exec vas-aht-AHT000 systemctl status field-signal.service
docker exec vas-aht-AHT000 curl -s -o /dev/null -w "%{http_code}" http://localhost:5000/
docker exec vas-aht-AHT000 journalctl -u field-signal --no-pager -n 30

field-telemetry.service is enabled in Docker Bench to support the health event pipeline from vehicle telemetry to the in-cab display. It subscribes to telemetry data via DDS (domain 51), extracts health events, and stores them in Redis for consumption by field-signal.

Communication path:

• DDS domain 51 bound to 127.* (localhost) - subscribes to /telemetry topic on the field domain
• HTTP (Kestrel) bound to localhost:4200 (set via systemd drop-in to avoid port 5000 conflict with field-signal)
• Redis connection to localhost:6379 - stores health events at LocalAsset:FieldHealth
• /opt/field-telemetry/ is explicitly excluded from IP address remapping

Operating mode: DDS input mode (UseDdsForTelemetryData: true) is the production default. No RaptorCore hardware required.

Dependencies: redis-server.service (hard)

Expected behaviour (normal in standalone Docker Bench):

• DDS participant on domain 51 subscribes to /telemetry topic
• Redis key LocalAsset:FieldHealth exists but is empty (ZCARD = 0) because telemetry data arrives on VAS domain 30 via VAS Bridge but field-telemetry subscribes on field domain 51; no domain 30→51 bridge exists in standalone Docker Bench
• Health events would be populated when running with Imperium local-env (which bridges telemetry to domain 51)
• No hardware errors, no gRPC errors, no RabbitMQ dependency

Verification:

docker exec vas-aht-AHT000 systemctl status field-telemetry.service
docker exec vas-aht-AHT000 curl -s -o /dev/null -w "%{http_code}" http://localhost:4200/
docker exec vas-aht-AHT000 journalctl -u field-telemetry --no-pager -n 30

The remaining disabled service (field-commissioning-ui) is not required in Docker Bench.

In Imperium, the FMS-to-VAS integration path is: FMS Core -> Asset Shadow (separate container) -> DDS via avi-radio NAT -> VAS:AHT. Field services are handled externally by dedicated Imperium containers on local-fms-network.

VAS:AHT Dockerfile Setup

1. Installs Netskope CA certificates
2. Copies shared overlay files (SSH keys, QoS profiles, spoof scripts) to /persistent/
3. Sets up SSH access for root and rtkuser accounts
4. Installs and enables overwrite-ip-address-config.service
5. Patches 20-wired.network to replace glob Name=enp* with exact Name=enp0s29f1 (FX-8560 — the base image uses a glob that apply-systemd-networkd-config.service reads literally)
6. Sets registration_state="Measured" in /etc/robot_model/robot_model.xml (base image ships with "Default", causing nodes to exit)
7. Sets robot_class="DumpTruck" in /etc/robot_model/robot_model.xml (base image ships with "Undefined")
8. Disables field-commissioning-ui.service (not required in Docker Bench)
9. Fixes field-sync Redis port (base image uses 6380, VAS:AHT Redis runs on 6379)
10. Fixes field-telemetry Kestrel port (base image defaults to 5000, conflicts with field-signal; set to 4200 via systemd drop-in)

Troubleshooting Workflows

Bench Won't Start

1. Check Docker version: requires >= 27.0.3. Run docker --version.
2. Verify the Docker CAN plugin is installed: docker plugin ls. The run_tests.sh script installs it automatically.
3. Check that base images are available: docker images | grep rtk_os_docker_bench.
4. Verify .env has valid AMT_DOCKER_BENCH_VERSION (e.g. latest-master).
5. Check for port conflicts on the host: ss -tlnp | grep -E '8009|8123|2020|2021|2022'.

Container Crashes or Won't Stay Running

1. Check logs: docker compose logs <service> (e.g. docker compose logs rpk1).
2. Check exit code: docker inspect <container> --format='{{.State.ExitCode}}'.
3. Verify the overwrite-ip-address-config service ran successfully — IP mismatches cause failures downstream.
4. For rpk2/rpk3: check that the CIC/RPK tarball matched the FIND_PATTERN and installed correctly.
5. Shell into a failed container interactively: docker run -it --entrypoint /bin/bash <image>.

Services Not Ready (run_tests.sh Timeout)

The test script waits for these processes:

rpk1: rtk_launcher, rtk_log_recorder, amt_diagnostics_watchdog, mine_model_adapter, task_spooler, blob_sync_node, fms_bridge

platform-sim: /usr/bin/t264_simulator

If the readiness check times out:

1. Shell into the container: docker compose exec rpk1 bash.
2. Check systemctl status amt.service (rpk1) or the relevant service.
3. Check journalctl -u amt.service --no-pager -n 50 for errors.
4. Verify DDS licence is present and valid (RTI Connext).

Networking Issues Between Containers

1. Verify containers are on the correct networks: docker network inspect <network>.
2. Test connectivity: docker compose exec rpk1 ping 10.10.10.111 (rpk2 onboard).
3. Check that applications bind to 0.0.0.0, not 127.0.0.1.
4. For DDS issues: verify QoS profiles match across containers and domain ID is consistent.
5. For FMS connectivity: check avi-radio NAT rules with docker compose exec avi-radio iptables -t nat -L -n.
6. If an interface has an unexpected name (e.g. enp* with a literal asterisk), check /etc/systemd/network/ for glob patterns in Name=. The apply-systemd-networkd-config.service reads Name= literally — use exact names only. See the Interface Naming System section.

CAN Bus Issues

1. Verify the Docker CAN plugin is running: docker plugin ls.
2. Check CAN interfaces inside containers: docker compose exec rpk2 ip link show.
3. Decode CAN messages from platform-sim:

   docker compose exec platform-sim bash -l -c "candump vcan10 | cantools decode --single-line /t264-sim/libraries/dbc/DBW_OAL.dbc"
   

Build Failures

1. Check if base images exist: the Dockerfiles expect pre-built rtk_os_docker_bench_* images.
2. Before changing AMT_DOCKER_BENCH_VERSION, consult the Docker Bench Compatibility Matrix section above. Fetch the matrix from Confluence and pick a known-compatible combination (newest compatible entries are at the bottom of the table).
3. Verify Netskope CA certificate is accessible (used for SSL in all containers).
4. For test-manager: ensure USER_GITHUB_PAT is set and the GitHub token has repo, read:org, read:user permissions with SSO enabled.
5. Check that overlay files referenced in COPY directives exist at the expected paths.

Useful Commands

Task	Command
Start the bench	cd docker_bench && docker compose up --build -d
Start (force pull latest images)	cd docker_bench && docker compose up --build --pull always -d
Start with test-manager	docker compose --profile test-manager up --build -d
Stop the bench	docker compose down
View all logs	docker compose logs -f
View one service's logs	docker compose logs -f rpk1
Run automated tests	./run_tests.sh
SSH into rpk1	ssh -p 2020 rtkuser@localhost or docker compose exec rpk1 bash
SSH into rpk2	ssh -p 2021 rtkuser@localhost
SSH into rpk3	ssh -p 2022 rtkuser@localhost
SSH into vas-aht (motium)	ssh -p 2025 rtkuser@localhost or docker compose exec vas-aht bash
CIC web UI	Open http://localhost:8009
T264 Sim web UI	Open http://localhost:8123
Run ARCU tests manually	docker exec -itu rtkuser rpk1-AHT000 /opt/amt/bin/arcu -d 20 -i /opt/amt/share/arcu/arcu_config.xml
Check container readiness	docker ps --format 'table {{.Names}}\t{{.Status}}'
Inspect NAT rules	docker exec avi-radio-AHT000 iptables -t nat -L -n

Imperium Integration

Docker Bench can integrate with the Imperium local environment for end-to-end testing:

• Use startAsset.ps1 -useDockerBench to connect Imperium to the bench
• The asset ID must be configured in .env to match the Imperium environment (e.g., ASSET_ID=AHT001) with IsAutonomousAsset set to true
• The Imperium startup script (startAssetDockerBench.ps1) generates an .env-asset-{ASSET_ID}-docker-bench file with asset-specific network settings from assets.json
• An Asset Shadow container (on local-fms-network) bridges FMS Core services to Docker Bench via DDS through avi-radio
• VAS:AHT integrates via the same avi-radio NAT path — its offboard IP (OFFBOARD__MOTIUM__IP) is SNAT/DNAT'd to a dedicated FMS IP (LOCAL_FMS_NETWORK__AVI_RADIO_MOTIUM__IP)
• SSH access to VAS:AHT is available on port 2025 via avi-radio port forwarding

Known Bugs

RPK Map Lane Breach Stuck / Unclearable (APADA-14702)

Details: https://fmg-lme-autonomy.atlassian.net/browse/APADA-14702

Observed with: FMS 4.1.0-rc.130 (April 2026)

Symptom: CIC raises a repeated "RPK map lane breach" diagnostic that activates and deactivates repeatedly while the truck is driving manually inside a locked AOZ. Once the AOZ is unlocked, the diagnostic clears from CIC — but the FMS retains the health event as unclearable (IsClearable = false, event type BOUNDARY_INTERACTION). The "RPK map lane breach" persists in the Asset Details panel and survives mode transitions (Manual → Autonomy). Standard RCS Health Reset and latching/unlatching do not clear it.

Root cause: Under investigation. The health event published by CIC/RPK is recorded in RCS with IsClearable = false. Once in this state, the normal "Clear to Proceed" action is not available in the RCS UI.

Workaround (requires FMS database access):

1. Stop the RCS service:

   docker service scale fms_robot-control-service=0
   

2. Clear restrictions in the RCS database:

   UPDATE [RobotHealth_Eli_Int].[dbo].[RobotControls]
   SET [RestrictionState] = N'{"Restrictions":[]}'
   WHERE [AssetIdentifier] = '<ASSET_ID>';
   

3. Clear health events in the RCS database:

   UPDATE [RobotHealth_Eli_Int].[dbo].[HealthEvents]
   SET [IsClearable] = 1
   WHERE [AssetIdentifier] = '<ASSET_ID>'
     AND [EventType] = 'BOUNDARY_INTERACTION';
   

4. Restart the RCS service:

   docker service scale fms_robot-control-service=1
   

Notes:

• Replace <ASSET_ID> with the actual asset identifier (e.g. DT5395)
• A simpler but less targeted alternative is to spoof a clearable diagnostic via RCS with "isResettable": true and "responseAction": "Proceed" — but this was observed to not work for this specific event
• This issue may also appear during Docker Bench integration testing with Imperium when AOZ locking scenarios are tested

FMS Integration Lessons (FX-8520)

Lessons learned from the FX-8520 refuelling integration test development, relevant to any Docker Bench + FMS testing:

Eliwana Seed Data Issues

The Eliwana mine model seed data (seedFMS.ps1) has several problems for refuelling tests:

Problem	Impact	Workaround
ELI_REFUEL area type is RefuelBay (enum=1)	TM's IsDynamicArea() filter ignores it entirely	Delete and recreate as Refuel (enum=6)
Entry/exit lanes share the same endpoint (co-located)	Drive-through path planning is mathematically impossible	Create new N→S drive-through lanes via mine model DEBUG API
Area polygon ~52 m × 26 m	Fails minimum inscribed circle requirement (20 m radius) for Refuel type	Use 120 m × 120 m boundary

Coordinate System

All mine model coordinates are in centimetres (mine grid). NTS geometry functions divide by 100 internally when computing metric distances. When creating lanes or boundaries via the API, all coordinate values must be in centimetres.

Traffic Manager Seeding Behaviour

• TM seeds its area read model once on startup using Polly retry (exponential backoff, 15s cap)
• TM's health check passes independently of seeding completion — a healthy TM does NOT mean areas are loaded
• If seedFMS.ps1 runs AFTER TM starts, TM never ingests the areas. Always restart TM after bulk seeding:

  docker restart traffic-manager
  

• Poll GET /areas/{areaId}/state for up to 180s to confirm ingestion

Spatial Association Requires State Change

The spatial-association-service only fires entry/exit events on position changes. If an asset was already at the target coordinates from a previous run:

1. Teleport asset outside the area (offset by ≥ 200 m)
2. Wait 3-5 seconds for the exit event
3. Teleport asset to the target position inside the area
4. Wait for the entry event

Without this pattern, the asset is never detected in the area.

Mine Model DEBUG API Notes

• POST /DEBUG/Lane creates a lane AND auto-generates child LaneSegment(s)
• The response structure is .lane.segments[] — NOT .laneSegments[]
• Lane segment IDs in the response are what you pass to POST /Area as entry/exit segment references
• NTS Polygon.Contains() uses DE-9IM Interior model: points exactly on the boundary are excluded. Queue points must be offset ≥ 1 m (100 cm) from edges.
• POST /Area requires controlBehaviors as a JSON-encoded string field within the request body, not a nested object

PowerShell Serialization Gotchas

When scripting mine model REST calls from PowerShell:

• Invoke-RestMethod auto-deserializes and flattens nested JSON (GeoJSON, lane segments). Use Invoke-WebRequest + ConvertFrom-Json for complex responses.
• GeoJSON coordinate arrays must be built as here-strings, NOT via ConvertTo-Json (which double-serializes nested arrays).
• The contentType parameter must be "application/json" — PowerShell does not default to this.

-dockerBenchUseLatest Artifactory Risk

The -dockerBenchUseLatest parameter queries the Artifactory Storage REST API, not the Docker registry. docker login success does NOT guarantee Storage API access (different backends). The API frequently returns 504 Gateway Timeout during CI load.

Recommendation: Always use hardcoded default versions from the compatibility matrix.

ACI V1.0 → V1.1 Breaking Change Boundary

FMS Version Threshold	Minimum AMT Version	ACI Version
< 4.2.1-master.278	Any AMT version	V1.0
>= 4.2.1-master.278	0.34.0-master.24	V1.1

FMS and AMT versions must be updated together when crossing this boundary. The default test versions (FMS .207 + AMT .16) are below the threshold (V1.0).

VAS:AHT Mode-Gated Telemetry

VAS:AHT only forwards fuel telemetry to FMS when the truck is in MANUAL mode:

• In AUTONOMY mode: fuel telemetry is processed but NOT forwarded (mode gate blocks it)
• In MANUAL mode: fuel telemetry flows VAS:AHT → DDS domain 50 → Asset Shadow → FMS

This is intentional — autonomous refuelling detection requires the truck to be stationary in MANUAL mode. If fuel telemetry never appears in FMS, verify the truck's robot mode state is MANUAL.

SQL Workarounds for Docker Bench FMS

• seedFMS.ps1 does NOT reliably set PasswordHash — write it directly via SQL UPDATE
• SET QUOTED_IDENTIFIER ON is required before any UPDATE on AspNetUsers (filtered indexes)
• stopAll.ps1 does NOT remove Docker networks — orphaned local-fms-network blocks subsequent startSql.ps1. Clean up with docker network rm local-fms-network

References

• Docker Bench Documentation
• AMT T264 Docker Bench (Confluence)
• Imperium Local Environment
• Docker official documentation
• Dockerfile reference
• Docker Compose reference