# Physical AI Toolchain
<!-- markdownlint-disable MD013 -->
[](https://github.com/microsoft/physical-ai-toolchain/actions/workflows/main.yml)
[](https://github.com/microsoft/physical-ai-toolchain/actions/workflows/codeql-analysis.yml)
[](https://scorecard.dev/viewer/?uri=github.com/microsoft/physical-ai-toolchain)
[](https://www.bestpractices.dev/projects/12195)
[](./LICENSE)
[](https://microsoft.github.io/physical-ai-toolchain/)
<!-- markdownlint-enable MD013 -->
## Overview
Physical AI and robotics are moving from headlines and experimentation into real-world industrial deployment. The shift creates practical implications for how human-robot-AI collaboration becomes an operational capability in manufacturing, logistics, healthcare, and autonomous systems. Operationalizing physical intelligence at scale, across fleets and federations of intelligent systems, is a challenge no single OEM or software vendor can deliver alone.
Physical AI is the strategic inflection point for AI platforms, and robotics is the hero use case. It sits at the intersection of cloud, edge, data, and agentic AI.
**Physical AI Toolchain** is an open-source, production-ready framework that integrates [Microsoft Azure](https://azure.microsoft.com/) cloud services with [NVIDIA's](https://developer.nvidia.com/) physical AI stack, accelerating robotics and physical AI developers to automate and scale data curation, augmentation, evaluation across perception, mobility, imitation learning, and reinforcement learning pipelines, along with efficient inference. It provides:
- **Accelerate physical AI innovation.** From edge data capture on NVIDIA Jetson devices through cloud-based training on GPU clusters to model deployment at the edge, every stage of the physical AI lifecycle is addressed with tested, repeatable automation.
- **Operationalize physical intelligence.** Built on Azure Machine Learning, Azure Kubernetes Service, Azure Arc, and Azure Storage with Entra ID authentication, managed identities, and Infrastructure as Code, so workloads meet the security, compliance, and governance requirements of production environments.
- **Scale through ecosystem collaboration.** Native support for NVIDIA Isaac Sim and Isaac Lab for simulation and reinforcement learning, NVIDIA OSMO for workflow orchestration, and the NVIDIA Jetson platform for edge inference provides a hardware-accelerated path from research to deployment, enabled by deep partnership across the ecosystem.
- **Efficient Inference.** Seamless capability to offload inference out of the robot to an on-premise GPU with minimal overhead using Kubernetes operators, towards benefits of energy/battery lifetime as well as cost.
- **Human-robot-AI agent collaboration.** Agentic engineering lets teams move from isolated machines to coordinated, instruction-driven workflows. AI agents can turn high-level instructions into executed pipelines, but they are a convenience layer, not a requirement. Start with manual workflows, introduce agents when you are ready, and customize their behavior to match your team's trust boundaries.
- **Broad physical AI applicability.** While robotics is the hero use case, the architecture supports any physical AI workload that follows the simulate → train → evaluate → deploy pattern, including autonomous mobile robots, robotic manipulation, industrial inspection, and embodied AI research.
Whether you are evaluating Azure and NVIDIA as a platform for physical AI, planning a proof of concept, or scaling to production, this toolchain provides a tested solution and working code to accelerate your timeline.
### Who This Is For
- **Robotics researchers** moving from Isaac Sim prototypes to production-grade training and deployment pipelines
- **Platform engineers** standardizing physical AI pipelines across teams with Infrastructure as Code and repeatable workflows
- **Enterprise teams** piloting Jetson + Azure deployments and need security, compliance, and scalability from day one
> [!NOTE]
> **Who it's not for (yet):** This toolchain targets production and pre-production workloads. It is not designed for hobbyist projects, ROS beginners learning the basics, or general-purpose desktop demos. T0 is a first-class single-robot entry point when the goal is a production-oriented capture -> train -> validate -> run workflow.
<!-- -->
> [!TIP]
> **Start on a laptop, not in the cloud.** The default starting path (T0 — Dev)
> closes the full capture -> train -> validate -> run loop on **one laptop and one
> robot**, with **zero cloud and no required Kubernetes**. Plain local processes are
> the baseline; a single-laptop kind cluster is an optional local orchestration profile
> for GPU offload. Cloud, Arc, and fleet components are tier-gated additions you adopt
> only when your scale demands them, not prerequisites. See
> [Start on a Laptop (T0 — Dev)](#-start-on-a-laptop-t0--dev) below.
## 🛫 Start on a Laptop (T0 — Dev)
Adoption is modeled as six graduated tiers (T0-T5). Each tier states the minimum edge and cloud infrastructure required to complete the full training lifecycle: capture demonstrations on a robot, train an imitation policy, validate it, and run that policy back on the robot. Each tier is a legitimate stopping point. You adopt only the infrastructure your scale actually demands; the heavy components are opt-in additions, not a baseline you must stand up first.
**T0 — Dev is the default starting path.** One laptop, one robot, and the full loop:
- **Capture:** ROS 2 bag recording to local disk. No Arc, no ACSA, no PVC.
- **Move data:** `cp` or `rsync` from robot to laptop.
- **Curate:** the dataviewer in `local` mode on the laptop.
- **Train:** `lerobot-train` on the laptop, on CPU or a local GPU.
- **Track:** training writes checkpoints and logs to local disk; hosted tracking enters at T2.
- **Validate:** `run-local-lerobot-eval.py` / `play.py` locally.
- **Run on robot:** the inference node as a plain process or container. No Flux, no gating, no GitOps.
**Edge infra:** ROS 2 and Docker; local kind and Helm are optional. **Cloud infra:** none.
| Tier | When to use it | Quick start |
|---------------------|--------------------------------------------------------------------------|------------------------------------------------------------------------|
| **T0 — Dev** ⭐ | Default. One laptop, one robot; zero cloud and no required Kubernetes. | [Tier 0 — Dev recipe](docs/recipes/tier-0-dev/README.md) |
| **T1 — Lab** | One site, a few robots, a shared GPU box. First cloud: storage. | [Tier 1 — Lab recipe](docs/recipes/tier-1-lab/README.md) |
| **T2 — Pilot** ✅ | Recommended production. One site at scale; cloud training default. | [Tier 2 — Pilot recipe](docs/recipes/tier-2-pilot/README.md) |
| **T3 — Production** | Advanced. Single-site declarative deployment (local k3s + Flux, no Arc). | [Tier 3 — Production recipe](docs/recipes/tier-3-production/README.md) |
| **T4 — Scale** | Advanced. Multi-site **fleet delivery**; Arc as reachability broker. | [Tier 4 — Scale recipe](docs/recipes/tier-4-scale/README.md) |
| **T5 — Operate** | Roadmap. **Fleet intelligence** for drift detection and retraining. | [Tier 5 — Operate recipe](docs/recipes/tier-5-operate/README.md) |
⭐ default · ✅ recommended production
Cloud training, model registries, production Kubernetes (k3s/AKS), Azure Arc, and
the fleet delivery/intelligence planes enter the picture only at the tier where they
earn their keep. An optional single-laptop kind cluster remains a T0 local tool, not
production infrastructure. Pick your tier in
[Getting Started → Choose Your Tier](docs/getting-started/README.md#choose-your-tier),
read the tier-by-tier infrastructure boundaries in the
[Architecture Overview](docs/contributing/architecture.md), and consult the canonical
[Tier Model](docs/design/tier-model.md) for the authoritative tier table and vocabulary.
> [!NOTE]
> **Roadmap honesty.** T5 (Operate / fleet intelligence) is on the roadmap and not yet available. The fleet-intelligence domain is currently specified, with implementation planned. Today's shipping capability spans T0-T4.
## What's Inside
| Capability | Description |
|---------------------------------|-------------------------------------------------------------------------------------------------------------|
| **Simulation & Synthetic Data** | Isaac Sim and Isaac Lab environments for RL task training and synthetic data generation |
| **Edge Data Capture** | ROS 2 demonstration recording on Jetson with chunking, compression, and cloud upload |
| **Cloud Data Pipeline** | Automated ROS-to-LeRobot conversion, quality validation, and event-driven orchestration |
| **Training Infrastructure** | OSMO + Azure ML integration for scalable RL and IL training with experiment tracking |
| **Model Evaluation** | Offline replay evaluation, Isaac Sim evaluation, and evaluation dashboards |
| **Model Deployment** | ONNX/TensorRT conversion, container packaging, and GitOps-based edge deployment |
| **Edge Inference** | Offloaded inference to remote GPU containers on on-premise edge |
| **Agentic Workflows** | Instruction-driven agents that orchestrate data collection, training, evaluation, and deployment end-to-end |
| **Hybrid Architecture** | Azure Arc, air-gapped training support, and MQTT telemetry for connected and disconnected sites |
## Key Features
- **Infrastructure as Code:** Terraform modules for reproducible Azure deployments
- **Containerized Workflows:** Docker-based Isaac Lab training with NVIDIA GPU support
- **MLflow Integration:** Automatic experiment tracking and model versioning
- **Scalable Compute:** Auto-scaling GPU nodes with pay-per-use cost optimization
- **Enterprise Security:** Entra ID integration with managed identities
- **CI/CD Integration:** Automated deployment pipelines with GitHub Actions
- **Edge-to-Cloud Data Pipeline:** Automated capture, upload, conversion, and validation
- **Offloaded Inference:** Automatic GPU container creation, deployment, and inference over RPCs
- **Multi-Modal Training:** Support for reinforcement learning and imitation learning workflows
- **Agentic Pipeline Orchestration:** Describe a task; agents handle data collection through policy deployment
## Quick Start
```bash
./setup-dev.sh
```
The setup script installs Python 3.12 via [uv](https://docs.astral.sh/uv/), creates a virtual environment, and installs training dependencies. This is all the default path (T0 — Dev) needs: train, track, and validate locally on a laptop GPU with no Azure subscription.
Follow the [Quickstart Guide](docs/getting-started/quickstart.md) for the default local-first walkthrough, then [Choose Your Tier](docs/getting-started/README.md#choose-your-tier) when you are ready to graduate to cloud training (T2 — Pilot) or beyond. The cloud, Kubernetes, and fleet steps are opt-in additions layered onto this working local baseline.
## Documentation
Full documentation is available in the [docs/](docs/README.md) directory.
| Guide | Description |
|---------------------------------------------------|----------------------------------------------------------------|
| [Getting Started](docs/getting-started/README.md) | Prerequisites, quickstart, and first training job |
| [Deployment](docs/infrastructure/README.md) | Infrastructure provisioning and setup |
| [Training](docs/training/README.md) | RL and IL training workflows, MLflow, and checkpointing |
| [Security](docs/security/README.md) | Threat model, security guide, deployment responsibilities |
| [Recipes](docs/recipes/README.md) | Guides that take you from a standing start to a working result |
| [Contributing](docs/contributing/README.md) | Architecture, style guides, contribution workflow |
## Architecture
The architecture is organized as graduated tiers (T0-T5): each component enters at the tier where it earns its keep, rather than as a baseline prerequisite. The default path (T0 — Dev) needs only the local components; everything below is an opt-in addition.
- **NVIDIA Isaac Sim & Isaac Lab:** Physics simulation and RL task environments
- **NVIDIA Jetson:** Edge inference and demonstration data capture
- **Azure Storage:** Cloud data and checkpoint storage (opt-in from T1 — Lab)
- **Azure Machine Learning:** Cloud training, experiment tracking, and model registry (default from T2 — Pilot)
- **NVIDIA OSMO:** Workflow orchestration and job scheduling (T2 — Pilot)
- **Local k3s + FluxCD:** Single-site declarative GitOps deployment, no Arc required (T3 — Production)
- **Azure Arc + AKS:** Cross-site reachability and identity broker for multi-site **fleet delivery** (T4 — Scale)
- **Azure IoT Operations & Fabric:** Telemetry aggregation and **fleet intelligence** (T5 — Operate, roadmap)
See the [Architecture Overview](docs/contributing/architecture.md) for the per-tier infrastructure boundaries and the canonical [Tier Model](docs/design/tier-model.md) for the authoritative tier table.
## Agentic Workflows
The toolchain includes agent-driven automation that collapses multi-stage physical AI pipelines into simple, instruction-level interactions.
**How it works:**
1. **Describe the objective.** Provide a natural-language instruction such as "collect 50 demonstrations of an inspection and sorting task and train an IL policy."
2. **Agent plans and executes.** The agent decomposes the objective into pipeline stages: data collection, conversion, training configuration, compute provisioning, and training launch. It then executes each stage using the toolchain's APIs and infrastructure.
3. **Evaluate and iterate.** The agent runs evaluation (simulation replay, success-rate metrics) and presents results. If the policy does not meet acceptance criteria, the agent adjusts hyperparameters or collects additional data and re-trains.
4. **Deploy.** Once a policy passes evaluation, the agent packages it (ONNX/TensorRT), builds a container image, and triggers GitOps deployment to target edge devices.
**What agents can do today:**
| Capability | Description |
|------------------------|------------------------------------------------------------------------------------|
| Sample data collection | Configure Isaac Sim scenes and collect synthetic demonstration datasets |
| RL pipeline execution | Set up Isaac Lab tasks, launch OSMO training jobs, and track experiments in MLflow |
| IL pipeline execution | Convert demonstration data to LeRobot format, run imitation learning training |
| Policy evaluation | Execute offline replay and simulation-based evaluation against success criteria |
| Deployment promotion | Convert checkpoints, package containers, and push to edge via GitOps |
Agents operate within the same security boundaries, managed identities, and RBAC controls as manual workflows. All agent actions are logged and auditable.
### Guardrails and Control
| Question | Answer |
|--------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Are agents required? | No. Every pipeline stage has a manual CLI and API path. Agents are opt-in. |
| Can I use agents for some stages but not others? | Yes. Agents are composable: use them for data collection but run training manually, or vice versa. |
| Are agents opinionated or customizable? | Customizable. Agent behavior is driven by configuration files you control: which stages to automate, compute budgets, approval gates, and evaluation thresholds. |
| What happens if an agent makes a mistake? | Agents request human approval before destructive actions (deploying to production, deleting data). All intermediate artifacts are versioned and recoverable. |
| How are agent actions audited? | Every agent action is logged with the initiating instruction, parameters, and outcome. Logs integrate with Azure Monitor and MLflow. |
## For Developers
### Repository Structure
| Directory | Purpose |
|------------|--------------------------------------------------------------------|
| `src/` | Core Python modules: conversion, validation, training utilities |
| `infra/` | Terraform and Bicep templates for Azure resource provisioning |
| `config/` | YAML configuration schemas for recording, training, and deployment |
| `scripts/` | Setup, benchmarking, and operational helper scripts |
| `tests/` | Unit, integration, and end-to-end test suites |
| `docs/` | All project documentation |
### Development Environment
Prerequisites:
- Python 3.12+
- Docker with NVIDIA Container Toolkit
- Terraform 1.5+ (for infrastructure deployment)
- Azure CLI with an active subscription
- NVIDIA GPU (local development) or Azure GPU VM
Run the test suite (the four component suites mirror the CI split):
```bash
# Run every component at once (uses testpaths from pyproject.toml)
uv run pytest
# Or run a single component
uv run pytest training/tests -v
uv run pytest data-management/tools/tests -v
uv run pytest data-pipeline/capture/tests -v
uv run pytest fleet-deployment/inference/tests -v
```
See [prerequisites](docs/contributing/prerequisites.md) for the complete setup guide.
## Contributing
Contributions are welcome. Whether fixing documentation or adding new training tasks:
1. Read the [Contributing Guide](CONTRIBUTING.md)
2. Review [open issues](https://github.com/microsoft/physical-ai-toolchain/issues)
3. See the [prerequisites](docs/contributing/prerequisites.md) for required tools
## Verifying Git Tags
All release tags are signed. Verify a release tag before using it in production workflows:
```bash
git fetch --tags
git tag -v v1.0.0
```
> [!NOTE]
> `git tag -v` confirms cryptographic integrity and the Rekor entry but does not validate the signer identity. CI gates each `v*` tag with constrained `gitsign verify-tag`, binding the signature to the pinned workflow identity. For identity-constrained local verification, run `gitsign verify-tag --certificate-identity 'https://github.com/microsoft/physical-ai-toolchain/.github/workflows/main.yml@refs/heads/main' --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' v1.0.0`.
This repository uses Sigstore `gitsign` keyless signing for release tags. For tag signing policy and maintainer guidance, see [CONTRIBUTING.md](CONTRIBUTING.md#release-tag-signing).
## Roadmap
See the [project roadmap](docs/contributing/ROADMAP.md) for priorities, timelines, and success metrics.
## Acknowledgments
This toolchain builds upon:
- [NVIDIA Isaac Lab](https://github.com/isaac-sim/IsaacLab): RL task framework
- [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac-sim): physics simulation
- [NVIDIA OSMO](https://developer.nvidia.com/osmo): workflow orchestration
- [LeRobot](https://github.com/huggingface/lerobot): imitation learning dataset format
- Built with [HVE Core](https://github.com/microsoft/hve-core)
- [Offload or Overload: A Platform Measurement Study of Mobile Robotic Manipulation Workloads](https://arxiv.org/abs/2603.18284)
## 🤖 Responsible AI
Microsoft encourages customers to review its Responsible AI Standard when developing AI-enabled systems to ensure ethical, safe, and inclusive AI practices. Learn more at [Microsoft's Responsible AI](https://www.microsoft.com/ai/responsible-ai).
## ⚠️ Deprecations
No interfaces are currently deprecated. When deprecations are announced, they appear here with migration guidance and removal timelines.
See the [Deprecation Policy](docs/deprecation-policy.md) for how interface changes are communicated and managed.
## Legal
This project is licensed under the [MIT License](./LICENSE).
See [SECURITY.md](./SECURITY.md) for the security policy and vulnerability reporting.
See [GOVERNANCE.md](./GOVERNANCE.md) for the project governance model.
See [SUPPORT.md](./SUPPORT.md) for support options and issue reporting.
## Trademark Notice
> This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
> trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft
> trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
> Any use of third-party trademarks or logos are subject to those third-party's policies.
---
<!-- markdownlint-disable MD036 -->
*🤖 Crafted with precision by ✨Copilot following brilliant human instruction,
then carefully refined by our team of discerning human reviewers.*
<!-- markdownlint-enable MD036 -->