matlab-integrate-pytorch-vision

name	matlab-integrate-pytorch-vision
description	Creates MATLAB interfaces to Python image processing and computer vision models from GitHub repositories or pip-installable packages using MPyReq. Use when asked to interface MATLAB with a Python CV/image model (segmentation, depth estimation, object detection, image generation, super-resolution, etc.), given a GitHub repo URL for an image/vision model, or asked to create an MPyReq demo for a deep-learning vision pipeline. Do NOT use for general-purpose Python-MATLAB interfacing, non-vision models (NLP, tabular, audio), model deployment/serving, or MATLAB-only image processing workflows.
license	MathWorks BSD-3-Clause
metadata	{"author":"MathWorks","version":"1.0"}

MPyReq MATLAB Interface Builder

Build a MATLAB interface to a Python/PyTorch model repository using the MPyReq framework.

When to Use

• User asks to interface MATLAB with a Python image processing or computer vision model (segmentation, depth estimation, object detection, image generation, super-resolution, pose estimation, optical flow, salient object detection, etc.)
• User provides a GitHub repository URL for a vision/image model and wants to call it from MATLAB
• User asks to "create an MPyReq wrapper" or "MPyReq demo" for an image/CV model
• User wants to run a pip-installable vision model library (e.g., Cellpose, SAM2, Depth-Pro, BiRefNet, StarDist) from MATLAB

When Not to Use

• General-purpose Python-MATLAB interfacing (no vision/image model involved)
• Non-vision models: NLP, audio, tabular, reinforcement learning, time-series
• Model deployment, containerization, or inference servers
• Pure MATLAB image processing workflows with no Python dependency
• Creating Python code (this skill creates MATLAB code that calls Python)

Prerequisites: MPyReq on the MATLAB Path

Before generating any demo script, verify that MPyReq is available. Run which MPyReq via the MATLAB MCP server (if available) or ask the user to confirm.

If MPyReq is NOT on the MATLAB path:

1. Download MPyReq from the MATLAB File Exchange: https://mathworks.com/matlabcentral/fileexchange/182230-matlab-based-python-requirements-manager
2. Install it — either:
   • Open the downloaded .mltbx file in MATLAB (double-click), which installs it as a MATLAB Add-On automatically, or
   • Extract the files and add the folder containing MPyReq.m to the MATLAB path:

     addpath("/path/to/mpyreq");
     savepath; % persist across sessions
     

3. Verify by running which MPyReq in MATLAB — it should return the path to MPyReq.m.

Do not proceed with demo generation until MPyReq is confirmed on the path.

Input

Ask the user for:

1. GitHub repository URL — the Python model repository to interface with
2. What the model does (optional) — to help identify the right inference example

Step 1: Analyze the Repository

Fetch and analyze the GitHub repository to determine:

• Python version requirement — check setup.py, setup.cfg, pyproject.toml, or README for the required Python version. Default to "3.12" if not specified. Use "3.11" if the project needs older compatibility.
• Installation method — determine how the project is installed:
  • If it uses torch.hub.load(): only need torch and torchvision as pip packages (model downloads automatically)
  • If it's a pip-installable package: use MPyReq.pipPackage()
  • If it's a non-packaged git repo: use MPyReq.gitrepo() + MPyReq.requirementTextFile() if a requirements.txt exists
  • If it needs pip install git+<url>: use MPyReq.pipPackage("git+<url>", Name="<ProjectName>")
• Additional dependencies — any extra pip packages needed (e.g., torch, torchvision, etc.)
• Model weights — determine how weights are loaded:
  • torch.hub.load() — weights download automatically, no MPyReq.weights() needed
  • Direct URL download — use MPyReq.weights() with the checkpoint URL
  • HuggingFace .from_pretrained() — weights download automatically via the library
• Inference example — locate the primary inference/prediction code in the README or example scripts
• Preprocessing requirements — check if the model requires specific input normalization (e.g., ImageNet mean/std), resizing, or center cropping

Step 2: Generate the MPyReq Setup Script

Create a MATLAB .m file that sets up the Python environment. Follow these patterns from the demo files:

MANDATORY: Installation folder setup

Every generated script MUST begin with MPyReq.setInstallFolder(). This tells MPyReq where to download Python, packages, and model weights. Without this, MPyReq will show a GUI dialog which blocks non-interactive execution. Also include MPyReq.autoAcceptDownloadPrompts(true) to avoid interactive confirmation prompts.

% Set installation folder (SSD recommended, ~15+ GB free space)
% Change this path to a suitable location on your machine
MPyReq.setInstallFolder(fullfile(tempdir, "MPyReq"));
MPyReq.autoAcceptDownloadPrompts(true);

Pattern A: Simple pip package (like Cellpose)

MPyReq.setInstallFolder(fullfile(tempdir, "MPyReq"));
MPyReq.autoAcceptDownloadPrompts(true);
MPyReq.python("3.12");
MPyReq.pipPackage("<package_name>");

Pattern B: Git repo as pip package (like SAM2)

MPyReq.setInstallFolder(fullfile(tempdir, "MPyReq"));
MPyReq.autoAcceptDownloadPrompts(true);
MPyReq.python("3.12");
MPyReq.pipPackage("git+https://github.com/<org>/<repo>.git", Name="<RepoName>");

Pattern C: Git repo + requirements.txt (like VGGT, BiRefNet)

MPyReq.setInstallFolder(fullfile(tempdir, "MPyReq"));
MPyReq.autoAcceptDownloadPrompts(true);
MPyReq.python("3.11");
MPyReq.gitrepo("https://github.com/<org>/<repo>.git");
reqTxt = MPyReq.pathTo("<repo>") + filesep + "requirements.txt";
MPyReq.requirementTextFile(reqTxt, Name="<repo>Packages");

Pattern D: torch.hub model (like DINOv2, ResNet, etc.)

When the model uses torch.hub.load(), no git clone or weights download is needed — just install torch/torchvision:

MPyReq.setInstallFolder(fullfile(tempdir, "MPyReq"));
MPyReq.autoAcceptDownloadPrompts(true);
MPyReq.python("3.12");
MPyReq.pipPackage("torch");
MPyReq.pipPackage("torchvision");
% Model loads automatically via torch.hub:
model = py.torch.hub.load('org/repo', 'model_name');

Weights download pattern

Only needed when weights are NOT handled by torch.hub.load() or .from_pretrained():

MPyReq.weights("<weights_url>", DownloadTo=MPyReq.pathTo("<RepoName>") + filesep + "checkpoints");

Step 3: Create the MATLAB Inference Interface

Translate the Python inference example to MATLAB. Refer to these resource files for conversion rules and patterns:

• references/python_matlab_conversions.md — Python-to-MATLAB syntax conversion table
• references/image_conversion_patterns.md — Image tensor conversions (MATLAB to Python and back), dimension reordering, display patterns

Step 4: Assemble the Final Script

Create a single demo<ModelName>.m file with clear sections:

%% Setup Python Environment
% Start with clean state (only if switching projects)
% terminate(pyenv); clear MPyReq

% Set installation folder (SSD recommended, ~15+ GB free space)
% Change this path to a suitable location on your machine
MPyReq.setInstallFolder(fullfile(tempdir, "MPyReq"));
MPyReq.autoAcceptDownloadPrompts(true);
MPyReq.python("<version>");
% ... package installation calls ...

%% Reference Python Code
%{
<paste the original Python inference code as a comment block>
%}

%% Load Model
% ... model loading code ...

%% Run Inference
% ... load input, run model, extract results ...

%% Visualize Results
% ... display/plot results ...

Step 5: Test with MATLAB MCP Server

Check if a MATLAB MCP server tool is available in the current session (look for MCP tools like matlabRunCode, matlab_run, or similar).

If MATLAB MCP server IS available:

1. Run the setup section — execute the MPyReq.python() and package installation calls through the MATLAB MCP server to verify the Python environment installs correctly.
2. Run the inference section — execute the model loading and inference code to verify end-to-end functionality.
3. Iterate on errors — if any step fails, read the error output, fix the generated code, and re-run.

Attempt Limit and Graceful Fallback

Track each fix-and-retry cycle as one attempt. Stop after a maximum of 5 attempts (combined across setup and inference). If the code is not fully working after 5 attempts:

1. Stop iterating. Do not continue retrying the same or similar approaches.
2. Save the best version of demo<ModelName>.m — the version that got furthest (e.g., setup succeeded but inference failed, or partial inference ran).
3. Return the script to the user with a structured handoff:

## What Works
- <list sections/steps that executed successfully>

## What Needs Attention
- <describe the remaining failure: error message, which line/section fails>
- <root cause hypothesis if known>

## Recommended Next Steps
1. <most likely fix — e.g., "Try Python 3.11 instead of 3.12 due to package compatibility">
2. <alternative approach — e.g., "Install system dependency X before running">
3. <manual verification — e.g., "Run `pip install <pkg>` in the MPyReq venv directly to check build logs">

## Environment Details
- Python version attempted: <version>
- Platform: <OS>
- Errors encountered: <brief summary of distinct errors across attempts>

4. Mark clearly in the script which sections are verified vs. unverified using comments:

%% Setup Python Environment — VERIFIED
% ... (code that ran successfully) ...

%% Run Inference — NEEDS MANUAL VERIFICATION
% The following section encountered errors during automated testing.
% See recommended next steps above.
% ... (best-effort code) ...

Early exit conditions (stop before 5 attempts):

• Same error repeats 2+ times with no new information — stop immediately
• Environment/platform issue outside MATLAB's control (e.g., missing system library, network block, GPU driver mismatch) — stop and report
• Package build failure requiring system-level intervention (e.g., C compiler missing, CUDA version mismatch) — stop and report

If MATLAB MCP server is NOT available:

1. Return the generated demo<ModelName>.m file to the user.
2. Provide setup instructions summarizing:
   • Prerequisites (MATLAB version, MPyReq on path)
   • The MPyReq commands that will run and what they install
   • Expected first-run behavior (downloads Python, packages — may take several minutes)
   • How to run: open the script in MATLAB and run section-by-section (Ctrl+Enter)
3. Note any platform-specific considerations (e.g., Windows CUDA setup, UV_EXTRA_INDEX_URL).

Important Notes

• NEVER modify the cloned repository's source code (e.g., editing config files, patching Python modules) without explicitly asking the user for permission first. If a workaround requires source edits, describe the change and let the user decide.
• NEVER use Python dunder methods (__enter__, __exit__, __init__, etc.) in MATLAB — double underscores are invalid MATLAB syntax. For context managers like torch.no_grad() or torch.inference_mode(), use the equivalent functional API (e.g., py.torch.set_grad_enabled(false/true)) instead of the with statement pattern.
• Always permute outputs back to MATLAB dimension ordering — PyTorch uses NCHW (batch first, channels second). MATLAB expects batch last and channels second-to-last (H x W x C x B). Use permute to reorder, then squeeze to remove singleton batch dims.
• Convert bounding boxes to MATLAB format — Python models typically return [x1, y1, x2, y2] (corner pairs). MATLAB expects M x 4 as [startX, startY, width, height]. Convert with: boxes(:,3) - boxes(:,1) for width, boxes(:,4) - boxes(:,2) for height.
• Always use imread for loading images — never use Python image libraries (PIL, OpenCV, etc.). Keep image I/O on the MATLAB side and convert to tensors for Python.
• Always include the original Python inference code as a %{ %} comment block for reference
• Use py.importlib.import_module() when direct py.module.submodule doesn't work (deep nesting)
• For Windows GPU/CUDA support, note that UV_EXTRA_INDEX_URL may need to be set
• If the repo requires cd to a specific directory (e.g., for relative config paths), use cd(MPyReq.pathTo("<Name>"))
• Test that the generated code handles both GPU and CPU paths where applicable
• Keep the MATLAB interface idiomatic — use MATLAB image types, display functions, and data structures

For additional gotchas (class ID mapping, opencv-python issues, insertObjectAnnotation tips), see references/common_gotchas.md.

Reference Files

The following reference files are bundled with this skill. Read them from the skill's references/ directory:

Demo examples (expected output format)

• references/demo-cellpose.md — simple pip package pattern
• references/demo-sam2.md — git+pip pattern with weights
• references/demo-depthpro.md — git repo + pip install pattern with weights
• references/demo-birefnet.md — git repo + requirements.txt + weights
• references/demo-stardist.md — complex dependency management
• references/demo-vggt.md — git repo + requirements.txt, point cloud output

Conversion guides

• references/python_matlab_conversions.md — Python-to-MATLAB syntax conversion table
• references/image_conversion_patterns.md — Image tensor conversions and display patterns
• references/common_gotchas.md — Class ID mapping, opencv-python issues, annotation tips

MPyReq documentation

• references/readme.md — MPyReq documentation overview

The MPyReq class is in the project's mpyreq/MPyReq.m.

Skill Feedback

If this skill's guidance was wrong, incomplete, or didn't work, emit a plain-text feedback marker in your response, then continue with the task.

Format: IPCV-SKILL-FEEDBACK [matlab-integrate-pytorch-vision] [category]: what failed; what you did instead Categories: wrong-command (incorrect/outdated), missing-guidance (omitted step), discovery-failure (suggested approach didn't work), platform-issue (environment-specific)

---

Copyright 2026 The MathWorks, Inc.

---