AI interactions increasingly span multiple modalities — text, images, voice, code, tools, and structured data. Designing how these modalities work together is orchestration.

Modality Selection

Each modality has strengths:

Text: Precise, editable, referenceable. Best for instructions, explanations, and nuanced content.
Image: Spatial, holistic, immediate. Best for layouts, diagrams, and visual concepts.
Voice: Natural, hands-free, emotional. Best for conversational flow and accessibility.
Code: Executable, precise, verifiable. Best for technical specifications and automation.
Structured data: Tables, forms, JSON. Best for comparison, configuration, and data entry.
Tool use: Actions in external systems. Best for execution, not just generation. The designer decides which modality the AI uses for each part of a response — and which modality the user uses for each input.

Cross-Modal Transitions

When the interaction switches modalities, design the transition:

Text to image: "Here's what that layout could look like" — the AI generates a visual from a text description
Image to text: The user uploads a screenshot, the AI describes or critiques it
Text to tool: The AI writes a plan, then executes it via tool use
Voice to text: Spoken conversation captured and summarised as structured notes Transitions should feel seamless. The user shouldn't have to manually switch modes.

Modality Conflicts

Sometimes modalities compete:

Text says one thing, the image shows another
Voice tone contradicts text content
Structured output doesn't match the conversational context Design for consistency across modalities. Establish a primary modality for each interaction type and treat others as supporting.

Design Artefacts

Modality maps showing which modality is used at each interaction point
Cross-modal transition specifications
Input/output modality matrices per feature
Fallback definitions when a modality is unavailable

Más de este repositorio

mismo repositorio

designer-role

wanghaisheng/openaiworkhorse-design-team

扮演设计师角色，完成从用户研究、UX策略、UI设计到交互设计的全流程设计任务。支持设计研究、设计系统、原型测试、设计运营等多维度职责。

2026-04-241

design-debate

wanghaisheng/openaiworkhorse-design-team

Use when a design direction is uncertain, when the team could go multiple ways, or when the user wants to see competing approaches argued before committing — orchestrates structured debate between agents who advocate for different directions

2026-04-241

design-memory

wanghaisheng/openaiworkhorse-design-team

Use when starting a new project or when taste decisions are made — accumulates the user's aesthetic preferences, recurring patterns, and design instincts across projects so each new project starts with what the system already knows about their taste

2026-04-241

design-retrospective

wanghaisheng/openaiworkhorse-design-team

Use after shipping or completing a design project — structured reflection on what worked, what didn't, what taste decisions landed, and what to carry forward. Feeds learnings back into design-memory so the next project is sharper

2026-04-241

harm-anticipation

wanghaisheng/openaiworkhorse-design-team

Proactively identifying failure modes, misuse, and unintended consequences.

2026-04-241

state-management

wanghaisheng/openaiworkhorse-design-team

Managing shared context, memory, and state across multiple agents.

2026-04-241

Multimodal Orchestration

AI interactions increasingly span multiple modalities — text, images, voice, code, tools, and structured data. Designing how these modalities work together is orchestration.

Modality Selection

Each modality has strengths:

Text: Precise, editable, referenceable. Best for instructions, explanations, and nuanced content.

Image: Spatial, holistic, immediate. Best for layouts, diagrams, and visual concepts.

Voice: Natural, hands-free, emotional. Best for conversational flow and accessibility.

Code: Executable, precise, verifiable. Best for technical specifications and automation.

Structured data: Tables, forms, JSON. Best for comparison, configuration, and data entry.

Tool use: Actions in external systems. Best for execution, not just generation. The designer decides which modality the AI uses for each part of a response — and which modality the user uses for each input.

Cross-Modal Transitions

When the interaction switches modalities, design the transition:

Text to image: "Here's what that layout could look like" — the AI generates a visual from a text description

Image to text: The user uploads a screenshot, the AI describes or critiques it

Text to tool: The AI writes a plan, then executes it via tool use

Voice to text: Spoken conversation captured and summarised as structured notes Transitions should feel seamless. The user shouldn't have to manually switch modes.

Modality Conflicts

Sometimes modalities compete:

Text says one thing, the image shows another

Voice tone contradicts text content

Structured output doesn't match the conversational context Design for consistency across modalities. Establish a primary modality for each interaction type and treat others as supporting.

Design Artefacts

Modality maps showing which modality is used at each interaction point

Cross-modal transition specifications

Input/output modality matrices per feature

Fallback definitions when a modality is unavailable