Executing sequences of agent, skill, and tool operations
Handling failures and retries gracefully
Tracking execution status and results
Ensuring critical bootstrap procedures run even when the server is offline
Prior to this extension, users had to manually manage these workflows using custom scripts or complex prompt engineering.
Solution Overview
The flow-engine extension introduces:
Workflow Definitions: JSON-based definitions of workflows consisting of nodes (agent, skill, tool)
Execution Engine: Reliable execution of workflows with dependency resolution
Persistence Layer: Storage of workflow definitions, runs, and task instances in js-doc-store-server
Monitoring Tools: Commands to check workflow status and list available workflows
Bootstrap Automation: Automatic setup of essential database tables
TUI Integration: Widget displaying flow engine readiness status
Key Features
Workflow Node Types
agent: Execute Pi subagents with input/output mapping
skill: Load and invoke Pi skills from the skill registry
tool: Execute Pi built-in or custom tools (read, write, bash, etc.)
Dependency Resolution
Automatic topological sort of workflow nodes based on dependsOn relationships
Prevents circular dependency detection and provides clear error messages
Input/Output Mapping
Use ${variable} syntax to map outputs from previous nodes as inputs to subsequent nodes
Support for literal values and complex object mapping
Flexible output storage (raw output under node ID or mapped to context variables)
Error Handling & Retries
Configurable maximum retry attempts per node
Automatic retry with exponential backoff (leveraging Pi's retry system)
Detailed error logging and tracking via Pi's message system
Persistence
All workflow definitions stored in flows table
All workflow executions stored in flow_runs table
Individual task executions stored in task_instances table
Automatic table creation via run_bootstrap tool
TUI Integration
Session startup widget showing "🟢 Flow Engine Ready" when operational
Extensible for future enhancement with detailed workflow visualization
Installation
The extension is installed as a standard Pi extension:
# From anywhere, install the extension
pi install git:https://github.com/MauricioPerera/pi-extension-data-architect
# Or install locally if you have the repo cloned
pi install -l /path/to/pi-extension-data-architect
Usage
Once installed, the extension provides several custom tools:
flow_create
Store a new workflow definition.
Parameters:
name (string, required): Unique name for the workflow
definition (string, required): JSON string representing the workflow definition
runId (string, required): ID of the workflow run to check
Example:
/tool:flow_status
{
"runId": "abc123def456"
}
run_bootstrap
Ensure essential tables exist for the flow engine to operate.
Parameters: None
Example:
/tool:run_bootstrap
# Creates tables: flows, flow_runs, task_instances, skills, messages if they don't exist
Workflow Definition Format
A workflow definition is a JSON object with the following structure:
{"nodes":[{"id":"unique_node_id","type":"agent|skill|tool","agent":"agent_name",// Required for agent type"skill":"skill_name",// Required for skill type"tool":"tool_name",// Required for tool type"dependsOn":["node_id1","node_id2"],// Optional: array of node IDs this node depends on"inputMapping":{// Optional: mapping of inputs for this node"param1":"${previous_node.output}","param2":"literal_value","param3":"${context.variable}"},"outputMapping":{// Optional: mapping of outputs to context"context_key":"${node.output}","another_key":"literal_value"},"maxRetries":3// Optional: maximum retry attempts (default: 0)}]}