coact-1-coding-agents

name	coact-1-coding-agents
title	CoAct-1 - Computer-using Agents with Coding as Actions
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2508.03923
keywords	["computer-agents","code-execution","multi-agent","orchestration"]
description	Hybrid multi-agent architecture where orchestrator delegates tasks to GUI Operator or Programmer agent. Coding enables efficiency on computational tasks, achieving 60.76% on OSWorld with 33% fewer steps.

CoAct-1: Computer-using Agents with Coding as Actions

Core Concept

CoAct-1 overcomes the fundamental limitation of GUI-only computer agents by enabling agents to write and execute code for computational tasks. A central Orchestrator analyzes each subtask and delegates to either a GUI Operator for visual interaction or a Programmer agent that writes executable scripts. This hybrid approach dramatically improves efficiency on complex computer automation tasks.

Architecture Overview

• Orchestrator Agent: Analyzes subtasks and routes to GUI Operator or Programmer
• GUI Operator Agent: Interacts with desktop through visual recognition and clicking
• Programmer Agent: Writes and executes Python/Bash scripts for computational tasks
• Task Decomposition: Breaks complex tasks into actionable subtasks
• Multi-agent Coordination: Seamless handoff between interaction modes

Implementation Steps

Step 1: Build Task Analyzer and Orchestrator

Create the central coordinator that decides execution strategy for each subtask.

from enum import Enum
from typing import Dict, List, Tuple

class ExecutionMode(Enum):
    GUI = "gui"
    CODE = "code"
    HYBRID = "hybrid"

class TaskOrchestrator:
    """
    Analyzes subtasks and routes to GUI or Programmer agent.
    """

    def __init__(self, model_name="gpt-4"):
        self.model = model_name

    def analyze_subtask(self, subtask: str, context: Dict) -> Tuple[ExecutionMode, str]:
        """
        Determine optimal execution mode for subtask.

        Args:
            subtask: Subtask description
            context: Current context (files, applications, etc.)

        Returns:
            (execution_mode, routing_reason)
        """
        analysis_prompt = f"""
        Subtask: {subtask}
        Current context: {context}

        For this subtask, should we use:
        1. GUI: Click buttons, fill forms, visual interaction
        2. CODE: Write Python/Bash script for automation
        3. HYBRID: Use both approaches

        Consider:
        - Is this a computational/file operation? (favor CODE)
        - Does this need visual interaction? (favor GUI)
        - Can this be automated programmatically? (favor CODE)

        Respond with JSON:
        {{
            "mode": "GUI|CODE|HYBRID",
            "reasoning": "brief explanation",
            "complexity": 1-5
        }}
        """

        response = self.model.generate(analysis_prompt)
        result = self._parse_json_response(response)

        mode_map = {"GUI": ExecutionMode.GUI, : ExecutionMode.CODE, : ExecutionMode.HYBRID}
        mode = mode_map.get(result[], ExecutionMode.HYBRID)

         mode, result[]

     () -> []:
        
        decomposition_prompt = 

        response = .model.generate(decomposition_prompt)
        subtasks = ._parse_json_response(response)

         subtasks

     () -> :
        
        
        subtasks = .decompose_task(task)

        execution_log = []
        context = {}

         idx, subtask_desc  (subtasks):
            ()

            
            mode, reasoning = .analyze_subtask(subtask_desc[], context)

            ()

            
             mode == ExecutionMode.GUI:
                result = gui_agent.execute(subtask_desc[], context)

             mode == ExecutionMode.CODE:
                result = code_agent.execute(subtask_desc[], context)

             mode == ExecutionMode.HYBRID:
                
                :
                    result = code_agent.execute(subtask_desc[], context)
                 Exception  e:
                    ()
                    result = gui_agent.execute(subtask_desc[], context)

            
            context.update(result.get(, {}))

            execution_log.append({
                : subtask_desc[],
                : mode.value,
                : result[],
                : result
            })

              result[]:
                ()
                

         {
            : task,
            : (log[]  log  execution_log),
            : (execution_log),
            : execution_log
        }

     () -> :
        
         json
         re
         = re.search(, response, re.DOTALL)
         :
             json.loads(.group())
         {}

Step 2: Implement Programmer Agent

Create agent that writes and executes code.

class ProgrammerAgent:
    """
    Executes subtasks by writing and executing code.
    """

    def __init__(self, model_name="gpt-4", sandbox=True):
        self.model = model_name
        self.sandbox = sandbox
        self.execution_history = []

    def execute(self, subtask: str, context: Dict) -> Dict:
        """
        Execute subtask via code generation and execution.

        Args:
            subtask: Task description
            context: Current context (files, variables, etc.)

        Returns:
            Execution result with success flag and outputs
        """
        # Generate code for subtask
        code = self.generate_code(subtask, context)

        if not code:
            return {"success": False, "error": "Could not generate code"}

        # Execute code
        try:
            result = self.execute_code(code, context)
            return {
                "success": True,
                "code": code,
                "output": result,
                "context_updates": self._extract_context_updates(result)
            }
        except Exception as e:
            return {
                "success": ,
                : code,
                : (e),
                : {}
            }

     () -> :
        
        code_prompt = 

        response = .model.generate(code_prompt)

        
         re
        code_match = re.search(, response, re.DOTALL)
         code_match:
             code_match.group()

         response

     () -> :
        
         .sandbox:
            result = ._execute_sandboxed(code, context)
        :
            result = ._execute_unsafe(code, context)

        .execution_history.append({
            : code,
            : result
        })

         result

     () -> :
        
         tempfile
         subprocess
         json

        
         tempfile.NamedTemporaryFile(mode=, suffix=, delete=)  f:
            
            f.write()

            
            f.write(code)

            script_path = f.name

        :
            
            result = subprocess.run(
                [, script_path],
                capture_output=,
                timeout=,
                text=
            )

             {
                : result.stdout,
                : result.stderr,
                : result.returncode,
                : result.returncode == 
            }

         subprocess.TimeoutExpired:
             {
                : ,
                : 
            }

     () -> :
        
        :
            exec_globals = {: {}}
            exec_globals.update(context)

            (code, exec_globals)

             {
                : exec_globals,
                : 
            }

         Exception  e:
             {
                : (e),
                : 
            }

     () -> :
        
        
         {
            : result.get(, ),
            : result.get(, )
        }

Step 3: Implement GUI Operator Agent

Create agent for visual interaction.

class GUIOperatorAgent:
    """
    Executes subtasks via GUI interaction.
    """

    def __init__(self, browser_controller=None):
        self.browser = browser_controller
        self.interaction_log = []

    def execute(self, subtask: str, context: Dict) -> Dict:
        """
        Execute subtask via GUI interaction.

        Args:
            subtask: Task description
            context: Current GUI context

        Returns:
            Execution result
        """
        # Take screenshot
        screenshot = self.browser.take_screenshot()

        # Analyze what to do
        action_plan = self.plan_actions(subtask, screenshot, context)

        # Execute actions
        try:
            for action in action_plan:
                self.execute_action(action, screenshot)
                screenshot = self.browser.take_screenshot()

            return {
                "success": True,
                "final_screenshot": screenshot,
                "actions": action_plan,
                "context_updates": {}
            }

        except Exception as e:
            return {
                "success": False,
                "error": str(e),
                "context_updates": {}
            }

     () -> []:
        
        
        visual_analysis = ._analyze_screenshot(screenshot_bytes)

        planning_prompt = 

        
        response = .model.generate(planning_prompt)

        actions = ._parse_action_list(response)

         actions

     ():
        
        action_type = action.get()

         action_type == :
            .browser.click_element(action[])

         action_type == :
            .browser.type_text(action[], action[])

         action_type == :
            .browser.scroll(action[])

     () -> :
        
        
        

     () -> []:
        
         json
         re
         = re.search(, response, re.DOTALL)
         :
             json.loads(.group())
         []

Step 4: Integrate Components

Create end-to-end orchestration system.

def run_coact_agent(task: str, browser_controller) -> Dict:
    """
    Run CoAct-1 agent on a task.

    Args:
        task: Task description
        browser_controller: Browser control interface

    Returns:
        Execution results
    """
    # Initialize agents
    orchestrator = TaskOrchestrator()
    programmer = ProgrammerAgent()
    gui_operator = GUIOperatorAgent(browser_controller)

    # Orchestrate execution
    result = orchestrator.coordinate_execution(
        task,
        gui_operator,
        programmer
    )

    # Print summary
    print(f"\nTask: {task}")
    print(f"Result: {'SUCCESS' if result['completed'] else 'FAILED'}")
    print(f"Steps: {result['steps']}")

    return result

Practical Guidance

When to Use CoAct-1

• Complex computer automation: Multi-step tasks requiring both GUI and computation
• File/data processing: Scripts handle these more efficiently than GUI clicks
• Mixed-mode interactions: Tasks needing visual components and computation
• Efficiency-critical workflows: Code execution 10x faster than GUI simulation

When NOT to Use CoAct-1

• High visual complexity: UI changes frequently or requires deep understanding
• Security restrictions: Code execution may be restricted
• Real-time responsiveness: GUI-only simpler for predictable delays
• Fully visual tasks: No computational component to automate

Hyperparameter Recommendations

• Code execution timeout: 30-60 seconds per script
• Max subtasks: 10-20 per main task
• Sandbox restrictions: Disable file system access if untrusted
• Vision model: GPT-4V or Claude Vision for screenshot analysis

Key Insights

The critical insight is recognizing that many computer automation tasks have computational components that GUI interaction handles inefficiently. By enabling code execution alongside GUI interaction, CoAct-1 exploits this asymmetry. The Orchestrator's routing decision is key: it must identify when coding is faster and safer than GUI simulation.

Reference

CoAct-1: Computer-using Agents with Coding as Actions (arXiv:2508.03923)

Introduces hybrid multi-agent architecture where Programmer writes code for computational tasks while GUI Operator handles visual interaction. Achieves 60.76% on OSWorld with 33% reduction in execution steps through intelligent task routing.