page-agent

name

page-agent

description

Expert guidance for implementing Alibaba's PageAgent — a client-side GUI agent framework that enables natural language control of web applications through text-based DOM manipulation. **PROACTIVE ACTIVATION**: Use this skill when the user asks to add AI-powered natural language automation to a web page, integrate a GUI agent, implement PageAgent, or enable LLM-driven DOM interaction on any website. **DETECTION**: At the start of a session, check for `page-agent` in the project's `package.json` dependencies. If found, apply PageAgent patterns proactively. Also activate when the user mentions "page-agent", "PageAgent", "GUI agent on web page", or "natural language web automation". **USE CASES**: Installing and configuring PageAgent, connecting LLM providers (OpenAI, Qwen, Claude, DeepSeek, Gemini, Ollama), creating custom tools with Zod schemas, implementing data masking, writing custom instructions, building custom UIs with PageAgentCore, configuring PageController, securing operations with allowlists, integrating with third-party agents, deploying via NPM/CDN/Chrome Extension, and troubleshooting common issues.

Alibaba PageAgent — Client-Side GUI Agent Framework

Auto-activation: This skill activates when page-agent is detected in project dependencies or when implementing natural language web automation.

What is PageAgent?

PageAgent is a purely client-side web automation framework powered by LLMs. It converts natural language commands into DOM operations through an iterative Re-act (Reasoning + Acting) loop. Unlike server-side tools (Selenium, Playwright) or screenshot-based approaches, PageAgent:

• Runs entirely within the webpage as JavaScript code
• Uses text-based DOM analysis (no screenshots, no vision models)
• Requires zero backend infrastructure
• Supports any OpenAI-compatible LLM provider

┌─────────────────────────────────────────────────────────────────┐
│                     PageAgent Architecture                       │
│                                                                  │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────────┐ │
│  │  User Input  │────▶│ PageAgent    │────▶│  LLM Provider    │ │
│  │  (Natural    │     │ Core         │◀────│  (OpenAI, Qwen,  │ │
│  │   Language)  │     │ (Re-act Loop)│     │   Claude, etc.)  │ │
│  └──────────────┘     └──────┬───────┘     └──────────────────┘ │
│                              │                                   │
│                       ┌──────▼───────┐                          │
│                       │ Page         │                          │
│                       │ Controller   │                          │
│                       │ (DOM ↔ HTML) │                          │
│                       └──────┬───────┘                          │
│                              │                                   │
│                       ┌──────▼───────┐                          │
│                       │  Live DOM    │                          │
│                       │  (Browser)   │                          │
│                       └──────────────┘                          │
└─────────────────────────────────────────────────────────────────┘

Key Concepts — The Re-act Loop

Each agent step follows this cycle:

1. Observe: PageController.getBrowserState() extracts DOM state → simplified indexed HTML
2. Think: System prompt + user task + history + browser state → sent to LLM via MacroTool (forces reflection-before-action structure: evaluation_previous_goal, memory, next_goal, then action)
3. Act: Selected tool executed against the live DOM (click, type, scroll, etc.)
4. Update: Result recorded in history, events emitted, lifecycle hooks called
5. Check: If action is done, return result; otherwise loop back to Observe

The MacroTool pattern wraps all tools into a single "macro" tool that forces the LLM to reflect before acting — outputting evaluation, memory, and next_goal fields before selecting any action. This significantly reduces impulsive errors.

Quick Start

Option 1: NPM Install (Recommended for Production)

npm install page-agent

import { PageAgent } from 'page-agent'

const agent = new PageAgent({
  // LLM Configuration (required)
  model: 'gpt-4.1-mini',           // or 'qwen3.5-plus', 'claude-haiku-4.5', etc.
  baseURL: 'https://api.openai.com/v1',  // OpenAI-compatible endpoint
  apiKey: 'YOUR_API_KEY',

  // Optional Configuration
  language: 'en-US',               // UI language
  maxSteps: 20,                    // Max reasoning steps (default: 20)
  temperature: 0.1,                // LLM temperature
  maxRetries: 3,                   // Retry on LLM failures
})

// Show the built-in chat panel
agent.panel.show()

// Or execute tasks programmatically
const result = await agent.execute('Click the login button and fill username as "admin"')
console.log(result.success, result.data)

Option 2: CDN Script (Quick Testing Only)

<!-- ⚠️ Demo only — uses shared testing API key with rate limits -->
<script src="https://cdn.jsdelivr.net/npm/page-agent@1.6.0/dist/iife/page-agent.demo.js" crossorigin="true"></script>

Option 3: Chrome Extension (Multi-Tab)

The Chrome extension (@page-agent/ext) enables multi-tab workflows, accessible via window.PAGE_AGENT_EXT. Useful when tasks span across multiple pages. Requires a token for authorization.

Package Architecture (Monorepo)

Package	NPM Name	Purpose
page-agent	page-agent	Complete package with UI panel (use this for standard integration)
@page-agent/core	@page-agent/core	Headless agent core — Re-act loop, tools, state management (no UI)
@page-agent/llms	@page-agent/llms	LLM client — OpenAI-compatible API, retry logic, model patches
@page-agent/page-controller	@page-agent/page-controller	DOM extraction, simplified HTML generation, action execution
@page-agent/ui	@page-agent/ui	React-based Panel component for human-in-the-loop interaction
@page-agent/ext	Private	Chrome extension for multi-tab automation

Dependency chain: page-agent → @page-agent/core → @page-agent/llms + @page-agent/page-controller

Supported LLM Models

Models must support OpenAI-compatible API and tool calls (function calling).

Provider	Tested Models	Recommended
Qwen (Alibaba)	qwen3.5-plus, qwen3.5-flash, qwen-3-max	⭐ qwen3.5-plus, qwen3.5-flash
OpenAI	gpt-5.1, gpt-4.1-mini	⭐ gpt-5.1
DeepSeek	deepseek-3.2	⭐ deepseek-3.2
Google	gemini-3-flash	⭐ gemini-3-flash
Anthropic	claude-haiku-4.5	⭐ claude-haiku-4.5
xAI	Grok (via patches)	—
Ollama	qwen3:14b (local)	Use with 64K context

Ollama Local Setup

# Required environment variables
export OLLAMA_CONTEXT_LENGTH=64000
export OLLAMA_HOST=0.0.0.0:11434
export OLLAMA_ORIGINS="*"

ollama pull qwen3:14b

const agent = new PageAgent({
  model: 'qwen3:14b',
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama',  // any non-empty string
})

LLM Configuration

interface LLMConfig {
  model: string       // Model name (e.g., 'gpt-4.1-mini')
  baseURL: string     // OpenAI-compatible API endpoint
  apiKey: string      // API key (⚠️ exposed client-side — use proxy in production)
  temperature?: number // Default: model-dependent
  maxTokens?: number   // Max response tokens
  maxRetries?: number  // Retry count on failure (default: 3)
}

🔐 Production Security: Backend Proxy Pattern

CRITICAL: Never expose API keys in client-side code for production. Use a backend proxy:

const agent = new PageAgent({
  model: 'gpt-4.1-mini',
  baseURL: '/api/llm-proxy',  // Your backend endpoint
  apiKey: '',                 // Empty — backend handles auth
})

Backend proxy options:

1. Reverse proxy through your existing API server
2. Edge functions (Vercel Edge, Netlify Edge)
3. Serverless functions (AWS Lambda, Cloud Functions)

The proxy receives the OpenAI-compatible request, injects the real API key, and forwards to the LLM provider.

Custom Tools — Extending Agent Capabilities

Custom tools are the primary way to make PageAgent do domain-specific work. Tools use Zod schemas for type-safe input validation.

Zod Version

PageAgent uses Zod for tool schemas. Both Zod 3 (≥3.25.0) and Zod 4 are supported. Always import from zod/v4 subpath. Zod Mini is NOT supported.

import { z } from 'zod/v4'  // ✅ Always use this import

Defining Custom Tools

import { z } from 'zod/v4'
import { PageAgent, tool } from 'page-agent'

const agent = new PageAgent({
  // ... LLM config ...
  customTools: {
    // Add a new tool
    add_to_cart: tool({
      description: 'Add a product to the shopping cart by its product ID.',
      inputSchema: z.object({
        productId: z.string().describe('The unique product identifier'),
        quantity: z.number().min(1).default(1).describe('Number of items to add'),
      }),
      execute: async function (input) {
        // `this` is the PageAgent instance — access DOM state, panel, etc.
        await fetch('/api/cart', {
          method: 'POST',
          body: JSON.stringify(input),
        })
        return `✅ Added ${input.quantity}x ${input.productId} to cart.`
      },
    }),
  },
})

Tool Anatomy

interface PageAgentTool {
  name: string              // Auto-set from the key in customTools record
  description: string       // Shown to LLM — be specific about when to use
  inputSchema: ZodSchema    // Zod validation schema (also generates JSON Schema for LLM)
  execute: (this: PageAgent, input: ValidatedInput) => Promise<string> | string
  pageFilter?: {            // (Beta) Restrict tool to specific URL patterns
    include?: string[]      // Glob patterns — tool only available on matching pages
    exclude?: string[]      // Glob patterns — tool hidden on matching pages
  }
}

Overriding & Removing Built-in Tools

const agent = new PageAgent({
  customTools: {
    scroll: null,               // Remove the scroll tool entirely
    execute_javascript: null,   // Remove JS execution for security
    click_element: tool({ ... }), // Override the built-in click with custom implementation
  },
})

Runtime Tool Registration

// Add tools dynamically after initialization
agent.registerTool(tool({
  name: 'submit_feedback',
  description: 'Submit user feedback to the backend',
  inputSchema: z.object({ rating: z.number().min(1).max(5), comment: z.string() }),
  execute: async function (input) {
    await fetch('/api/feedback', { method: 'POST', body: JSON.stringify(input) })
    return '✅ Feedback submitted successfully.'
  },
}))

Page Filters (Beta) — Context-Aware Tools

Restrict tools to specific pages to reduce LLM token usage and improve accuracy:

const agent = new PageAgent({
  customTools: {
    approve_order: tool({
      description: 'Approve a pending order.',
      inputSchema: z.object({ orderId: z.string() }),
      pageFilter: {
        include: ['/admin/orders', '/admin/orders/*'],
        exclude: ['/admin/orders/archived'],
      },
      execute: async function (input) {
        await fetch(`/api/orders/${input.orderId}/approve`, { method: 'POST' })
        return `✅ Order ${input.orderId} approved.`
      },
    }),
  },
})

Best Practices for Custom Tools

1. Description Writing: Start with action verbs ("Add...", "Submit...", "Navigate to..."). Be specific about when to use.
2. Schema Design: Use .describe() on Zod fields — these appear in the LLM's tool schema. Use defaults for common values.
3. Error Handling: Catch exceptions, return descriptive error messages with emoji prefixes (✅, ❌, ⏳). Never expose sensitive data.
4. Return Values: Return strings that help the LLM understand what happened and decide next steps.
5. Performance: Keep tool execution fast. The agent waits for tool results before the next reasoning step.

Custom Instructions — Guiding Agent Behavior

System Instructions (Global)

const agent = new PageAgent({
  instructions: {
    system: `You are a professional e-commerce assistant.
    Guidelines:
    - Always confirm before placing orders
    - Never modify payment information
    - Use polite, professional language`
  },
})

Page-Specific Instructions (Dynamic)

const agent = new PageAgent({
  instructions: {
    system: 'You are an order management assistant.',
    getPageInstructions: (url) => {
      if (url.includes('/checkout')) {
        return `This is the checkout page. Important fields: shipping address, payment method, promo code.
        Never auto-submit — always ask user to confirm before clicking "Place Order".`
      }
      if (url.includes('/products')) {
        return `This is the product listing page. Help users find products by name, category, or price range.`
      }
      return undefined  // No special instructions for other pages
    },
  },
})

Data Masking — Protecting Sensitive Information

Use transformPageContent to sanitize the DOM content before it's sent to the LLM:

const agent = new PageAgent({
  transformPageContent: async (content) => {
    // Mask phone numbers: 13812345678 → 138****5678
    content = content.replace(/\b(1[3-9]\d)(\d{4})(\d{4})\b/g, '$1****$3')
    
    // Mask emails: [e-mail] → j***@example.com
    content = content.replace(
      /\b([a-zA-Z0-9._%+-])[^@]*(@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})\b/g,
      '$1***$2'
    )
    
    // Mask credit card numbers
    content = content.replace(/\b(\d{4})\s?\d{4}\s?\d{4}\s?(\d{4})\b/g, '$1 **** **** $2')
    
    return content
  },
})

PageController Configuration

The PageController handles all DOM interaction. Configure it for optimal element detection:

const agent = new PageAgent({
  // PageController options
  enableMask: true,           // Show blocking overlay during automation (default: true via PageAgent)
  viewportExpansion: -1,      // -1 = index ALL elements (full page), 0 = viewport only, >0 = viewport + N pixels
  highlightOpacity: 0.0,     // Visual highlight on elements (0 = transparent, 1 = opaque)
  highlightLabelOpacity: 0.1,// Label opacity for indexed elements

  // Element filtering
  interactiveBlacklist: [     // Elements AI should NEVER interact with
    document.getElementById('admin-panel'),
    () => document.querySelector('.dangerous-button'),
  ],
  interactiveWhitelist: [],   // Force elements to be interactive even if not detected
  
  // HTML attributes to include in simplified DOM
  includeAttributes: ['aria-label', 'data-testid', 'placeholder', 'title', 'alt'],
})

Key: How PageController Represents the DOM

PageController converts the live DOM into simplified, indexed HTML for the LLM:

[0]<a aria-label="Home">Home</a>
[1]<button>Click Me</button>
[2]<input type="text" placeholder="Search...">
Welcome to the page
[3]<select>
  <option>Option A</option>
  <option>Option B</option>
</select>

The LLM references elements by numeric index (not CSS selectors), making it:

• Deterministic: Indices are stable within a single DOM snapshot
• LLM-friendly: Simple numeric references in responses
• No selector fragility: No XPath or CSS selector generation

BrowserState Output Format

Each step, the LLM sees a structured prompt including:

Current Page: [Example Site](https://example.com)
Page info: 1920x1080px viewport, 1920x2400px total page size, 0.0 pages above, 1.2 pages below
Interactive elements from top layer of the current page inside the viewport:
[Start of page]
[0]<a aria-label="Home">Home</a>
[1]<button>Click Me</button>
[2]<input type="text" placeholder="Search...">
Welcome to the page
... 1320 pixels below (1.2 pages) - scroll to see more ...

Using PageAgentCore (Headless / Custom UI)

For custom UIs or headless automation, use PageAgentCore directly:

import { PageAgentCore } from '@page-agent/core'
import { PageController } from '@page-agent/page-controller'

// Create the controller manually
const pageController = new PageController({
  enableMask: false,
  viewportExpansion: -1,
})

// Create headless agent
const agent = new PageAgentCore(pageController, {
  model: 'gpt-4.1-mini',
  baseURL: 'https://api.openai.com/v1',
  apiKey: 'YOUR_API_KEY',
})

// Listen to events for custom UI
agent.addEventListener('statuschange', (e) => {
  console.log('Agent status:', e.detail.status)  // 'idle' | 'running' | 'paused' | 'done' | 'error'
})

agent.addEventListener('historychange', (e) => {
  // Historical events — persisted and sent to LLM as context
  const history = e.detail.history
  console.log('Latest step:', history[history.length - 1])
})

agent.addEventListener('activity', (e) => {
  // Transient activity events — real-time UI updates
  console.log('Activity:', e.detail.type, e.detail.data)
})

// Execute
const result = await agent.execute('Find and click the Submit button')

React Hook Example for Custom UI

import { useEffect, useState } from 'react'
import { PageAgentCore } from '@page-agent/core'
import { PageController } from '@page-agent/page-controller'

function useAgent(config) {
  const [agent] = useState(() => {
    const pc = new PageController({ enableMask: false, viewportExpansion: -1 })
    return new PageAgentCore(pc, config)
  })
  const [status, setStatus] = useState('idle')
  const [history, setHistory] = useState([])

  useEffect(() => {
    const onStatus = (e) => setStatus(e.detail.status)
    const onHistory = (e) => setHistory([...e.detail.history])
    
    agent.addEventListener('statuschange', onStatus)
    agent.addEventListener('historychange', onHistory)
    
    return () => {
      agent.removeEventListener('statuschange', onStatus)
      agent.removeEventListener('historychange', onHistory)
    }
  }, [agent])

  return { agent, status, history }
}

Lifecycle Hooks

const agent = new PageAgent({
  onBeforeStep: async ({ step, browserState }) => {
    console.log(`Step ${step}: Current URL = ${browserState.url}`)
    // Return false to skip this step
  },
  onAfterStep: async ({ step, action, result }) => {
    console.log(`Step ${step}: Action = ${action.name}, Result = ${result}`)
    // Useful for logging, analytics, or custom state management
  },
})

Integrating PageAgent as a Tool in an Existing Agent

// Use PageAgent as a tool within your own AI agent/chatbot
const pageAgentTool = {
  name: 'page_agent',
  description: 'Execute web page operations via natural language instructions',
  execute: async (params) => {
    const result = await pageAgent.execute(params.instruction)
    return { success: result.success, message: result.data }
  },
}

// Register this tool in your external agent framework (LangChain, AutoGen, etc.)

Interaction Capabilities

✅ Supported Actions

• Click elements (buttons, links, checkboxes, radio buttons)
• Text input (type into fields, textareas)
• Select from dropdowns
• Scroll (vertical and horizontal)
• Form submission
• Focus elements
• Same-origin iframe (single level only)
• Execute JavaScript (opt-in — can be disabled for security)

❌ NOT Supported

• Hover, drag & drop, right-click
• Keyboard shortcuts (Ctrl+C, etc.)
• Position-based control (x,y coordinates)
• Nested iframes, cross-origin iframes
• Canvas drawing / WebGL
• Monaco, CodeMirror, and other code editors requiring JS instance access
• Image/screenshot recognition (text-based only)

Security Permissions (Beta)

Interaction Allowlists/Blocklists

Prevent AI from interacting with dangerous elements:

const agent = new PageAgent({
  interactiveBlacklist: [
    document.getElementById('delete-account-btn'),
    () => document.querySelector('.payment-submit'),
  ],
  instructions: {
    system: `FORBIDDEN ACTIONS:
    - Never click buttons labeled "Delete" or "Remove"
    - Never modify payment information
    - Always ask for confirmation before submitting forms
    REQUIRED: Ask user for confirmation before any destructive action.`
  },
})

Deployment Decision Guide

Method	When to Use	Pros	Cons
NPM Package	Production SPA integration	Full control, tree-shakeable, secure with proxy	Requires build step
CDN Script	Quick demos, prototyping	Zero setup, instant	Exposed credentials, rate-limited demo API
Chrome Extension	Multi-tab workflows, any site	Works on any website, tab management	Requires user install

Troubleshooting Quick Reference

Problem	Likely Cause	Solution
Model returns malformed responses	Model doesn't support tool calls	Switch to a tested model (see supported list)
Agent can't find elements	Poor HTML semantics	Add aria-label, improve semantic HTML, check includeAttributes
Agent clicks wrong element	Elements not properly indexed	Set viewportExpansion: -1 for full page indexing
API key exposed in network tab	Client-side API call	Implement backend proxy pattern
Low task success rate	Complex instructions	Decompose into simpler steps, add instructions context
customFetch needed	Non-OpenAI-spec provider	Wrap fetch to adapt request/response format

Code Generation Guidelines

When generating PageAgent integration code:

1. Always secure API keys — Use backend proxy for production
2. Import Zod from zod/v4 — Never zod directly
3. Write descriptive tool descriptions — Start with action verbs
4. Add .describe() to Zod fields — Helps the LLM understand parameters
5. Return informative strings from tools — Use emoji prefixes (✅, ❌, ⏳)
6. Set viewportExpansion: -1 for complex pages — Ensures all elements are indexed
7. Add instructions — Provide context about the website's purpose and layout
8. Disable dangerous tools — Set execute_javascript: null for untrusted environments
9. Use transformPageContent — Mask sensitive data before LLM receives it
10. Handle errors gracefully — Set maxRetries and use lifecycle hooks for monitoring

Additional Resources

• For complete API reference, see REFERENCE.md
• For implementation patterns and recipes, see PATTERNS.md
• For debugging and troubleshooting, see TROUBLESHOOTING.md