testdriver-find

name	testdriver:find
description	Locate UI elements using natural language

Overview

Locate UI elements on screen using AI-powered natural language descriptions. Returns an Element object that can be interacted with.

Syntax

const element = await testdriver.find(description)
const element = await testdriver.find(description, options)

Parameters

Natural language description of the element to find Optional configuration for finding and caching Custom cache key for storing element location. Use this to prevent cache pollution when using dynamic variables in prompts, or to share cache across tests.

<ParamField path="cacheThreshold" type="number" default={0.05}>
  Similarity threshold (0-1) for cache matching. Lower values require more similarity. Set to -1 to disable cache.
</ParamField>

<ParamField path="timeout" type="number" default={10000}>
  Maximum time in milliseconds to poll for the element. Retries every 5 seconds until found or timeout expires. Defaults to `10000` (10 seconds). Set to `0` to disable polling and make a single attempt.
</ParamField>

<ParamField path="confidence" type="number">
  Minimum confidence threshold (0-1). If the AI's confidence score for the found element is below this value, the find will be treated as a failure (`element.found()` returns `false`). Useful for ensuring high-quality matches in critical test steps.
</ParamField>

<ParamField path="type" type="string">
  Element type hint that wraps the description for better matching. Accepted values:
  - `"text"` — Wraps the prompt as `The text "..."`
  - `"image"` — Wraps the prompt as `The image "..."`
  - `"ui"` — Wraps the prompt as `The UI element "..."`
  - `"any"` — No wrapping, uses the description as-is (default behavior)
</ParamField>

<ParamField path="zoom" type="boolean" default={true}>
  Two-phase zoom mode for better precision in crowded UIs with many similar elements. Enabled by default.
</ParamField>

<ParamField path="ai" type="object">
  AI sampling configuration for this find call (overrides global `ai` config from constructor).
  
  <Expandable title="properties">
    <ParamField path="temperature" type="number">
      Controls randomness. `0` = deterministic. Default: `0` for find verification.
    </ParamField>
    
    <ParamField path="top" type="object">
      Sampling parameters
      
      <Expandable title="properties">
        <ParamField path="p" type="number">
          Top-P (nucleus sampling). Range: 0-1.
        </ParamField>
        
        <ParamField path="k" type="number">
          Top-K sampling. `1` = most deterministic.
        </ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

Returns

Promise<Element> - Element instance that has been automatically located

Examples

Basic Element Finding

// Find by role
const button = await testdriver.find('submit button');
const input = await testdriver.find('email input field');

// Find by text content
const link = await testdriver.find('Contact Us link');
const heading = await testdriver.find('Welcome heading');

// Find by visual appearance
const icon = await testdriver.find('red warning icon');
const image = await testdriver.find('company logo image');

Finding with Context

// Provide location context
const field = await testdriver.find('username input in the login form');
const button = await testdriver.find('delete button in the top right corner');

// Describe nearby elements
const input = await testdriver.find('input field below the email label');
const checkbox = await testdriver.find('checkbox next to "Remember me"');

// Describe visual position
const menu = await testdriver.find('hamburger menu icon in the top left');

Interacting with Found Elements

// Find and click
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

// Find and verify
const message = await testdriver.find('success message');
if (message.found()) {
  console.log('Success message appeared');
}

// Find and extract info
const price = await testdriver.find('product price');
console.log('Price location:', price.coordinates);
console.log('Price text:', price.text);

Element Object

The returned Element object provides:

Methods

• found() - Check if element was located
• click(action) - Click the element
• hover() - Hover over the element
• doubleClick() - Double-click the element
• rightClick() - Right-click the element
• find(newDescription) - Re-locate with optional new description

Properties

• coordinates - Element position {x, y, centerX, centerY}
• x, y - Top-left coordinates
• centerX, centerY - Center coordinates
• text - Text content (if available)
• screenshot - Base64 screenshot (if available)
• confidence - AI confidence score
• width, height - Element dimensions
• boundingBox - Complete bounding box

See Elements Reference for complete details.

JSON Serialization

Elements can be safely serialized using JSON.stringify() for logging and debugging. Circular references are automatically removed:

const element = await testdriver.find('login button');

// Safe to stringify - no circular reference errors
console.log(JSON.stringify(element, null, 2));

// Output includes useful debugging info:
// {
//   "description": "login button",
//   "coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
//   "found": true,
//   "threshold": 0.01,
//   "x": 100,
//   "y": 200,
//   "cache": {
//     "hit": true,
//     "strategy": "pixel-diff",
//     "createdAt": "2025-12-09T10:30:00Z",
//     "diffPercent": 0.0023,
//     "imageUrl": "https://..."
//   },
//   "similarity": 0.98,
//   "confidence": 0.95,
//   "selector": "button#login",
//   "aiResponse": "Found the blue login button..."
// }

This is useful for:

• Debugging element detection issues
• Logging test execution details
• Sharing element information across processes
• Analyzing cache performance

Best Practices

**Be specific in descriptions**

More specific descriptions improve accuracy:

// ✅ Good
await testdriver.find('blue submit button below the email field');

// ❌ Too vague
await testdriver.find('button');

**Always check if found**

Verify elements were located before interacting:

const element = await testdriver.find('login button');
if (!element.found()) {
  throw new Error('Login button not found');
}
await element.click();

**Include visual or positional context**

// Include color
await testdriver.find('red error icon');

// Include position
await testdriver.find('search button in the top navigation bar');

// Include nearby text
await testdriver.find('checkbox next to "I agree to terms"');

Confidence Threshold

Require a minimum AI confidence score for element matches. If the confidence is below the threshold, find() treats the result as not found:

// Require at least 90% confidence
const element = await testdriver.find('submit button', { confidence: 0.9 });

if (!element.found()) {
  // AI found something but wasn't confident enough
  throw new Error('Could not confidently locate submit button');
}

await element.click();

This is useful for:

• Critical test steps where an incorrect click could cause cascading failures
• Distinguishing between similar elements (e.g., multiple buttons)
• Failing fast when the UI has changed unexpectedly

// Combine with timeout for robust polling with confidence gate
const element = await testdriver.find('success notification', {
  confidence: 0.85,
  timeout: 15000,
});

The `confidence` value is a float between 0 and 1 (e.g., `0.9` = 90%). The AI returns its confidence with each find result, which you can also read from `element.confidence` after a successful find. ## Element Type

Use the type option to hint what kind of element you're looking for. This wraps your description into a more specific prompt for the AI, improving match accuracy — especially when users provide short or ambiguous descriptions.

// Find text on the page
const label = await testdriver.find('Sign In', { type: 'text' });
// AI prompt becomes: The text "Sign In"

// Find an image
const logo = await testdriver.find('company logo', { type: 'image' });
// AI prompt becomes: The image "company logo"

// Find a UI element (button, input, checkbox, etc.)
const btn = await testdriver.find('Submit', { type: 'ui' });
// AI prompt becomes: The UI element "Submit"

// No wrapping — same as omitting the option
const el = await testdriver.find('the blue submit button', { type: 'any' });

Type	Prompt sent to AI
"text"	The text "..."
"image"	The image "..."
"ui"	The UI element "..."
"any"	Original description (no wrapping)

This is particularly useful for short descriptions like `"Submit"` or `"Login"` where the AI may not know whether to look for a button, a link, or visible text. Specifying `type` removes the ambiguity. ## Polling for Dynamic Elements

By default, find() polls for up to 10 seconds (retrying every 5 seconds) until the element is found. You can customize this with the timeout option:

// Uses default 10s timeout - polls every 5 seconds
const element = await testdriver.find('login button');
await element.click();

// Custom timeout - wait up to 30 seconds
const element = await testdriver.find('login button', { timeout: 30000 });
await element.click();

// Disable polling - single attempt only
const element = await testdriver.find('login button', { timeout: 0 });

The timeout option:

• Defaults to 10000 (10 seconds)
• Retries finding the element every 5 seconds
• Stops when the element is found or the timeout expires
• Logs progress during polling
• Returns the element (check element.found() if not throwing on failure)
• Set to 0 to disable polling and make a single attempt

Zoom Mode

Zoom mode is enabled by default. It uses a two-phase approach for better precision when locating elements, especially in crowded UIs with many similar elements.

To disable zoom for a specific find call, pass zoom: false:

// Zoom is on by default — no option needed
const extensionsBtn = await testdriver.find('extensions puzzle icon in Chrome toolbar');
await extensionsBtn.click();

// Disable zoom for a specific call if needed
const largeButton = await testdriver.find('big hero button', { zoom: false });

How Zoom Mode Works

1. Phase 1: AI identifies the approximate location of the element
2. Phase 2: A 30% crop of the screen is created around that location
3. Phase 3: AI performs precise location on the zoomed/cropped image
4. Result: Coordinates are converted back to absolute screen position

This two-phase approach gives the AI a higher-resolution view of the target area, improving accuracy when multiple similar elements are close together.

You may want to disable zoom with `zoom: false` when: - Targeting large, isolated elements where the extra precision isn't needed - You want to speed up find calls in simple UIs

Cache Options

Control caching behavior to optimize performance, especially when using dynamic variables in prompts.

Custom Cache Key

Use cacheKey to prevent cache pollution when prompts contain variables:

// ❌ Without cacheKey - creates new cache entry for each email value
const email = '[e-mail]';
await testdriver.find(`input for ${email}`); // Cache miss every time

// ✅ With cacheKey - reuses cache regardless of variable
const email = '[e-mail]';
await testdriver.find(`input for ${email}`, {
  cacheKey: 'email-input'
});

// Also useful for dynamic IDs, names, or other changing data
const orderId = generateOrderId();
await testdriver.find(`order ${orderId} status`, {
  cacheKey: 'order-status'  // Same cache for all orders
});

Cache Threshold

Control how similar a cached result must be to reuse it:

// Default: 95% similarity required
await testdriver.find('submit button');

// Strict threshold - 99% similarity required
await testdriver.find('submit button', {
  cacheThreshold: 0.01
});

// Disable cache entirely for this call
await testdriver.find('submit button', {
  cacheThreshold: -1
});

// Combine cacheKey with threshold
await testdriver.find('submit button', {
  cacheKey: 'submit-btn',
  cacheThreshold: 0.01
});

By default, TestDriver auto-generates a cache key from the SHA-256 hash of your test file. When you modify your test file, the hash changes automatically, invalidating stale cache entries.

Manual Polling (Alternative)

If you need custom polling logic:

async function waitForElement(testdriver, description, timeout = 30000) {
  const startTime = Date.now();
  
  while (Date.now() - startTime < timeout) {
    const element = await testdriver.find(description);
    if (element.found()) return element;
    await new Promise(r => setTimeout(r, 1000));
  }
  
  throw new Error(`Element "${description}" not found after ${timeout}ms`);
}

// Usage
const button = await waitForElement(testdriver, 'submit button', 10000);
await button.click();

Use Cases

```javascript const emailField = await testdriver.find('email input field'); await emailField.click(); await testdriver.type('[e-mail]');

const passwordField = await testdriver.find('password input');
await passwordField.click();
await testdriver.type('MyP@ssw0rd');
```

```javascript const submitBtn = await testdriver.find('submit button'); await submitBtn.click();

const cancelLink = await testdriver.find('cancel link');
await cancelLink.click();

const menuIcon = await testdriver.find('hamburger menu icon');
await menuIcon.click();
```

```javascript // Wait for loading to complete let content; for (let i = 0; i < 30; i++) { content = await testdriver.find('results table'); if (content.found()) break; await new Promise(r => setTimeout(r, 1000)); }

// Interact with loaded content
const firstRow = await testdriver.find('first row in the results table');
await firstRow.click();
```

```javascript // Modals and dialogs const modal = await testdriver.find('confirmation dialog'); if (modal.found()) { const confirmBtn = await testdriver.find('confirm button in the dialog'); await confirmBtn.click(); }

// Dropdown menus
const dropdown = await testdriver.find('country dropdown');
await dropdown.click();

const option = await testdriver.find('United States option');
await option.click();
```

Complete Example

import { beforeAll, afterAll, describe, it, expect } from 'vitest';
import TestDriver from 'testdriverai';

describe('Element Finding', () => {
  let testdriver;

  beforeAll(async () => {
    client = new TestDriver(process.env.TD_API_KEY);
    await testdriver.auth();
    await testdriver.connect();
  });

  afterAll(async () => {
    await testdriver.disconnect();
  });

  it('should find and interact with elements', async () => {
    await testdriver.focusApplication('Google Chrome');
    
    // Find login form elements
    const usernameField = await testdriver.find('username input field');
    expect(usernameField.found()).toBe(true);
    
    await usernameField.click();
    await testdriver.type('testuser');
    
    // Find with context
    const passwordField = await testdriver.find('password input below username');
    await passwordField.click();
    await testdriver.type('password123');
    
    // Find button
    const submitBtn = await testdriver.find('green submit button');
    expect(submitBtn.found()).toBe(true);
    
    console.log('Button location:', submitBtn.centerX, submitBtn.centerY);
    
    await submitBtn.click();
    
    // Wait for success message
    let successMsg;
    for (let i = 0; i < 10; i++) {
      successMsg = await testdriver.find('success notification');
      if (successMsg.found()) break;
      await new Promise(r => setTimeout(r, 1000));
    }
    
    expect(successMsg.found()).toBe(true);
  });
});

Related Methods

• click() - Click on found elements
• hover() - Hover over elements
• assert() - Verify element states
• Elements Reference - Complete Element API

---

findAll()

Locate all elements matching a description, rather than just one.

Syntax

const elements = await testdriver.findAll(description, options)

Parameters

Natural language description of elements to find Optional cache options (same as `find()`) Cache key for storing element location

<ParamField path="cacheThreshold" type="number" default={-1}>
  Similarity threshold (0-1) for cache matching. Set to -1 to disable cache.
</ParamField>

Returns

Promise<Element[]> - Array of Element instances

Examples

Basic Usage

// Find all matching elements
const buttons = await testdriver.findAll('button');
console.log(`Found ${buttons.length} buttons`);

// Interact with specific element
if (buttons.length > 0) {
  await buttons[0].click(); // Click first button
}

// Iterate over all
for (const button of buttons) {
  console.log(`Button at (${button.x}, ${button.y})`);
}

Finding Multiple Items

// Find all list items
const items = await testdriver.findAll('list item');

// Find specific item by index
const thirdItem = items[2];
await thirdItem.click();

// Check all items
for (let i = 0; i < items.length; i++) {
  console.log(`Item ${i + 1}: ${items[i].text || 'No text'}`);
}

With Caching

// Cache element locations for faster subsequent runs
const menuItems = await testdriver.findAll('menu item', {
  cacheKey: 'main-menu-items'
});

// First run: ~2-3 seconds (AI call)
// Subsequent runs: ~100ms (cache hit)

Empty Results

// Returns empty array if nothing found (doesn't throw error)
const errors = await testdriver.findAll('error message');

if (errors.length === 0) {
  console.log('No errors found - test passed!');
} else {
  console.log(`Found ${errors.length} errors`);
}

Differences from find()

Feature	find()	findAll()
Return type	Single Element	Array of Element[]
If nothing found	Throws ElementNotFoundError	Returns empty array []
Chainable	✅ Yes: await find('button').click()	❌ No (returns array)
Use case	One specific element	Multiple similar elements
Cache support	✅ Yes	✅ Yes

Use Cases

```javascript // Find all rows in a table const rows = await testdriver.findAll('table row');

// Click every row
for (const row of rows) {
  await row.click();
  await new Promise(r => setTimeout(r, 500)); // Wait between clicks
}

// Or click specific row
await rows[2].click(); // Click third row
```

```javascript // Find all checkboxes const checkboxes = await testdriver.findAll('checkbox');

// Check all boxes
for (const checkbox of checkboxes) {
  await checkbox.click();
}

// Or select first unchecked
const unchecked = checkboxes[0];
await unchecked.click();
```

```javascript // Find all navigation links const navLinks = await testdriver.findAll('navigation link');

// Validate all are present
expect(navLinks.length).toBeGreaterThan(0);

// Click specific link by text
const homeLink = navLinks.find(link => 
  link.text?.toLowerCase().includes('home')
);

if (homeLink) {
  await homeLink.click();
}
```

```javascript // Check if any error messages exist const errors = await testdriver.findAll('error message');

if (errors.length > 0) {
  console.log(`Found ${errors.length} validation errors`);
  
  // Log each error location
  errors.forEach((error, i) => {
    console.log(`Error ${i + 1} at (${error.x}, ${error.y})`);
  });
} else {
  console.log('Form validation passed!');
}
```

Complete Example

import { test, expect } from 'vitest';
import { chrome } from 'testdriverai/presets';

test('select multiple items from list', async (context) => {
  const { testdriver } = await chrome(context, {
    url: 'https://example.com/products'
  });
  
  // Find all product cards
  const products = await testdriver.findAll('product card');
  
  expect(products.length).toBeGreaterThan(0);
  console.log(`Found ${products.length} products`);
  
  // Click first 3 products
  const productsToSelect = Math.min(3, products.length);
  
  for (let i = 0; i < productsToSelect; i++) {
    await products[i].click();
    console.log(`Selected product ${i + 1}`);
    await new Promise(r => setTimeout(r, 500)); // Brief pause
  }
  
  // Verify selections
  const selectedBadges = await testdriver.findAll('selected badge');
  expect(selectedBadges.length).toBe(productsToSelect);
});

Best Practices

**Handle empty arrays gracefully**

// ✅ Good - check length first
const items = await testdriver.findAll('list item');
if (items.length > 0) {
  await items[0].click();
}

// ❌ Bad - may throw error
const items = await testdriver.findAll('list item');
await items[0].click(); // Error if array is empty!

**Use find() for single elements**

// ✅ Use find() when you need exactly one
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

// ❌ Unnecessary - findAll() returns array
const buttons = await testdriver.findAll('submit button');
await buttons[0].click(); // Extra array handling

**Cache for performance**

// First run - slow (AI call)
const items = await testdriver.findAll('menu item', {
  cacheKey: 'menu-items'
});

// Subsequent runs - fast (cache hit)
// ~10-20x faster than without cache