Testdriver:find

Overview

Find UI elements on the screen with natural language descriptions and AI. This returns an Element object. You can interact with the object.

Syntax

const element = await testdriver.find(description)
const element = await testdriver.find(description, options)

Parameters

<ParamField path="cacheThreshold" type="number" default={0.05}>
  The similarity threshold (0-1) for a cache match. A lower value needs more similarity. Set it to -1 to disable the cache.
</ParamField>

<ParamField path="timeout" type="number" default={10000}>
  The maximum time in milliseconds to poll for the element. TestDriver tries again each 5 seconds until it finds the element or the timeout ends. The default is `10000` (10 seconds). Set it to `0` to disable the poll and try one time.
</ParamField>

<ParamField path="confidence" type="number">
  The minimum confidence threshold (0-1). If the confidence score of the found element is less than this value, the find is a failure (`element.found()` returns `false`). Use this to make sure of good matches in critical test steps.
</ParamField>

<ParamField path="type" type="string">
  Element type hint that wraps the description for better matching. Accepted values:
  - `"text"` — Wraps the prompt as `The text "..."`
  - `"image"` — Wraps the prompt as `The image "..."`
  - `"ui"` — Wraps the prompt as `The UI element "..."`
  - `"any"` — No wrapping, uses the description as-is (default behavior)
</ParamField>

<ParamField path="zoom" type="boolean" default={false}>
  A two-phase zoom mode for more precision in full UIs that have many similar elements. It is disabled by default.
</ParamField>

<ParamField path="verify" type="boolean" default={false}>
  This enables AI verification of the found element. When `true`, a second AI call makes sure that the coordinates agree with the correct element. This catches incorrect positions. It is disabled by default for less latency. When you do not set it for each call, it uses the global `verify` option from the [SDK constructor](/client).
</ParamField>

<ParamField path="ai" type="object">
  AI sampling configuration for this find call (overrides global `ai` config from constructor).
  
  <Expandable title="properties">
    <ParamField path="temperature" type="number">
      Controls randomness. `0` = deterministic. Default: `0` for find verification.
    </ParamField>
    
    <ParamField path="top" type="object">
      Sampling parameters
      
      <Expandable title="properties">
        <ParamField path="p" type="number">
          Top-P (nucleus sampling). Range: 0-1.
        </ParamField>
        
        <ParamField path="k" type="number">
          Top-K sampling. `1` = most deterministic.
        </ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

Returns

Promise<Element> - The Element instance that TestDriver found automatically.

Examples

Basic Element Finding

// Find by role
const button = await testdriver.find('submit button');
const input = await testdriver.find('email input field');

// Find by text content
const link = await testdriver.find('Contact Us link');
const heading = await testdriver.find('Welcome heading');

// Find by visual appearance
const icon = await testdriver.find('red warning icon');
const image = await testdriver.find('company logo image');

Finding with Context

// Provide location context
const field = await testdriver.find('username input in the login form');
const button = await testdriver.find('delete button in the top right corner');

// Describe nearby elements
const input = await testdriver.find('input field below the email label');
const checkbox = await testdriver.find('checkbox next to "Remember me"');

// Describe visual position
const menu = await testdriver.find('hamburger menu icon in the top left');

Interacting with Found Elements

// Find and click
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

// Find and verify
const message = await testdriver.find('success message');
if (message.found()) {
  console.log('Success message appeared');
}

// Find and extract info
const price = await testdriver.find('product price');
console.log('Price location:', price.coordinates);
console.log('Price text:', price.text);

Element Object

The Element object that TestDriver returns gives these:

Methods

• found() - Make a check if TestDriver found the element
• click(action) - Click the element
• hover() - Put the cursor on the element
• doubleClick() - Double-click the element
• rightClick() - Right-click the element
• find(newDescription) - Find the element again with an optional new description

Properties

• coordinates - Element position {x, y, centerX, centerY}
• x, y - Top-left coordinates
• centerX, centerY - Center coordinates
• text - Text content (if available)
• screenshot - Base64 screenshot (if available)
• confidence - AI confidence score
• width, height - Element dimensions
• boundingBox - Complete bounding box

See Elements Reference for complete details.

JSON Serialization

You can serialize elements safely with JSON.stringify() for logs and for debug. TestDriver removes circular references automatically:

const element = await testdriver.find('login button');

// Safe to stringify - no circular reference errors
console.log(JSON.stringify(element, null, 2));

// Output includes useful debugging info:
// {
//   "description": "login button",
//   "coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
//   "found": true,
//   "threshold": 0.01,
//   "x": 100,
//   "y": 200,
//   "cache": {
//     "hit": true,
//     "strategy": "pixel-diff",
//     "createdAt": "2025-12-09T10:30:00Z",
//     "diffPercent": 0.0023,
//     "imageUrl": "https://..."
//   },
//   "similarity": 0.98,
//   "confidence": 0.95,
//   "selector": "button#login",
//   "aiResponse": "Found the blue login button..."
// }

Use this for these:

• To debug problems with element detection
• To log the details of the test
• To share element data between processes
• To examine the cache performance

Best Practices

More specific descriptions make the accuracy better:

// ✅ Good
await testdriver.find('blue submit button below the email field');

// ❌ Too vague
await testdriver.find('button');

Make sure that TestDriver found the elements before you interact with them:

const element = await testdriver.find('login button');
if (!element.found()) {
  throw new Error('Login button not found');
}
await element.click();

// Include color
await testdriver.find('red error icon');

// Include position
await testdriver.find('search button in the top navigation bar');

// Include nearby text
await testdriver.find('checkbox next to "I agree to terms"');

Confidence Threshold

Set a minimum confidence score for element matches. If the confidence is less than the threshold, find() makes the result "not found":

// Require at least 90% confidence
const element = await testdriver.find('submit button', { confidence: 0.9 });

if (!element.found()) {
  // AI found something but wasn't confident enough
  throw new Error('Could not confidently locate submit button');
}

await element.click();

Use this for these:

• Critical test steps. An incorrect click can cause more failures.
• To tell the difference between similar elements (for example, many buttons)
• To fail quickly when the UI changed

// Combine with timeout for robust polling with confidence gate
const element = await testdriver.find('success notification', {
  confidence: 0.85,
  timeout: 15000,
});

Use the type option to show which kind of element you look for. This puts your description into a more specific prompt for the AI. It makes the match accuracy better, primarily when a description is short or not clear.

// Find text on the page
const label = await testdriver.find('Sign In', { type: 'text' });
// AI prompt becomes: The text "Sign In"

// Find an image
const logo = await testdriver.find('company logo', { type: 'image' });
// AI prompt becomes: The image "company logo"

// Find a UI element (button, input, checkbox, etc.)
const btn = await testdriver.find('Submit', { type: 'ui' });
// AI prompt becomes: The UI element "Submit"

// No wrapping — same as omitting the option
const el = await testdriver.find('the blue submit button', { type: 'any' });

By default, find() polls for a maximum of 10 seconds (it tries again each 5 seconds) until it finds the element. You can change this with the timeout option:

// Uses default 10s timeout - polls every 5 seconds
const element = await testdriver.find('login button');
await element.click();

// Custom timeout - wait up to 30 seconds
const element = await testdriver.find('login button', { timeout: 30000 });
await element.click();

// Disable polling - single attempt only
const element = await testdriver.find('login button', { timeout: 0 });

The timeout option:

• Has a default of 10000 (10 seconds)
• Tries to find the element again each 5 seconds
• Stops when it finds the element or the timeout ends
• Logs the progress during the poll
• Returns the element (make a check with element.found() if it does not throw an error on a failure)
• Set it to 0 to disable the poll and try one time

Zoom Mode

Zoom mode is disabled by default. It uses a two-phase method for more precision when it finds elements, primarily in full UIs that have many similar elements.

To enable zoom for a specific find call, give zoom: true:

// Enable zoom for better precision in crowded UIs
const extensionsBtn = await testdriver.find('extensions puzzle icon in Chrome toolbar', { zoom: true });
await extensionsBtn.click();

// Without zoom (default)
const largeButton = await testdriver.find('big hero button');

How Zoom Mode Works

1. Phase 1: The AI finds the approximate location of the element.
2. Phase 2: TestDriver makes a 30% crop of the screen around that location.
3. Phase 3: The AI does the precise location on the cropped image.
4. Result: TestDriver changes the coordinates back to the absolute screen position.

This two-phase method gives the AI a higher-resolution view of the target area. It makes the accuracy better when many similar elements are near together.

Verify Mode

Verify mode is disabled by default. When it is enabled, a second AI call makes sure that the coordinates from find() agree with the correct element. This catches incorrect positions.

// Enable verification for critical interactions
const deleteBtn = await testdriver.find('delete account button', { verify: true });
await deleteBtn.click();

How Verify Mode Works

1. Phase 1: The AI finds the element and returns coordinates.
2. Phase 2: A second AI call looks at the screenshot at those coordinates. It makes sure that the element agrees with the description.
3. Result: If the verification fails, TestDriver tries the find again or marks it "not found".

Combining Zoom and Verify

For the maximum accuracy, enable zoom and verify together. Use this for critical interactions. A click on the wrong element can cause more failures:

// Maximum accuracy: zoom for precision + verify to catch hallucinations
const element = await testdriver.find('small cancel icon next to the subscription', {
  zoom: true,
  verify: true,
});
await element.click();

Cache Options

When a test completes correctly, TestDriver caches the result of each find(). On later runs, TestDriver uses the cached match again. It does not make a new AI call. This finds the same element much more quickly. The cache is in your dashboard. TestDriver shares it between runs. Read the Cache page to see how the match, the thresholds, and the invalidation work.

Control the cache to make the performance better, primarily when you use dynamic variables in prompts.

Custom Cache Key

Use cacheKey to keep the cache clean when prompts have variables:

// ❌ Without cacheKey - creates new cache entry for each email value
const email = 'user@example.com';
await testdriver.find(`input for ${email}`); // Cache miss every time

// ✅ With cacheKey - reuses cache regardless of variable
const email = 'user@example.com';
await testdriver.find(`input for ${email}`, {
  cacheKey: 'email-input'
});

// Also useful for dynamic IDs, names, or other changing data
const orderId = generateOrderId();
await testdriver.find(`order ${orderId} status`, {
  cacheKey: 'order-status'  // Same cache for all orders
});

Cache Threshold

Control how similar a cached result must be before TestDriver uses it again:

// Default: 95% similarity required
await testdriver.find('submit button');

// Strict threshold - 99% similarity required
await testdriver.find('submit button', {
  cacheThreshold: 0.01
});

// Disable cache entirely for this call
await testdriver.find('submit button', {
  cacheThreshold: -1
});

// Combine cacheKey with threshold
await testdriver.find('submit button', {
  cacheKey: 'submit-btn',
  cacheThreshold: 0.01
});

Manual Polling (Alternative)

If you need custom poll logic:

async function waitForElement(testdriver, description, timeout = 30000) {
  const startTime = Date.now();
  
  while (Date.now() - startTime < timeout) {
    const element = await testdriver.find(description);
    if (element.found()) return element;
    await new Promise(r => setTimeout(r, 1000));
  }
  
  throw new Error(`Element "${description}" not found after ${timeout}ms`);
}

// Usage
const button = await waitForElement(testdriver, 'submit button', 10000);
await button.click();

Use Cases

const passwordField = await testdriver.find('password input');
await passwordField.click();
await testdriver.type('MyP@ssw0rd');
```

const cancelLink = await testdriver.find('cancel link');
await cancelLink.click();

const menuIcon = await testdriver.find('hamburger menu icon');
await menuIcon.click();
```

// Interact with loaded content
const firstRow = await testdriver.find('first row in the results table');
await firstRow.click();
```

// Dropdown menus
const dropdown = await testdriver.find('country dropdown');
await dropdown.click();

const option = await testdriver.find('United States option');
await option.click();
```

Complete Example

import { beforeAll, afterAll, describe, it, expect } from 'vitest';
import TestDriver from 'testdriverai';

describe('Element Finding', () => {
  let testdriver;

  beforeAll(async () => {
    client = new TestDriver(process.env.TD_API_KEY);
    await testdriver.auth();
    await testdriver.connect();
  });

  afterAll(async () => {
    await testdriver.disconnect();
  });

  it('should find and interact with elements', async () => {
    await testdriver.focusApplication('Google Chrome');
    
    // Find login form elements
    const usernameField = await testdriver.find('username input field');
    expect(usernameField.found()).toBe(true);
    
    await usernameField.click();
    await testdriver.type('testuser');
    
    // Find with context
    const passwordField = await testdriver.find('password input below username');
    await passwordField.click();
    await testdriver.type('password123');
    
    // Find button
    const submitBtn = await testdriver.find('green submit button');
    expect(submitBtn.found()).toBe(true);
    
    console.log('Button location:', submitBtn.centerX, submitBtn.centerY);
    
    await submitBtn.click();
    
    // Wait for success message
    let successMsg;
    for (let i = 0; i < 10; i++) {
      successMsg = await testdriver.find('success notification');
      if (successMsg.found()) break;
      await new Promise(r => setTimeout(r, 1000));
    }
    
    expect(successMsg.found()).toBe(true);
  });
});

Related Methods

• click() - Click on found elements
• hover() - Hover over elements
• assert() - Verify element states
• Elements Reference - Complete Element API

findAll()

Locate all elements matching a description, rather than just one.

Syntax

const elements = await testdriver.findAll(description, options)

Parameters

<ParamField path="cacheThreshold" type="number" default={-1}>
  Similarity threshold (0-1) for cache matching. Set to -1 to disable cache.
</ParamField>

Returns

Promise<Element[]> - Array of Element instances

Examples

Basic Usage

// Find all matching elements
const buttons = await testdriver.findAll('button');
console.log(`Found ${buttons.length} buttons`);

// Interact with specific element
if (buttons.length > 0) {
  await buttons[0].click(); // Click first button
}

// Iterate over all
for (const button of buttons) {
  console.log(`Button at (${button.x}, ${button.y})`);
}

Finding Multiple Items

// Find all list items
const items = await testdriver.findAll('list item');

// Find specific item by index
const thirdItem = items[2];
await thirdItem.click();

// Check all items
for (let i = 0; i < items.length; i++) {
  console.log(`Item ${i + 1}: ${items[i].text || 'No text'}`);
}

With Caching

// Cache element locations for faster subsequent runs
const menuItems = await testdriver.findAll('menu item', {
  cacheKey: 'main-menu-items'
});

// First run: ~2-3 seconds (AI call)
// Subsequent runs: ~100ms (cache hit)

Empty Results

// Returns empty array if nothing found (doesn't throw error)
const errors = await testdriver.findAll('error message');

if (errors.length === 0) {
  console.log('No errors found - test passed!');
} else {
  console.log(`Found ${errors.length} errors`);
}

Differences from find()

Use Cases

// Click every row
for (const row of rows) {
  await row.click();
  await new Promise(r => setTimeout(r, 500)); // Wait between clicks
}

// Or click specific row
await rows[2].click(); // Click third row
```

// Check all boxes
for (const checkbox of checkboxes) {
  await checkbox.click();
}

// Or select first unchecked
const unchecked = checkboxes[0];
await unchecked.click();
```

// Validate all are present
expect(navLinks.length).toBeGreaterThan(0);

// Click specific link by text
const homeLink = navLinks.find(link => 
  link.text?.toLowerCase().includes('home')
);

if (homeLink) {
  await homeLink.click();
}
```

if (errors.length > 0) {
  console.log(`Found ${errors.length} validation errors`);
  
  // Log each error location
  errors.forEach((error, i) => {
    console.log(`Error ${i + 1} at (${error.x}, ${error.y})`);
  });
} else {
  console.log('Form validation passed!');
}
```

Complete Example

import { test, expect } from 'vitest';
import { chrome } from 'testdriverai/presets';

test('select multiple items from list', async (context) => {
  const { testdriver } = await chrome(context, {
    url: 'https://example.com/products'
  });
  
  // Find all product cards
  const products = await testdriver.findAll('product card');
  
  expect(products.length).toBeGreaterThan(0);
  console.log(`Found ${products.length} products`);
  
  // Click first 3 products
  const productsToSelect = Math.min(3, products.length);
  
  for (let i = 0; i < productsToSelect; i++) {
    await products[i].click();
    console.log(`Selected product ${i + 1}`);
    await new Promise(r => setTimeout(r, 500)); // Brief pause
  }
  
  // Verify selections
  const selectedBadges = await testdriver.findAll('selected badge');
  expect(selectedBadges.length).toBe(productsToSelect);
});

Best Practices

// ✅ Good - check length first
const items = await testdriver.findAll('list item');
if (items.length > 0) {
  await items[0].click();
}

// ❌ Bad - may throw error
const items = await testdriver.findAll('list item');
await items[0].click(); // Error if array is empty!

// ✅ Use find() when you need exactly one
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

// ❌ Unnecessary - findAll() returns array
const buttons = await testdriver.findAll('submit button');
await buttons[0].click(); // Extra array handling

// First run - slow (AI call)
const items = await testdriver.findAll('menu item', {
  cacheKey: 'menu-items'
});

// Subsequent runs - fast (cache hit)
// ~10-20x faster than without cache