Screenshot Analysis

Screenshot Analysis Skill

This skill guides you on how to effectively use the Screenshot Analyzer MCP server to validate visual elements, UI states, game mechanics, and more through AI-powered vision analysis.

When to Use Screenshot Analysis

Screenshot analysis is valuable for:

• UI Validation: Verify colors, text, layout, and component visibility
• Game State Checks: Validate player position, game mechanics, UI elements in games
• Visual Regression Testing: Compare screenshots across versions or browsers
• Accessibility Checks: Verify contrast, text readability, and visual hierarchy
• Cross-Browser Validation: Ensure consistent appearance across browsers
• Dynamic Content Verification: Check content that loads asynchronously
• Error State Detection: Identify error messages, warnings, or unexpected states

Available Tools

analyze_screenshot

Analyzes a screenshot using OpenAI Vision API.

Key Parameters:

• screenshot (required): The full file path to the screenshot image.
• prompt (required): Your custom analysis prompt.
• model: OpenAI model (default: "gpt-4o")
• responseFormat: "text" or "json_object" (default: "text")
• maxTokens: Maximum tokens in response
• temperature: Generation temperature (default: 0)

Prompt Crafting Best Practices

1. Be Specific and Contextual

Bad:

"What do you see?"

Good:

"Check if the login button in the top-right corner is visible and what color it is. Also verify if the text says 'Sign In' or 'Log In'."

2. Include Location and Context

Always specify:

• Where to look (top-left, center, specific region)
• What to look for (button, text, color, layout)
• Expected state (visible, hidden, specific value)

Example:

"In the header navigation bar, verify that the 'Cart' button is visible and displays the number '3' in a red badge. The button should be positioned on the right side."

3. Request Structured Output When Needed

For programmatic use, request JSON format:

"Analyze this UI and return a JSON object with: { 'loginButtonVisible': boolean, 'loginButtonColor': string, 'headerText': string, 'errorMessages': string[] }"

4. Handle Dynamic Content

Account for:

• Loading states ("Check if the loading spinner is visible")
• Animations ("Wait for animations to complete, then check...")
• Dynamic data ("Verify the user's name appears in the profile section")

5. Cost Optimization

• Use gpt-4o-mini for simple checks (colors, visibility, text)
• Use gpt-4o for complex analysis (layout analysis, multiple elements, detailed descriptions)
• Set maxTokens to limit response length when appropriate

6. Multi-Step Validation

Break complex validations into steps:

1. Capture screenshot
2. Analyze for specific element
3. Based on result, capture another screenshot or analyze further

Common Use Cases

UI Element Validation

Prompt Example:

"Verify that the submit button is visible, enabled (not grayed out), and displays the text 'Submit Order'. Check if it's positioned below the form fields."

Color and Styling Checks

Prompt Example:

"Check the color of the primary action button. It should be blue (#0066CC). Also verify the text is white and the button has rounded corners."

Text Content Verification

Prompt Example:

"Read the heading text at the top of the page. It should say 'Welcome to Dashboard'. Also check if there's any error message displayed in red text."

Layout and Positioning

Prompt Example:

"Verify the layout: the sidebar should be on the left (200px wide), the main content area should be in the center, and the header should span the full width at the top."

Game State Validation

Prompt Example:

"Check the game state: verify the player's health bar shows 75%, the score displays '1,250', and there are 3 enemies visible on screen. The pause button should be in the top-right corner."

Visual Regression

Prompt Example:

"Compare this screenshot to the expected design. Check for: 1) Header height matches (should be 60px), 2) Logo is positioned correctly, 3) Navigation items are aligned, 4) No unexpected elements or spacing issues."

Workflow Patterns

Pattern 1: Simple Validation

1. Obtain screenshot
2. Analyze with specific prompt
3. Parse result and validate

Pattern 2: Multi-Element Check

1. Obtain screenshot
2. Analyze with structured JSON prompt requesting multiple elements
3. Parse JSON and validate all elements

Pattern 3: Conditional Validation

1. Obtain initial screenshot
2. Analyze to determine current state
3. Based on state, obtain additional screenshots or perform actions
4. Validate final state

Pattern 4: Comparison Validation

1. Obtain screenshot
2. Obtain screenshot of current state
3. Analyze both (or use comparison logic)
4. Validate differences are expected

Error Handling

• Analysis Failures: Verify API key, check prompt clarity, ensure screenshot is a valid full file path
• Invalid Format: Ensure screenshot is a valid file path

Tips for Effective Analysis

1. Start Broad, Then Narrow: First verify overall page state, then focus on specific elements
2. Use Descriptive Prompts: Include visual cues (colors, positions, sizes) in your prompts
3. Leverage Structured Output: Use JSON format for programmatic validation
4. Combine with Other Tools: Use screenshot analysis alongside DOM inspection or API checks
5. Ensure Base64 Format: Screenshots must be provided as base64-encoded strings
6. Iterate on Prompts: Refine prompts based on analysis results

Example Workflow

// 1. Obtain screenshot (from external source)
// Screenshot should be provided as a valid full file path
const screenshotBase64 = "...";

// 2. Analyze for specific elements
const analysis = await analyze_screenshot({
  screenshot: screenshotBase64,
  prompt:
    "Return JSON with: { 'headerVisible': boolean, 'userName': string, 'notificationCount': number, 'primaryButtonColor': string }",
  responseFormat: "json_object",
  model: "gpt-4o",
});

// 3. Validate results
const result = JSON.parse(analysis.analysis);
if (!result.headerVisible) {
  throw new Error("Header is not visible");
}
if (result.notificationCount !== 3) {
  throw new Error(
    `Expected 3 notifications, found ${result.notificationCount}`,
  );
}

See Also

• examples/prompt-examples.md - Detailed prompt examples for various scenarios
• MCP Server README for technical details and configuration