IOS Testing Patterns

iOS Testing Patterns Skill

Comprehensive guide to XCTest and XCUITest execution, analysis, and flaky test detection

Overview

This skill provides patterns and workflows for executing iOS tests using XCTest and XCUITest frameworks. It covers test execution strategies, result analysis, flaky test detection, and troubleshooting common test failures.

Use this skill when:

• Running unit tests or UI tests
• Analyzing test failures
• Detecting and debugging flaky tests
• Optimizing test execution performance
• Setting up CI/CD test pipelines

Quick Reference

When to Use This Skill

Use ios-testing-patterns when:

• Executing XCTest unit tests
• Running XCUITest UI automation tests
• Investigating test failures
• Detecting intermittent test failures (flaky tests)
• Analyzing test performance and execution time
• Setting up test schemes and test plans
• Configuring parallel test execution
• Debugging test-specific simulator issues

Related Skills:

• xcode-workflows: For general build and test operations
• simulator-workflows: For managing test simulators
• ui-automation-workflows: For UI test interaction patterns

Key Concepts

XCTest vs XCUITest

XCTest (Unit/Integration Tests):

• Fast execution (milliseconds to seconds)
• Tests individual components in isolation
• Direct access to app internals
• No UI interaction required
• Runs in same process as app code

XCUITest (UI Tests):

• Slower execution (seconds to minutes)
• Tests user-facing workflows
• Black-box testing (no app internals)
• Requires simulator/device UI
• Runs in separate process

Test Execution Strategies

Sequential Execution:

• Tests run one after another
• Predictable, repeatable results
• Slower overall execution
• Better for debugging

Parallel Execution:

• Tests run simultaneously on multiple simulators
• Faster overall execution
• Requires more system resources
• May expose race conditions

Test Sharding:

• Split test suite across multiple destinations
• Optimal for CI/CD pipelines
• Reduces total execution time
• Requires careful test isolation

Workflows

1. Running Unit Tests

Basic Unit Test Execution:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15,OS=18.0"
}

Note: The destination parameter now supports auto-resolution! You can omit the OS version (e.g., "platform=iOS Simulator,name=iPhone 15") and the tool will automatically select the latest available OS version for that device.

Run Specific Test Class:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15,OS=18.0",
  "options": {
    "only_testing": ["MyAppTests/NetworkTests"]
  }
}

Run Specific Test Method:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15,OS=18.0",
  "options": {
    "only_testing": ["MyAppTests/NetworkTests/testAPIRequest"]
  }
}

2. Running UI Tests

Basic UI Test Execution:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPhone 15"
}

UI Tests with Specific Device:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPad Pro (12.9-inch)"
}

UI Tests with Test Plan:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "test_plan": "SmokeTests"
  }
}

3. Parallel Test Execution

Multiple Destinations:

Run tests on multiple simulators simultaneously:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": [
    "platform=iOS Simulator,name=iPhone 15",
    "platform=iOS Simulator,name=iPhone SE (3rd generation)"
  ],
  "options": {
    "parallel": true
  }
}

Parallel Test Benefits:

• Reduces total execution time
• Tests across multiple device types
• Exposes device-specific issues
• Better CI/CD performance

Parallel Test Considerations:

• Requires multiple simulators
• Higher CPU/memory usage
• May expose race conditions
• Test isolation critical

4. Test Filtering

Skip Specific Tests:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "skip_testing": [
      "MyAppUITests/SlowTests",
      "MyAppTests/NetworkTests/testLargeDownload"
    ]
  }
}

Run Only Fast Tests:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "only_testing": ["MyAppTests/UnitTests"],
    "skip_testing": ["MyAppTests/UnitTests/testSlowOperation"]
  }
}

5. Test Result Analysis

Success Response:

{
  "success": true,
  "tests_run": 45,
  "tests_passed": 45,
  "tests_failed": 0,
  "execution_time": "12.4s",
  "scheme": "MyApp"
}

Failure Response:

{
  "success": false,
  "tests_run": 45,
  "tests_passed": 43,
  "tests_failed": 2,
  "failures": [
    {
      "test": "MyAppTests.LoginTests.testInvalidPassword",
      "message": "XCTAssertEqual failed: (\"error\") is not equal to (\"invalid\")",
      "file": "LoginTests.swift",
      "line": 42
    },
    {
      "test": "MyAppUITests.CheckoutTests.testPaymentFlow",
      "message": "Failed to find button \"Confirm\"",
      "file": "CheckoutTests.swift",
      "line": 78
    }
  ],
  "execution_time": "18.7s"
}

Analyzing Test Results:

1. Check success status
2. Review tests_failed count
3. Examine failures array for details
4. Check execution_time for performance issues
5. Look for patterns in failure messages

6. Flaky Test Detection

What are Flaky Tests?

Tests that intermittently pass or fail without code changes:

• Timing-dependent assertions
• Race conditions
• Async operation issues
• Shared state between tests
• Network-dependent tests
• UI animation timing issues

Detection Strategy 1: Multiple Runs

Run tests multiple times to detect flakiness:

# Run tests 5 times and compare results
for i in {1..5}; do
  execute_xcode_command(test, scheme, destination)
  # Record results
done

Detection Strategy 2: Pattern Analysis

Look for common flaky test patterns:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "only_testing": ["MyAppTests/SuspectedFlakyTest"]
  }
}

Flaky Test Indicators:

1. Timeout failures: "Exceeded timeout of 30 seconds"
2. Element not found: "Failed to find button..." (UI tests)
3. Race conditions: "Expected 5 but got 4"
4. Async timing: "Asynchronous wait failed"
5. Network failures: "Request timed out"

Flaky Test Workflow:

1. Identify suspect test (intermittent failures)
2. Run test 10+ times in isolation
3. Analyze failure patterns
4. Check for:
   - Hard-coded waits (sleep/wait)
   - Expectation timeouts too short
   - Shared mutable state
   - Network dependencies
   - Animation timing assumptions
5. Fix root cause
6. Verify with repeated runs

7. Test Isolation

Clean State Before Tests:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "clean_before_build": true
  }
}

Reset Simulator State:

# Use simulator-workflows skill
execute_simulator_command(operation: "erase", udid: <device-udid>)

Test Isolation Best Practices:

1. Use setUp() and tearDown() methods
2. Clear app state between tests
3. Use unique test data per test
4. Avoid shared static state
5. Reset UserDefaults in setUp
6. Clear keychain data
7. Reset app permissions

Common Patterns

Test Scheme Configuration

Typical Test Scheme Setup:

• Build Configuration: Debug (faster builds)
• Test Plans: Separate unit/UI/smoke tests
• Coverage: Enable code coverage collection
• Diagnostics: Enable thread sanitizer for threading issues

Multiple Test Plans:

UnitTests.xctestplan       → Fast unit tests only
UITests.xctestplan         → UI automation tests
SmokeTests.xctestplan      → Critical path tests
RegressionTests.xctestplan → Full test suite

Test Data Management

Pattern 1: Test Fixtures

class TestData {
    static let validUser = User(name: "Test", email: "test@example.com")
    static let invalidUser = User(name: "", email: "invalid")
}

Pattern 2: Factory Methods

extension User {
    static func makeTestUser(name: String = "Test") -> User {
        return User(name: name, email: "\(name)@test.com")
    }
}

Pattern 3: Test Database

class TestDatabase {
    func setupTestData() {
        // Create known test data
    }

    func teardownTestData() {
        // Clean up after tests
    }
}

Screenshot Capture on Failure

UI Test Screenshot Pattern:

override func setUp() {
    continueAfterFailure = false
}

override func tearDown() {
    if let testRun = testRun, testRun.failureCount > 0 {
        let screenshot = XCUIScreen.main.screenshot()
        let attachment = XCTAttachment(screenshot: screenshot)
        attachment.lifetime = .keepAlways
        add(attachment)
    }
}

Execute Tests with Screenshots:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "result_bundle_path": "./TestResults.xcresult"
  }
}

Performance Testing

XCTest Performance Measurement:

func testPerformance() {
    measure {
        // Code to measure
        performExpensiveOperation()
    }
}

Performance Test Execution:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "only_testing": ["MyAppTests/PerformanceTests"]
  }
}

Troubleshooting

Common Test Failures

1. "Test crashed with signal SIGABRT"

Causes:

• Force unwrapping nil values
• Array index out of bounds
• Precondition/assertion failures
• Fatal errors in app code

Solutions:

1. Check crash logs for stack trace
2. Run test in debugger (Xcode)
3. Add breakpoints at crash location
4. Review force unwraps in test path

2. "Failed to find element/button"

Causes:

• Element not yet rendered
• Animation not complete
• Incorrect accessibility identifier
• Element off-screen

Solutions:

1. Increase timeout for element queries
2. Wait for animations to complete
3. Verify accessibility identifiers
4. Scroll element into view
5. Check element existence before interaction

Example Fix:

// Bad: Immediate query may fail
app.buttons["Submit"].tap()

// Good: Wait for element
let submitButton = app.buttons["Submit"]
XCTAssert(submitButton.waitForExistence(timeout: 5))
submitButton.tap()

3. "Exceeded timeout of X seconds"

Causes:

• Network requests too slow
• Async operations not completing
• Deadlocks or infinite loops
• Expectations never fulfilled

Solutions:

1. Increase expectation timeout
2. Mock network requests
3. Check for deadlocks
4. Verify expectation fulfillment
5. Use proper async testing patterns

Example Fix:

// Bad: May timeout
wait(for: [expectation], timeout: 1.0)

// Good: Reasonable timeout
wait(for: [expectation], timeout: 10.0)

4. "Test failed due to app termination"

Causes:

• Memory pressure
• Watchdog timeout (app hung)
• Background task limits
• System resource constraints

Solutions:

1. Profile memory usage
2. Optimize test efficiency
3. Break up long-running tests
4. Check for memory leaks
5. Reduce simulator load

Timeout Issues

UI Test Timeouts:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "test_timeout": 600
  }
}

Common Timeout Scenarios:

1. App Launch Timeout: App takes too long to launch

   • Solution: Optimize app startup, increase timeout

2. Element Query Timeout: UI element not found

   • Solution: Wait for element existence, check identifiers

3. Network Timeout: API requests fail

   • Solution: Mock network, increase timeout, fix connectivity

4. Animation Timeout: Waiting for animation

   • Solution: Disable animations in tests, use proper waits

Simulator State Problems

Symptom: Tests fail due to simulator state

Common Issues:

1. Previous app installation interfering
2. Simulator storage full
3. Corrupt simulator data
4. Multiple tests modifying shared state

Solutions:

# Erase simulator before tests
execute_simulator_command({
  "operation": "erase",
  "udid": "<simulator-udid>"
})

# Then run tests
execute_xcode_command({
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15"
})

Test Dependency Issues

Symptom: Tests pass individually but fail in suite

Causes:

• Test execution order dependencies
• Shared mutable state
• Singleton pollution
• Database state carryover

Solutions:

1. Ensure Test Isolation:

override func setUp() {
    super.setUp()
    // Reset all shared state
    AppState.shared.reset()
    clearDatabase()
}

2. Avoid Singletons:

// Bad: Shared state
class APIManager {
    static let shared = APIManager()
}

// Good: Dependency injection
class APIManager {
    // Inject per test
}

3. Randomize Test Order:

{
  "operation": "test",
  "scheme": "MyApp",
  "options": {
    "test_order": "random"
  }
}

Best Practices

1. Test Isolation

Always isolate tests:

• Each test should run independently
• No shared state between tests
• Clean up in tearDown()
• Use unique test data

Example:

class UserTests: XCTestCase {
    var sut: UserManager!

    override func setUp() {
        super.setUp()
        sut = UserManager()
    }

    override func tearDown() {
        sut = nil
        super.tearDown()
    }

    func testCreateUser() {
        // Test uses fresh sut instance
    }
}

2. Fast Test Execution

Optimize test speed:

• Mock network requests
• Use in-memory databases
• Disable animations in UI tests
• Avoid actual file I/O
• Use test doubles (mocks/stubs)

UI Test Speed:

// Disable animations for faster tests
app.launchArguments += ["DISABLE_ANIMATIONS"]

3. Readable Test Names

Use descriptive test names:

// Bad
func test1() { }

// Good
func testLoginWithValidCredentialsSucceeds() { }
func testLoginWithInvalidPasswordShowsError() { }
func testCheckoutWithEmptyCartDisplaysWarning() { }

4. Arrange-Act-Assert Pattern

Structure tests clearly:

func testAddItemToCart() {
    // Arrange
    let cart = ShoppingCart()
    let item = Product(name: "Test", price: 10)

    // Act
    cart.add(item)

    // Assert
    XCTAssertEqual(cart.items.count, 1)
    XCTAssertEqual(cart.total, 10)
}

5. Test Data Management

Use consistent test data:

struct TestFixtures {
    static let validEmail = "test@example.com"
    static let invalidEmail = "invalid"
    static let testUser = User(name: "Test", email: validEmail)
}

6. CI/CD Integration

CI Test Workflow:

1. Clean build environment
2. Build app for testing
3. Boot fresh simulator
4. Run test suite
5. Collect code coverage
6. Parse test results
7. Archive test artifacts
8. Report results

CI Test Command:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "clean_before_build": true,
    "code_coverage": true,
    "result_bundle_path": "./TestResults.xcresult"
  }
}

7. Test Pyramid

Maintain proper test distribution:

      /\
     /UI\      ← Few, slow, broad
    /----\
   /Integ\     ← Some, medium, focused
  /------\
 /  Unit  \    ← Many, fast, isolated
/----------\

Recommended ratio:

• 70% Unit tests
• 20% Integration tests
• 10% UI tests

8. Code Coverage

Enable coverage collection:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "code_coverage": true
  }
}

Coverage goals:

• Critical paths: 90%+ coverage
• Business logic: 80%+ coverage
• UI code: 60%+ coverage
• Overall: 75%+ coverage

9. Continuous Test Monitoring

Track test metrics:

• Test execution time trends
• Flaky test frequency
• Test failure rate
• Code coverage trends
• Test suite growth

When to investigate:

• Test execution time increases >20%
• Flaky tests appear
• Coverage drops
• Failure rate increases

Related Skills

• xcode-workflows: General build and test operations
• simulator-workflows: Managing test simulators and devices
• ui-automation-workflows: Advanced UI test interaction patterns
• crash-debugging: Analyzing test crashes

Related Resources

• xc://operations/test: Complete test operations reference
• xc://reference/xctest: XCTest framework documentation
• xc://reference/xcuitest: XCUITest framework documentation
• xc://patterns/flaky-tests: Flaky test detection patterns

Tip: Isolate tests, mock dependencies, and run tests frequently during development.