Load Testing

Load Testing

Comprehensive guide for load testing, stress testing, and performance benchmarking of APIs, web services, and distributed systems.

Load Testing Fundamentals

Types of Performance Tests

Key Metrics

• Throughput: Requests per second (RPS) the system handles
• Latency Percentiles: p50, p90, p95, p99 response times
• Error Rate: Percentage of failed requests (target < 1%)
• Concurrent Users: Number of simultaneous active connections
• Apdex Score: Application Performance Index (0 to 1)

Test Phases

       Load
        ^
        |      ___________
        |     /           \        Steady State
        |    /             \       (measure here)
        |   /               \
        |  /                 \
        | /                   \
        |/                     \
        +-------------------------> Time
        Ramp-Up    Hold    Ramp-Down

k6 (Grafana k6) - Modern Load Testing

k6 is the recommended tool for most load testing scenarios. It uses JavaScript for test scripts and is built for developer productivity.

Installation

# macOS
brew install k6

# Linux (Debian/Ubuntu)
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
  --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D68
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update && sudo apt-get install k6

# Docker
docker run --rm -i grafana/k6 run - <script.js

# Windows
choco install k6

Basic k6 Load Test

// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 50 },   // Ramp up to 50 users over 2 minutes
    { duration: '5m', target: 50 },   // Stay at 50 users for 5 minutes
    { duration: '2m', target: 0 },    // Ramp down to 0 users
  ],
  thresholds: {
    http_req_duration: ['p(95)<500', 'p(99)<1000'],  // 95% under 500ms, 99% under 1s
    http_req_failed: ['rate<0.01'],                   // Error rate under 1%
    http_reqs: ['rate>100'],                          // Throughput above 100 RPS
  },
};

export default function () {
  const res = http.get('https://api.example.com/health');

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
    'body contains expected field': (r) => JSON.parse(r.body).status === 'ok',
  });

  sleep(1); // Think time between requests
}

# Run the test
k6 run load-test.js

# Run with environment variables
k6 run -e BASE_URL=https://staging.example.com load-test.js

# Run with custom VUs and duration (overrides script options)
k6 run --vus 100 --duration 30s load-test.js

k6 Advanced Scenarios

// advanced-scenarios.js
import http from 'k6/http';
import { check, group, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

// Custom metrics
const errorRate = new Rate('errors');
const loginDuration = new Trend('login_duration');

export const options = {
  scenarios: {
    // Scenario 1: Constant arrival rate (open model)
    constant_request_rate: {
      executor: 'constant-arrival-rate',
      rate: 100,              // 100 requests per timeUnit
      timeUnit: '1s',         // per second = 100 RPS
      duration: '5m',
      preAllocatedVUs: 50,
      maxVUs: 200,
    },
    // Scenario 2: Ramping arrival rate
    ramping_request_rate: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      stages: [
        { target: 50, duration: '2m' },
        { target: 200, duration: '3m' },
        { target: 50, duration: '1m' },
      ],
      preAllocatedVUs: 100,
      maxVUs: 500,
      startTime: '6m',       // Start after first scenario
    },
    // Scenario 3: Spike test
    spike: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '10s', target: 500 },  // Instant spike
        { duration: '1m', target: 500 },   // Hold spike
        { duration: '10s', target: 0 },    // Drop back
      ],
      startTime: '12m',
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<500'],
    errors: ['rate<0.05'],
    login_duration: ['p(95)<2000'],
  },
};

export default function () {
  group('API Health Check', function () {
    const res = http.get(`${__ENV.BASE_URL}/api/health`);
    check(res, { 'health ok': (r) => r.status === 200 });
    errorRate.add(res.status !== 200);
  });

  group('User Login Flow', function () {
    const loginStart = Date.now();
    const loginRes = http.post(`${__ENV.BASE_URL}/api/auth/login`, JSON.stringify({
      email: `user${__VU}@example.com`,
      password: 'testpassword123',
    }), {
      headers: { 'Content-Type': 'application/json' },
    });
    loginDuration.add(Date.now() - loginStart);

    check(loginRes, {
      'login successful': (r) => r.status === 200,
      'has auth token': (r) => JSON.parse(r.body).token !== undefined,
    });

    if (loginRes.status === 200) {
      const token = JSON.parse(loginRes.body).token;

      // Authenticated request
      const profileRes = http.get(`${__ENV.BASE_URL}/api/profile`, {
        headers: { Authorization: `Bearer ${token}` },
      });
      check(profileRes, { 'profile loaded': (r) => r.status === 200 });
    }
  });

  sleep(Math.random() * 3 + 1); // Random think time 1-4s
}

k6 File Upload and Multipart Testing

// file-upload-test.js
import http from 'k6/http';
import { check } from 'k6';

const binFile = open('/path/to/file.png', 'b');

export default function () {
  const res = http.post('https://api.example.com/upload', {
    file: http.file(binFile, 'test.png', 'image/png'),
    description: 'Load test upload',
  });

  check(res, {
    'upload successful': (r) => r.status === 201,
    'response time ok': (r) => r.timings.duration < 5000,
  });
}

k6 WebSocket Testing

// websocket-test.js
import ws from 'k6/ws';
import { check } from 'k6';

export const options = {
  vus: 50,
  duration: '5m',
};

export default function () {
  const url = 'wss://api.example.com/ws';
  const params = { headers: { Authorization: 'Bearer token123' } };

  const res = ws.connect(url, params, function (socket) {
    socket.on('open', () => {
      console.log('WebSocket connected');
      socket.send(JSON.stringify({ type: 'subscribe', channel: 'updates' }));
    });

    socket.on('message', (msg) => {
      const data = JSON.parse(msg);
      check(data, {
        'message has type': (d) => d.type !== undefined,
        'message has payload': (d) => d.payload !== undefined,
      });
    });

    socket.on('close', () => console.log('WebSocket disconnected'));
    socket.on('error', (e) => console.error('WebSocket error:', e.error()));

    // Keep connection alive for 30 seconds
    socket.setTimeout(function () {
      socket.close();
    }, 30000);
  });

  check(res, { 'ws status is 101': (r) => r && r.status === 101 });
}

k6 gRPC Testing

// grpc-test.js
import grpc from 'k6/net/grpc';
import { check, sleep } from 'k6';

const client = new grpc.Client();
client.load(['definitions'], 'service.proto');

export const options = {
  vus: 20,
  duration: '3m',
  thresholds: {
    grpc_req_duration: ['p(95)<300'],
  },
};

export default function () {
  client.connect('grpc.example.com:443', { plaintext: false });

  const response = client.invoke('api.v1.UserService/GetUser', {
    user_id: '12345',
  });

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
    'has user data': (r) => r && r.message.name !== '',
  });

  client.close();
  sleep(1);
}

k6 Output and Reporting

# Output to JSON
k6 run --out json=results.json load-test.js

# Output to CSV
k6 run --out csv=results.csv load-test.js

# Output to InfluxDB (for Grafana dashboards)
k6 run --out influxdb=http://localhost:8086/k6 load-test.js

# Output to Prometheus via Remote Write
k6 run --out experimental-prometheus-rw load-test.js

# Multiple outputs simultaneously
k6 run --out json=results.json --out influxdb=http://localhost:8086/k6 load-test.js

Artillery - YAML-Based Load Testing

Artillery is great for teams who prefer YAML configuration and need built-in protocol support for HTTP, WebSocket, Socket.IO, and more.

Installation

npm install -g artillery
# or
npx artillery

Basic Artillery Config

# artillery-config.yml
config:
  target: "https://api.example.com"
  phases:
    - duration: 120        # 2 minutes
      arrivalRate: 10      # 10 new users per second
      name: "Warm up"
    - duration: 300        # 5 minutes
      arrivalRate: 50      # 50 new users per second
      name: "Sustained load"
    - duration: 60         # 1 minute
      arrivalRate: 100     # 100 new users per second
      name: "Peak load"
  defaults:
    headers:
      Content-Type: "application/json"
      Authorization: "Bearer {{ $processEnvironment.API_TOKEN }}"
  ensure:
    thresholds:
      - http.response_time.p95: 500
      - http.response_time.p99: 1000
      - http.codes.500: 0

scenarios:
  - name: "Browse and Purchase"
    weight: 70              # 70% of virtual users run this
    flow:
      - get:
          url: "/api/products"
          capture:
            - json: "$.data[0].id"
              as: "productId"
      - think: 2
      - get:
          url: "/api/products/{{ productId }}"
          expect:
            - statusCode: 200
      - think: 1
      - post:
          url: "/api/cart"
          json:
            productId: "{{ productId }}"
            quantity: 1
          expect:
            - statusCode: 201

  - name: "Search Flow"
    weight: 30              # 30% of virtual users run this
    flow:
      - get:
          url: "/api/search?q=laptop&page=1"
          expect:
            - statusCode: 200
            - hasProperty: "data.results"
      - think: 3
      - get:
          url: "/api/search?q=laptop&page=2"

Run Artillery Tests

# Run test
artillery run artillery-config.yml

# Run with environment override
artillery run --target https://staging.example.com artillery-config.yml

# Generate HTML report
artillery run --output report.json artillery-config.yml
artillery report report.json --output report.html

# Quick test (no config file needed)
artillery quick --count 100 --num 10 https://api.example.com/health

Artillery with Custom JavaScript

# artillery-custom.yml
config:
  target: "https://api.example.com"
  processor: "./custom-functions.js"
  phases:
    - duration: 300
      arrivalRate: 20

scenarios:
  - name: "Authenticated flow"
    flow:
      - function: "generateUser"
      - post:
          url: "/api/auth/login"
          json:
            email: "{{ email }}"
            password: "{{ password }}"
          capture:
            - json: "$.token"
              as: "authToken"
      - get:
          url: "/api/dashboard"
          headers:
            Authorization: "Bearer {{ authToken }}"

// custom-functions.js
module.exports = {
  generateUser: function (context, events, done) {
    const id = Math.floor(Math.random() * 10000);
    context.vars.email = `loadtest_user_${id}@example.com`;
    context.vars.password = 'TestPassword123!';
    return done();
  },
};

JMeter Basics

Apache JMeter is a Java-based tool for complex test plans with a GUI for design and CLI for execution.

CLI Execution (Recommended for CI/CD)

# Run test plan in non-GUI mode
jmeter -n -t test-plan.jmx -l results.jtl -e -o report/

# With properties
jmeter -n -t test-plan.jmx \
  -Jthreads=100 \
  -Jrampup=60 \
  -Jduration=300 \
  -Jhost=api.example.com \
  -l results.jtl

# Generate HTML report from results
jmeter -g results.jtl -o html-report/

JMeter Test Plan Structure

Test Plan
  +-- Thread Group (users=100, ramp-up=60s, loops=forever, duration=300s)
       +-- HTTP Request Defaults (server, port, protocol)
       +-- HTTP Header Manager (Content-Type, Authorization)
       +-- CSV Data Set Config (test data file)
       +-- HTTP Request: GET /api/health
       +-- HTTP Request: POST /api/login
       +-- Response Assertion (status code = 200)
       +-- JSON Extractor (extract token)
       +-- HTTP Request: GET /api/data (with token)
       +-- Summary Report
       +-- Aggregate Report
       +-- View Results Tree (debug only)

Locust (Python-Based)

Locust is ideal for Python teams. Define user behavior in Python code.

Installation and Basic Test

pip install locust

# locustfile.py
from locust import HttpUser, task, between, tag

class APIUser(HttpUser):
    wait_time = between(1, 5)  # Think time between 1-5 seconds
    host = "https://api.example.com"

    def on_start(self):
        """Called once per user on start."""
        response = self.client.post("/api/auth/login", json={
            "email": "testuser@example.com",
            "password": "password123"
        })
        self.token = response.json().get("token", "")

    @tag("read")
    @task(3)  # Weight: 3x more likely than weight-1 tasks
    def get_products(self):
        self.client.get("/api/products", headers={
            "Authorization": f"Bearer {self.token}"
        })

    @tag("read")
    @task(2)
    def get_product_detail(self):
        product_id = 42
        self.client.get(f"/api/products/{product_id}", headers={
            "Authorization": f"Bearer {self.token}"
        })

    @tag("write")
    @task(1)
    def create_order(self):
        self.client.post("/api/orders", json={
            "product_id": 42,
            "quantity": 1
        }, headers={
            "Authorization": f"Bearer {self.token}"
        })

# Run with web UI
locust -f locustfile.py

# Run headless (CI/CD mode)
locust -f locustfile.py --headless -u 100 -r 10 --run-time 5m \
  --host https://api.example.com --csv results

# Run specific tags only
locust -f locustfile.py --tags read --headless -u 50 -r 5 --run-time 3m

# Distributed mode (master)
locust -f locustfile.py --master

# Distributed mode (worker)
locust -f locustfile.py --worker --master-host=192.168.1.10

wrk and wrk2 - HTTP Benchmarking

wrk and wrk2 are lightweight, high-performance HTTP benchmarking tools for maximum throughput testing.

wrk

# Install
brew install wrk  # macOS

# Basic benchmark: 12 threads, 400 connections, 30 seconds
wrk -t12 -c400 -d30s https://api.example.com/health

# With Lua script for custom requests
wrk -t8 -c200 -d60s -s post.lua https://api.example.com/api/data

# With latency distribution
wrk -t4 -c100 -d30s --latency https://api.example.com/health

-- post.lua: Custom POST requests with wrk
wrk.method = "POST"
wrk.headers["Content-Type"] = "application/json"
wrk.body = '{"name": "load-test", "value": 42}'

-- Dynamic request generation
request = function()
  local id = math.random(1, 10000)
  local body = string.format('{"user_id": %d, "action": "test"}', id)
  return wrk.format("POST", "/api/events", nil, body)
end

-- Report custom stats
done = function(summary, latency, requests)
  io.write("------------------------------\n")
  io.write(string.format("Total requests: %d\n", summary.requests))
  io.write(string.format("Total errors:   %d\n", summary.errors.status))
  io.write(string.format("Avg latency:    %.2fms\n", latency.mean / 1000))
  io.write(string.format("Max latency:    %.2fms\n", latency.max / 1000))
end

wrk2 (Constant Throughput)

# wrk2 maintains constant request rate (unlike wrk which pushes max throughput)
# 1000 RPS, 8 threads, 100 connections, 2 minutes
wrk2 -t8 -c100 -d120s -R1000 --latency https://api.example.com/health

ab (Apache Bench) - Quick Tests

ab is pre-installed on most systems and ideal for quick, simple benchmarks.

# 10000 requests, 100 concurrent
ab -n 10000 -c 100 https://api.example.com/health

# POST with JSON body
ab -n 5000 -c 50 -p payload.json -T application/json https://api.example.com/api/data

# With keep-alive and custom header
ab -n 10000 -c 200 -k -H "Authorization: Bearer token123" https://api.example.com/api/users

# Quick smoke test
ab -n 100 -c 10 https://api.example.com/health

Thresholds and SLOs

Defining Performance Budgets

# performance-budget.yml
thresholds:
  response_time:
    p50: 100ms     # Median response time
    p90: 250ms     # 90th percentile
    p95: 500ms     # 95th percentile (primary SLO)
    p99: 1000ms    # 99th percentile
    max: 5000ms    # Absolute maximum

  throughput:
    minimum_rps: 500    # Minimum requests per second
    target_rps: 1000    # Target under normal load

  error_rate:
    threshold: 0.1%     # Max 0.1% error rate
    5xx_threshold: 0%   # Zero server errors

  availability:
    target: 99.95%      # Four nines availability

  apdex:
    target: 0.9         # Satisfied threshold
    t_value: 500ms      # Apdex T value

k6 Threshold Examples

export const options = {
  thresholds: {
    // HTTP request duration thresholds
    http_req_duration: [
      'p(50)<100',          // Median under 100ms
      'p(90)<250',          // p90 under 250ms
      'p(95)<500',          // p95 under 500ms (SLO)
      'p(99)<1000',         // p99 under 1 second
      'max<5000',           // No request over 5 seconds
    ],

    // Error rate thresholds
    http_req_failed: ['rate<0.001'],  // Less than 0.1% errors

    // Throughput thresholds
    http_reqs: ['rate>100'],          // More than 100 RPS

    // Custom metric thresholds
    'http_req_duration{name:login}': ['p(95)<2000'],   // Login endpoint SLO
    'http_req_duration{name:search}': ['p(95)<300'],   // Search endpoint SLO

    // Group duration thresholds
    'group_duration{group:::checkout flow}': ['p(95)<10000'],
  },
};

CI/CD Integration

GitHub Actions

# .github/workflows/load-test.yml
name: Load Test
on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 6 * * 1'   # Weekly Monday 6 AM

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install k6
        run: |
          sudo gpg -k
          sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
            --keyserver hkp://keyserver.ubuntu.com:80 \
            --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D68
          echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" \
            | sudo tee /etc/apt/sources.list.d/k6.list
          sudo apt-get update && sudo apt-get install k6

      - name: Run load tests
        run: k6 run --out json=results.json tests/load/api-load-test.js
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}
          API_TOKEN: ${{ secrets.LOAD_TEST_TOKEN }}

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: k6-results
          path: results.json

      - name: Check thresholds
        run: |
          if k6 run --quiet tests/load/api-load-test.js 2>&1 | grep -q "thresholds on metrics.*crossed"; then
            echo "Performance regression detected!"
            exit 1
          fi

GitLab CI

# .gitlab-ci.yml
load_test:
  stage: test
  image: grafana/k6:latest
  script:
    - k6 run --out json=results.json tests/load/api-load-test.js
  artifacts:
    paths:
      - results.json
    when: always
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

Distributed Load Testing

k6 with Kubernetes Operator

# k6-distributed-test.yml
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: api-load-test
spec:
  parallelism: 4          # 4 worker pods
  script:
    configMap:
      name: k6-test-script
      file: load-test.js
  arguments: --out influxdb=http://influxdb:8086/k6
  runner:
    resources:
      limits:
        cpu: "1"
        memory: "1Gi"
      requests:
        cpu: "500m"
        memory: "512Mi"

Locust Distributed with Docker Compose

# docker-compose.yml
version: '3'
services:
  master:
    image: locustio/locust
    ports:
      - "8089:8089"
    volumes:
      - ./:/mnt/locust
    command: -f /mnt/locust/locustfile.py --master -H https://api.example.com

  worker:
    image: locustio/locust
    volumes:
      - ./:/mnt/locust
    command: -f /mnt/locust/locustfile.py --worker --master-host master
    deploy:
      replicas: 4

Interpreting Results

Reading k6 Output

          /\      |------| /\
     /\  /  \     |  oo  |/  \
    /  \/    \    |      |    \
   /          \   |______|     \

  execution: local
     script: load-test.js
     output: -

  scenarios: (100.00%) 1 scenario, 50 max VUs, 9m30s max duration
           default: Up to 50 looping VUs for 9m0s

     data_received..................: 45 MB  83 kB/s
     data_sent......................: 3.2 MB 5.9 kB/s
     http_req_blocked...............: avg=2.3ms   p(90)=4.5ms   p(95)=6.1ms
     http_req_connecting............: avg=1.8ms   p(90)=3.2ms   p(95)=4.5ms
   ✓ http_req_duration..............: avg=125ms   p(90)=210ms   p(95)=350ms    <-- KEY METRIC
     http_req_failed................: 0.12%  ✓ 15  ✗ 12485                     <-- ERROR RATE
     http_req_receiving.............: avg=0.8ms   p(90)=1.5ms   p(95)=2.1ms
     http_req_sending...............: avg=0.3ms   p(90)=0.5ms   p(95)=0.7ms
     http_req_tls_handshaking.......: avg=5.2ms   p(90)=8.1ms   p(95)=10.3ms
     http_req_waiting...............: avg=124ms   p(90)=208ms   p(95)=348ms
     http_reqs......................: 12500  23.1/s                             <-- THROUGHPUT
     iteration_duration.............: avg=1.13s   p(90)=1.22s   p(95)=1.36s
     iterations.....................: 12500  23.1/s
     vus............................: 1      min=1  max=50
     vus_max........................: 50     min=50 max=50

What to Look For

Common Bottleneck Patterns

CPU Bottleneck

• Symptoms: High CPU, linear latency increase with load, throughput plateau
• Common causes: Inefficient algorithms, excessive serialization/deserialization, regex backtracking
• Diagnosis: Profile with perf, py-spy, or async-profiler

Memory Bottleneck

• Symptoms: Growing memory usage over time, GC pauses, OOM kills during soak tests
• Common causes: Memory leaks, large in-memory caches, unbounded queues
• Diagnosis: Heap dumps, memory profilers, monitor RSS over soak tests

Database Connection Pool Exhaustion

• Symptoms: Latency spikes at specific concurrency, connection timeout errors
• Common causes: Slow queries holding connections, pool too small, missing connection release
• Diagnosis: Monitor active/idle connections, query duration distribution

Network Bottleneck

• Symptoms: High bandwidth utilization, TCP connection errors, retransmissions
• Common causes: Large payloads, missing compression, connection limits
• Diagnosis: netstat, ss, network monitoring, check payload sizes

Thread/Worker Pool Exhaustion

• Symptoms: Request queuing, latency increases while CPU stays low
• Common causes: Blocking I/O in async code, insufficient worker count, thread contention
• Diagnosis: Thread dumps, worker pool metrics, request queue depth

Realistic Test Data Generation

k6 Data-Driven Testing

// Use SharedArray for efficient data loading
import { SharedArray } from 'k6/data';
import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';

const users = new SharedArray('users', function () {
  return papaparse.parse(open('./test-users.csv'), { header: true }).data;
});

export default function () {
  const user = users[Math.floor(Math.random() * users.length)];
  const res = http.post(`${__ENV.BASE_URL}/api/login`, JSON.stringify({
    email: user.email,
    password: user.password,
  }), {
    headers: { 'Content-Type': 'application/json' },
  });
}

Generating Test Data

# Generate CSV of test users
python3 -c "
import csv, random, string
with open('test-users.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerow(['email', 'password', 'name'])
    for i in range(10000):
        name = ''.join(random.choices(string.ascii_lowercase, k=8))
        writer.writerow([f'{name}_{i}@loadtest.example.com', 'TestPass123!', name])
print('Generated 10000 test users')
"

Performance Testing Best Practices

Before Testing

1. Define objectives: What are the SLOs? What load is expected?
2. Baseline metrics: Measure current performance before changes
3. Isolate the environment: Use a dedicated staging environment
4. Prepare test data: Use realistic, diverse datasets
5. Warm up caches: Run a warm-up phase before measuring

During Testing

1. Monitor the system: CPU, memory, disk I/O, network, DB connections
2. Watch for errors: Check application and server logs in real time
3. Correlate metrics: Cross-reference load generator data with server metrics
4. Test incrementally: Start small, increase load gradually
5. Document everything: Record test parameters, environment, and results

After Testing

1. Analyze percentiles: Focus on p95/p99, not averages
2. Compare to baseline: Look for regressions, not just absolute values
3. Identify bottlenecks: Use profiling data to find root causes
4. Share results: Publish dashboards and reports for the team
5. Automate regression detection: Integrate into CI/CD pipeline

Anti-Patterns to Avoid

• Testing from the same machine as the server
• Using only averages instead of percentiles
• Running tests on shared/noisy environments
• Ignoring think time (unrealistic constant hammering)
• Testing with a single endpoint only
• Not monitoring the system under test
• Running tests without a warm-up phase
• Using hardcoded test data instead of realistic distributions

References

• k6 Documentation: https://grafana.com/docs/k6/latest/
• Artillery Documentation: https://www.artillery.io/docs
• Apache JMeter: https://jmeter.apache.org/
• Locust Documentation: https://docs.locust.io/
• wrk GitHub: https://github.com/wg/wrk
• Google SRE Book - Load Testing: https://sre.google/sre-book/
• Performance Testing Guidance: https://learn.microsoft.com/en-us/azure/well-architected/performance/