# Write Agent Health E2e

> Create an E2E lifecycle test for a new health platform issue in test/new-e2e/tests/agent-health/

- Skill: `datadog/write-agent-health-e2e` (Agent Skill)
- Install (CLI): `npx skillmds@latest add datadog/write-agent-health-e2e`
- Raw SKILL.md: https://api.skillmd.com/api/skills/datadog/write-agent-health-e2e/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: datadog (https://skillmd.com/u/datadog)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/datadog/write-agent-health-e2e

---


## Parse the argument

`$ARGUMENTS` must be the name of an existing issue module directory under `comp/healthplatform/issues/`.

**If `$ARGUMENTS` is empty or not provided, stop immediately and ask:**
> Which health platform issue module should I write the E2E test for?
> Available modules: run `ls comp/healthplatform/issues/` to list them.

Once you have the module name, set:
- `MODULE` = `$ARGUMENTS` (e.g. `rofspermissions`)
- `MODULE_PATH` = `comp/healthplatform/issues/$MODULE/`
- `TEST_FILE` = `test/new-e2e/tests/agent-health/{module}_test.go`

Verify `$MODULE_PATH` exists before proceeding:
```bash
ls comp/healthplatform/issues/$MODULE/
```
If the directory does not exist, stop and tell the user which modules are available.

---

## Step 1 — Read the issue module

Read both files before writing anything:

**`$MODULE_PATH/module.go`** — extract:
- `IssueID` constant → used as `const issueID` in the test
- `IssueName` constant → asserted as `issue.IssueName` in the test
- `IssueType` constant → asserted as `issue.IssueType` in the test. It is a fixed const set explicitly by the module (`IssueName` lowercased, spaces replaced by underscores) — not computed by the agent at runtime, so read it directly rather than deriving it yourself.
- Whether `BuiltInPeriodicHealthCheck()` returns non-nil → agent runs the check on a schedule
- Whether `BuiltInStartupHealthCheck()` returns non-nil → check runs once at startup
- Both returning `nil` → issues are pushed externally by another component (like `admissionprobe`)

**`$MODULE_PATH/issue.go`** — extract the exact values returned by `BuildIssue()`:
- `Category`, `Source`, `Location`, `Severity`, `Tags` → assert these in `IssueDetection`
- Whether `Remediation` is non-nil and has `Summary`/`Steps` → assert those too

If a `check.go` file exists in the module, read it to understand how the issue is triggered (what system condition causes it).

---

## Step 2 — Choose the environment

| Scenario | Environment |
|---|---|
| Standard host issue (most cases) | `environments.Host` + `awshost.Provisioner` |
| Issue requires Docker daemon | Custom env struct with `Docker *components.RemoteHostDocker` + Pulumi provisioner |

Do **not** implement `Diagnose()` on the env — all assertions go through fakeintake only.

Check whether another test file in this package already embeds a suitable base agent config (e.g. a short forwarder interval) before adding your own — reuse it if so, following the fixture rules in Step 4 otherwise.

---

## Step 3 — Write `$TEST_FILE`

```go
// Unless explicitly stated otherwise all files in this repository are licensed
// under the Apache License Version 2.0.
// This product includes software developed at Datadog (https://www.datadoghq.com/).
// Copyright 2025-present Datadog, Inc.

package agenthealth

import (
    "testing"
    "time"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"

    "github.com/DataDog/agent-payload/v5/healthplatform"

    "github.com/DataDog/datadog-agent/test/e2e-framework/components/datadog/agentparams"
    "github.com/DataDog/datadog-agent/test/e2e-framework/scenarios/aws/ec2"
    "github.com/DataDog/datadog-agent/test/e2e-framework/testing/e2e"
    "github.com/DataDog/datadog-agent/test/e2e-framework/testing/environments"
    awshost "github.com/DataDog/datadog-agent/test/e2e-framework/testing/provisioners/aws/host"
)

type {module}Suite struct {
    e2e.BaseSuite[environments.Host]
}

func Test{Module}Suite(t *testing.T) {
    t.Parallel()
    e2e.Run(t, &{module}Suite{},
        e2e.WithProvisioner(awshost.Provisioner(
            awshost.WithRunOptions(
                ec2.WithAgentOptions(
                    agentparams.WithAgentConfig({module}AgentConfig),
                    // add agentparams here to pre-configure the issue trigger if needed
                ),
            ),
        )),
    )
}

// Test{Module}IssueLifecycle verifies that <describe trigger condition> is detected
// in fakeintake as NEW, and that <describe fix> causes the issue to stop being
// reported (or be reported as RESOLVED).
//
// Cross-restart persistence is tested separately in TestResilienceSuite.
func (suite *{module}Suite) Test{Module}IssueLifecycle() {
    // only declare host/agent if needed for runtime trigger/fix steps
    fakeIntake := suite.Env().FakeIntake.Client()

    const issueID = "<IssueID from module.go>"

    suite.T().Run("IssueDetection", func(t *testing.T) {
        // if the issue is not pre-configured by the provisioner, trigger it here:
        // suite.Env().RemoteHost.MustExecute("sudo ...")

        var issues []*healthplatform.Issue
        require.EventuallyWithT(t, func(ct *assert.CollectT) {
            payloads, err := fakeIntake.GetAgentHealth()
            assert.NoError(ct, err)
            issues = nil
            for _, p := range payloads {
                for _, iss := range findIssuesByID(t, p, issueID) {
                    if iss.PersistedIssue != nil && iss.PersistedIssue.State == healthplatform.IssueState_ISSUE_STATE_ACTIVE {
                        issues = append(issues, iss)
                    }
                }
            }
            assert.NotEmpty(ct, issues, "issue not found as ACTIVE in fakeintake")
        }, defaultIssueTimeout, defaultIssuePollInterval, "issue not detected as ACTIVE in fakeintake")

        require.NotEmpty(t, issues)
        issue := issues[0]
        // assert values read from issue.go:
        assert.Equal(t, "<IssueName>", issue.IssueName)
        assert.Equal(t, "<issue_type>", issue.IssueType) // IssueName lowercased, spaces -> underscores
        assert.Equal(t, "<Category>", issue.Category)
        assert.Equal(t, "<Source>", issue.Source)
        // assert.Equal(t, "<Location>", issue.Location)  // if Location is set
        assert.Contains(t, issue.Tags, "<tag>")
        require.NotNil(t, issue.Remediation)
        assert.NotEmpty(t, issue.Remediation.Summary)
    })

    suite.T().Run("Resolution", func(t *testing.T) {
        // Option A — fix via config change (preferred): UpdateEnv handles restart + wait
        suite.UpdateEnv(awshost.Provisioner(
            awshost.WithRunOptions(
                ec2.WithAgentOptions(
                    agentparams.WithAgentConfig({module}AgentConfig),
                    // agentparams that represent the fixed state
                ),
            ),
        ))
        require.NoError(t, fakeIntake.FlushServerAndResetAggregators())

        // Option B — fix via runtime action (e.g. chmod), then manual restart:
        // suite.Env().RemoteHost.MustExecute("sudo ...")
        // require.NoError(t, suite.Env().Agent.Client.Restart())
        // require.EventuallyWithT(t, func(ct *assert.CollectT) {
        //     assert.True(ct, suite.Env().Agent.Client.IsReady())
        // }, 2*time.Minute, 10*time.Second, "agent not ready after fix")
        // require.NoError(t, fakeIntake.FlushServerAndResetAggregators())

        require.Never(t, func() bool {
            payloads, _ := fakeIntake.GetAgentHealth()
            for _, p := range payloads {
                for _, iss := range findIssuesByID(t, p, issueID) {
                    if iss.PersistedIssue == nil || iss.PersistedIssue.State != healthplatform.IssueState_ISSUE_STATE_RESOLVED {
                        return true
                    }
                }
            }
            return false
        }, defaultIssueAbsenceWindow, defaultIssuePollInterval,
            "issue still reported as non-resolved after fix")
    })
}
```

**If the issue ID includes a runtime-generated suffix** (e.g. an issue ID with a hash appended per instance):
add a `findIssuesByPrefix` helper to `helpers_test.go` (mirroring `findIssuesByID` but using `strings.HasPrefix`) and use it in place of `findIssuesByID(t, p, issueID)`.

---

## Step 4 — Fixture rules

| Content | How to include |
|---|---|
| Short YAML (≤ 10 lines) | `const` string literal inline in the test file |
| Python file | `//go:embed fixtures/{module}.py` → `var {module}Py string` |
| Long YAML | `//go:embed fixtures/{module}_config.yaml` → `var {module}Config string` |

Before adding a new agent config fixture, check whether an existing one in the package already fits — reuse it instead of declaring a near-duplicate.

---

## Step 5 — Rules (never break these)

- **Flush order**: restart or `UpdateEnv` first → wait for agent ready → `FlushServerAndResetAggregators()`. Never flush before restarting.
- **No resilience phase**: `RestartResilience` is covered once in `TestResilienceSuite`. Do not add it here.
- **No diagnose assertions**: assert only through fakeintake state (`ISSUE_STATE_ACTIVE`, `ISSUE_STATE_RESOLVED`).
- **No shared lifecycle driver**: phases (`IssueDetection`, `Resolution`) are always inline in the test method.
- **No agent-ready wait at suite start**: the framework guarantees the agent is ready before the first test method runs.
- **No `Diagnose()` on the env struct**.
- **Always call `t.Parallel()`**: the first line of every `Test{Module}Suite` function must be `t.Parallel()` so the suite can run concurrently with others.

---

## Step 6 — Verify and report

```bash
dda inv linter.go --targets=test/new-e2e/tests/agent-health
dda inv new-e2e-tests.run --targets=./tests/agent-health/... --run=^Test{Module}Suite$
```

Tell the user:
- The file that was created
- How to run the test locally: `dda inv new-e2e-tests.run --targets=./tests/agent-health/... --run=^Test{Module}Suite$`
- Any fixtures that need to be created in `fixtures/`