Build Service Behavioural Test Analysis

Build service behavioural test analysis

Produce analysis.md and a minimal cloud-api interface sketch before writing feature files or Go implementations. Implementation is covered by build-features-and-cloud-api.

When to use

• A new or updated catalogs/<category>/<service>/controls.yaml needs behavioural test coverage.
• You are asked to plan how to test assessment requirements (ARs) on AWS, Azure, and GCP.
• You need to decide which methods belong on the service interface vs logging.Service vs generic.Service.

Outputs

Create only analysis.md. Do not create catalog subdirectories, .feature files, Go code, terraform, privateer YAML, integration_calls.csv, or actions-config in this skill unless the user explicitly asks.

Do not create:

• Placeholder README.md files under <CatalogId>/ or CCC.Core/ (e.g. CCC.VM/README.md)
• Empty catalog subdirectories “for later”
• .gitkeep or other scaffold files
• Updates to modules/features/README.md routing rules (that belongs in the implementation skill)

The analysis document describes the planned feature tree, terraform fixture roles, and config vars; physical artifacts are created in the implementation skill.

Do not create .feature files or implement Go in this skill unless the user explicitly asks to continue to implementation.

Workflow

Copy and track progress:

Analysis progress:
- [ ] Step 1: Ingest catalog + metadata
- [ ] Step 2: Inventory generic/ + plan layout (reuse table + new-only paths)
- [ ] Step 3: Classify every AR (testable / not / inherited / reuse generic)
- [ ] Step 4: Draft per-AR test approach
- [ ] Step 5: Design minimal cloud-api interface(s)
- [ ] Step 6: Cross-cloud implementation notes (AWS / Azure / GCP)
- [ ] Step 7: Plan fixtures, config vars, and integration CSV rows
- [ ] Step 8: Write analysis.md
- [ ] Step 9: Review — interface count, AR coverage, gaps

Step 1: Ingest the control catalog

Read both:

1. catalogs/<path>/controls.yaml — controls, ARs, imported-controls
2. catalogs/<path>/metadata.yaml — catalog id (e.g. CCC.VPC, CCC.SecMgmt), CSP service names, docs URLs

From controls.yaml extract for each AR:

From imported-controls list inherited CCC.Core (or other) ARs that apply to this service but are defined elsewhere.

Reuse modules/features/generic/ first. That folder holds shared @PerService and @PerPort Core scenarios (CN01, CN03, CN04, CN05, CN07, CN10, …) that already use {service-type} or port probes. For a new service, the default plan is to add a service tag (e.g. @virtual-machines) to existing generic scenarios — not to copy feature files into <service-folder>/CCC.Core/. Only plan new feature files when generic steps cannot express the AR (service-specific cloud-api methods, probes that differ from @PerPort patterns). See modules/features/virtual-machines/analysis.md for a reuse table example.

Parse the AR sentence. Most ARs follow:

When <trigger condition>, the service MUST <expected behaviour>.

Map to test shape:

Step 2: Inventory generic features and plan layout

Before planning new files, read modules/features/generic/CCC.Core/ and note which inherited ARs already have scenarios there (or in another shared folder such as vpc/CCC.Core/ for CN06).

Document in analysis.md:

1. A Feature reuse from generic table: Core AR → existing generic (or shared) feature path → action (add @<service> tag vs new feature under <service-folder>/).
2. The intended on-disk layout for new-only scenarios.
3. Whether the service needs port/ (PerPort TLS/SSH/TCP) catalog dirs in runner discovery — see BasicServiceRunner.go: today port/ is loaded for object-storage and virtual-machines; vpc/ for virtual-machines and serverless-computing.

Do not create directories or files on disk — only analysis.md is written in this skill.

Reference modules/features/README.md for naming and routing:

modules/features/
  generic/                    # shared Core — tag new services here when steps are generic
    CCC.Core/
      CCC-Core-CN04-AR01.feature   # @PerService + {service-type}
  <service-folder>/
    analysis.md               # sole file created by this skill
    <CatalogId>/              # planned — native ARs only, typically
      <AR-id>.feature
    CCC.Core/                 # planned — ONLY when generic steps do not fit (rare)
  port/                       # @PerPort TLS/SSH/TCP probes (CN01, CN12-style)

Reuse rules:

Service folder naming (kebab-case, plural where existing):

Step 3: Classify each AR

For every AR (native + inherited that you will cover), assign one primary disposition:

Document gaps explicitly in analysis.md (e.g. “AR text says ‘all relevant information’ but test only checks log-status=OK”).

Step 4: Per-AR test approach

For each Behavioural or Destructive AR, write a short subsection in analysis.md:

1. Requirement (quote) — verbatim text from catalog
2. Reuse — generic/shared feature path, or “new under <service-folder>/” with reason
3. Interpretation — what “when” and “must” mean operationally
4. Approach — the steps you would take to test the service
5. Fixtures / config — what must exist in modules/cloud-api-test/terraform and privateer vars (no discovery or resource creation in Go)

For ARs covered by tag-only reuse, a brief service implementation note under the generic feature is enough — do not repeat full Gherkin steps already in generic/.

Notes:

• Prefer the two-service logging pattern for any “must log / capture” AR:

  • trigger the loggable activity (TriggerDataWrite, GenerateTestTraffic, UpdateResourcePolicy, …)
  • use the logging service: QueryLogs(resourceID, logType, lookbackMinutes) with explicit sink config in privateer vars
  • Do not embed log-query logic on the resource service interface — it belongs on logging.Service.

• Prefer test identities for access-denial ARs: GetServiceAPIWithIdentity + test-user-no-access / test-user-read from privateer test-identities — never ProvisionUserWithAccess in features.

• Service interaction: where the service under test interacts with another service (IAM, logging, object storage, etc.), assume it is available via the cloud-api factory.

Step 5: Minimal cloud-api interface

Follow the same interface design as other services (see modules/cloud-api/logging/logging.go for an example).

Rules:

1. Do not add a method if an existing generic.Service method fits (check generic/service.go — UpdateResourcePolicy, TriggerDataWrite, TriggerDataRead, GetResourceRegion, etc.).
2. Do not add a method if the scenario can call an existing method with different arguments.
3. Return maps for exploratory ops, typed structs for stable domain objects (see object-storage Bucket / Object). This allows cloud-agnostic feature steps.
4. Every method must appear in at least one planned scenario — otherwise omit.
5. Design the smallest interface that supports all planned scenarios.

List methods in a table:

If zero service-specific methods are needed (only generic + logging), say so — implement Service as generic.Service embed only.

Step 6: Cross-cloud implementation matrix

For each method in your API, cover how implementable it is for the three main cloud APIs:

No magical discovery. Every sink, workspace, trail, or account name must come from config (see types.LoggingConfig in types/config.go). If a cloud cannot support an AR without a specific prerequisite (e.g. Traffic Analytics for Azure flow logs), document it under Prerequisites.

Mark unsupported cells explicitly (— + rationale), not silent omission.

Step 7: Write analysis.md

Use this template:

# Behavioural test analysis: <Service display name>

- **Catalog**: `catalogs/<path>/controls.yaml`
- **Catalog id**: `<metadata.id>` (e.g. CCC.VPC)
- **Features root**: `modules/features/<service-folder>/`
- **Cloud-api package**: `modules/cloud-api/<package>/` (existing or new)
- **Factory service id**: `<service-id>`
- **Date**: <ISO date>

## Summary

<2–4 sentences: scope, number of ARs, how many behavioural vs not testable, **how many reuse generic/ vs need new features>

## Feature reuse from generic

| Core control | Generic (or shared) feature | Action for this service |
|--------------|----------------------------|-------------------------|
| CCC.Core.CN04 | `generic/CCC.Core/CCC-Core-CN04-AR01.feature` | Add `@<service>`; `{service-type}` in config |
| … | … | … |

List **new-only** ARs separately (native controls + Core ARs that generic steps cannot cover).

## Imported controls

| Reference | Action |
|-----------|--------|
| CCC.Core.CN04 | Reuse `generic/…`; add `@<service>` — implement generic embed methods only |
| CCC.Core.CN02 | **New** feature — service-specific `Get…Status` method |

## Assessment requirements

### <AR-ID> — <short title>

- **Requirement**: > <quoted catalog text>
- **Disposition**: Behavioural | Destructive | Not testable | Policy-deferred
- **Applicability**: tlp-…
- **Trigger**: …
- **Observation**: …
- **Feature sketch**: (bullet steps, not full Gherkin)
- **Config / fixtures**: …
- **Gaps / honesty notes**: …

(repeat per AR)

## Cloud-api interface (minimal)

### `<package>.Service`

<method table>

### `logging.Service` (if used, or any other service e.g. `iam.Service`)

| logType | AR(s) | resourceID meaning |
|---------|-------|-------------------|

### `generic.Service` methods used

| Method | AR(s) |
|--------|-------|

## Cross-cloud implementation

### `<MethodName>`

#### AWS
…

#### Azure
…

#### GCP
…

(repeat per method)

## Terraform fixtures (planned)

| Fixture name | Role | AR(s) | Cloud(s) |
|--------------|------|-------|----------|
| `finos-ccc-integration-<role>` | main test resource | … | aws, azure, gcp |

Note vpc good/bad/CN03 peers if applicable. Submodule path: `modules/cloud-api-test/terraform/<cloud>/modules/<service>/`.

## Integration test coverage (planned)

| api | method | cloud | expect_error | arg1 | Notes |
|-----|--------|-------|--------------|------|-------|
| `<service-id>` | `ExampleMethod` | all | | `finos-ccc-integration-…` | … |
| `logging` | `QueryLogs` | all | | `<resource>`, `admin`, `60` | … |

Vars for `modules/cloud-api-test/privateer-config/*.yml`: …

## Privateer config (planned vars)

### Behavioural (`cfi-testing/privateer-config/finos-integration/<service>/`)

| Var | Purpose | Example |
|-----|---------|---------|
| `service` / `service-type` | factory id | `virtual-machines` |
| `tags` | scenario filter | `@Behavioural @virtual-machines` |
| `resource` | resource filter (Name tag) | `finos-ccc-integration-vm-main` |
| `test-identities` | CN05 | same shape as object-storage |

### Integration (`modules/cloud-api-test/privateer-config/<cloud>.yml`)

| Var | Purpose | Example |
|-----|---------|---------|
| `resource` | CSV / GetOrProvision filter | `finos-ccc-integration-vm-main` |
| `aws-flow-log-group-name` | logging | from terraform output |

## CI actions-config (planned)

| File | `privateer-service` | `test-configuration` |
|------|---------------------|----------------------|
| `cfi-testing/actions-config/aws-<service>-finos.yaml` | `aws<Service>` | `../privateer-config/finos-integration/<service>/aws-….yml` |

## Open questions

- …

Step 9: Review checklist

Before finishing:

• Every native AR in controls.yaml appears in analysis.md
• Feature reuse from generic table lists each inherited Core AR with path + tag-only vs new-file decision
• No planned duplication of feature files that already exist under modules/features/generic/
• Each behavioural AR has trigger + observation + fixtures
• Interface method count is minimal; prefer generic.Service embed over new methods; no duplicate “query logs” on service interface
• AWS / Azure / GCP columns filled or marked unsupported with reason
• Inherited Core ARs either point at generic/shared features or justify new service-specific scenarios
• Subscription-init / alert / MFA-at-account-layer ARs not falsely marked Behavioural
• Terraform fixtures use finos-ccc-integration-* naming; one main resource per service type (vpc exception documented)
• Integration test coverage table lists every new method + planned expect_error where honest
• Privateer config split documented: finos-integration (behavioural) vs cloud-api-test/privateer-config (integration)
• actions-config entry planned with path under modules/cloud-api-test/terraform/<cloud>
• Only modules/features/<service-folder>/analysis.md was created — no placeholder READMEs, empty catalog dirs, or .feature files

Related skills