Octo Operator

OctoMesh Communication Operator

Purpose

Developer-facing skill for the Octo Communication Operator (octo-communication-operator) and its companion chart repo (octo-helm-core). It provides the architecture map, key testing seams, the Helm values model, the CRD-generation workflow, and the local kind developer loop so you can extend or debug the operator without reading the full repo CLAUDE.md cold.

Repos:

• C:\dev\meshmakers\octo-communication-operator — the operator (.NET 10, KubeOps).
• C:\dev\meshmakers\octo-helm-core — the official Helm charts (CRDs, operator, core services, demo app).

Cross-references:

• octo-devtools skill — owns the local kind bring-up cmdlets (Install-OctoKubernetes, Deploy-OctoOperator, Get-OctoKubernetesStatus, Uninstall-OctoKubernetes). Do not duplicate them here; route there.
• octo-agent skill — debugging/build-chain investigation (DebugL NuGet, service health).

Verify-before-claim: confirm CLI/cmdlet behavior against source before asserting it — operator C# in src/CommunicationOperator/, chart YAML in octo-helm-core/src/, cmdlets in octo-tools/modules/*.psm1.

What the Operator Is

A Kubernetes operator built on the KubeOps framework (dotnet-operator-sdk). It:

1. Watches CommunicationPool custom resources (CRD group octo-mesh.meshmakers.io, version v1alpha1, kind CommunicationPool; the C# entity is V1CommunicationPoolEntity).
2. Connects to the Communication Controller's /operatorHub SignalR hub (always, when a controller URI is configured — see modes below).
3. On WorkloadDeployedAsync events, runs helm upgrade --install per Adapter and Application workload; on WorkloadUndeployedAsync, runs helm uninstall.

There is no raw-K8s AdapterReconciler path anymore — every Adapter and Application is deployed exclusively as a Helm release via WorkloadReconciler.

The CRD itself is not generated in this repo's output by default; it ships from the octo-mesh-crds chart in octo-helm-core. See CRD Generation.

Deployment Modes

The operator runs in two modes, distinguished by OPERATOR__AUTOMANAGEPOOLS and where it runs.

CRITICAL — the flag is narrow: AutoManagePools gates only the side effect of auto-creating/-deleting CommunicationPool CRs and broker secrets. The SignalR connection, pool register/unregister round-trip, and the WorkloadDeployedAsync → helm path run in both modes whenever OPERATOR__COMMUNICATIONCONTROLLERURI is set. A previous early-return on !AutoManagePools caused edge pools to show as Unregistered forever. When CommunicationControllerUri is empty, OperatorHubService logs a warning and exits, and RegisterPoolAsync becomes a no-op — the CR reconciles locally but the controller never sees the pool.

For multiple edge operators on one cluster, each must set OPERATOR__WATCHNAMESPACE.

Reconciliation Flow

1. A CommunicationPool CR is created (manually, or by OperatorHubService when central).
2. PoolService registers the pool with the Controller via the PoolHub SignalR client.
3. The Controller fans out WorkloadDeployedAsync on /operatorHub per Adapter/Application.
4. WorkloadReconciler materializes secret-flagged values into an operator-owned Secret, ensures the chart repo, then runs helm upgrade --install (preceded by a dry-run pre-flight — see below).
5. On pool deletion, matching WorkloadUndeployedAsync events trigger helm uninstall per release.

Soft-failure on unregister: PoolService.UnRegisterPoolAsync (from CommunicationPoolController.DeletedAsync) treats any HubException from the controller as a soft failure — logs only. The CR is already gone when DeletedAsync fires and the tenant may no longer exist controller-side; re-throwing would trap the entity in the KubeOps retry queue forever. The local connection is stopped and the pool removed regardless.

Helm Values Layering Model

WorkloadReconciler writes up to three values files to a temp dir and passes them via -f in this order (Helm's later -f wins, so order = precedence, lowest → highest):

WorkloadContextValuesBuilder projects only the fields that are set — an edge operator with an empty DTO context passes no context layer at all. Secrets are deliberately not handled in the context builder; they flow through WorkloadOverrideYamlBuilder and the per-release Secret. Secret-flagged override entries become a valueFrom: secretKeyRef envelope pointing at the operator-owned {release}-octo-secrets Secret; non-secret entries are literal values; dotted paths (e.g. oauth.clientSecret) become nested maps.

Release name: {tenantId}-{workloadRtId}, DNS-sanitised via K8sNaming.DnsName and truncated to Helm's 53-char limit. The runtime entity id (24-char lowercase hex) is used — not the user-facing WorkloadName — so renaming the workload in Studio does not orphan the helm release. WorkloadReconciler.ReleaseName / SecretName / RepoAlias are the deterministic helpers (exposed to tests via InternalsVisibleTo).

Empty version: UpgradeInstallAsync omits --version when blank so helm picks the newest tag — the contract for System.Communication.MainLatest on dev/test clusters. Pass a non-blank value to pin a chart.

Full builder mechanics and the IHelmRunner / IHelmProcessInvoker argument contracts are in references/operator-internals.md.

Secret Injection Tiers

Before the override builder runs, WorkloadReconciler calls AppendClusterSecrets with two distinct tiers — this is a foot-gun, get it right:

1. secrets.rabbitmq (from OperatorOptions.BrokerPassword) — injected UNCONDITIONALLY whenever BrokerPassword is set. RabbitMQ is the controller↔adapter command bus; every adapter needs it. Gating this behind the cluster-secrets opt-in previously broke pure edge adapters (Modbus/Loxone) that failed the chart's mandatory secrets.rabbitmq check.
2. Data-store secrets (secrets.databaseUser, secrets.databaseAdmin, secrets.streamDataPassword, from OperatorOptions.ClusterSecrets.*) — injected only when the workload's WorkloadDeployedDto.ReceivesClusterSecrets flag is true (set by the controller from the Adapter CK entity's ReceivesClusterSecrets attribute). Pure edge adapters leave this false; the chart's features.mongo / features.streamData gates then skip the matching env blocks.

Injected entries are prepended so any entity-supplied override on the same path still wins, then flow through the normal secret pipeline into {release}-octo-secrets. Each adapter chart's secrets.* block must accept both plaintext strings (legacy) and valueFrom maps — see the octo-mesh.secretEnv helper in octo-mesh-adapter / octo-eda-adapter templates.

Pre-flight, Watcher, Cancellation

Three mechanisms surface the real failure reason behind helm upgrade --install --atomic (which otherwise collapses everything into one opaque context deadline exceeded stderr line):

• Pre-flight dry-run (UpgradeInstallDryRunAsync, --dry-run=server, no --atomic) runs before the real install in WorkloadReconciler.DeployAsync. Admission webhooks, OpenAPI schema validation, and RBAC run server-side without creating resources, catching schema/Gatekeeper/RBAC errors in <2s. On failure it throws HelmException and the real install is skipped.
• Live Deploy Watcher (WorkloadDeployWatcher) polls IWorkloadDiagnosticsCollector.CollectAsync every ~3s during the real install and pushes non-empty, changed snapshots to the controller via ReportWorkloadDeploymentProgressAsync (state stays Pending — helm may still recover). Closes the 5-minute gap before the post-failure collector runs. Older controllers reject the new hub method with HubException; the service logs one warning and degrades silently.
• Post-failure diagnostics (IWorkloadDiagnosticsCollector) scrapes pod container statuses and namespace Warning events on HelmException, merging the result into the rethrown exception's stderr. Events outlive pods, so ImagePullBackOff / FailedScheduling / FailedMount are caught even after atomic rollback.

Cancellable deploys: WorkloadReconciler._inFlightDeploys tracks running deploys by release. UndeployAsync cancels an in-flight deploy's CTS, waits a 2s grace for atomic rollback, then helm uninstall --ignore-not-found. Cancellation works end-to-end only because HelmProcessInvoker.InvokeAsync does process.Kill(entireProcessTree: true) on cancel — WaitForExitAsync(ct) alone leaves helm holding the release lock.

Reconciler exceptions are logged but not propagated to the hub — one bad workload must not crash the SignalR connection.

Reverse-Sync on Reconnect

After re-registering owned CRs on reconnect, a central operator (AutoManagePools=true) calls IOperatorHub.ReportDeployedStateAsync(reports) to restore DeploymentState=Deployed on pools whose state drifted while offline. Two coupled paths run it: bulk on reconnect (OperatorHubService.onReconnect, covers controller-restart) and per-pool on register (PoolService.RegisterPoolAsync, covers operator-restart race where KubeOps repopulates _pools after the bulk snapshot). Per-pool is idempotent (restore-only-when-changed). Edge operators skip it (controller rejects them); empty owned-pool list skips it; call failure is logged, not propagated. Workloads are not yet reverse-synced (no persistent helm-release→workload-rtId map survives a pod restart). See docs/DEPLOYMENT-MANAGEMENT-CONCEPT.md for the contract.

Key Testing Seams

The operator is built to be testable at abstraction boundaries, not against the k8s SDK. Mock these:

Test sync point for OperatorHubService: use client.When(c => c.EnableReconnect(...)).Do(_ => tcs.TrySetResult()) — once EnableReconnect ran, the connect callback finished and the service is parked in Task.Delay(Infinite, ...). Asserting earlier races.

CommunicationPoolKubernetesGateway itself and OperatorHubClientFactory are not unit-tested (thin pass-throughs, covered by E2E).

Test Conventions (.NET 10 / TUnit / MTP)

Framework: TUnit ([Test] attribute, Assert.That(...).IsXxx(...) fluent API) + NSubstitute. Tests mirror the source folder structure; namespaces Meshmakers.Octo.Communication.Operator.Tests.<Area>.

CRITICAL — .NET 10 uses Microsoft.Testing.Platform, not VSTest. The legacy VSTest path is rejected. Two pieces opt in: global.json at the repo root sets "test": { "runner": "Microsoft.Testing.Platform" }, and the test csproj references Microsoft.Testing.Extensions.TrxReport. Under MTP, the project/solution is a flag (--project/--solution, not positional), and reporter/filter args go after --.

Verified commands (run from the repo root; build only ever in DebugL):

Test-project gotchas: ThrowsAsync(...) (not Throws(...)) for Task-returning substitutes (NS5003 is an error); implement IDisposable + dispose any disposable SUT (TUnit0023); inject options with fully-qualified Microsoft.Extensions.Options.Options.Create(new OperatorOptions { ... }); set <EnablePreviewFeatures>true</EnablePreviewFeatures> (KubeOps APIs are [RequiresPreviewFeatures]). Target framework net10.0, three configs Debug/Release/DebugL.

CRD Generation Workflow

The CRD is shipped from octo-helm-core (octo-mesh-crds chart), not from this repo's build output. To regenerate operator + CRD YAML into config/ (run from src/CommunicationOperator/, where install.ps1 lives):

# Read-only generation — writes YAML manifests to ./config, clears stale output.
dotnet kubeops g op meshmakers-octo-communication-operator ./CommunicationOperator.csproj --out config --clear-out

This is a codegen command (writes files under config/), not a cluster mutation. Apply with kubectl apply -k config/install only deliberately on a dev cluster. When the CRD schema changes, propagate the generated CRD into the octo-mesh-crds chart in octo-helm-core.

Local Kind Developer Loop

The full kind bring-up is owned by octo-tools cmdlets — route to the octo-devtools skill for these; do not re-implement. Verified entry points (from octo-tools/modules/*.psm1):

In-repo helpers (run via PowerShell): src/scripts/Create-KindTestCluster.ps1 (+ Remove-KindTestCluster.ps1), and start-operator.ps1 at the repo root runs a locally-built operator in central mode on ports 5022/5023 (ASPNETCORE_ENVIRONMENT=Development). It is intentionally not named octo-start.ps1 so Start-Octo does not auto-launch it.

In DEBUG/DEBUGL builds the operator auto-picks the first non-loopback IPv4 for its dev webhook endpoint and logs it (Dev webhook endpoint: https://192.168.x.y:6001). Override with Operator:DevWebhookHost/Port in appsettings.Development.json or OCTO_OPERATOR__DEVWEBHOOKHOST/PORT.

Manual E2E runbook: docs/E2E-SMOKE-TEST.md (uses Install-OctoKubernetes, Start-Octo, start-operator.ps1).

octo-helm-core Charts

octo-helm-core is the official Helm chart repo, published to https://meshmakers.github.io/charts (GitHub Pages). Add it with helm repo add meshmakers https://meshmakers.github.io/charts.

Charts live under src/<chart>/ (each has its own Chart.yaml and values.yaml):

Adapter charts (octo-mesh-adapter, octo-eda-adapter) now live in their own adapter repos and publish from there, not from octo-helm-core.

References

• references/operator-internals.md — OperatorOptions full config table (env-var keys, central vs edge requirements), Helm builder mechanics, IHelmRunner/IHelmProcessInvoker argument contracts, CI pipeline shape, solution layout, and the full unit-test coverage map.