# Solve Migration Diff Issues > Skill: Solve Migration Diff Issues for Direct Takeover - Skill: `googlecloudplatform/solve-migration-diff-issues` (Agent Skill, multi-file: 6 files) - Install (CLI): `npx skillmds@latest add googlecloudplatform/solve-migration-diff-issues` - Raw SKILL.md: https://api.skillmd.com/api/skills/googlecloudplatform/solve-migration-diff-issues/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: GoogleCloudPlatform (https://skillmd.com/u/googlecloudplatform) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/googlecloudplatform/solve-migration-diff-issues --- # Skill: Solve Migration Diff Issues for Direct Takeover This skill guides an automated agent through diagnosing, explaining, and fixing "takeover diff" issues when migrating a Kubernetes controller in KCC from Terraform/DCL to the Direct approach. --- ## ⚠️ MANDATORY WORKFLOW REQUIREMENT You MUST run the **4-Step Development Sequence** in exact sequential order (Step 1 ➔ Step 2 ➔ Step 3 ➔ Step 4). * **NEVER skip Step 1.** You must always run the migration test against real GCP first. This is required to exercise the actual GCP APIs, generate GCP project audit logs, and produce a baseline `_migration_diffs.json` from real GCP behavior. **Note that the automated agent has full access to a real GCP project in its environment (pre-configured inside the runtime container/sandbox via Application Default Credentials or environment variables) and is fully expected and required to run `./hack/record-gcp` against live GCP.** * **NEVER jump straight to Step 3 or Step 4.** Writing code or testing solely against MockGCP without first diagnosing on real GCP violates this workflow. * **WHENEVER A TEST CASE IS UPDATED, WE MUST RECORD REAL GCP LOGS AGAIN.** If you make any modifications to a test case configuration, manifest files (such as `create.yaml`, `update.yaml`, or `dependencies.yaml`), or the controller's runtime mapping configuration, you **MUST** run the test case against real GCP (`hack/record-gcp` or with `E2E_GCP_TARGET=real`) to regenerate the authentic logs baseline before comparing or committing any mock log changes. Do not attempt to manually edit the logs or bypass recording live traffic. --- ## The 4 Migration Test Phases (How the Test Runs) The migration test (`TestMigrationToDirect` in `tests/e2e/migration_test.go`) executes in four distinct phases automatically: * **Phase 1 (Legacy Create):** Provisions the initial resource using the legacy (TF or DCL) controller, recording API traffic to `_http_migration_phase1_legacy_create.log`. * **Phase 2 (Legacy Re-reconcile):** Applies a no-op annotation (`re-reconcile-legacy-v1`) while the legacy controller is still active to test re-reconciliation behavior and record any unexpected legacy traffic or diffs to `_http_migration_phase2_legacy_re-reconciliation.log`. (Note: if the legacy controller makes no API requests during re-reconciliation, `_http_migration_phase2_legacy_re-reconciliation.log` will not be created or will remain empty). * **Phase 3 (Direct Takeover):** Applies the `reconciler: direct` annotation to force the Direct controller to adopt the resource. During this takeover, the Direct controller must adopt the resource with a clean **0-write no-op reconciliation**, recording read-only verification traffic to `_http_migration_phase3_direct_takeover.log`. Any diff detected during this phase is considered a bug. * **Phase 4 (Direct Re-reconcile):** Applies another no-op annotation (`re-reconcile-direct-v1`) while the Direct controller is active to verify re-reconciliation behavior. The Direct controller must perform a clean **0-write no-op reconciliation**, recording read-only verification traffic to `_http_migration_phase4_direct_re-reconciliation.log`. --- ## The 4-Step Development Sequence (How to Fix Issues) ### Step 0: Remove from Ratcheting Exclusions (MANDATORY Make-up) The primary focus of this skill is diagnosing and fixing takeover diffs in `TestMigrationToDirect`. Removing the target resource from the ratcheting exclusion list in `tests/e2e/ratcheting.go` is ideally handled during the direct controller logic phase (Step 1.5 in the brownfield logic skill). If the resource has not yet been removed from `tests/e2e/ratcheting.go`, performing the removal here in Step 0 acts as a **make-up for a previous miss** from the controller logic phase. If you perform this make-up removal here, you **MUST** ensure that: 1. The standard fixtures (`TestAllInSeries`) are validated and re-recorded against real GCP (using `./hack/record-gcp` to verify 0-write re-reconciliation), and 2. The migration diff tests (`TestMigrationToDirect`) are successfully validated. To remove the resource from the exclusions: 1. Open `tests/e2e/ratcheting.go`. 2. Locate the function `ShouldTestRereconiliation`. 3. Locate the `switch` statement that checks `primaryResource.GroupVersionKind()`. 4. If there is a `case` block for your target resource's `GroupKind`, remove that `case` line from the switch statement. ### Step 1: Diagnose the Takeover Diff (MANDATORY - RUN ON REAL GCP FIRST) Before writing any code or making any changes, you must diagnose the behavior against real GCP and document it in a separate commit. 1. Run the E2E migration test suite against real GCP for the specific fixture to exercise the APIs and generate audit logs: ```bash ./hack/record-gcp TestMigrationToDirect/fixtures/$ ``` 2. Open the recorded structured diff file: `pkg/test/resourcefixture/testdata/basic/////_migration_diffs.json` 3. Look for blocks containing `"isNewObject": false`. Note that diffs may appear from legacy re-reconciliation (Phase 2), Direct takeover (Phase 3), or Direct re-reconciliation (Phase 4). Diffs occurring during **Phase 3** or **Phase 4** indicate mismatches or re-reconciliation bugs where the Direct controller attempts to modify fields on an unchanged resource. 4. Analyze the diff. A diff of the form `"old": , "new": null` (or vice-versa) indicates a mismatch where a field is populated in one state but not the other on real GCP. 5. **Commit the Baseline Analysis:** Commit the generated diffs/logs from Step 1 into a **separate commit** so the reviewer can clearly understand the problem. Add your detailed analysis of the issue and diffs directly to the commit message. ### Step 2: Identify the Root Cause Using the diff produced in Step 1, identify why the Direct controller sees a difference between KRM and GCP: * **Derived/Computed Fields:** In GCP, some fields (e.g., a BigQuery view's `schema`, default database settings, or server-generated metadata) are automatically computed/derived by the server. These are omitted in the KRM spec (desired is `nil`) but populated by GCP (actual is non-nil). * **Casing & Aliases:** Strings returned by the GCP API might have different casing (e.g. `INT64` vs `INTEGER`, `true` vs `TRUE`) or format (e.g. fully-qualified URIs vs relative paths). * **Default Values:** The old controller might have applied a default value that the new Direct controller does not apply, or vice-versa. * **Diff Suppression in Legacy Controller:** The TF or DCL controller might have custom diff suppression functions (`DiffSuppressFunc` or similar) to suppress diffs for specific fields. Check the legacy controller for this logic to see if it needs to be replicated in the Direct controller. * **Legacy Controller Bugs (Phase 2 Writes):** If there is an unexpected write in Phase 2, that means there is a bug in the legacy controller. If Phase 3 and Phase 4 have the same write as Phase 2, it is not a breaking change. ### Step 3: Formulate the Fix in the Direct Controller 1. Locate the comparison logic for the resource: * For most resources, the comparison is done directly inside the `Update` method (or helper functions) in the controller file: `pkg/controller/direct//_controller.go`. * For extremely large or complex resources (like BigQuery Table), it may be split out into a separate file: `pkg/controller/direct//_compare.go`. 2. **Prevent Parameter Swap Bugs:** When writing or editing comparison functions, **always** explicitly name the parameters `actual` and `desired` instead of `a` and `b`. This prevents accidentally swapping them during comparison and diff reporting. 3. **Ignore Undesired Optional Fields:** If a field is optional in KRM and is omitted from the spec (desired is `nil`), the comparison logic should **ignore** the actual value returned by GCP rather than attempting to delete it. * *Implementation Pattern:* ```go func compareFieldEq(actual, desired *Type, prefix string, diff *structuredreporting.Diff) (bool, error) { if desired == nil { // If the desired state is not specified in the KRM spec, we do not enforce it. return true, nil } if actual == nil { // Desired is specified, but actual is nil. This is a diff. diff.AddField(prefix, actual, desired) return false, nil } // Perform deep comparison... } ``` 4. **Normalize Values before Comparison:** If the diff is due to formatting or casing differences, implement normalization helpers to format both `actual` and `desired` identically before calling `reflect.DeepEqual`. 5. **Create a Clean Commit:** After the fix is created, create a commit with the fix alone so it is easy to review. ### Step 4: Validate the Fix (MANDATORY) 1. Run the E2E migration test against **real GCP** to verify alignment and update golden files: ```bash ./hack/record-gcp TestMigrationToDirect/fixtures/$ ``` 2. Verify that the `"isNewObject": false` block in `_migration_diffs.json` is **completely gone**, indicating a clean 0-write takeover on real GCP. 3. Confirm that the test passes with a perfect green status and there are no unexpected write calls to the GCP API in the real GCP traffic. 4. **Iterate if Necessary:** If there are still bugs, diffs, or unexpected writes at this step, repeat Step 2 and Step 3 to fix the issue until validation passes cleanly. --- ## Step 5: Format and Clean Up 1. Run `make fmt` and `go vet ./pkg/controller/direct/...` to ensure formatting and compilation are perfect. 2. Clean up any accidentally generated untracked `.log` and `.json` files in other fixture directories: ```bash rm -f pkg/test/resourcefixture/testdata/basic//v1beta1//*/_http_migration_phase*.log rm -f pkg/test/resourcefixture/testdata/basic//v1beta1//*/_migration_diffs.json ```