Match Mockgcp With Realgcp

Skill: Match Mock behaviour with real GCP api

This skill provides a structured workflow for matching the mock {service}{resource} behaviour with the real GCP.

When the golden tests for K8s Config Connector mock output diverge from real GCP output, we need to inspect the discrepancies and fix either the mock implementation or the normalizers. This ensures that hack/compare-mock runs cleanly and accurately represents GCP API contracts.

Critical Rules

• Real GCP Baseline Required: You must always generate the initial baseline _http.log by running hack/record-gcp against a real GCP project and commit it first.
• Basic Test Case Naming: Ensure that the basic test case directory/folder names always contain the resource kind name (e.g., use computesubnetwork-reservedinternalrange instead of reservedinternalrange).
• Handling record-gcp Failures: If record-gcp fails, the bot should attempt to debug and fix the test case configuration (e.g., resolving IP address range conflicts, API/service enablement, or configuration schema issues). If it still fails after 3 distinct retry attempts, the bot must halt, escalate the failure to the human reviewer, and clearly explain the blocker/error in a comment.
• No New Missing Fields: We should not add new missing fields to the exceptions file (tests/apichecks/testdata/exceptions/missingfields.txt). If fixing or renaming an existing test case exposes/causes more missing fields, you must extend the test case configuration or add a new test case (or more than one if needed) to cover those missing fields.
• Do NOT Generate Golden Logs from Mock: You are strictly forbidden from generating or updating _http.log files against the mock in this step. You MUST first establish a baseline against real GCP with E2E_GCP_TARGET=real E2E_KUBE_TARGET=envtest with WRITE_GOLDEN_OUTPUT=1.
• Do NOT Manually Edit HTTP Logs: You are strictly forbidden from using text-editing tools (like replace, write_file, or sed) to modify any _http.log files. The ONLY valid way to update these files is by executing the test scripts (hack/record-gcp or hack/compare-mock) via the shell.
• Provide Timestamps: When starting and finishing running hack/record-gcp command, you must capture and provide the exact timestamps in PDT in the summary comment after addressing the comments so that the reviewer can check the GCP logs to verify the work.
• Do NOT Modify Test Cases Post-Recording: Do not update/modify the test case configuration (e.g., changing location/zone) after you have committed the real GCP logs. If you must change the test case, you must run hack/record-gcp again to record the real GCP logs.
• WHENEVER A TEST CASE IS UPDATED, WE MUST RECORD REAL GCP LOGS AGAIN: If you make any modifications to a test case configuration, manifest files (such as create.yaml, update.yaml, or dependencies.yaml), or the controller's runtime mapping configuration, you MUST run the test case against real GCP (hack/record-gcp) to regenerate the authentic _http.log baseline before comparing or committing any mock log changes. Do not attempt to manually edit the logs or bypass recording live traffic.
• Do NOT Update Legacy Controller: Do not update or modify the legacy/Terraform-based controller (such as code under pkg/krmtotf/references.go) unless absolutely necessary. Keep the legacy controller logic unchanged and avoid modifications to common or shared legacy framework code.
• Focus on Target CRUD Requests: When aligning the HTTP logs, we only care about the requests made to the CRUD endpoint of the target resource. Non-target/dependent resource requests in the log do not need to be matched as strictly.

Workflow

Step 0: Remove from Ratcheting Exclusions (MANDATORY)

Before running the test cases against real or mock GCP, you MUST ensure the target resource is removed from the ratcheting exclusion list in tests/e2e/ratcheting.go. This enables the re-reconciliation test step, which is a fundamental use case KCC resources must support.

1. Open tests/e2e/ratcheting.go.
2. Locate the function ShouldTestRereconiliation.
3. Locate the switch statement that checks primaryResource.GroupVersionKind().
4. If there is a case block for your target resource's GroupKind, remove that case line from the switch statement.

Step 1: Record GCP logs

1. Run hack/record-gcp "fixtures/^<testname>$" to capture real GCP behavior.
   • Troubleshooting Service Not Enabled: If hack/record-gcp fails because a GCP service is not enabled (e.g., error mentions that the API is disabled or has not been used in the project before), enable the service using gcloud and try again:

     gcloud services enable <service-name>.googleapis.com
     

     (For example: gcloud services enable compute.googleapis.com or gcloud services enable run.googleapis.com)
2. Please commit the real gcp logs generated by running the command.

Important:

• It is important to commit the files modified by running realgcp tests in its own commit.
• This is for the human reviewer to compare the diff in the test artifacts when running real and mockgcp.

Step 2: Match mockgcp behaviour with real GCP

1. Run hack/compare-mock "fixtures/^<testname>$" to check mock behavior.
2. Iteratively fix discrepancies in the mock implementation or normalize.go.

Tips for fixing the discrepancies:

1. Look closely at the compare-mock HTTP log differences (typically mock on left => real on right).
2. For missing default values (e.g., <missing> => REGIONAL), add a populateDefaultsFor<Resource> function to the mock service's file (e.g. mockgcp/mockcompute/networksv1.go). Make sure it is called on Insert and Get.
3. For generated IDs or volatile values (e.g. IPs, resource URLs) where real GCP generates dynamically but mockgcp outputs something static, you need to update the normalizer mockgcp/mock<service>/normalize.go.
4. Run hack/compare-mock "fixtures/^<testname>$" to see the diff and overwrite _http.log.
5. Run git diff on the test fixtures to ensure that the golden _http.log accurately replaces volatile data with placeholder variables (e.g., ${ipAddress}).
6. Certain operation metadata values (e.g. done: <missing> => false) can be safely ignored as mock operations are generally simpler.