Webstatus Ingestion

webstatus-ingestion

This skill provides guidance for developing and deploying the scheduled data ingestion workflows (Cloud Run Jobs) in the workflows/ directory.

Architecture

• Location: Workflows are stand-alone Go applications located in workflows/steps/services/.
• Varied Consumers: Specialized workflows (BCD, WPT, UMA, Mapping, Dev Signals) are defined in infra/ingestion/workflows.tf.
• Pattern: Most workflows follow a "Downloader -> Parser -> Processor" separation of concerns.
• Trigger: When ingestion flows complete, they trigger the event_producer to begin the notification diffing process.

Architecture

For a detailed map of data sources, Spanner target tables, and job orchestration patterns, see references/architecture.md.

Infrastructure Abstraction (The Adapter Pattern)

Ingestion jobs must be decoupled from the core DB logic and the "Backend" API.

• Consumer Adapters: Each workflow uses a purpose-built spanneradapter (e.g., BCDConsumerAdapter).
• Why: This prevents ingestion logic from breaking the Public API if the data schema changes. It also allows mocking the database during parser tests.

Guides

• Add a New Scheduled Workflow: Steps for creation, scheduling, and Terraform integration.
• Ingestion Patterns: Choosing between Sync, Batch Upsert, and Simple Insert.

General Do's and Don'ts

• DO cross-reference all code against the official Google Go Style Guide. If you are unsure about a specific style rule, DO NOT assume; you MUST ask the user for clarification.
• DO use consumer-specific spanneradapters (e.g. BCDConsumer).
• DON'T call the Backend spanner adapter from a workflow.
• DO separate data fetching/parsing from the main workflow processor (use pkg/data/downloader.go and parser.go).
• DO use web_features_mapping_consumer when syncing browser-specific implementation keys or "implementer" metadata.
• DO use uma_export for any changes involving Chromium usage metrics or histograms.
• DO use intermediate types in lib/ (e.g. lib/webdxfeaturetypes) to decouple logic from external source schemas.
• Tip: While make dev_workflows exists to pull live data, it is generally preferred to use make dev_fake_data for UI and Backend development to ensure stability.
• DO use manifests/job.yaml for workflows (scheduled jobs), unlike workers which use pod.yaml.

Testing & Linting

• Precommit Suite: Run make precommit to execute the full suite of Go tests, formatting, and linting.
• Linting: Run make go-lint to lint all Go code using golangci-lint.
• Quick Test Iteration: Because this project uses a multi-module workspace (go.work), to run tests quickly for a single package without running the whole suite, execute go test from within the specific module directory:

  cd workflows/steps/services/<workflow_name> && go test -v ./...
  

Documentation Updates

When you add a new workflow or change the ingestion patterns:

• Update docs/ARCHITECTURE.md to reflect the new external source or data flow.
• Trigger the "Updating the Knowledge Base" prompt in GEMINI.md to ensure I am aware of the changes.
• Update these very skills files if you introduce new structural patterns.