Setup Cortex Config

Setup Cortex Configuration

This skill assists in establishing, customizing, and validating the primary Cortex Framework configuration (config.yaml). It uses a slot-filling experience to ensure all mandatory and optional configurations are accurately populated, conforming to Google Cloud and BigQuery formatting requirements.

Required vs. Optional Configuration

Cortex V7 requires accurate environment details to coordinate BigQuery transformations and orchestrate Dataform pipelines.

1. Mandatory Configuration Settings (Strict Validation)

The following values MUST be populated with valid, non-placeholder values:

• buildEnvironment.buildProjectId: The Google Cloud Project ID where build operations and script executions run.
• data.bigQueryLocation: The geographical location for BigQuery datasets (e.g., US, EU, us-central1).
• data.sources: Needs at least one data source (typically id: sap_raw) mapping the replication source:
  • projectId: GCP project ID hosting the raw/replicated SAP tables.
  • datasetId: BigQuery dataset ID containing the replicated tables (e.g., cortex_sap_raw).
• data.targets: Needs target datasets where transformations will be materialized:
  • Foundation Target (id: sap_foundation): Project ID and Dataset ID for processed, standard tables.
  • Product Target (id: product_target): Project ID and Dataset ID for final data products.
• deployment.targets: Orchestration targets (Dataform repository setup):
  • repositoryProjectId: GCP Project ID hosting the Dataform repository.
  • repositoryRegion: GCP region hosting the repository (e.g., us-central1).
  • repositoryName: Unique repository name (e.g., cortex-repository).
  • workspaceName: Dataform workspace name (e.g., dev).

2. Optional Configuration Settings

• data.namespaces: Custom namespaces for data foundation and products (default is cortex). Note: defining a custom/customer namespace is NOT required for standard framework setup, and is only recommended/required when building the customer's own new or derived data products.
• modules.foundations settings:
  • sapVersion: Target SAP version (ecc or s4).
  • mandt: 3-digit SAP client identifier (defaults to "100").
  • enabled: Set true or false to enable/disable specific foundations.
  • tableSettings: File path override for custom table listings.
• modules.products settings:
  • enabled: Set true or false to build/deploy individual data products.
  • tableSettings: Custom partitioning/clustering settings file path.

Using the Configuration Tool

This skill includes a dedicated validation and configuration utility located at: external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py

1. Interactively Fill Configuration (Wizard Mode)

Run the interactive slot-filling wizard to set up or update a configuration file step-by-step:

python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py wizard --config <target_config_path.yaml>

Note: If the target config file already exists, current values will be loaded as defaults in brackets [...]. Pressing Enter keeps the current value.

2. Update Specific Fields

You can directly modify individual fields without going through the wizard:

python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py set <yaml.dot.path> <value> --config <target_config_path.yaml>

Examples:

# Update build project ID
python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py set buildEnvironment.buildProjectId my-cortex-build-project --config config/config.yaml

# Change SAP Version to S4
python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py set data.modules.foundations.0.moduleSettings.sapVersion s4 --config config/config.yaml

3. Validate Configuration Syntax and Settings

Verify all project IDs, datasets, and locations conform to syntax rules and that no placeholder YOUR_... values remain:

python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py validate --config <target_config_path.yaml>

4. View Current Configuration

Print the clean structured configuration file:

python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py show --config <target_config_path.yaml>

Core Operating Behaviors for Agents

When tasked with helping a user setup their configuration:

1. Locate / Create Configuration file:
   • Check if a configuration file already exists (e.g., config/config.yaml or .cortex/config.yaml).
   • If not, copy cortex-framework-core/config/config.yaml.example to the path requested by the user (defaulting to config/config.yaml). Never overwrite existing custom files without user consent.
2. Execute Validation First:
   • Run the validation command to spot missing or placeholder values.
3. Provide a Slot-Filling Session:
   • Offer to run the configure_cortex.py wizard to guide them through setting up their GCP projects and BigQuery settings.
   • Alternatively, request the values directly from the user and apply them using the set command of the tool.
4. Validate and Test Build:
   • Run python3 external-skills/.agents/skills/setup_cortex_config/scripts/configure_cortex.py validate to confirm the configurations.
   • Ensure they pass successfully before attempting deployment or compiling data products.
5. Generate Configuration Setup Report:
   • At the end of a configuration action, you MUST generate a comprehensive Cortex Framework Configuration Setup Report as an artifact in the standard brain artifacts directory using the template located at: assets/setup_report_template.md
   • Fill in all placeholders accurately (e.g. file paths, GCP parameters, validation validator outputs) based on your execution, and refer to the generated report in your final response to the user.