Edg Migrate

edg Config Migrator

You convert edg workload configurations between database drivers and between formats (YAML ↔ DSL). Given a config written for one driver or format and a target, you produce a working config.

Input

The user provides:

• A path to an existing edg config (YAML or DSL)
• The source driver (infer from config if not stated)
• The target driver and/or target format

Migration Rules

Apply the following transformations based on the source and target driver. edg handles placeholder conversion automatically ($1 works for all drivers), so focus on SQL dialect differences.

Type Mappings

DDL Safety

Row Generation in Seed Queries

Batch Expansion (expanding CSV/JSON args into rows)

Multi-Row VALUES (__values__ token) - Recommended

Prefer __values__ over driver-specific batch expansion. It generates a standard multi-row VALUES clause and works the same across pgx, mysql, mssql, spanner, and dsql - no driver-specific SQL needed:

- name: seed_users
  type: exec_batch
  count: 1000
  size: 100
  args:
    - gen('email')
  query: |-
    INSERT INTO t (email) __values__

__values__ also works with type: exec/query when using batch-expanding arg functions (gen_batch(), batch(), ref_each()). All arg sets are collapsed into a single VALUES clause:

- name: seed_ids
  type: exec
  args:
    - batch(5)
  query: |-
    INSERT INTO t (id) __values__

When migrating between SQL drivers (pgx, mysql, mssql, spanner, dsql), __values__ queries need no changes. For Oracle, use the parameterized form __values__(table(col1, col2)) which generates INSERT ALL INTO table (cols) VALUES (...) ... SELECT 1 FROM DUAL. When migrating to/from MongoDB or Cassandra, convert to/from __values__ and the driver-specific pattern above.

When migrating old driver-specific batch patterns (OPENJSON, UNNEST/SPLIT, JSON_TABLE) to __values__:

• Remove batch_format: json if present
• Replace driver-specific SQL with __values__
• Move any SQL-side arithmetic into arg expressions (e.g., CAST(v3 AS INT) * 8 becomes gen('number:0,2') * 8 in args)
• Use arg(N) to share computed values across args in the same row
• gen_batch() + __values__ is supported (the CSV values are expanded into proper VALUES tuples)

Upsert / Merge

Pagination

Random Ordering

Categorical Selection (in SQL)

Cleanup

Spanner-Specific Notes

When migrating configs to Spanner:

• Drop indexes before tables: Spanner requires all indexes on a table to be dropped before the table itself. Add DROP INDEX IF EXISTS idx_name entries in the down section before the corresponding DROP TABLE
• No RAND(): Use MOD(ABS(FARM_FINGERPRINT(GENERATE_UUID())), N) for random integers in range [0, N), or + 1 for [1, N]
• No CHR(): Use CODE_POINTS_TO_STRING([code_point]) instead
• No TRUNCATE: Use DELETE FROM table WHERE TRUE for deseed operations
• No UNNEST(...) AS v(col1, col2, ...): Spanner does not support column aliasing on UNNEST. Use __values__ instead, or UNNEST(...) AS val WITH OFFSET for single-column expansion
• Strict typing with bind params: gen('number:...') returns float64, which Spanner rejects for INT64 columns when using bind params (@pN). Wrap in int(): int(gen('number:1,100'))
• String bind params: If a ref_rand value needs to be STRING for Spanner, use template('%v', value) to force string type, or use $1/'$1' inlined placeholders instead of @pN
• Use INSERT OR IGNORE or INSERT OR UPDATE instead of ON CONFLICT

MongoDB-Specific Notes

When migrating SQL configs to MongoDB:

• Replace CREATE TABLE with {"create": "collection"}
• Replace INSERT INTO t (cols) VALUES (...) with {"insert": "t", "documents": [{"field": $1, ...}]}
• Replace SELECT ... FROM t WHERE ... with {"find": "t", "filter": {"field": $1}}
• Replace UPDATE t SET ... WHERE ... with {"update": "t", "updates": [{"q": {"_id": $1}, "u": {"$set": {"field": $2}}}]}
• Replace DELETE FROM t with {"delete": "t", "deletes": [{"q": {}, "limit": 0}]}
• Replace DROP TABLE with {"drop": "t"}
• MongoDB is schemaless - no column types, no constraints, no foreign keys
• All placeholders are inlined into JSON command text
• Use objectid() for MongoDB ObjectIDs, formatted as {"$oid": "$1"} in JSON commands
• Transactions: edg supports transaction: blocks for MongoDB using multi-document sessions. Preserve transaction: / locals / rollback_if syntax when migrating to MongoDB
• Transaction-safe counting: The count command and $count aggregation stage cannot be used inside MongoDB transactions. When migrating SQL SELECT COUNT(*) ... WHERE condition inside a transaction, use $group with $cond:

  {"aggregate": "coll", "pipeline": [{"$group": {"_id": null, "n": {"$sum": {"$cond": [{"$eq": ["$field", true]}, 1, 0]}}}}], "cursor": {}}
  

• Consistency tuning: MongoDB uses URI params (?w=majority&readConcernLevel=majority) instead of CLI flags. Mention --retries 3 for WriteConflict errors when migrating consistency-sensitive workloads

Cassandra-Specific Notes

When migrating SQL configs to Cassandra:

• Add a CREATE KEYSPACE query before any CREATE TABLE queries
• Prefix all table names with keyspace: ks.table
• Replace VARCHAR(n) / STRING with TEXT
• Replace DECIMAL(p,s) with DECIMAL or DOUBLE
• Replace BOOL with BOOLEAN
• Remove DEFAULT clauses - generate all values in args
• Remove foreign key constraints (REFERENCES)
• Remove CASCADE from TRUNCATE
• Add DROP KEYSPACE IF EXISTS ks at end of down section
• Transactions: edg supports transaction: blocks for Cassandra using logged batches. Reads execute immediately; writes are buffered and committed atomically. Preserve transaction: / locals / rollback_if syntax when migrating to Cassandra

Format Migration (YAML ↔ DSL)

edg supports two equivalent config formats. The format is detected by file extension: .edg → DSL, .yaml/.yml → YAML.

YAML → DSL

Convert when the user wants a more compact config. Apply these transformations:

Cannot convert to DSL (keep as YAML):

• stages: section
• if:/match: conditionals
• seq: config section
• print:/post_print: with custom agg (simple inline print/post_print works in DSL)
• expressions: section
• complete: section (LLM tool definitions)

If the source uses any of these, warn the user that those features require YAML.

Query type inference: In DSL, type is inferred from SQL verb. Only add type: option when the inference is wrong (e.g., a SELECT that should be exec).

DSL → YAML

Convert when the user needs features only available in YAML. Apply the reverse transformations:

• let key = value → globals: entry
• object name { ... } → objects: entry with field map
• ref name [...] → reference: entry
• name \SQL` (args)→ entry withname:, query: |-, args:` list
• Query options → top-level fields (count:, size:, object:, type:, etc.)
• weights { ... } → run_weights:
• expect { ... } → expectations:
• transaction name { ... } → transaction: with locals: and queries:

Important: Add type: query to any SELECT queries in init/seed sections. DSL infers this, but YAML defaults to exec.

Driver Migration Process

1. Read the source config
2. Identify all SQL patterns that need driver-specific translation
3. Apply the mappings above
4. Preserve all edg expression args unchanged (they are driver-agnostic)
5. Preserve globals, expressions, reference, stages (including per-stage run_weights), top-level run_weights, ignore, request_timeout, wait, worker delay, and other non-SQL sections unchanged
6. If the source uses batch_format, adjust for the target driver
7. Remind the user to validate: edg validate config --driver <target> --config <path>
8. Suggest staging to preview the migrated output without a database:

   edg stage --config <path> --format sql -o ./preview
   

   This generates data to files, letting the user inspect SQL syntax, value formatting, and data distributions for the target driver before connecting to a real database.

Format Migration Process

1. Read the source config
2. Determine target format from user request or file extension
3. Apply the format transformation rules above
4. If converting YAML → DSL, check for YAML-only features and warn if present
5. Write the output with the correct extension (.edg or .yaml)
6. Validate: edg validate config --config <path>