Writing Data Transfer Config

Writing a @webiny/data-transfer config

A config is a .ts file that export defaults one of the two factory calls:

• createDdbConfig(...) — DDB primary table (+ S3 files). Covers CMS + security + file manager + tenancy.
• createOsConfig(...) — OpenSearch companion DDB table. CMS entries only, gzipped records.

Both validate with Zod at import time — invalid configs fail fast with a useful message, before any AWS call.

Minimal shape

import {
    loadEnv,
    createDdbConfig,
    fromAwsProfile,
    fromEnv,
    numberFromEnv
} from "@webiny/data-transfer";

loadEnv(import.meta.url);

export default createDdbConfig({
    source: {
        region: fromEnv("SOURCE_REGION", "us-east-1"),
        credentials: fromAwsProfile({ profile: fromEnv("SOURCE_PROFILE", "default") }),
        dynamodb: { tableName: fromEnv("SOURCE_DDB_TABLE") },
        s3: { bucket: fromEnv("SOURCE_S3_BUCKET") }
    },
    target: {
        region: fromEnv("TARGET_REGION", "us-east-1"),
        credentials: fromAwsProfile({ profile: fromEnv("TARGET_PROFILE", "default") }),
        dynamodb: { tableName: fromEnv("TARGET_DDB_TABLE") },
        s3: { bucket: fromEnv("TARGET_S3_BUCKET") }
    },
    pipeline: {
        preset: "v5-to-v6-ddb", // built-in, OR "./presets/my-preset.ts"
        segments: numberFromEnv("SEGMENTS", 4)
    }
});

Credentials — three accepted shapes

The credentials field accepts any of the three. Pick what matches where the transfer runs; mixing is fine across source/target.

A) Profile — explicit (fromAwsProfile)

credentials: fromAwsProfile({ profile: fromEnv("SOURCE_PROFILE", "default") })

Reads ~/.aws/credentials using the named profile (re-export of fromIni; renamed because "ini" leaks the implementation detail). Use locally when you have multiple AWS accounts — explicit profile selection prevents a stray AWS_ACCESS_KEY_ID in your shell from silently overriding the wrong account.

B) Default credential chain — flexible (fromAwsCredentialChain)

credentials: fromAwsCredentialChain()

Runs the AWS SDK's default resolution: env vars → shared credentials file → SSO / web identity → EC2/ECS IAM role. Use in CI / cloud where the credentials source depends on the deploy target, or when you want one config that "just works" locally AND on a build agent AND on an IAM-instance-profile box.

C) Literal credentials from env — explicit strings

credentials: {
    accessKeyId: fromEnv("SOURCE_AWS_ACCESS_KEY_ID"),
    secretAccessKey: fromEnv("SOURCE_AWS_SECRET_ACCESS_KEY"),
    // sessionToken: fromEnv("SOURCE_AWS_SESSION_TOKEN")  // only for temporary STS creds
}

Use when your CI injects credentials as env vars directly, when you have temporary STS credentials to pass in, or when you prefer creds in a single .env file over a shared profile.

Picking between them

fromEnv(name, default?)

Reads process.env[name] as a string. Throws if the variable is unset OR empty when no default is provided. Empty-string counts as missing because KEY= in a .env file is almost always a forgotten value.

region: fromEnv("SOURCE_REGION", "us-east-1"),   // has default
tableName: fromEnv("SOURCE_DDB_TABLE"),           // no default → throws if missing

numberFromEnv(name, default?)

Same contract, but parses via Number(...). Throws if the variable is set but not parseable (SEGMENTS=four → named error, not silent NaN).

segments: numberFromEnv("SEGMENTS", 4),

loadEnv(import.meta.url)

Loads the .env file next to the config file (not the one at the repo root). Using import.meta.url anchors the lookup to THIS file's directory — so running from the repo root with --config=./projects/X/... still loads projects/X/.env. Every project should have its own .env.

Config validation rules

Enforced by Zod at build time:

• All string fields are trimmed (region, tableName, bucket, endpoint, creds, preset). A trailing-space paste error doesn't silently corrupt anything.
• Whitespace-only rejected — empty-after-trim is treated as missing.
• Source/target collision guard:
  • Same S3 bucket on both sides → rejected (would overwrite source files).
  • Same region + same DDB / OS-DDB table name → rejected (would read and write to the same table). Same table name across DIFFERENT regions is allowed — distinct physical tables.

Pointing at a preset

pipeline.preset takes one of:

• A built-in name: "v5-to-v6-ddb" or "v5-to-v6-os" (filename in src/presets/ without extension). The runner auto-discovers built-ins.
• A file path: "./presets/my-preset.ts" or "../shared/presets/foo.ts". Resolved relative to the CONFIG file's directory.

pipeline.modelsDir — CMS model definitions

Required by the OS preset (v5-to-v6-os) and by built-in transformers that inspect field types (fixBrokenStorageKeys, transformRichText, addLiveField). Point at a directory of exported model definitions.

pipeline: {
    preset: "v5-to-v6-os",
    modelsDir: fromEnv("MODELS_DIR", "./models")
}

Three JSON shapes are accepted and can be mixed freely in the same directory:

The Webiny admin panel's Export button produces the {groups, models} shape. JSON models override DB-loaded models (user-provided definition takes precedence over what was scanned from the source DDB table).

Snapshot / debug (optional)

debug: {
    snapshot: true
    // or: snapshot: { dir: "./my-snapshot", compress: false }
}

Dumps every record the pipeline touches to local JSONL files. Useful for diffing source vs post-transform on a specific record without re-scanning AWS. Layout (one file per shard per pipeline per category):

• <dir>/<pipelineName>/segment-<n>.source.jsonl.gz — post-filter, pre-transform records.
• <dir>/<pipelineName>/segment-<n>.post-transform.jsonl.gz — after the transformer chain + onEnd.
• <dir>/<pipelineName>/segment-<n>.commands.jsonl.gz — every emitted command (PutRecord, S3Copy, custom).
• <dir>/dropped/segment-<n>.jsonl.gz — records that matched no pipeline filter.

Default dir: .transfer/<runId>/snapshot. Default compress: true. Best-effort — write errors log warn but never fail the transfer.

Snapshot files may contain production data. The default .transfer/ location is gitignored in this repo. If you override dir to a path outside .transfer/, add your override path to .gitignore yourself — these files typically contain full source records + transformed records + emitted commands, which are usually not things you want committed.

Persistent log file (debug.logFile)

debug: {
    logFile: true                // default: .transfer/<runId>/logs/<orchestrator|segment-N>.log
    // or: logFile: "./my-transfer.log"
}

Writes raw pino JSONL to disk in addition to stdout. With true, each process gets its own file (orchestrator + one per worker), so concurrent appends can't interleave. With a string path, all processes append to the same file — fine for low-throughput orchestrator logs, risky when worker parallelism is high.

Post-run inspection: cat .transfer/<runId>/logs/*.log | pino-pretty. Default path is under .transfer/ (gitignored in this repo). Custom paths are the user's gitignore responsibility.

Tuning (optional)

tuning: {
    flushEvery: numberFromEnv("FLUSH_EVERY", 500), // records per shard flush — bounds peak memory
    ddb: { maxRetries: 3, initialBackoffMs: 100 },
    s3:  { concurrency: 10, maxRetries: 3, initialBackoffMs: 100 },
    os:  {
        maxRetries: 3,
        retryScheduleMs: [5000, 10000, 20000, 30000, 30000],
        gzipConcurrency: 16
    }
}

All optional; absent = built-in defaults. AWS SDK retryMode: "adaptive" is always on for DDB + S3 — it self-tunes backoff based on real throttle signals, so you usually don't need to tune these.

flushEvery caps peak per-shard memory. The runner calls processor.execute() every N records and resets the pending-commands buffer. Default 500 (≈ 5 MB at a 10 KB average record). Lower to 100 for tables with very large records (approaching the 400 KB DDB max).

Running it

From the user project root:

yarn transfer --config=./projects/<name>/ddb.transfer.config.ts

Or with a specific AWS profile pre-set in .env:

SOURCE_PROFILE=prod-reader
TARGET_PROFILE=staging-writer

Common patterns

• DDB first, then OS — run them as separate transfers with separate config files. They don't share state. DDB uses v5-to-v6-ddb; OS uses v5-to-v6-os. Both can share the same .env.
• Multiple target environments — duplicate the project folder under projects/ with different .env. Configs stay identical.
• Custom preset — if the built-in doesn't match your needs, write one (see writing-data-transfer-preset skill) and point pipeline.preset at its file path.

Anti-patterns

• process.env.X! bang-casting — loses the friendly "variable not set" error, and silently passes undefined to the AWS SDK on typos. Use fromEnv(name) or fromEnv(name, default) instead.
• Hardcoded credentials in the config file — .env* is gitignored; put secrets there. If .env ever grows a key that shouldn't be in source, make sure .gitignore covers it.
• Source and target point at the same table — rejected at build time, but worth re-checking manually before a real run: different regions OR different table names.