Migrate Meltano Connector

Context: This plugin is for the Fivetran Connector SDK (CSDK). "CSDK" is shorthand for "Connector SDK".

Migrate a Meltano Extractor or Singer Tap to CSDK

FIRST: Read sdk-reference.md from the plugin directory to load SDK rules, operation names, schema constraints, configuration rules, and testing requirements.

Use this workflow to convert Meltano extractor logic into a Connector SDK project. Meltano migration references:

• Meltano project/config docs: https://docs.meltano.com/concepts/project/
• Meltano plugin docs: https://docs.meltano.com/concepts/plugins/
• Singer spec reference: https://hub.meltano.com/singer/spec/
• Meltano Singer SDK docs: https://sdk.meltano.com/
• Fivetran CLI for pipeline resources: https://pypi.org/project/fivetran-cli/

When validating the migration workflow itself, use representative public projects such as Matatika/example-github-analytics, which exercises:

• A Meltano project with tap-github, target-postgres, dbt transforms, a utility plugin, a job, and a schedule.
• Selector rules like *!.* followed by explicitly selected streams.
• Array settings such as repositories.
• A third-party Singer SDK tap whose source code must be consulted for schemas, primary keys, parent-child stream context, pagination, and state behavior.

Step 1: Locate the Source and Target

If the user did not provide a source path, ask for one of:

• A Meltano project directory containing meltano.yml.
• A custom Singer tap package or local tap source directory.
• Singer artifacts: config.json, catalog.json or properties.json, state.json, and sample output.

Find or create the target CSDK project:

• If the current directory already has connector.py, configuration.json, and requirements.txt, migrate in place.
• If no CSDK project exists, ask for the target directory name, then scaffold it with fivetran init before editing files.
• Do not overwrite unrelated files.

Scope the migration:

• Migrate Meltano extractors/Singer taps only.
• Do not port Meltano loaders/targets, dbt transforms, schedules, orchestration, or environments into CSDK connector code.
• For pipeline resources that correspond to Fivetran platform objects, create a follow-up plan using fivetran-cli instead of embedding that behavior in connector.py.
• Use loaders, transforms, schedules, and environments as context for what the extractor produced, how it was configured, and which Fivetran CLI resources may need to be created or updated.

Step 2: Inventory the Meltano Source

Read source files and identify:

• Meltano project files: meltano.yml, lock files under plugins/, .env templates, catalog files, state files, and any local tap source.
• Extractor name, namespace, executable, variant, pip_url, capabilities, and settings.
• Whether the tap is a local/custom tap, a third-party package, or a MeltanoHub discoverable extractor.
• Selected streams and fields from select, catalog metadata, or catalog selected flags.
• Selection rule semantics and precedence. For example, *!.* followed by issues.* means start with all streams deselected, then include the named streams.
• Stream schema from discovery/catalog JSON, tap source classes, or sample Singer SCHEMA messages.
• Primary keys from key_properties, table-key-properties, SDK stream primary_keys, or source code.
• Replication method and replication keys from metadata, SDK stream settings, or tap source code.
• State/bookmark shape from state.json, Singer STATE messages, or tap source code.
• Tap configuration fields, defaults, env aliases, and which fields are credentials.
• Complex settings such as arrays and objects. Prefer separate connector deployments for multi-entity settings; record JSON-encoded string handling only when it is needed for parity with the source tap.
• API/database/file access logic, pagination, rate limiting, retries, and error handling.
• Parent-child stream dependencies. Some selected child streams require parent stream API calls for context even when the parent stream is not selected for output.
• Pipeline intent from loaders, targets, transformations, schedules, and environments. Record this separately from extractor migration as potential fivetran-cli follow-up work.

Do not ask the user to paste credentials in chat. Do not print .env, config.json, or Meltano config values. Replace real values with placeholders if they appear in source files.

Step 3: Map Meltano/Singer Concepts to CSDK

Use this mapping:

Map JSON Schema types conservatively:

Full-table behavior needs explicit review. Singer taps commonly emit current rows without an explicit delete stream. If the old Meltano pipeline relied on a target to replace/truncate data for full-table streams, document that behavior and only use op.truncate(table=...) when the intended CSDK behavior is to mark missing prior rows as deleted before reloading a complete snapshot.

Step 4: Port the Implementation

Edit the CSDK project files:

connector.py

• Keep the scaffolded structure where present: validate_configuration(), schema(), update(), global connector = Connector(...), and if __name__ == "__main__": connector.debug().
• Port extraction logic directly into Python helper functions.
• If the source tap is already Python, reuse small pure extraction helpers where practical, but remove Singer CLI, stdout message emission, and Meltano runtime assumptions.
• If only a Meltano plugin declaration exists and no tap source is available, treat it as a behavior/configuration specification and reimplement from the source API/database/file protocol.
• Replace tap config access with configuration.
• Replace Singer state/bookmarks with the CSDK state dict. Preserve the original bookmark shape when it makes parity easier.
• Emit rows using SDK operations directly. Do not yield operations and do not print Singer JSON messages.
• Call op.checkpoint(state=state) after each safe page, batch, or stream boundary.
• Preserve selected stream behavior; do not silently add streams that were disabled in Meltano.
• Preserve incremental cursor semantics, including start-date behavior for empty state.
• Add retry handling for HTTP 429 and transient 5xx responses.
• Keep type hints simple: use dict and list; do not import Dict, Any, or use op.Operation.

configuration.json

• Keep flat string key/value pairs only.
• Include fields needed by the connector; preserve supplied values and use obvious placeholders for unresolved fields.
• Convert nested Meltano settings to clear flat names, and document any rename.
• Prefer separate connector deployments for multi-entity configs (see sdk-reference.md). Only when the source tap truly requires multi-item selection, represent Meltano arrays or objects as JSON-encoded string placeholders, such as "repositories": "[\"owner/repo\"]" or "searches": "[{\"name\":\"example\",\"query\":\"repo:owner/repo\"}]", then parse and validate them with json.loads() in connector.py.
• Do not include real credentials from .env, config.json, or Meltano config.
• Do not use arrays or nested objects.

requirements.txt

• Include only source-specific dependencies not already available in the CSDK environment.
• Do not include Meltano, Singer SDK, Singer target packages, or fivetran_connector_sdk.
• Do not include loader/target/dbt/orchestrator dependencies.

README.md

• Explain that this connector was migrated from a Meltano extractor/Singer tap.
• Document source extractor name, selected streams, primary keys, and cursor fields.
• Document configuration fields with placeholders only.
• Document any behavior changes, especially full-table replacement/truncate semantics, stream selection changes, and removed Meltano loader/transform/orchestration behavior.
• Document any pipeline resources that were intentionally left out of connector code and should be handled with fivetran-cli, such as destination/connection setup, transformations, or transformation projects.
• Follow Configuration entry in sdk-reference.md; use the SDK form or ordinary JSON values.

fivetran-cli follow-up notes

• If the Meltano pipeline includes loader/target configuration that maps to Fivetran destinations or connections, document the intended Fivetran CLI resource commands to run after connector migration.
• If the Meltano pipeline includes dbt transformations or transformation projects, document the intended Fivetran CLI transformation or transformation-project commands to run after connector migration.
• Do not execute account-changing fivetran-cli commands unless the user explicitly asks for deployment/setup and provides the required non-secret identifiers. Never ask the user to paste API keys in chat.

Step 5: Validate the Migration

Check behavior before testing:

• Every selected stream has a CSDK schema entry with a primary key.
• Disabled streams are not emitted as tables. If a disabled parent stream is needed for child context, it is used only as an internal helper path.
• Every Singer RECORD stream maps to op.upsert.
• Every incremental stream checkpoints after a safe cursor boundary.
• Array/object configuration fields are documented as JSON-encoded strings and parsed safely.
• Full-table streams have an explicit documented decision: upsert-only snapshot or op.truncate plus reload.
• JSON Schema to CSDK column type mapping is conservative and does not invent unsupported SDK types.
• No Singer stdout message generation remains.
• No Meltano loader, target, dbt, schedule, or environment code is required for local CSDK execution.
• Any Meltano pipeline resources that should survive migration are listed as fivetran-cli follow-up work instead of being embedded in connector code.
• No real credentials are present in source files, README, or chat.

Then follow the secure test flow from test-connector:

• Run the secure runner, not fivetran debug directly.
• If configuration values are missing, follow Configuration entry in sdk-reference.md; preserve supplied and existing values.
• Do not inspect or print configuration values.

Step 6: Report Results

Summarize:

• Source Meltano project or tap package migrated.
• Streams migrated, primary keys, and cursor fields.
• Configuration fields created or renamed.
• State/bookmark mapping.
• Full-table replacement/truncate decision.
• Meltano pieces intentionally not migrated into connector code, such as loaders, transforms, schedules, and environments.
• fivetran-cli follow-up plan for pipeline resources, if applicable.
• Remaining manual checks, especially API credentials, endpoint access, and data parity against the original Meltano run.