# Gx Configure Data Source

> Set up Great Expectations data access end to end — connect a data source, define a data asset, add a batch definition, and verify it by actually reading data through it. Use when a user asks to connect Great Expectations to their data (files, a SQL database or warehouse, or an in-memory dataframe), to add an asset or batch definition to a project they already have, or when other Great Expectations work is blocked because no working batch definition exists yet.

- Skill: `fivetran/gx-configure-data-source` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add fivetran/gx-configure-data-source`
- Raw SKILL.md: https://api.skillmd.com/api/skills/fivetran/gx-configure-data-source/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- License: Apache-2.0
- Author: fivetran (https://skillmd.com/u/fivetran)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/fivetran/gx-configure-data-source

---


# Configure a Great Expectations data source

This skill takes a user from "here is my data" to a **verified working batch
definition** — a named, saved way of pulling a specific slice of their data
that you have proven works by actually reading through it.

Three objects, in order, each built on the one before:

- A **data source** holds the connection: a directory, a connection string, or
  an in-memory handle.
- A **data asset** names the logical collection of data within it: a table, a
  query, a family of files, a dataframe.
- A **batch definition** selects how much of that asset a single operation
  reads: the whole thing, or one time window of it.

Everything here goes through Great Expectations' public configuration API and
produces ordinary project artifacts. Nothing depends on this skill being
present afterwards.

## The flow

1. **Preflight** — find out which project (or in-memory session) you are
   operating on, and tell the user. See `references/preflight.md`.
2. **Elicit** — the source type, the connection details, the asset, and the
   batching cadence.
3. **Configure** — data source, then asset, then batch definition, each with
   the reuse-first pattern below.
4. **Verify** — retrieve a batch and probe it. Retrieval alone proves nothing.
5. **Report** — say what was built and where it lives; offer write-out if the
   session is in memory.

Do not skip step 1, and do not stop before step 4. A configuration that was
never read through is not a result worth reporting.

## What this skill will not do without being asked

<!-- consent-gate: install -->
- **Install, upgrade, or remove a package, or otherwise modify the
  interpreter, virtual environment, environment variables, or shell state.**
  Not in response to an error, not preemptively while checking whether
  something is present, and not by announcing an intention to install with a
  chance to decline attached — none of those is the user asking. When
  something is missing, name it and hand over the command; running it is a
  separate act that starts only from the user's own instruction, later.
<!-- consent-gate: project -->
- **Create a project directory on the user's disk.** Only after the user has
  agreed to write the session out and named where.

## Step 1 — Preflight

Follow `references/preflight.md` in full before configuring anything. It
establishes whether you are working against a project on disk or an in-memory
session, tells you what to announce to the user, and covers the environment
problems that silently masquerade as "no project found".

The outcome you carry forward is a `context` object and one fact: whether the
session is file-backed or in memory. Both are fully supported paths.

If anything here, or in step 3 below, turns up a missing driver or client
library, report it and hand over the install command per the standing rule
above. Do not install it yourself — not now, before anything has failed, and
not later without the user's own go-ahead.

## Step 2 — Elicit what you need

Four things, in this order. Ask for them together where you can; don't
interrogate the user one field at a time.

**1. Which data source type.** Read the shipped catalog rather than working
from memory — see `references/datasource-catalog.md`. It gives you the exact
factory name for each type, the arguments that type accepts, and which of them
are mandatory. It also covers how to steer the conversation when the user
describes their data rather than naming a backend.

**2. The connection details** the chosen type's schema marks as required. Read
the secrets rule below before you write any of them down.

**3. What the asset is.** A table name, a SQL query, a file-name pattern, or a
dataframe variable in the user's session. Which of these are available depends
on the type; the catalog reference enumerates them.

**4. The batching cadence.** Whole collection, or sliced by day, month, or
year. Ask what the natural unit of the user's data is — if they check data
daily, a daily batch definition matches how they work.

**A user who expresses no preference has not chosen the whole collection.**
Don't let silence settle it. Carry the question forward to step 4 instead,
where the probe's own output tells you whether the data can carry a time slice
at all, and an open question becomes a recommendation: "`ordered_at` looks
like your time column — monthly?" is something a user can answer, where "how
would you like to batch this?" mostly is not.

If nothing in the data can carry a time slice, the whole collection is the
right answer and there is nothing to apologize for — say that you looked and
found nothing, so the user hears a finding rather than a default.

### Secrets: templates only, never values

Connection strings routinely carry passwords, tokens, and account
identifiers. Great Expectations already has a mechanism for this, and it is
the only one to use.

**Put a `${VARIABLE_NAME}` reference in the configuration; never the literal
value.** The reference is what gets stored, and it is resolved at connection
time:

```python
connection_string = "postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@warehouse.internal:5432/analytics"
```

Three rules follow from this, and none of them bend:

- **Never write a literal secret into a configuration, a file, or a message
  back to the user.** Not as a placeholder, not "just for testing", not
  inlined to get past an error. If you are holding a secret value, the only
  correct thing to do with it is tell the user which environment variable to
  put it in.
- **Never echo a resolved secret value.** Report the template
  (`${DB_PASSWORD}`), never what it resolved to.
- **If a required credential is missing, say which variable is missing and how
  to set it — then stop.** Do not guess a value, do not substitute a default,
  and do not fall back to an unauthenticated connection.

**Where the variable is read from depends on the session**, and the split
matters:

- An **in-memory session** resolves `${VARIABLE_NAME}` from process
  environment variables only. It has no project on disk, so it has no
  uncommitted config file to read from. If the user wants to supply the value,
  it has to be an environment variable.
- A **file-backed project** reads environment variables *and* an uncommitted
  config-variables file inside the project directory. Either works; the
  uncommitted file is the option for values that should be available to
  anyone opening that project.

Name whichever applies when you ask for a credential, so the user knows where
to put it. `references/write-out.md` covers what changes when an in-memory
session later becomes a project.

## Step 3 — Configure, reusing what already exists

The user may be adding to a project that already has some of this configured —
possibly from a teammate, a previous conversation, or an earlier run of this
same flow. **Fetch each object before creating it**, and create only what is
missing.

Adding a duplicate is not an option the API offers: `add_<type>_asset` and
`add_batch_definition_*` raise `ValueError: "<name>" already exists` on a
second call with the same name. There is no `add_or_update_*` variant for
assets or batch definitions.

All three fetch calls signal absence with a `LookupError` subclass, so a
single `except LookupError` is the correct catch for each:

| Fetch | Raises when missing |
| --- | --- |
| `context.data_sources.get(name)` | `KeyError` (a `LookupError`) |
| `datasource.get_asset(name)` | `LookupError` |
| `asset.get_batch_definition(name)` | `KeyError` (a `LookupError`) |

### Never "update" a data source that already exists

**`add_or_update_<type>` replaces a data source wholesale.** It does not merge.
Calling it against a name that already exists drops every asset and batch
definition attached to that data source — including ones this session never
created. A run that "updates" a data source in order to add one asset to it
destroys the user's other assets silently: no error, no warning, and the flow
carries on and reports success.

Verified directly: a data source carrying assets `['orders', 'products']` has
`[]` assets immediately after a second `add_or_update_sqlite` under the same
name.

So:

- Call the data-source factory **at most once per flow**, before any asset
  step, and **only when the data source does not already exist**.
- If the user genuinely wants to change the connection configuration of an
  existing data source, tell them first, in plain terms, that every asset and
  batch definition on it will be dropped and have to be rebuilt — then let
  them decide.

Note also that the factory **tests the connection as part of the call**, so it
can be slow or hang on an unreachable host. Run it inside the duration-tracked
wrapper in `references/robustness.md` like any other data-touching call.

That connection test is exactly where a missing driver or client library
surfaces. Report it and hand over the install command — do not install it
yourself to get the connection working, and do not offer to.

### The pattern

```python executable
DATASOURCE_NAME, ASSET_NAME, BATCH_DEFINITION_NAME = "warehouse", "orders", "by_month"

# --- data source: create only if absent; never replace an existing one ---
try:
    datasource = context.data_sources.get(DATASOURCE_NAME)
except LookupError:
    datasource = context.data_sources.add_or_update_sqlite(
        name=DATASOURCE_NAME,
        connection_string="sqlite:///${WAREHOUSE_PATH}",
    )

# --- asset: reuse, or delete and re-add when its configuration must change ---
replace_asset = False  # set True only if the user wants different asset config
try:
    asset = datasource.get_asset(ASSET_NAME)
    asset_exists = True
except LookupError:
    asset_exists = False

if asset_exists and replace_asset:
    datasource.delete_asset(ASSET_NAME)
    asset_exists = False
if not asset_exists:
    asset = datasource.add_table_asset(name=ASSET_NAME, table_name="orders")

# --- batch definition: same shape ---
replace_batch_definition = False
try:
    batch_definition = asset.get_batch_definition(BATCH_DEFINITION_NAME)
    batch_definition_exists = True
except LookupError:
    batch_definition_exists = False

if batch_definition_exists and replace_batch_definition:
    asset.delete_batch_definition(BATCH_DEFINITION_NAME)
    batch_definition_exists = False
if not batch_definition_exists:
    batch_definition = asset.add_batch_definition_monthly(
        name=BATCH_DEFINITION_NAME, column="ordered_at"
    )
```

Two things to keep intact when you adapt this:

- **The fetch and the create are separate statements, not a create inside the
  `except` of the fetch's own `try`.** Keeping them apart means a failure in
  the create is reported as a create failure rather than being swallowed by
  the same handler.
- **Deleting and re-adding an asset affects only that asset.** Its siblings on
  the same data source are untouched — this is exactly why the delete-and-add
  pattern is safe for assets and batch definitions while the data-source
  factory is not.

Set `replace_asset` / `replace_batch_definition` from what the user actually
asked for. If they want a different table, different files, or a different
partitioning column than what is already configured under that name, the
existing object has to be replaced. If they described what is already there,
reuse it and say so.

## Step 4 — Verify by reading through it

**Retrieving a batch does not prove the configuration works.** For a SQL query
or table asset, building the batch touches nothing — `get_batch()` returns a
real `Batch` object even when the table does not exist. Reporting success on
that basis means telling the user their setup works when it does not.

Always follow retrieval with a probe that actually reads data:

```python executable
batch = batch_definition.get_batch()  # add batch_parameters=... for a partitioned definition
head = batch.head(n_rows=5)           # this is the step that touches the data
```

`head` is a small object with a `.data` attribute holding a pandas DataFrame
of the sampled rows; printing it renders the rows directly.

**Run this probe inside the duration-tracked, exception-catching wrapper in
`references/robustness.md`.** Do not write your own — that reference already
handles the parts that are easy to get wrong: checking in with the user while
a slow query is still running rather than after it returns, never cancelling
work that continues to run and bill on the data platform, and recovering the
real database error out of the bare `KeyError` a broken probe raises.

Four outcomes, and three of them are not failures:

- **The probe returns rows** — the batch definition works. Proceed to step 5.
- **The probe returns zero rows**, with the expected column names — the
  configuration works and the underlying collection is empty. This is what an
  empty table or empty dataframe looks like: `head` returns a frame with the
  right columns and no rows. Say both things plainly. Do not report it as a
  configuration failure, and do not start changing the configuration to make
  rows appear.
- **`get_batch()` raises `NoAvailableBatchesError: No available batches
  found.`** — the configuration works, but the *window you asked for* holds no
  data. Both partitioned SQL assets and file-based assets fail this way when
  the requested year/month matches nothing, and it is the normal answer for a
  window outside the data's range. Report it as an empty window, name the
  batch parameters you used, and offer to try a window that exists — not as a
  broken setup.
- **The probe raises anything else** — the configuration does not work. Report
  it per `references/robustness.md`'s rules: what failed, why as far as it is
  known, one concrete next step. A broken table or query surfaces here as a
  bare `KeyError` with no readable message; that reference explains how to
  recover the real database error behind it. Do not report success, and do not
  retry with different parameters hoping something sticks.

**If the cadence is still open from step 2, close it here.** The probe's frame
is the survey you needed: `head.data.dtypes` names every column and its type,
so a date or timestamp among them is the recommendation to bring back. A
whole-collection definition standing where a time column exists should be
something the user chose, not something they were never offered — say what you
found, say what it would slice by, and let them decide. Switching is the
`replace_batch_definition` path in step 3, which leaves the asset and every
other batch definition on it untouched.

## Step 5 — Report, and offer write-out when in memory

Tell the user, concretely:

- The names of the data source, asset, and batch definition, and which of them
  you created versus reused.
- Where the configuration lives: the project's configuration directory for a
  file-backed session, or a plain statement that the session is in memory and
  nothing is saved yet.
- That the batch definition is verified, and what the probe returned — a row
  count and the columns is usually enough. Never paste a secret or a resolved
  credential into this report.
- How to retrieve a batch again, including the `batch_parameters` the batch
  definition needs.

<!-- consent-gate: project -->
**If the session is in memory, offer to write it out** to a real project so
the work survives — see `references/write-out.md` for the procedure and for
what the user needs to know about dataframe assets, which carry configuration
but no data. Offer it; don't do it unprompted, and don't pick the location.

**The offer ends this flow.** Write-out creates a project on the user's disk,
so it starts only from their reply — a separate run, after they have agreed
and named a directory. Reporting the result and then writing it out in the
same breath means they were never asked; putting the write-out call in the
same program as the configure and verify calls means the same thing, because
there was no point in it where an answer could have arrived.
`references/write-out.md` opens with the gate this depends on.

## Where this flow ends

**The verified batch definition is the end state.** Do not build a suite of
"smoke test" expectations to prove the setup works — the probe in step 4 has
already proven it, an invented expectation asserts something the user never
asked for, and a failing one would report a data-quality problem that is
really just a guess of yours.

When the user wants to assert things about their data, that is expectation
work: hand off to the `gx-configure-expectations` skill with the batch
definition you just verified. If they ask for both in one breath, finish this
flow, report it, and then move on.

## Worked examples

Each of these shows the object chain — data source, asset, batch definition,
verified batch. Preflight comes first in all three.

All three carry step 3's fetch-first guard on the data-source call, because
these are the blocks that get copied. A bare `add_or_update_<type>` sitting in
an example reads as the shape to follow, and followed against a name that
already exists it silently drops every asset on that data source — the caveat
in the prose above it does not survive the copy. The asset and
batch-definition calls are left bare to keep the per-backend differences
legible: those raise on a name that already exists rather than destroying
anything, and step 3 carries the reuse pattern for them.

Take the factory names and arguments for the user's actual backend from
`references/datasource-catalog.md`.

### A file-based source: monthly CSV files

The date lives in the file name, so the monthly batch definition takes a
`regex` with named groups. **Batch parameters are integers**, and they are not
padded to match the file name: `{"month": 2}` selects `sales_2024-02.csv`, even
though the regex reads that group as two zero-padded digits.

```python
try:
    datasource = context.data_sources.get("sales_files")
except LookupError:
    datasource = context.data_sources.add_or_update_pandas_filesystem(
        name="sales_files",
        base_directory="/data/sales",
    )
asset = datasource.add_csv_asset(name="monthly_sales")
batch_definition = asset.add_batch_definition_monthly(
    name="by_month",
    regex=r"sales_(?P<year>\d{4})-(?P<month>\d{2})\.csv",
)

batch = batch_definition.get_batch(batch_parameters={"year": 2024, "month": 2})
print(batch.head(n_rows=5))
```

### A SQL source: a table partitioned by month

The date lives in a column, so the monthly batch definition takes `column`.
Batch parameters are integers here too — the two families take the same window,
so one set of parameters drives both. The credential-bearing part of the
connection string is a `${VARIABLE_NAME}` reference, never a literal.

```python
try:
    datasource = context.data_sources.get("warehouse")
except LookupError:
    datasource = context.data_sources.add_or_update_postgres(
        name="warehouse",
        connection_string="postgresql+psycopg2://${DB_USER}:${DB_PASSWORD}@warehouse.internal:5432/analytics",
    )
asset = datasource.add_table_asset(name="orders", table_name="orders")
batch_definition = asset.add_batch_definition_monthly(name="by_month", column="ordered_at")

batch = batch_definition.get_batch(batch_parameters={"year": 2024, "month": 3})
print(batch.head(n_rows=5))
```

A query asset is the alternative when the logical collection is not a whole
table — `datasource.add_query_asset(name=..., query=...)`. Use it to express
what the data *is*, not to bolt a `LIMIT` onto a slow table; see
`references/robustness.md` for why row-limited query assets are a last-resort
exploration tool rather than a batching mechanism.

### An in-memory dataframe

A dataframe asset stores configuration only — the data itself is handed over
at retrieval time, every time, in this session and in every future one:

```python executable
try:
    datasource = context.data_sources.get("in_memory")
except LookupError:
    datasource = context.data_sources.add_or_update_pandas(name="in_memory")
asset = datasource.add_dataframe_asset(name="customers")
batch_definition = asset.add_batch_definition_whole_dataframe(name="all_rows")

batch = batch_definition.get_batch(batch_parameters={"dataframe": df})
print(batch.head(n_rows=5))
```

Say this out loud to the user when you configure one: the dataframe is not
saved, and anything reading this asset later must supply a dataframe itself.

## References

- `references/preflight.md` — establishing and announcing the session context.
- `references/datasource-catalog.md` — every configurable type, its factory,
  its arguments, and its asset and batch-definition surface, read from the
  shipped catalog.
- `references/robustness.md` — the time-budget wrapper, scope-reduction
  levers, and how to report a failure helpfully.
- `references/write-out.md` — turning an in-memory session into a real
  project.