Datasource Connectors

Datasource Connectors

Guide the user through loading data from any common source into a working dataset. This module is a decision framework, not a library. It tells the LLM what to ask, what to watch for, and how to handle each format's quirks.

When to Use

• The brief names a data source that is not yet loaded
• The user says "the data is in X" where X is a file, database, or API
• The orchestrator's plan includes an Ingest step

When to Skip

• Data is already loaded (inline table, prior step output, user pasted it)
• The brief says "data is pre-loaded" or "use this dataframe"

Connector Selection

Ask the user what format the data is in, or infer from the path/URL:

If ambiguous, ask. Do not guess the format.

CSV / TSV

What to ask

• Does the file have a header row? (default: yes)
• What is the delimiter? (auto-detect: comma, tab, semicolon, pipe)
• What is the encoding? (default: UTF-8; watch for: Latin-1, Windows-1252)

Quirks to handle

Output

A table with typed columns. Report: row count, column count, detected delimiter, detected encoding.

JSON / JSONL

What to ask

• Is it a single JSON object, an array, or newline-delimited (JSONL)?
• What path holds the data? (e.g., data.results[], root array)

Quirks to handle

Output

A flat table. Nested objects flattened to dot-notation columns. Report: record count, nesting depth, keys found.

REST API

What to ask

1. What is the endpoint URL?
2. What authentication is needed? (none, API key, OAuth, bearer token)
3. Does it paginate? (offset, cursor, link header, or single page)
4. What response path holds the data? (e.g., results[])

Authentication patterns

Never hardcode credentials. Use environment variables or secret storage.

Pagination patterns

Set a maximum page limit (default: 100 pages) to prevent runaway loops.

Retry and timeout

Output

A flat table from all pages combined. Report: total records, pages fetched, any errors skipped.

SQL

What to ask

1. What database engine? (PostgreSQL, SQL Server, MySQL, SQLite)
2. Connection string or host/port/database?
3. What table or query?

Security rules

Query patterns

-- Simple table read
SELECT * FROM schema.table_name LIMIT 1000;

-- Filtered read (parameterized)
SELECT * FROM sales WHERE region = @region AND date >= @start_date;

-- Aggregated read (push aggregation to the database when possible)
SELECT region, month, SUM(revenue) AS revenue
FROM sales
GROUP BY region, month;

Push aggregation and filtering to the database when possible. Pulling raw data and aggregating in the LLM wastes tokens and is slower.

Output

A table. Report: row count, column count, query execution time, database engine.

Excel

What to ask

1. Which sheet? (default: first sheet)
2. Does the data start at A1 or is there a header region to skip?
3. Are there named ranges?

Quirks to handle

Output

A table from the specified sheet and range. Report: sheet name, row count, column count, any merged cells detected.

Parquet / Arrow

What to ask

• Path to the file or directory (partitioned datasets use directories)

Advantages

Parquet is self-describing: schema, types, and encoding are embedded. No delimiter guessing, no encoding issues, no header detection.

Quirks to handle

Output

A typed table. Report: row count, column count, schema summary (types from the Parquet metadata).

Error Handling (all connectors)

Never fail silently. Every error must produce a specific, actionable message.

Output Contract

Pass to the next module (data-preparation):

• The loaded dataset (or a reference to it)
• A one-line summary: format, row count, column count, source path
• Any warnings (encoding fallback, partial data, skipped errors)