Datasource Creator

Datasource Creator

Create OData or flow-based datasource configurations for Datex Studio.

References

• ../datex-studio-shared/branch-setup.md -- Branch & connection selection (shared across skills)
• references/parameter-strategies.md -- Parameter strategies, linked datasources, quoting rules, cascading params
• references/datasources.md -- Datasource taxonomy: component variant (-datasource.json vs -footprintDatasource.json) × query type (OData vs flow), calling convention, tier restrictions, embedded datasources
• references/odata-datasources.md -- OData query-type platform reference: queryOptions tree, filter expressions, expands, result type shape, pre-flight schema validation, canonical skeleton
• references/flow-datasources.md -- Flow query-type platform reference: paginated vs single-result shapes, getFlow/getListFlow/getByKeysFlow slots, callsite syntax, entity-definition contract, canonical skeletons

Dependencies

REQUIRED BACKGROUND: Read the schema-explorer skill for OData entity discovery and the odata-execution skill for query building and verification.

Schema exploration is mandatory for ODATA datasources — The type definition must be built from validated schema, not from examples or templates alone. Flow datasources might aggregate data from OData queries under the hood but if it works with already existing datasources, we assume they are already verified.

Footprint connections — consult footprint-entity-expert first. If the datasource will query Footprint WMS entities (Tasks, Shipments, ArchivedShippingLicensePlateContents, LicensePlate, Lot, Material, etc.), check footprint-entity-expert before schema exploration. It documents which entity backs a given business concept, the correct navigation chain to Owner/Project, weight-calc branching, and pagination obligations — context that the OData metadata doesn't surface. Skip for non-Footprint connections (custom apps, non-WMS data).

Prerequisites check

Before starting schema exploration or datasource generation, check whether a requirements brief already exists in the conversation context (produced by requirements-gathering or report-creator).

• Requirements brief exists → use it. The brief provides the field list, semantic roles, and business rules that drive which entities to explore and which fields to include. Verify datasource output against the brief.
• No requirements brief exists → invoke the requirements-gathering skill first. This happens when datasource-creator is invoked standalone (not from report-creator). The brief ensures you don't miss fields, map entities incorrectly, or skip calculated fields.

Do NOT skip this check. Building a datasource without understanding what fields are needed and what they mean leads to incomplete configs and rework.

Input/Output Contract

Mode defaults to owned. If the caller didn't specify one, author the datasource owned and say so in the return summary. Switch to standalone only when an override applies — a second consumer genuinely needs the same query, the consumer type can't hold an owned datasource (selectors, lists, calendars, cards, widgets), a flow datasource must call it via $datasources.<Package>.<name>, or it ships in a library package. The rule and the full override list live in references/datasources.md → Owned by Default.

Steps by Mode

For a grid, editor, or form the owned column continues into the splice procedure in references/datasources.md → Creating an Owned Datasource; the host is validated in place of the standalone upsert.

Workflow

[requirements brief in context?]
        |
  +-----+-----+
  |            |
 YES          NO
  |            |
 use it    invoke `requirements-gathering` skill
  |            |
  +-----+------+
        |
[determine type: OData or Flow]
        |
  +-----+-----+
  |            |
 OData       Flow
  |            |
 schema ->   schema -> (REQUIRED: validate entities/properties
 query ->     the flow code will query against the connection)
 generate    create standalone OData datasources (generate + dxs configuration upsert datasource each)
  |          write type-def YAML (from validated schema, NOT from examples)
  |          write flow TS code (referencing $datasources.<RepoName>.<ref_name>)
  |          generate-flow
  |            |
  +-----+------+
        |
  context (dxs datasource context - get type defs)
  validate (dxs datasource validate - both modes)
        |
  +-----+-----+
  |            |
Standalone   Owned
  |            |
 dxs configuration upsert datasource
              return JSON file path
 fields       + ref name
 test data
 return ref

When to use OData vs Flow

How to tell: Run dxs report datasource-fields <ref> --branch <id> (or --report <ref> for owned). If the output has a collections: section with fields you need in standalone textboxes (not tables), use a flow datasource. Collections in OData datasources cannot be bound as flat DataSet fields — they silently resolve to blank.

The production pattern: All existing Datex Studio reports with complex navigation (packing slips, BOLs, master BOLs) use flow datasources for their header/detail data. The flow code fetches the OData entities and flattens collections into scalar fields. OData datasources are used for simple list queries (line items, lookup tables).

Flow datasources and schema exploration: Flow datasources are NOT a shortcut around schema exploration. A flow datasource's JavaScript code typically queries one or more OData entities and reshapes the data. Before writing the type definition or flow code, you MUST use schema-explorer to validate that the target connection has the expected entities and properties. Existing report examples and templates show what has worked on some connection — they are starting hypotheses, not validated designs for the current connection.

Return to Caller

After completing the workflow (either standalone or owned), return this structured summary to the calling skill or user:

1. Reference name — the -r value (e.g., ds_shipment_bol)
2. File path — the -o output path (e.g., reports/bol/ds_shipment_bol.json)
3. Mode — owned (local file, to splice into the host's datasources[] or register with --owned) or standalone (upserted). State which, and if standalone, which override justified it
4. Result type — single or collection (from the generated config's resultIsCollection)
5. in_params — list of input parameter names and types (from the config's inParams), or empty if none
6. Field summary — read the generated JSON config and extract the field tree from queryOptionsObjectTypeDef. List fields as name: type with dot-notation for nested objects. Mark collections. This gives the caller a machine-derived field list without re-reading the field-mapping artifact.

Example return:

Datasource: ds_shipment_bol
File: reports/bol/ds_shipment_bol.json
Mode: owned
Result type: single
in_params: shipmentId (number)
Fields:
  Id: number (key)
  BillOfLading: string
  LookupCode: string
  Carrier.Name: string
  Carrier.ScacCode: string
  Status.Name: string
  ShipmentLines [collection]:
    LineNumber: number
    OrderLine.Material.LookupCode: string
    OrderLine.Material.Description: string

Collection fields in the return require caller action. When the field summary contains [collection] markers, the caller must choose:

1. Flow datasource (preferred): Rewrite the datasource as a flow that flattens collections into scalar fields. The caller gets a flat field list with no collections.
2. Child datasets (alternative): Create a separate DataSet in the report with CommandText: "$.ds_name.result.CollectionPath.*" and use =First(Fields!Field.Value, "child_dataset") in standalone textboxes. This works but adds complexity.
3. Separate OData datasource: Query the collection entity directly (e.g., ShipmentOrderLookups?$filter=ShipmentId eq {id}) as its own datasource. Only viable when the entity supports direct filtering.

Never pass collection-path fields (e.g., OrderLookups.Order.OwnerReference) to dxs report dataset add --field as flat fields on a single-result DataSet. They will silently render blank.

OData Datasource Generation

Platform reference: references/odata-datasources.md covers the underlying Datex Studio file shape — queryOptions tree, filter expression syntax, expands, result type contract, single-object vs collection traps, and the canonical skeleton. Consult it when reasoning about why a generated config has the structure it does, when authoring or hand-editing JSON outside the generator, or when debugging import errors.

Generate an OData datasource config with dxs datasource generate:

dxs datasource generate \
  -c <connection_id> \
  -q '<odata_query>' \
  -r <reference_name> \
  -t "<reference_name>" \
  -d "<description>" \
  --api-setting-name <app_level_name> \
  -o ds_name.json \
  --branch <branch_id>

Key Flags

Parameter Strategy

Flow Datasource Generation

Platform reference: references/flow-datasources.md covers the underlying Datex Studio file shape — paginated vs single-result execution shapes, the getFlow/getListFlow/getByKeysFlow slots, callsite syntax via referenceName, the entity-definition output contract, and the canonical skeletons for both shapes (including the enum-dropdown pattern). Consult it when reasoning about which shape a use case needs, when authoring JSON outside the generator, or when fields are silently undefined at runtime.

Flow Runtime Model (CRITICAL)

A flow datasource draws its data from one of two legitimate sources, never from a raw OData query string: (a) standalone OData datasources already on the branch, via $datasources.<RepoName>.<ref_name> (covered below); or (b) cloud storage, via the function-tier $db.<Package>.<storage_referenceName> predicate API — the pattern that backs a grid or selector over a *-storage.json component. The $db path has its own authoring rules (paging, dynamic filter/sort, getQuery() factory); see ../db-query/references/flow-db-datasources.md. The rest of this section covers source (a).

Flow datasource code does NOT execute raw OData queries. Instead, flows reference standalone OData datasources that already exist on the branch, using the $datasources object. Standalone datasources are scoped under their repository module name (the repo's uniqueIdentifier name, e.g., PurchaseOrders, AsnOrders), so the path is $datasources.<RepoName>.<ref_name>:

// CORRECT: Reference standalone OData datasources with module scope, unwrap .result
const shipmentResp = await $datasources.PurchaseOrders.ds_shipment.get({ shipmentId: $flow.inParams.shipmentId });
const shipment = shipmentResp.result;
// For collection datasources, .result is an array:
const contactsResp = await $datasources.PurchaseOrders.ds_wh_contacts.get({ warehouseId: shipment.ActualWarehouse.Id });
const firstContact = contactsResp.result?.[0]?.Contact;
// Set output via $flow.outParams.result
$flow.outParams.result = { Name: shipment.Name, Phone: firstContact?.PrimaryTelephone };

// WRONG: Missing module scope — datasources are not at the root level
const result = await $datasources.ds_shipment.get({ shipmentId: 123 });

// WRONG: Raw OData queries in flow code — this is NOT how flows work
const result = await $datasource.getList({ query: 'Shipments(123)?$expand=Carrier' });

Flow code runtime variables:

Unwrapping responses: All $datasources calls return { result?: ... }. For --param-keys datasources, result is a single object. For collection datasources, result is an array. Always access .result before navigating into fields.

The workflow for building a flow datasource:

1. Create standalone OData datasources for each query the flow needs — use dxs datasource generate + dxs configuration upsert datasource for each
2. Write the flow code referencing those datasources via $datasources.<RepoName>.<ref_name>.get() or $datasources.<RepoName>.<ref_name>.getList() (where <RepoName> is the repository's name from dxs source repo list)
3. Generate the flow config with dxs datasource generate-flow, which embeds the flow code and type definition
4. The flow datasource itself is owned by default (embedded in its host), but its OData dependencies must be standalone on the branch — $datasources.<RepoName>.<ref_name> resolves only for standalone configs, so an embedded dependency is unreachable from flow code. This is one of the standing overrides in references/datasources.md → Owned by Default.

$datasources API:

Pass input parameters as an object — the keys must match the OData datasource's inParams exactly.

Pagination for high-volume collection datasources

OData responses are capped at 5,000 records per request. A flow that aggregates Tasks, ArchivedShippingLicensePlateContents, Shipments, or any other entity over a date range or multi-warehouse scope must paginate explicitly — there's no automatic continuation, and the truncation is silent.

To paginate, the standalone OData datasource the flow calls must declare a skip inParam, and its query must use $top=5000&$skip=${$datasource.inParams.skip}. Then the flow loops with getList({ ..., skip }), breaking when a short page comes back.

See ../datex-studio-shared/flow-code-patterns.md#odata-pagination--the-5000-record-cap for the full pattern — datasource query shape, the --detect-params requirement, and the canonical fetch loop. If you're building a flow datasource whose underlying queries could exceed 5k rows, build the underlying OData datasources with pagination wired from the start; retrofitting later requires regenerating both the datasource and the flow code.

--param-keys creates named params, NOT a keys array. When an OData datasource uses --param-keys (e.g., Shipments(0)), the generator creates inParams named after the entity key (e.g., shipmentId). Always check the generated config's inParams to get the exact param name. Never use { keys: [value] } — that pattern does not exist.

// CORRECT: Use the actual inParam name from the generated config
const resp = await $datasources.ds_shipment.get({ shipmentId: $flow.inParams.shipmentId });

// WRONG: There is no "keys" parameter — this causes TS compilation errors
const resp = await $datasources.ds_shipment.get({ keys: [$flow.inParams.shipmentId] });

Generation Command

Generate a flow datasource config with dxs datasource generate-flow:

dxs datasource generate-flow \
  -r ds_lookup -t "ds_lookup" -d "Custom lookup datasource" \
  --type-def types.yaml \
  --get-flow get.ts \
  --in-param id:number \
  -o ds_lookup.json --branch <branch_id>

Required: At least one flow method (--get-flow, --get-list-flow, or --get-by-keys-flow) and a type definition file (--type-def).

Key Flags

Type Definition YAML Format

- id: Id
  type: number
- id: Name
  type: string
- id: Items
  type: object
  isCollection: true
  objectTypeDef:
    - id: LineNumber
      type: number

Valid types: string, number, boolean, date, object, union, blob.

Enhancement flags (--dynamic-filter, --linked, --custom-column, etc.) work the same as OData datasources.

File size limit: All code files and the type definition file are limited to 512 KB.

Server-Tier Variant (FPDS — -footprintDatasource.json, typeId 19)

Everything above produces the cloud -datasource.json (configurationTypeId: 6), callable by functions and selectors. When the datasource must be called by an action (Footprint server tier), author the FootprintDatasource (FPDS) variant instead. Both OData and flow query types support it; the body shape is identical to the cloud variant — only the variant fields change.

The generator has no FPDS flag — generate the config exactly as above, then edit the produced JSON:

• OData FPDS: set configurationTypeId: 19. apiSettingName already names the branch's Footprint API connection (set by generate) — leave it. (One-field change.)
• Flow FPDS: set configurationTypeId: 19 and set apiSettingName to the branch's Footprint API connection setting name (generate-flow leaves it null; get the name from dxs source branch settings <branch_id> — conventionally FootprintApi, but branches may differ, e.g. fpapiconn).

Everything else — queryOptions / flow slots, outParams, keyDef, and the explicit null slots — is identical to the cloud variant. See references/datasources.md → Structural Deltas Between Variants.

Validate and push with the footprintdatasource CLI type (the dxs datasource … / dxs configuration upsert datasource path is hardwired to the cloud datasource/6 endpoint and cannot emit typeId 19):

dxs configuration validate footprintdatasource -b <branch_id> -D ds_name.json   # exit 1 = errors found
dxs configuration upsert  footprintdatasource -b <branch_id> -D ds_name.json

upsert resolves by referenceName (creates or updates). Delete with dxs configuration delete footprintdatasource <id> -b <branch_id> -y.

Selectors must never be backed by an FPDS — a selector backing must be the cloud -datasource.json variant (see references/datasources.md). FPDS targets are for action-tier callers.

Context Command

See ../datex-studio-shared/context-navigation.md for the full guide on retrieving and reading context responses, including backend vs frontend symbol filtering. To quickly confirm a $types.<Package>.* custom type or enum member exists without parsing the full context blob, use the nomenclature registry described there.

dxs -O json datasource context <file.json> --branch <branch_id>

Returns designer type definitions for writing expressions or TypeScript code. Works for both OData and flow datasources. Run after generate / generate-flow to understand the available fields before writing custom columns or flow code.

For datasources, the primary scope symbols are $entity, $ccentity, and $datasource (in flowContext or linkedDatasourcesContext). The appContext contains additional services — read defaultContext.imports to determine which ones are available (see the shared reference).

Validation

dxs datasource validate <file.json> --branch <branch_id>

Validates the datasource config against the branch. Run for BOTH standalone and owned modes before proceeding to upsert or report upload. This command validates the datasource in isolation — it does not catch consumer-shape mismatches, so a datasource whose shape no consumer can use still passes here.

The branch runs two separate server-side gates. Both fire at publish and via dxs configuration validate; this command only partly stands in for them:

• Usage gate — grid/selector require getList + getByKeys; editor/form require get on a single result; each linkedDatasources entry must match its link type (see references/flow-datasources.md → Linked datasource link types). It only fires when the consumer is validated (dxs configuration validate <consumer type>), so it says nothing about a datasource you validate on its own.
• Isolation gate — checks the datasource's own shape, so a mis-shaped datasource with no consumer at all no longer publishes clean the way it used to. It reaches FootprintDatasource configs too. The flow-slot rules are the part this repo documents, and this command's local lint reproduces them offline — see references/flow-datasources.md → Three Execution Shapes. The gate also checks OData query options and the outputs contract; those rules are not documented here yet.

Two things this command cannot check on a filtered query, both worth a minute before upsert: whether the filter column is indexed, and whether the predicate that bounds the query can vanish at runtime — every generate flag that parameterizes a filter wraps it in a $utils.isDefined() guard, so an absent parameter fails open against the whole table. See references/odata-datasources.md → Also check: is the filter column indexed? and → The index check is undone by a guarded scoping filter.

Reading the report

The command runs a local flow-shape lint and merges its findings with the server's, so one run reports both. Every item carries four fields:

Exit codes:

• Any error, local or server → exit 1, with validation_errors[] carrying errors and warnings merged into one list. That non-zero exit means validation found errors — read the payload, fix the config, re-validate. It does not mean the CLI broke, and it is not a reason to halt or retry the command unchanged.
• Warnings alone → exit 0, with validation_result: {status: "valid", warnings: [...]}. Still read them; they just don't block.

dxs configuration validate datasource and dxs configuration validate footprintdatasource run the same local lint and report the same merged shape — the two commands are deliberately kept in sync. Other config types report server findings only. Full exit-code matrix across the CLI (including dxs function validate, which still exits 0 on errors): ../datex-studio-shared/configuration-roundtrip.md.

If the server call itself fails while local findings exist, the report keeps the local findings and adds a validate API call failed (…) item with origin: server — the local findings are still real and still worth fixing.

Standalone Completion

After validation passes, complete the standalone workflow:

1. Upsert

dxs configuration upsert datasource -D <file.json> --branch <branch_id>

2. Verify Fields

dxs report datasource-fields <reference_name> --branch <branch_id>

Check these in the output:

1. in_params names -- must match exactly in --datasource-param (don't assume id)
2. result_type -- single vs list affects report layout
3. collections -- nav-property collections available for table sections
4. Field paths -- exact dot-notation paths for expressions

3. Discover Test Data

Query the entity without template literal params, using the base filter + $top=5 + $orderby to find recent records:

dxs odata execute -c <id> \
  -q 'Entity?$top=5&$filter=<base_filters>&$select=Id,<param_fields>&$orderby=<date_field> desc'

Verify count with $count=true&$top=1:

dxs odata execute -c <id> \
  -q 'Entity?$count=true&$top=1&$filter=<full_filter_with_real_values>&$select=Id'

4. Deleting a Datasource

Delete by reference name or by config ID:

dxs datasource delete ds_my_report --branch <branch_id>
dxs datasource delete --id 42 --branch <branch_id>

Use --id when reference-name lookup returns 404 (can happen on branches with component modules). Get the ID from dxs datasource list.

Naming Convention (CRITICAL)

The datasource reference name (-r), display title (-t), the RDLX-JSON DataSet name, and the --owned alias must ALL be identical:

-r ds_my_report -t "ds_my_report"              # generate: -t = -r
DataSet.Name = "ds_my_report"                  # RDLX-JSON
--owned ds_my_report.json:ds_my_report         # report datasource add: file:alias

Rules:

• Use ds_ prefix convention
• Must be valid JS identifiers (start with letter/_/$, no spaces/hyphens, no leading digits)
• Examples: ds_shipment_bol, ds_orders, ds_inventory_summary

Key Rules

• Use schema batch to combine multiple schema discovery calls into 2-3 requests instead of 9+ sequential calls
• Check composite keys -- some entities have multi-field keys that affect --param-keys behavior
• Use $top=1 during query testing -- large queries timeout without limits
• Single quotes for $ values -- shell expands $ in double quotes; use single quotes for -q, --linked, --custom-column, --datasource-param, --linked-param
• Always include $select in $expand -- bare $expand without $select pulls all fields

Common Mistakes

After your edit, invoke post-edit-verification to surface description/JSON/schema violations. For a final review, invoke component-validator.