# Dolphindb Data Import

> DolphinDB data import guide for IoT and general scenarios. Covers CSV/TXT, JSON, binary, plugin-based (HDF5, Parquet, MySQL, ODBC), and IoT real-time ingestion (MQTT, OPC UA). Includes IOTDB engine, date/time conversion, array vector import, null handling, and binary record import. Does NOT cover database administration, permission management, stream compute engine configuration, or data export.

- Skill: `dolphindb/dolphindb-data-import` (Agent Skill, multi-file: 39 files)
- Install (CLI): `npx skillmds@latest add dolphindb/dolphindb-data-import`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dolphindb/dolphindb-data-import/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: dolphindb (https://skillmd.com/u/dolphindb)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dolphindb/dolphindb-data-import

---


## DolphinDB Data Import Guide

Use this skill when the user asks about importing, loading, ingesting, or
migrating data into DolphinDB, including CSV/text files, JSON, binary files,
external databases, Parquet/HDF5 files, or IoT real-time data streams.

### Import Method Decision Tree

1. **CSV / text file → memory table**: use `loadText` (single-thread) or `ploadText` (parallel, faster for files > 16 MB).
2. **CSV / text file → distributed table**: use `loadTextEx` (recommended, direct-to-disk, avoids OOM for large files).
3. **Very large CSV file (> memory)**: use `textChunkDS` + `mr` to chunk and import.
4. **JSON string**: use `fromStdJson` (standard JSON) or `parseJsonTable` (JSON to table).
5. **Binary file with string columns**: use `loadRecord(filename, schema)`. Schema must be provided by user.
6. **Binary file without string columns**: use `readRecord!(handle, holder)`. Preferred for pure numeric files — simpler and safer.
7. **Parquet file → memory table**: use `parquet::loadParquet(filePath, [schema], [columnsToLoad], [startRowGroup], [rowGroupNum])`. Supports partial column loading via `columnsToLoad` and row group selection via `startRowGroup`/`rowGroupNum`.
8. **Parquet file → distributed table**: use `parquet::loadParquetEx(dbHandle, tableName, partitionColumns, fileName, [schema], [columnsToLoad], [startRowGroup], [rowGroupNum], [transform])`. Note: no `sortColumns` parameter; if sortColumns is needed, use `parquet::loadParquet` + `replaceColumn!` + `append!`.
9. **HDF5 file → memory table**: use `hdf5::loadHDF5(fileName, datasetName, [schema], [startRow], [rowNum])`. Supports row range selection via `startRow` and `rowNum`.
10. **HDF5 file → distributed table**: use `hdf5::loadHDF5Ex(dbHandle, tableName, [partitionColumns], fileName, datasetName, [schema], [startRow], [rowNum], [transform])`. Note: no `sortColumns` parameter; if sortColumns is needed, use `hdf5::loadHDF5` + `replaceColumn!` + `pt.append!(data)`.
11. **External database (MySQL, ODBC, etc.)**: use the corresponding plugin.
12. **IoT real-time data (MQTT)**: use MQTT plugin to subscribe to broker topics and write to stream tables.
13. **IoT real-time data (OPC UA)**: use OPC UA plugin to subscribe to server nodes and write to stream tables.
14. **IoT point management**: use IOTDB engine with IOTANY column and latestKeyCache for millisecond-level latest-value queries.

### Quick Reference: Core Import Functions

| Function | Target | Key Feature |
|----------|--------|-------------|
| `loadText` | Memory table | Single-thread; schema, skipRows, delimiter, partial columns via schema.col |
| `ploadText` | Memory table | Parallel; faster for > 16 MB |
| `loadTextEx` | Database table | Direct-to-disk; transform, sortColumns, skipRows |
| `textChunkDS` | Chunks for mr | Splits large file; use with `mr` |
| `extractTextSchema` | Schema preview | Auto-detect column types and delimiter |
| `loadRecord` | Memory table | Binary import with string support; schema required |
| `readRecord!` | Memory table | Binary import, no string support; holder required |
| `parquet::loadParquet` | Memory table | columnsToLoad, startRowGroup, rowGroupNum |
| `parquet::loadParquetEx` | Database table | Direct-to-disk; columnsToLoad, startRowGroup, rowGroupNum, transform (no sortColumns) |
| `hdf5::loadHDF5` | Memory table | Schema override; startRow, rowNum |
| `hdf5::loadHDF5Ex` | Database table | Direct-to-disk; startRow, rowNum, transform (no sortColumns) |

### Critical Rules

- **loadText/loadTextEx do NOT support rowNum parameter** (⚠️ CRITICAL — #1 cause of wasted React rounds): `loadText` and `loadTextEx` do NOT have a `rowNum` parameter. To preview data, use `select top N * from loadText(...)` or load first then `select top N`. NEVER attempt to pass `rowNum` to these functions — it will cause an error and waste a React round.
- **Delimiter auto-detection**: When `extractTextSchema` returns all columns merged into a single STRING column, it means the delimiter was not correctly detected. Automatically try these delimiters in order: (1) Tab: `char(9)` (most common for TXT files), (2) Semicolon: `;`, (3) Pipe: `|`. Re-run `extractTextSchema(filename, delimiter=char(9))` etc. until columns are properly separated.
- **skipRows semantics**: `skipRows=N` skips the first N lines of the file, INCLUDING the header row. When using `skipRows > 0`, you MUST provide a schema parameter (from `extractTextSchema`) because the header row will be skipped. If the user wants to skip N data rows but keep the header, do NOT use `skipRows` — instead, load all data with `loadText`, then use `select * from t where rowNo(t) >= N` or filter after import. `skipRows` does NOT count the header row separately — it counts from line 1 of the file.
- **Array vector import**: When a CSV column contains delimited values (e.g., "1.0,2.0,3.0"), it should be imported as an array vector. Steps: (1) `extractTextSchema` → (2) modify schema type to `DOUBLE[]` (or `INT[]` etc.) → (3) use `loadText(path, schema=schema, arrayDelimiter=",")`. The `arrayDelimiter` parameter tells DolphinDB how to split the values within each cell.
- **temporalParse with insufficient digit count**: When converting INT time values (e.g., 93000000) to TIME/TIMESTAMP using `temporalParse`, the string representation must match the format pattern length exactly. If the INT value has fewer digits than the format pattern requires, use `lpad(string(col), N, "0")` to pad with leading zeros. `lpad` is safe: it only pads strings shorter than N, and leaves strings already at length N or longer unchanged. Example: INT `93000000` (8 digits) → `lpad(string(93000000), 9, "0")` → `"093000000"` → `temporalParse("093000000", "HHmmssSSS")` → 09:30:00.000. INT `100000000` (9 digits) → `lpad(string(100000000), 9, "0")` → `"100000000"` (unchanged) → `temporalParse("100000000", "HHmmssSSS")` → 10:00:00.000. Always check digit count before applying temporalParse.
- **File path MUST be provided first**: If the user's message does NOT contain a file path (no string matching patterns like `/path/`, `.csv`, `.txt`, `.parquet`, `.h5`, `.bin`), you MUST use `ask_user` to ask for the file path BEFORE doing anything else. Do NOT attempt to execute any DolphinDB scripts until you have a file path. This is the FIRST step of the workflow and cannot be skipped.
- **Always preview schema first**: use `extractTextSchema(filename)` to check auto-detected types before importing. If types are wrong, modify the schema and pass it to the import function.
- **Date literals in DolphinDB**: use `2024.01.01` format, NOT `'2024-01-01'`.
- **Long integer timestamps**: import as LONG first, then convert with `timestamp()` function. Do NOT specify LONG timestamps as TIMESTAMP type directly in schema — it will produce NULL values.
- **Timezone handling**: use `localtime()` or `convertTZ()` to convert UTC timestamps to local time after import.
- **Header with numbers**: if column names start with digits, set `containHeader=true` to preserve names (system prefixes "c").
- **Partition conflict**: avoid parallel writes to the same partition. Use serial import or set `atomic='CHUNK'`.
- **Partition size**: recommended 100 MB to 1 GB per partition (uncompressed).
- **Null value handling**: use `nullFill`/`nullFill!` (fill with specific value), `ffill` (forward fill), `bfill`/`bfill!` (backward fill), `interpolate` (linear/pad/nearest/krogh interpolation), or `lfill!` (linear fill between endpoints). In `loadTextEx`, use `transform` parameter for one-step null filling during import.
- **FLOAT and DOUBLE cannot be used as partition columns**; convert STRING to SYMBOL for partitioning.
- **Script blocks are not function libraries**: copy script blocks from examples and replace placeholder variables (file paths, database names, table names); do not run entire example files, do not define or call custom functions.
- **Interactive decision-making**: when creating a distributed table, use ask_user to confirm storage engine and partition scheme before proceeding. Provide options with a recommended default. Do NOT ask about things you can determine yourself (e.g., file format from extension, schema from extractTextSchema, import method from file size).
- **Partial column loading**: For text files, use schema's `col` column to specify column indices (must be ascending). For Parquet, use `columnsToLoad` parameter (integer vector of zero-based column indices). For HDF5, load all columns then select.
- **Row skipping**: For text files, use `skipRows` (0-1024). For binary files, use `skipBytes` in `loadRecord` or `offset`/`length` in `readRecord!`. For Parquet, use `startRowGroup` and `rowGroupNum` to read specific row groups. For HDF5, use `startRow` and `rowNum` parameters in `loadHDF5`/`loadHDF5Ex`.
- **Binary file schema is mandatory**: DolphinDB cannot auto-detect binary file structure. ALWAYS obtain schema from user before using `loadRecord` or `readRecord!`. If user provides the writeRecord script, derive schema from it.
- **Data import workflow**: Follow the 10-step workflow (file path → format → metadata → user confirm → import method → preprocessing → final form → execute → verify → summary). Skip steps only when information is already provided or can be reliably inferred.
- **Schema confirmation is MANDATORY (Step 4)**: NEVER skip schema confirmation, even when all types appear correct and no conversion is needed. The user may want to adjust types, and their confirmation takes priority.
- **Memory table variable name confirmation**: When importing to a memory table, you MUST ask the user to confirm the variable name (e.g., `t` in `t = loadText(...)`). Suggest a default name based on the file name (e.g., `trades` for `trades.csv`). Before assigning, check if the variable name already exists using `try { objByName("varName", false) } catch(ex) { ... }`. If the name exists: (i) Ask the user whether to overwrite (use `undef(\`varName)` to delete the old variable, then reassign) or rename (provide a new name and validate uniqueness again). (ii) NEVER overwrite a variable without user confirmation. **⚠️ API note**: `existsShareVariable` only checks shared variables — for local variables, use the `try { objByName(...) } catch(ex)` pattern. To delete a local variable, use `undef(\`varName)`.
- **Existing database/table handling (⚠️ CRITICAL)**: Before creating a distributed table, check if the database and table already exist. Use `existsDatabase("dfs://path")` to check database, then `existsTable("dfs://path", "tableName")` to check table. **⚠️ API pitfall**: `existsTable` and `listTables` take a **path string** (e.g., `"dfs://sensor_db"`), NOT a `database()` object. `db.listTables()` will cause "db isn't an instance of a class" error — always use `listTables("dfs://path")` instead. If the database exists: (i) If `existsTable` returns false (table name differs from any existing table), import directly into the existing database without dropping it. (ii) If `existsTable` returns true (table name matches an existing table), determine the import mode by analyzing the prompt content: **Append import mode** (prompt explicitly mentions "append"/"追加", OR the table existed before this test flow — e.g., user pre-created the table for historical data accumulation): verify schema consistency between source data and existing table, adjust source data structure to match existing table schema, then append data. If schema mismatch or import fails, create a new table (preferably in the same database) and import there. **New table import mode** (prompt does NOT mention append, OR the table was created by a previous executor attempt in the same test flow): use `truncate(dbPath, tableName)` to clear existing data before import to avoid duplication from repeated executor attempts. In interactive mode, ask the user whether to append or overwrite. NEVER drop an existing database or table without explicit user confirmation.
- **Non-interactive mode (⚠️ CRITICAL — for automated testing/batch processing)**: When the user's initial prompt provides ALL necessary information (file path, target form, database path, table name, engine, partition scheme, schema, type conversions, etc.), SKIP all `ask_user` confirmation steps and execute directly. If the target database already exists and the table name conflicts, **use a new database path** (e.g., append a suffix) instead of dropping the old database — this avoids irreversible operations. **⚠️ Testing platform restriction**: The DolphinX testing platform PROHIBITS `dropDatabase` and `dropTable` operations. NEVER use these functions in testing scenarios — always use `existsDatabase` + `existsTable` conditional creation logic, or use a unique database path to avoid conflicts. **⚠️ Import mode handling when table exists**: When the table already exists, determine the import mode by analyzing the prompt: (i) If the prompt explicitly mentions "append"/"追加" → **append import mode**: verify schema consistency between source data and existing table (column count, partition column inclusion, column type compatibility), adjust source data structure to match existing table schema, then append data. If schema mismatch or import fails, create a new table (preferably in the same database with a new name) and import there. (ii) If the prompt does NOT mention append → **new table import mode**: use `truncate(dbPath, tableName)` to clear existing data before import to avoid duplication from repeated executor attempts within the same test flow. `truncate` preserves the table structure and only clears data — it is NOT a drop operation and is safe to use on the testing platform. If `truncate` fails, append to existing data and report a warning.
- **DB/Table existence 4-branch handling (⚠️ CRITICAL)**: When using `loadTextEx` + `transform` (or any import function that requires a pre-created table), MUST handle all 4 combinations of database/table existence:
  ```sql
  dbExists = existsDatabase(dbPath)
  tableExists = iif(dbExists, existsTable(dbPath, tableName), false)
  if (!dbExists) {
      // Branch 1: DB doesn't exist — create new DB and table
      db = database(dbPath, VALUE, ..., engine="OLAP")
      db.createPartitionedTable(schemaTable, tableName, partitionColumns)
  } else if (dbExists && !tableExists) {
      // Branch 2: DB exists but table doesn't — create table in existing DB
      db = database(dbPath)
      db.createPartitionedTable(schemaTable, tableName, partitionColumns)
  } else {
      // Branch 3 & 4: DB and table both exist
      db = database(dbPath)
      // Determine import mode by analyzing prompt:
      // - Append import (table existed before this test flow): verify schema, then append
      // - New table import (table from previous executor attempt): truncate to avoid duplication
      // Non-interactive mode defaults:
      // - If prompt mentions "append"/"追加": append mode (verify schema, then append)
      // - Otherwise: truncate mode (clear data, then import fresh)
      try { truncate(dbPath, tableName) } catch(ex) { /* truncate failed, append to existing data */ }
  }
  loadTextEx(db, tableName, partitionColumns, filePath, schema=schema, transform=transformFunc)
  ```
  Without this 4-branch logic, repeated runs will fail with "transform requires pre-created table" or "database already exists" errors. **⚠️ Data duplication prevention**: In non-interactive mode, if the table already exists and the prompt does NOT explicitly request append import, use `truncate(dbPath, tableName)` to clear existing data before import. This prevents data duplication caused by repeated executor attempts within the same test flow. `truncate` preserves the table structure and only clears data — it is NOT a drop operation and is safe on the testing platform. If `truncate` fails (e.g., permission error), append to existing data and report a warning.
- **transform function mutable rules (⚠️ CRITICAL)**: The `transform` function parameter is a constant reference by default.
  - Using `update t set col = ...` syntax: NO `mutable` keyword needed (recommended, simpler)
  - Using `replaceColumn!(t, ...)` or other mutable functions: parameter MUST be declared as `mutable t`
  - Example 1 (update syntax): `def cleanTransform(t) { update t set volume = nullFill(volume, 0); return t }`
  - Example 2 (mutable syntax): `def convertMarketData(mutable t) { replaceColumn!(t, `ts, timestamp(exec ts from t)); return t }`
  - WRONG: `def convert(t) { replaceColumn!(t, `col, ...) }` — will fail with "Constant variable [t] can't be used as argument for mutable function replaceColumn!"
- **transform requires pre-created table (⚠️ CRITICAL)**: When using `transform` parameter with `loadTextEx`/`loadParquetEx`/`loadHDF5Ex`, the partitioned table MUST be created BEFORE calling the import function using `db.createPartitionedTable(schemaTable, tableName, partitionColumns, [sortColumns])`. `loadTextEx` with `transform` does NOT auto-create the table. Without `transform`, `loadTextEx` auto-creates the table.
- **TSDB engine requires sortColumns (⚠️ CRITICAL)**: When using `engine="TSDB"`, `sortColumns` parameter is MANDATORY in `createPartitionedTable` and `loadTextEx`. If `sortColumns` is not needed, use `engine="OLAP"` instead. **OLAP engine does NOT support array vector types** (DOUBLE[], INT[]) — for array vectors, MUST use TSDB with `sortColumns`.
- **Engine-aware import verification (⚠️ CRITICAL — row count equality check fails for PKEY / TSDB+LAST/FIRST)**: After import, NEVER verify data correctness by simply comparing `afterCount - beforeCount == expectedImportCount` for ALL engines — this equality check is ONLY valid for non-overwrite scenarios (OLAP engine, or TSDB with `keepDuplicates=ALL`). For PKEY engine or TSDB with `keepDuplicates=LAST`/`keepDuplicates=FIRST`, the engine OVERWRITES existing rows with the same primary key (PKEY Merge-on-Write) or DEDUPLICATES rows with the same `sortColumns` value (TSDB+LAST/FIRST keeps only the last/first occurrence), so `afterCount - beforeCount` may be LESS than `expectedImportCount` even though the import succeeded. **For these overwrite/deduplication scenarios, MUST use the three-step key-existence check (Strategy B)** detailed in Step 9 of the workflow: (B1) `afterCount >= beforeCount`; (B2) `afterCount - beforeCount <= expectedImportCount`; (B3) extract unique key combinations via `select distinct keyCols from sourceData`, extract the same key combinations from the target table, use `ej(uniqueKeys, targetKeys, `keyCols)` to find matched keys, and verify `count(matchedKeys) == count(uniqueKeys)`. **⚠️ NEVER rely on `tableInsert`/`append!`/`loadTextEx` return values for verification**: For PKEY engines and key-value tables, `tableInsert` return value only counts NEW rows inserted (NOT rows updated due to key conflicts); `append!` and `loadTextEx` do not return inserted count at all. Always use SQL `select count(*)` and key-existence checks instead.
- **Key columns for engine-aware verification**: Determine the key columns for Strategy B based on the storage engine used at table creation time. (i) **PKEY engine**: use the columns declared in the `primaryKey` parameter of `createPartitionedTable`. (ii) **TSDB engine with keepDuplicates=LAST/FIRST**: use the columns declared in the `sortColumns` parameter. If `sortColumns` contains only ONE column that coincides with the partition column, append the partition column to form a composite key. (iii) For OLAP engine or TSDB with keepDuplicates=ALL, Strategy A (row count difference equality) is sufficient — no key-existence check needed.
- **sortColumns format (⚠️ CRITICAL)**: `sortColumns` accepts column name vector, NOT comma-separated string.
  - Single column: `sortColumns=`date`` or `sortColumns="date"` (string scalar)
  - Multiple columns: `sortColumns=`date`sym`` (backtick vector, recommended)
  - Variable reference: `sortColumns=colName` (variable value is string or symbol)
  - WRONG: `sortColumns="date,sym"` (interpreted as single column name "date,sym")
  - WRONG: `sortColumns=["date","sym"]` (string array, should use backticks)
- **loadHDF5Ex partition column type CANNOT be transformed (⚠️ CRITICAL)**: `loadHDF5Ex` uses the HDF5 file's schema to create the partitioned table, so the partition column type is FIXED to the HDF5 type (e.g., INT). The `transform` function only transforms data AFTER it's loaded, but the partition scheme is determined at table creation time.
  - If HDF5 has INT `tradingDay` but database requires DATE partition: `loadHDF5Ex` will ALWAYS fail with "partitioning column type doesn't match"
  - Solution: MUST use `hdf5::loadHDF5` + `replaceColumn!` + `createPartitionedTable` (with correct DATE type) + `pt.append!`
  - Same rule applies to `parquet::loadParquetEx`
- **IOTDB create table syntax (⚠️ CRITICAL — #1 cause of IOTDB import failure)**: IOTANY column can ONLY be created via SQL `create table` statement, NOT via `table()` function or `createPartitionedTable`.
  - Full syntax (SQL statement executed directly in DolphinDB script):
    ```sql
    create table `dfs://iot_db`.`sensors` (
        deviceId INT,
        location SYMBOL,
        timestamp TIMESTAMP,
        value IOTANY
    )
    partitioned by deviceId, timestamp
    sortColumns=`deviceId`location`timestamp
    latestKeyCache=true
    ```
  - Identifiers MUST use backticks (\`), NOT double quotes (")
  - `partitioned by` is followed by comma-separated column names (NOT backtick vector)
  - `sortColumns` uses backtick vector
  - `latestKeyCache=true` is a key-value pair connected with `=`
  - Common error: `schema must be provided to create table` indicates syntax parsing failure — check backticks and keywords
- **HDF5 plugin: ls vs lsTable (⚠️ CRITICAL)**: `hdf5::ls` and `hdf5::lsTable` return DIFFERENT column structures:
  - `hdf5::ls(path)` returns table with columns [objName, objType] — for exploring file structure
  - `hdf5::lsTable(path)` returns table with column [tableName] — for listing datasets
  - DO NOT mix column names: `ls` has no `name` or `tableName` column; `lsTable` has no `objName` or `objType` column
  - Recommended: use `hdf5::lsTable(path)` to get dataset names, then use `tableName[0]` as scalar
- **getLoadedPlugins() column names (⚠️ CRITICAL)**: `getLoadedPlugins()` returns a TABLE with EXACTLY these columns: [plugin, version, user, time].
  - Correct: `exec plugin from getLoadedPlugins() where plugin = "HDF5"`
  - WRONG: `exec name from getLoadedPlugins()` (no `name` column)
  - WRONG: `exec pluginName from getLoadedPlugins()` (no `pluginName` column)
  - WRONG: `getLoadedPlugins().has_key("HDF5")` (returns table not dict)
- **HDF5 plugin: datasetName must be string scalar (⚠️ CRITICAL)**: `hdf5::extractHDF5Schema(path, datasetName)` and `hdf5::loadHDF5(path, datasetName)` require `datasetName` to be a STRING SCALAR, not a vector.
  - WRONG: `dsName = exec tableName from datasets limit 1` (returns vector)
  - Correct: `dsName = exec tableName from datasets limit 1; dsName = dsName[0]` (take first element as scalar)
  - Correct: `dsName = datasets.tableName[0]` (directly take first element)
- **dict() construction rules (⚠️ CRITICAL)**:
  - Syntax: `dict(keyVector, valueVector)` or `dict(\`key1\`key2, [val1, val2])`
  - key and value count MUST be equal
  - WRONG: `dict(["a","b"], [v1])` (count mismatch)
  - WRONG: `dict(["a","b"], [exec count(*) from pt, exec sum(x) from pt])` (embedding exec SQL causes parser confusion)
  - Correct: `rowCount = exec count(*) from pt; dict(["rows"], [rowCount])` (store in variable first)
  - WRONG: `dict(["a","b"] as \`a\`b, ...)` (as syntax not applicable to dict)
- **DolphinDB reserved words (⚠️ CRITICAL)**: DO NOT use these as variable names:
  - `name`, `type`, `tuple`, `dict`, `table`, `select`, `exec`, `from`, `where`, `by`, `update`, `insert`, `delete`
  - Use `colName` instead of `name`, `dataTuple` instead of `tuple`
  - In schema tables, `name` and `type` are column names, not variable names
- **DolphinDB has NO .limit() method (⚠️ CRITICAL)**: Tables in DolphinDB do NOT have a `.limit()` method. To get top N rows:
  - Correct: `select top 10 * from tableName`
  - Correct: `tableName[:10]` (take first 10 rows)
  - WRONG: `tableName.limit(10)` (no such method)
  - WRONG: `tableName.head(10)` (no such method)
- **DolphinDB function name pitfalls**: DolphinDB built-in functions are LOWERCASE and case-sensitive. Common mistakes:
  - `strLen` → should be `strlen`
  - `IsValid` → should be `isvalid` (but isvalid doesn't exist, use other methods)
  - `Count` → should be `count`
  - `Exists` → should be `existsDatabase`/`existsTable`/`existsShareVariable`
  - `valid()` function does NOT exist — use `t.size() > 0` or `count(t) > 0` to check vector emptiness
- **loadParquetEx parameter count (⚠️ CRITICAL)**: `parquet::loadParquetEx` accepts 4~9 arguments ONLY. Parameter order: `(dbHandle, tableName, partitionColumns, fileName, [schema], [columnsToLoad], [startRowGroup], [rowGroupNum], [transform])`. `transform` is the 9th (last) parameter. Do NOT pass more than 9 arguments.
- **schema() function usage**: `schema(table)` returns a schema object (NOT a table).
  - Correct: `schema(pt)` returns directly
  - Correct: `schema(pt).colDefs` gets column definition table
  - WRONG: `select name, type from schema(pt)` (schema() cannot be used in FROM clause)
- **extractTextSchema returns metadata table, NOT schema table (⚠️ CRITICAL)**: `extractTextSchema` returns a table with columns [name, type, ...] describing the schema. To use it with `createPartitionedTable`, convert it to an empty table first: `schemaTable = table(1:0, schema.name, schema.type)`. NEVER pass `extractTextSchema` result directly to `createPartitionedTable`.
- **Derived partition column**: When the source data has no suitable partition column but has a timestamp column, derive a DATE column from the timestamp for partitioning.
  - Example: `addColumn(t, `trade_date, DATE); t[`trade_date] = date(exec timestamp from t)`
  - Then use `trade_date` as the partition column
- **File deletion in DolphinDB**:
  - `rmdir(path)`: deletes EMPTY directory only, NOT files
  - `deleteFile(path)`: deletes a file (correct function for file deletion)
  - For temporary files, use `try { deleteFile(path) } catch(ex) {}` pattern
- **Plugin function parse-time recognition**: Plugin functions with `::` prefix (e.g., `parquet::extractParquetSchema`, `hdf5::lsTable`) are recognized at PARSE time. If the plugin is not loaded before parsing, the function call will fail with "Cannot recognize function".
  - Solution 1: Use `use pluginName` after `loadPlugin` to import namespace, then call bare function names
  - Solution 2: Ensure `loadPlugin` is called BEFORE the script block containing `::` functions
  - Solution 3: For compile probe, wrap plugin calls in try/catch or use `use` approach
- **JSON serialization limitations**: `toStdJson()` and `internalToStdJson` do NOT support all data forms:
  - Supported: STRING, INT, DOUBLE, BOOL, DATE, TIMESTAMP and other scalars and vectors
  - NOT supported: PAIR, MATRIX, DICT, SET and other complex data forms
  - Solution: convert complex types to simple vectors before serialization, or use `string()` conversion

### Confirmation Strategy

Data import involves irreversible operations. Follow these confirmation rules:

**Mandatory Confirmation (must stop and ask user before proceeding):**
- If the user has not provided a file path: ask via ask_user to provide one. If the user's message does NOT contain a file path (no string matching patterns like `/path/`, `.csv`, `.txt`, `.parquet`, `.h5`, `.bin`), you MUST use `ask_user` to ask for the file path BEFORE doing anything else. Do NOT attempt to execute any DolphinDB scripts until you have a file path. This is the FIRST step of the workflow and cannot be skipped. If the user does not provide a file path, end the workflow gracefully.
- If the user has not specified the target form (memory table vs distributed table): ask via ask_user. Default is memory table.
- **Memory table variable name confirmation**: When importing to a memory table, you MUST ask the user to confirm the variable name (e.g., `t` in `t = loadText(...)`). Suggest a default name based on the file name (e.g., `trades` for `trades.csv`). Before assigning, check if the variable name already exists using:
  ```sql
  try { objByName("varName", false); varExists = true } catch(ex) { varExists = false }
  ```
  If the name exists, ask the user whether to overwrite (use `undef(\`varName)` then reassign) or rename. NEVER overwrite a variable without user confirmation.
- Before creating a distributed table: output engine selection, partition scheme, schema mapping, and confirmation question; wait for user confirmation before executing CREATE DATABASE/TABLE. NEVER assume a database path — always ask the user to confirm.
- **Existing database/table handling (⚠️ CRITICAL)**: Before creating a distributed table, check both database and table existence. Use the following code pattern:
  ```sql
  dbExists = existsDatabase("dfs://path")
  tableExists = iif(dbExists, existsTable("dfs://path", "tableName"), false)
  ```
  **⚠️ API pitfall**: `existsTable("dfs://path", "tableName")` and `listTables("dfs://path")` take a **path string**, NOT a `database()` object. `db.listTables()` will cause "db isn't an instance of a class" error. If the database exists: (i) If `existsTable` returns false (table name differs from any existing table), import directly into the existing database without dropping it. (ii) If `existsTable` returns true (table name matches an existing table), ask the user whether to overwrite (drop and recreate) or append. NEVER drop an existing database or table without explicit user confirmation.
- Before binary file import: output inferred schema and data preview; wait for user confirmation that the schema is correct before executing loadRecord or readRecord!.
- **Schema confirmation is MANDATORY (Step 4)**: After parsing metadata, present the COMPLETE schema (with auto-detected time-type suggestions) to the user via ask_user and wait for confirmation. NEVER skip this step, even when all types appear correct and no conversion is needed — the user may want to adjust types, and their confirmation takes priority.

**Conditional Confirmation (stop and ask when unclear):**
- Storage engine is not specified and multiple reasonable choices exist.
- Partition scheme has multiple reasonable options — present specific choices based on data columns and selected engine (e.g., VALUE(time_col), HASH(category_col, N), COMPO(VALUE + HASH)). For IOTDB, only COMPO with time as LAST dimension.
- Column type is ambiguous: STRING columns containing delimited values that may be array vectors (e.g., "1.0,2.0,3.0"), or LONG/INT columns containing values that look like dates (8-digit integers) or timestamps (10-13 digit integers). Use ask_user to confirm the intended type and any conversion format.
- Date/time format cannot be inferred from data sample.
- Null handling strategy may affect business semantics.

**Can Continue But Must Explain:**
- File format inferred from extension — state the inference basis.
- Schema correction following skill rules (e.g., LONG timestamp → timestamp()) — state what was corrected.
- Import method auto-selected by file size — state the selection basis.

### Quick Path

- When user instruction already specifies engine, partition scheme, or target table, skip corresponding confirmation points and proceed directly.
- When user requests quick import, use defaults (OLAP engine + VALUE partition) and skip non-mandatory confirmations; state which confirmations were skipped.
- When user provides the writeRecord script that created a binary file, derive schema from that script directly; no need to ask about schema.

### Workflow

Follow these steps in order. For each step, if the information is already provided by the user or can be reliably inferred, proceed without asking. If confirmation is needed, use ask_user.

1. **Confirm file path**: Verify the data file path provided by the user. If the user's message does NOT contain a file path (no string matching patterns like `/path/`, `.csv`, `.txt`, `.parquet`, `.h5`, `.bin`), you MUST use `ask_user` to ask for the file path BEFORE doing anything else. Do NOT attempt to execute any DolphinDB scripts until you have a file path. This is the FIRST step of the workflow and cannot be skipped. If the user does not provide a file path, end the workflow gracefully.
2. **Identify file format**: Auto-detect format from file extension (.csv, .txt, .bin, .parquet, .h5/.hdf5) or file header. If ambiguous, ask the user.
3. **Parse metadata**: For text files, use `extractTextSchema(filename, [delimiter])` to preview column names, types, and delimiter. For Parquet, use `parquet::extractParquetSchema`. For HDF5, use `hdf5::extractHDF5Schema` (after loading plugin and exploring with `hdf5::lsTable`). For binary files, there is no auto-detection function — the schema MUST be provided by the user (see Binary File Schema below).
4. **Auto-detect time-type columns and present schema for user confirmation**: Apply the Time-Type Auto-Detection rules below to identify columns that may need type conversion. Present the COMPLETE schema (including auto-detected type suggestions) to the user via ask_user. The user MUST confirm the schema before proceeding. This step is MANDATORY — never skip it, even when all types appear correct and no conversion is needed. The user may want to adjust types, and their confirmation takes priority.
5. **Confirm target form**: If the user has not specified the target form, ask: memory table or distributed table? Default is memory table if not specified. If memory table:
   - Ask user to confirm the variable name (e.g., `t` in `t = loadText(...)`). Suggest a default name based on the file name (e.g., `trades` for `trades.csv`).
   - Before assigning, check if the variable name already exists using:
     ```sql
     try { objByName("varName", false); varExists = true } catch(ex) { varExists = false }
     ```
     If the name exists, ask the user whether to overwrite (use `undef(\`varName)` then reassign) or rename. NEVER overwrite a variable without user confirmation.
   If distributed table:
   - Ask user to confirm database path and table name (suggest defaults based on file name). NEVER assume a database path without user confirmation.
   - Before creating, check both database and table existence using:
     ```sql
     dbExists = existsDatabase("dfs://path")
     tableExists = iif(dbExists, existsTable("dfs://path", "tableName"), false)
     ```
     **⚠️ API pitfall**: `existsTable` and `listTables` take a **path string** (e.g., `"dfs://sensor_db"`), NOT a `database()` object. `db.listTables()` will cause "db isn't an instance of a class" error — always use `listTables("dfs://path")` instead.
     If the database exists: (i) If `existsTable` returns false (table name differs from any existing table), import directly into the existing database without dropping it. (ii) If `existsTable` returns true (table name matches an existing table), ask the user whether to overwrite (drop and recreate) or append via ask_user. NEVER drop an existing database or table without explicit user confirmation.
   - Ask user to confirm storage engine (OLAP/TSDB/PKEY/IOTDB) with recommended default.
   - Ask user to confirm partition scheme with specific options based on data columns.
6. **Confirm import method**: Based on file format and target, select the appropriate import function:
   - Text → memory table: `loadText` or `ploadText`
   - Text → distributed table: `loadTextEx`
   - Text → very large file: `textChunkDS` + `mr`
   - Binary with string columns: `loadRecord`
   - Binary without string columns: `readRecord!` (preferred)
   - Parquet → memory table: `parquet::loadParquet`
   - Parquet → distributed table (no sortColumns needed): `parquet::loadParquetEx`
   - Parquet → distributed table (sortColumns needed): `parquet::loadParquet` + `replaceColumn!` + `pt.append!(data)`
   - HDF5 → memory table: `hdf5::loadHDF5`
   - HDF5 → distributed table (no sortColumns needed): `hdf5::loadHDF5Ex`
   - HDF5 → distributed table (sortColumns needed): `hdf5::loadHDF5` + `replaceColumn!` + `pt.append!(data)`
7. **Data preprocessing**: Apply any user-specified or default preprocessing before or during import:
   - Specify date/time format for date columns (via schema `format` column or `replaceColumn!` after import)
   - Select specific columns to import (via schema `col` column for text files, `columnsToLoad` for Parquet)
   - Skip rows (via `skipRows` for text files, `skipBytes` for binary files)
   - Handle null values (via `transform` parameter with `nullFill!`, `ffill!`, etc.)
   - For Parquet: specify `startRowGroup` and `rowGroupNum` to read specific row groups; specify `columnsToLoad` to load specific columns
   - For HDF5: use `startRow` and `rowNum` parameters in `hdf5::loadHDF5`/`hdf5::loadHDF5Ex` for efficient row range selection
8. **Execute import**: Run the import script. If errors occur, diagnose and fix (e.g., schema mismatch, partition conflict, plugin not loaded).
9. **Verify result**: Execute `select count(*) from tableName` (NOT `tableName.count()`) and `select top 10 * from tableName`. For distributed tables, `tableName.count()` may return 0 or incorrect results — always use SQL `select count(*)`.

   **⚠️ Verification mode depends on import mode (determined by prompt content and table existence check — see "Existing database/table handling" and "Non-interactive mode" rules)**:

   **Non-append import mode (DEFAULT — new table OR truncate-then-import)**: After import, simply verify `exec count(*) from loadTable(dbPath, tableName) == expectedImportCount` and check `select top 10 * from loadTable(dbPath, tableName)` for data correctness. Since the table was either newly created or cleared with `truncate(dbPath, tableName)` before import, there are NO pre-existing rows to conflict with — the final row count should exactly equal the source data row count for ALL storage engines (OLAP, TSDB, PKEY, IOTDB). No beforeCount/afterCount tracking or key-existence check is needed in this mode.

   **Append import mode (⚠️ engine-aware verification REQUIRED — ONLY when prompt explicitly mentions "append"/"追加")**: When data is appended to an existing table that already contains rows, the simple row-count equality check may FAIL for overwrite/deduplication engines. In append mode, MUST use the engine-aware verification mechanism based on the storage engine, because `tableInsert`/`append!`/`loadTextEx` return values are unreliable for PKEY/key-value tables (PKEY engine only counts NEW rows, not updates from key conflicts; `append!` and `loadTextEx` do not return inserted count at all):

   **Strategy A — Row count difference check (non-overwrite engines)**: Use when the storage engine is `OLAP`, or `TSDB` with `keepDuplicates=ALL` (default). Every appended row is added without deduplication, so the row count difference accurately reflects imported rows:
   ```sql
   beforeCount = exec count(*) from loadTable(dbPath, tableName)
   // ... execute append import ...
   afterCount = exec count(*) from loadTable(dbPath, tableName)
   expectedImportCount = /* row count of source data */
   // Verify exact equality:
   if (afterCount - beforeCount == expectedImportCount) { /* success */ }
   ```

   **Strategy B — Key existence check (overwrite/deduplication engines)**: Use when the storage engine is `PKEY`, or `TSDB` with `keepDuplicates=LAST` or `keepDuplicates=FIRST`. In append mode, the engine may OVERWRITE existing rows with the same primary key (PKEY) or KEEP only the last/first occurrence (TSDB+LAST/FIRST), so `afterCount - beforeCount` may be LESS than `expectedImportCount` — the row count difference equality check (Strategy A) WILL FAIL. Use

…(truncated)