Clickzetta Data Science

clickzetta-data-science

See references/setup.md for environment setup and Jupyter kernel configuration. See references/zettapark-api.md for ZettaPark DataFrame API reference. See references/data-patterns.md for data discovery, quality, and feature engineering patterns. See references/stats-functions.md for statistical analysis SQL functions. See references/write-and-infer.md for writing results back to Lakehouse and model inference patterns. See references/bitmap-profile.md for BITMAP-based user profiling and audience segmentation.

Goal

Help the user complete their data science workflow end-to-end using ClickZetta Lakehouse as the data backend. Meet the user where they are — whether they're setting up for the first time or already mid-project.

Workflow

Assess state → Fill gaps → Execute work stage

Step 0: Assess where the user is

Before diving in, quickly determine:

• Environment ready? Python 3.10+, ZettaPark installed, Jupyter kernel configured?
• Which stage? Setup / data discovery / feature engineering / model inference?
• Experience level? First time with ZettaPark, or already familiar?

If environment is not ready, go to Setup first — ZettaPark requires Python 3.10+ and will silently fail on 3.9.

Step 1: Environment Setup (if needed)

See references/setup.md for full steps. Quick check:

python --version          # must be 3.10+
pip show clickzetta-zettapark  # check if installed

If not ready:

pip install clickzetta-zettapark jupyter

Then configure the Jupyter kernel to connect to Lakehouse (connection params: instance, workspace, vcluster, username, password).

Step 2: Project Structure

Use the Cookiecutter Data Science standard layout — keeps notebooks, data, and models organized:

{project}/
├── notebooks/          # Jupyter notebooks (exploration, EDA)
├── src/                # reusable Python modules
├── data/
│   ├── raw/            # original extracts (never modify)
│   └── processed/      # cleaned / feature tables
├── models/             # trained model artifacts
├── config.json         # Lakehouse connection config (gitignored)
└── .env                # secrets (gitignored)

Step 3: Data Discovery & Quality

Use ZettaPark or %%sql magic to explore tables. Key patterns in references/data-patterns.md:

• Schema and table discovery
• Row count, null rate, cardinality checks
• Distribution sampling with TABLESAMPLE
• Approximate statistics with approx_percentile, approx_count_distinct

Step 4: Feature Engineering

Two approaches — choose based on data volume:

• SQL-first (recommended for large tables): write feature logic as SQL, materialize as Lakehouse table
• ZettaPark DataFrame (for complex Python logic): use ZettaPark API, push computation to Lakehouse

See references/zettapark-api.md for DataFrame API patterns.

Step 5: Model Inference & Write-back

After training, write predictions back to Lakehouse. Three patterns in references/write-and-infer.md:

• UDF batch inference: register Python model as UDF, run inference in SQL
• BITMAP user profiling: high-performance audience segmentation — see references/bitmap-profile.md
• Vector search: store embeddings in Lakehouse, query with cosine similarity

Data Write Rules

• Always use CREATE TABLE IF NOT EXISTS — ClickZetta regular tables don't support CREATE OR REPLACE TABLE
• For incremental writes, use INSERT INTO or ZettaPark df.write.mode("append")
• For full refresh, TRUNCATE TABLE then INSERT INTO (safer than DROP + CREATE)

Unsupported ClickZetta SQL Syntax

For other syntax errors, load clickzetta-sql-migration to see Snowflake/Databricks/Spark vs. ClickZetta differences.

Routing