Omnioperator Expression Analysis

Expression Ecosystem Status Analysis Skill

Objective

Systematically scan three sources and produce OmniOperator/docs/expression-analysis/expression_analysis_report_<yyyymmdd>.md answering:

1. Which expressions has Omni implemented (both codegen and vectorization), and what types does each support?
2. Which Spark SQL expressions does Velox (industry reference) cover?
3. How many expressions does the Spark SQL standard expose, and in what categories?
4. What is the gap between Omni and Velox / Spark SQL standard, and how should it be prioritized?

Expression Data Flow

An expression travels through these layers from Spark to Omni execution:

Spark SQL standard expression catalog
        ↓  (Gluten maps Spark → Substrait)
ExpressionMappings.scala          ← which expressions Gluten attempts to push down
ExpressionTransformer.scala      ← Omni-specific transformations (timezone, seed, etc.)
OmniExpressionAdaptor.scala      ← Scala-layer runtime validation (whitelist, type checks)
        ↓  (pass → generate Substrait plan)
SubstraitToOmniPlan.cpp           ← C++ bridge: Substrait expression → Omni plan
        ↓  (Omni execution)
OmniOperator execution layer
  ├── vectorization/              ← ExprEval visitor + VectorFunction dispatch (SIMD)
  └── codegen/                    ← LLVM JIT compilation (native machine code)

Upstream reference:

velox/velox/functions/sparksql/   ← Velox Spark SQL expressions (industry baseline)

Key Methodology

Do not estimate coverage by file count. One Register*.cpp may register dozens of expression names (each with multiple type overloads), while one .h file may correspond to only one expression. The only reliable data source is the registration/ directory — extract actual RegisterFunction calls.

Type coverage is determined by registration helpers. RegistrationHelpers.h defines template helpers that batch-register type combinations. Whether a helper like RegisterUnaryDecimal is called determines if DECIMAL is supported:

Use global grep to confirm registration. To verify whether a specific expression is registered:

grep -r '"func_name"' OmniOperator/core/src/vectorization/registration/

No match = confirmed not registered.

Step 1: Set Up Output Directory

mkdir -p OmniOperator/docs/expression-analysis
REPORT=OmniOperator/docs/expression-analysis/expression_analysis_report_$(date +%Y%m%d).md

Step 2: Scan Omni Execution Layer

OmniOperator has two expression execution frameworks. Scan both to get complete coverage.

2a. Vectorization Framework (SIMD-based, ExprEval dispatch)

Primary source: OmniOperator/core/src/vectorization/registration/

This directory contains 18 Register*.cpp files, each binding a category of functions into the runtime registry.

ls OmniOperator/core/src/vectorization/registration/Register*.cpp

Read each RegisterXxx.cpp and extract RegisterFunction<...>(prefix + "func_name", ...) calls.

To confirm whether a specific expression is registered, use global grep (not file-by-file reading):

grep -r '"func_name"' OmniOperator/core/src/vectorization/registration/

Check ExprEval dispatch completeness — read OmniOperator/core/src/vectorization/ExprEval.cpp and find expression names in the FuncExpr dispatch block. This catches "registered but ExprEval has no dispatch entry" (incomplete wiring).

2b. Codegen Framework (LLVM JIT)

Source: OmniOperator/core/src/codegen/func_registry_*.cpp

The codegen framework has its own function registrations separate from vectorization. Scan these files to find expressions registered only in the JIT path:

ls OmniOperator/core/src/codegen/func_registry_*.cpp

Read each file and extract registered function names and type signatures. Note which expressions are:

• Only in codegen (not in vectorization)
• Only in vectorization (not in codegen)
• In both frameworks

2c. Aggregate Expressions

Read OmniOperator/core/src/vectorization/aggregator_factory.cpp (or similar path) for aggregate expression type variants. Aggregate expressions follow a separate dispatch path from scalar expressions.

Step 3: Scan Gluten Mapping Layer

This layer determines which Spark expressions are mapped to Omni, and what runtime restrictions apply.

3a. Expression Pushdown Intent

File: Gluten/gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala

This is the authoritative complete set of expressions Gluten attempts to push down. Three categories:

Extract: grep all Sig[ClassName](FUNCTION_NAME) lines, organize by category.

3b. Omni-Specific Expression Transforms

File: Gluten/backends-omni/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala

Contains Omni-specific transformers that inject extra parameters (timezone, seed, etc.) into the Substrait output. Check what transformations are applied and to which expressions.

3c. Runtime Restrictions (Scala Layer)

File: Gluten/backends-omni/src/main/scala/org/apache/gluten/expression/OmniExpressionAdaptor.scala

This file validates expressions at runtime. Extract:

• Whitelist-based expression validation logic
• Type restrictions (e.g. greatest limited to 2 parameters, concat only STRING type)
• Timezone/format restrictions (e.g. unix_timestamp/from_unixtime limited to GMT+08:00/Asia/Shanghai)

These restrictions cause "expression is registered but falls back for specific types at runtime."

3d. Substrait → Omni Bridge

grep -n "scalar_function\|functionName\|\"spark:" \
  Gluten/cpp-omni/src/substrait/SubstraitToOmniPlan.cpp | head -100

Check which Substrait-level expressions are wired to Omni implementations.

Step 4: Scan Velox Reference Baseline

Velox is Meta's vectorized execution engine and the industry reference for Spark SQL expression support.

Path: velox/velox/functions/sparksql/registration/

ls velox/velox/functions/sparksql/registration/Register*.cpp

Read each Register*.cpp and extract:

• registerFunction<FuncName, ReturnType, ArgTypes...>({"func_name"}) — type signature
• registerVectorFunction("func_name", signatures, ...) — vector expression
• Template parameters indicate supported types

Key files: RegisterMath.cpp, RegisterString.cpp, RegisterDateTime.cpp, RegisterArray.cpp, RegisterCollection.cpp, RegisterMap.cpp, RegisterHash.cpp, RegisterConversion.cpp, RegisterComparison.cpp, Register.cpp (top-level entry).

Step 5: Spark SQL Standard Expression Catalog

If spark/ exists locally:

grep -r "class \w\+\s\+extends.*Expression\|FunctionRegistration" \
  spark/sql/catalyst/src/main/scala/ --include="*.scala" -l | head -20

Otherwise, use the known standard categories as a reference framework:

Step 6: Build Comparison & Gap Analysis

6.1 Three-Way Expression Comparison

For each expression, mark its status across all sources:

Fill-in rules:

• Omni Impl: Step 2 scan results (note codegen vs vectorization)
• Gluten Mapping: Step 3a results
• Velox Impl: Step 4 scan results

6.2 Type Coverage Matrix

For expressions with type coverage gaps, list explicitly:

6.3 Gap Priority Analysis

🔴 High Priority — Gluten-mapped but Omni has no implementation

🟡 Medium Priority — Implemented but specific conditions cause runtime fallback

🟢 Low Priority — Not mapped by Gluten (Spark has it but doesn't attempt pushdown)

Step 7: Generate Report

Write the report to $REPORT using the language the user communicates in. Follow this structure:

# OmniOperator Expression Ecosystem Analysis Report
Generated: <date>
Scan branch: <branch> (OmniOperator / Gluten / Velox versions)
Commit: <short_commit_hash> (OmniOperator) / <short_commit_hash> (Gluten) / <short_commit_hash> (Velox)

## 1. Executive Summary
| Metric | Count | Data Source |
|--------|-------|-------------|
| Spark SQL standard expression count | ~XXX | Spark SQL standard |
| Expressions Gluten attempts to push down (scalar / aggregate / window) | XXX / XXX / XXX | ExpressionMappings.SCALAR/AGGREGATE/WINDOW_SIGS |
| Omni vectorized registered expressions (unique names) | XXX | vectorization/registration/*.cpp |
| Omni codegen registered expressions | XXX | codegen/func_registry_*.cpp |
| Omni vectorized total signatures (incl. type overloads) | XXX | Same |
| Omni aggregate expression type variants | XXX | aggregator_factory.cpp |
| Velox SparkSQL implemented expressions | XXX | velox/.../Register*.cpp |

Key conclusions (3-5 bullets highlighting the most important findings).

## 2. Scan Methodology
(Describe data sources — Omni execution layer, Gluten mapping layer, Velox reference layer)

## 3. Velox vs Omni Three-way Comparison
### 3.1 Velox Implemented Expressions (measured, by category)
### 3.2 Three-way Comparison Matrix (key difference expressions)

| Expression | Spark SQL | Gluten Mapping | Velox Impl | Omni Registered | Notes |
|------------|:---------:|:--------------:|:----------:|:---------------:|-------|

## 4. Gap Analysis
### 4.1 🔴 High Priority (Gluten-mapped but Omni has no implementation)
### 4.2 🟡 Medium Priority (implemented but specific conditions cause runtime fallback)
### 4.3 🟢 Low Priority (not mapped by Gluten — Spark has it but doesn't attempt pushdown)

## 5. Type Coverage Matrix
### 5.1 Scalar Expressions (vectorization framework, measured)
### 5.2 Aggregate Expressions (aggregator_factory, measured)

## 6. Priority Suggestions
### 6.1 High-value Fixes (impact production correctness)
### 6.2 Coverage Extensions (Velox has, Omni pending)

Caveats

1. File exists ≠ fully supported: a vectorization/functions/Foo.h file does not mean the ExprEval.cpp dispatch chain is wired, nor that SubstraitToOmniPlan.cpp handles the corresponding Substrait function name
2. Gluten mapping ≠ Omni implemented: ExpressionMappings.scala maps expressions to Substrait, but the C++ layer may have no corresponding conversion, causing runtime fallback
3. Type restrictions may be runtime: type checks in OmniExpressionAdaptor.scala throw UnsupportedOperationException at runtime
4. Two frameworks, different coverage: codegen and vectorization register expressions independently — an expression may exist in one but not the other
5. Helper function call chain determines type coverage: whether helpers like RegisterUnaryDecimal in RegistrationHelpers.h are actually called decides whether an expression supports DECIMAL — trace the full call chain