SAP HANA Cloud Data Intelligence Skill
Related Skills
- sap-hana-ml: Use for Python ML client workflows that run against SAP HANA
- sap-datasphere: Use for Datasphere modeling, replication, and analytics data warehousing
- sap-btp-connectivity: Use for cloud-to-on-premise connection and destination prerequisites
- sap-btp-integration-suite: Use for integration scenarios that should move out of Data Intelligence pipelines
This skill provides documentation-audited guidance for developing with SAP Data Intelligence Cloud, including pipeline creation, operator development, data integration, and machine learning scenarios. The last_verified date is intentionally stale until product lifecycle status and live graph/runtime behavior are checked against primary sources.
Table of Contents
When to Use This Skill
Use this skill when:
- Creating or modifying data processing graphs/pipelines
- Developing custom operators (Gen1 or Gen2)
- Integrating ABAP-based SAP systems (S/4HANA, BW)
- Building replication flows for data movement
- Developing ML scenarios with ML Scenario Manager
- Working with JupyterLab in Data Intelligence
- Using Data Transformation Language (DTL) functions
- Configuring subengines (Python, Node.js, C++)
- Working with structured data operators
Common Issues
| Issue |
First check |
| Graph fails after operator change |
Confirm all operators use the same generation and compatible subengine. |
| ABAP/S4HANA connection fails |
Verify Cloud Connector, destination, and credential configuration. |
| Python or Node operator behaves differently in runtime |
Check subengine version, package availability, and serialization boundaries. |
Core Concepts
Graphs (Pipelines)
Graphs are networks of operators connected via typed input/output ports for data transfer.
Two Generations:
- Gen1 Operators: Legacy operators, broad compatibility
- Gen2 Operators: Enhanced error recovery, state management, snapshots
Critical Rule: Graphs cannot mix Gen1 and Gen2 operators - choose one generation per graph.
Gen2 Advantages:
- Automatic error recovery with snapshots
- State management with periodic checkpoints
- Native multiplexing (one-to-many, many-to-one)
- Improved Python3 operator
Operators
Building blocks that process data within graphs. Each operator has:
- Ports: Typed input/output connections for data flow
- Configuration: Parameters that control behavior
- Runtime: Engine that executes the operator
Operator Categories:
- Messaging (Kafka, MQTT, NATS)
- Storage (Files, HDFS, S3, Azure, GCS)
- Database (HANA, SAP BW, SQL)
- Script (Python, JavaScript, R, Go)
- Data Processing (Transform, Anonymize, Validate)
- Machine Learning (TensorFlow, PyTorch, HANA ML)
- Integration (OData, REST, SAP CPI)
- Workflow (Pipeline, Data Workflow)
Subengines
Subengines enable operators to run on different runtimes within the same graph.
Supported Subengines:
- ABAP: For ABAP Pipeline Engine operators
- Python 3.9: For Python-based operators
- Node.js: For JavaScript-based operators
- C++: For high-performance native operators
Key Benefit: Connected operators on the same subengine run in a single OS process for optimal performance.
Trade-off: Cross-engine communication requires serialization/deserialization overhead.
Quick Start Patterns
Basic Graph Creation
1. Open SAP Data Intelligence Modeler
2. Create new graph
3. Add operators from repository
4. Connect operator ports (matching types)
5. Configure operator parameters
6. Validate graph
7. Execute and monitor
Replication Flow Pattern
1. Create replication flow in Modeler
2. Configure source connection (ABAP, HANA, etc.)
3. Configure target (HANA Cloud, S3, Kafka, etc.)
4. Add tasks with source objects
5. Define filters and mappings
6. Validate flow
7. Deploy to tenant repository
8. Run and monitor
Delivery Guarantees:
- Default: At-least-once (may have duplicates)
- With UPSERT to databases: Exactly-once
- For cloud storage: Use "Suppress Duplicates" option
ML Scenario Pattern
1. Open ML Scenario Manager from launchpad
2. Create new scenario
3. Add datasets (register data sources)
4. Create Jupyter notebooks for experiments
5. Build training pipelines
6. Track metrics with Metrics Explorer
7. Version scenario for reproducibility
8. Deploy model pipeline
Common Tasks
ABAP System Integration
For integrating ABAP-based SAP systems:
- Prerequisites: Configure Cloud Connector for on-premise systems
- Connection Setup: Create ABAP connection in Connection Management
- Metadata Access: Use Metadata Explorer for object discovery
- Data Sources: CDS Views, ODP (Operational Data Provisioning), Tables
Reference: See references/abap-integration.md for detailed setup.
Structured Data Processing
Use structured data operators for SQL-like transformations:
- Data Transform: Visual SQL editor for complex transformations
- Aggregation Node: GROUP BY with aggregation functions
- Join Node: INNER, LEFT, RIGHT, FULL joins
- Projection Node: Column selection and renaming
- Union Node: Combine multiple datasets
- Case Node: Conditional logic
Reference: See references/structured-data-operators.md for configuration.
Data Transformation Language
DTL provides SQL-like functions for data processing:
Function Categories:
- String: CONCAT, SUBSTRING, UPPER, LOWER, TRIM, REPLACE
- Numeric: ABS, CEIL, FLOOR, ROUND, MOD, POWER
- Date/Time: ADD_DAYS, MONTHS_BETWEEN, EXTRACT, CURRENT_UTCTIMESTAMP
- Conversion: TO_DATE, TO_STRING, TO_INTEGER, TO_DECIMAL
- Miscellaneous: CASE, COALESCE, IFNULL, NULLIF
Reference: See references/dtl-functions.md for complete reference.
Best Practices
Graph Design
- Choose Generation Early: Decide Gen1 vs Gen2 before building
- Minimize Cross-Engine Communication: Group operators by subengine
- Use Appropriate Port Types: Match data types for efficient transfer
- Enable Snapshots: For Gen2 graphs, enable auto-recovery
- Validate Before Execution: Always validate graphs
Operator Development
- Start with Built-in Operators: Use predefined operators first
- Extend When Needed: Create custom operators for specific needs
- Use Script Operators: For quick prototyping with Python/JS
- Version Your Operators: Track changes with operator versions
- Document Configuration: Describe all parameters
Replication Flows
- Plan Target Schema: Understand target structure requirements
- Use Filters: Reduce data volume with source filters
- Handle Duplicates: Configure for exactly-once when possible
- Monitor Execution: Track progress and errors
- Clean Up Artifacts: Remove source artifacts after completion
ML Scenarios
- Version Early: Create versions before major changes
- Track All Metrics: Use SDK for comprehensive tracking
- Use Notebooks for Exploration: JupyterLab for experimentation
- Productionize with Pipelines: Convert notebooks to pipelines
- Export/Import for Migration: Use ZIP export for transfers
Error Handling
Common Graph Errors
| Error |
Cause |
Solution |
| Port type mismatch |
Incompatible data types |
Use converter operator or matching types |
| Gen1/Gen2 mixing |
Combined operator generations |
Use single generation per graph |
| Resource exhaustion |
Insufficient memory/CPU |
Adjust resource requirements |
| Connection failure |
Network or credentials |
Verify connection settings |
| Validation errors |
Invalid configuration |
Review error messages, fix config |
Recovery Strategies
Gen2 Graphs:
- Enable automatic recovery in graph settings
- Configure snapshot intervals
- Monitor recovery status
Gen1 Graphs:
- Implement manual error handling in operators
- Use try-catch in script operators
- Configure retry logic
Reference Files
For detailed information, see:
references/operators-reference.md - Complete operator catalog (266 operators)
references/abap-integration.md - ABAP/S4HANA/BW integration with SAP Notes
references/structured-data-operators.md - Structured data processing
references/dtl-functions.md - Data Transformation Language (79 functions)
references/ml-scenario-manager.md - ML Scenario Manager, SDK, artifacts
references/subengines.md - Python, Node.js, C++ subengine development
references/graphs-pipelines.md - Graph execution, snapshots, recovery
references/replication-flows.md - Replication flows, cloud storage, Kafka
references/data-workflow.md - Data workflow operators, orchestration
references/security-cdc.md - Security, data protection, CDC methods
references/additional-features.md - Monitoring, cloud storage services, scenario templates, data types, Git terminal
references/modeling-advanced.md - Graph snippets, SAP cloud apps, configuration types, 141 graph templates
Templates
Starter templates are available in templates/:
templates/basic-graph.json - Simple data processing graph
templates/replication-flow.json - Data replication pattern
templates/ml-training-pipeline.json - ML training workflow
Documentation Links
Primary Sources:
Section-Specific:
Bundled Resources
Reference Documentation
references/abap-integration.md - ABAP system integration guide
references/ml-scenario-manager.md - Machine Learning scenario manager
references/replication-flows.md - Data replication flow configuration
references/operators-reference.md - Complete operators reference
references/dtl-functions.md - Data Transformation Language functions
references/modeling-advanced.md - Advanced modeling techniques
references/structured-data-operators.md - Structured data operators guide
Documentation Links
Version Information
- Last Updated: 2025-11-27
- Evidence Status: Stale docs-only guidance; source refresh and live tenant/runtime checks pending
- Documentation Source: SAP-docs/sap-hana-cloud-data-intelligence (GitHub)
1---
2name: sap-hana-cloud-data-intelligence
3description: |
4 Develops data processing pipelines, integrations, and machine learning scenarios in SAP Data Intelligence Cloud. Use when building graphs/pipelines with operators, integrating ABAP/S4HANA systems, creating replication flows, developing ML scenarios with JupyterLab, or using Data Transformation Language functions. Covers Gen1/Gen2 operators, subengines (Python, Node.js, C++), structured data operators, and repository objects.
5license: GPL-3.0
6metadata:
7 source: https://skillmd.com/skills/secondsky/sap-hana-cloud-data-intelligence
8 maintainer: "Eduard Jiglau"
9 maintainer_email: "hello@sap-ai-skills.com"
10 website: "https://sap-ai-skills.com"
11 version: "2.4.1"
12 last_verified: "2025-11-27"
13 evidence_status: "stale_docs_only_pending_refresh"
14---
15
16# SAP HANA Cloud Data Intelligence Skill
17
18## Related Skills
19
20- **sap-hana-ml**: Use for Python ML client workflows that run against SAP HANA
21- **sap-datasphere**: Use for Datasphere modeling, replication, and analytics data warehousing
22- **sap-btp-connectivity**: Use for cloud-to-on-premise connection and destination prerequisites
23- **sap-btp-integration-suite**: Use for integration scenarios that should move out of Data Intelligence pipelines
24
25This skill provides documentation-audited guidance for developing with SAP Data Intelligence Cloud, including pipeline creation, operator development, data integration, and machine learning scenarios. The `last_verified` date is intentionally stale until product lifecycle status and live graph/runtime behavior are checked against primary sources.
26
27## Table of Contents
28- [When to Use This Skill](#when-to-use-this-skill)
29- [Core Concepts](#core-concepts)
30- [Quick Start Patterns](#quick-start-patterns)
31- [Common Tasks](#common-tasks)
32- [Bundled Resources](#bundled-resources)
33
34## When to Use This Skill
35
36Use this skill when:
37
38- Creating or modifying data processing graphs/pipelines
39- Developing custom operators (Gen1 or Gen2)
40- Integrating ABAP-based SAP systems (S/4HANA, BW)
41- Building replication flows for data movement
42- Developing ML scenarios with ML Scenario Manager
43- Working with JupyterLab in Data Intelligence
44- Using Data Transformation Language (DTL) functions
45- Configuring subengines (Python, Node.js, C++)
46- Working with structured data operators
47
48## Common Issues
49
50| Issue | First check |
51|-------|-------------|
52| Graph fails after operator change | Confirm all operators use the same generation and compatible subengine. |
53| ABAP/S4HANA connection fails | Verify Cloud Connector, destination, and credential configuration. |
54| Python or Node operator behaves differently in runtime | Check subengine version, package availability, and serialization boundaries. |
55
56## Core Concepts
57
58### Graphs (Pipelines)
59
60Graphs are networks of operators connected via typed input/output ports for data transfer.
61
62**Two Generations:**
63- **Gen1 Operators**: Legacy operators, broad compatibility
64- **Gen2 Operators**: Enhanced error recovery, state management, snapshots
65
66**Critical Rule**: Graphs cannot mix Gen1 and Gen2 operators - choose one generation per graph.
67
68**Gen2 Advantages:**
69- Automatic error recovery with snapshots
70- State management with periodic checkpoints
71- Native multiplexing (one-to-many, many-to-one)
72- Improved Python3 operator
73
74### Operators
75
76Building blocks that process data within graphs. Each operator has:
77- **Ports**: Typed input/output connections for data flow
78- **Configuration**: Parameters that control behavior
79- **Runtime**: Engine that executes the operator
80
81**Operator Categories:**
821. Messaging (Kafka, MQTT, NATS)
832. Storage (Files, HDFS, S3, Azure, GCS)
843. Database (HANA, SAP BW, SQL)
854. Script (Python, JavaScript, R, Go)
865. Data Processing (Transform, Anonymize, Validate)
876. Machine Learning (TensorFlow, PyTorch, HANA ML)
887. Integration (OData, REST, SAP CPI)
898. Workflow (Pipeline, Data Workflow)
90
91### Subengines
92
93Subengines enable operators to run on different runtimes within the same graph.
94
95**Supported Subengines:**
96- **ABAP**: For ABAP Pipeline Engine operators
97- **Python 3.9**: For Python-based operators
98- **Node.js**: For JavaScript-based operators
99- **C++**: For high-performance native operators
100
101**Key Benefit**: Connected operators on the same subengine run in a single OS process for optimal performance.
102
103**Trade-off**: Cross-engine communication requires serialization/deserialization overhead.
104
105## Quick Start Patterns
106
107### Basic Graph Creation
108
109```
1101. Open SAP Data Intelligence Modeler
1112. Create new graph
1123. Add operators from repository
1134. Connect operator ports (matching types)
1145. Configure operator parameters
1156. Validate graph
1167. Execute and monitor
117```
118
119### Replication Flow Pattern
120
121```
1221. Create replication flow in Modeler
1232. Configure source connection (ABAP, HANA, etc.)
1243. Configure target (HANA Cloud, S3, Kafka, etc.)
1254. Add tasks with source objects
1265. Define filters and mappings
1276. Validate flow
1287. Deploy to tenant repository
1298. Run and monitor
130```
131
132**Delivery Guarantees:**
133- Default: At-least-once (may have duplicates)
134- With UPSERT to databases: Exactly-once
135- For cloud storage: Use "Suppress Duplicates" option
136
137### ML Scenario Pattern
138
139```
1401. Open ML Scenario Manager from launchpad
1412. Create new scenario
1423. Add datasets (register data sources)
1434. Create Jupyter notebooks for experiments
1445. Build training pipelines
1456. Track metrics with Metrics Explorer
1467. Version scenario for reproducibility
1478. Deploy model pipeline
148```
149
150## Common Tasks
151
152### ABAP System Integration
153
154For integrating ABAP-based SAP systems:
155
1561. **Prerequisites**: Configure Cloud Connector for on-premise systems
1572. **Connection Setup**: Create ABAP connection in Connection Management
1583. **Metadata Access**: Use Metadata Explorer for object discovery
1594. **Data Sources**: CDS Views, ODP (Operational Data Provisioning), Tables
160
161**Reference**: See `references/abap-integration.md` for detailed setup.
162
163### Structured Data Processing
164
165Use structured data operators for SQL-like transformations:
166
167- **Data Transform**: Visual SQL editor for complex transformations
168- **Aggregation Node**: GROUP BY with aggregation functions
169- **Join Node**: INNER, LEFT, RIGHT, FULL joins
170- **Projection Node**: Column selection and renaming
171- **Union Node**: Combine multiple datasets
172- **Case Node**: Conditional logic
173
174**Reference**: See `references/structured-data-operators.md` for configuration.
175
176### Data Transformation Language
177
178DTL provides SQL-like functions for data processing:
179
180**Function Categories:**
181- String: CONCAT, SUBSTRING, UPPER, LOWER, TRIM, REPLACE
182- Numeric: ABS, CEIL, FLOOR, ROUND, MOD, POWER
183- Date/Time: ADD_DAYS, MONTHS_BETWEEN, EXTRACT, CURRENT_UTCTIMESTAMP
184- Conversion: TO_DATE, TO_STRING, TO_INTEGER, TO_DECIMAL
185- Miscellaneous: CASE, COALESCE, IFNULL, NULLIF
186
187**Reference**: See `references/dtl-functions.md` for complete reference.
188
189## Best Practices
190
191### Graph Design
192
1931. **Choose Generation Early**: Decide Gen1 vs Gen2 before building
1942. **Minimize Cross-Engine Communication**: Group operators by subengine
1953. **Use Appropriate Port Types**: Match data types for efficient transfer
1964. **Enable Snapshots**: For Gen2 graphs, enable auto-recovery
1975. **Validate Before Execution**: Always validate graphs
198
199### Operator Development
200
2011. **Start with Built-in Operators**: Use predefined operators first
2022. **Extend When Needed**: Create custom operators for specific needs
2033. **Use Script Operators**: For quick prototyping with Python/JS
2044. **Version Your Operators**: Track changes with operator versions
2055. **Document Configuration**: Describe all parameters
206
207### Replication Flows
208
2091. **Plan Target Schema**: Understand target structure requirements
2102. **Use Filters**: Reduce data volume with source filters
2113. **Handle Duplicates**: Configure for exactly-once when possible
2124. **Monitor Execution**: Track progress and errors
2135. **Clean Up Artifacts**: Remove source artifacts after completion
214
215### ML Scenarios
216
2171. **Version Early**: Create versions before major changes
2182. **Track All Metrics**: Use SDK for comprehensive tracking
2193. **Use Notebooks for Exploration**: JupyterLab for experimentation
2204. **Productionize with Pipelines**: Convert notebooks to pipelines
2215. **Export/Import for Migration**: Use ZIP export for transfers
222
223## Error Handling
224
225### Common Graph Errors
226
227| Error | Cause | Solution |
228|-------|-------|----------|
229| Port type mismatch | Incompatible data types | Use converter operator or matching types |
230| Gen1/Gen2 mixing | Combined operator generations | Use single generation per graph |
231| Resource exhaustion | Insufficient memory/CPU | Adjust resource requirements |
232| Connection failure | Network or credentials | Verify connection settings |
233| Validation errors | Invalid configuration | Review error messages, fix config |
234
235### Recovery Strategies
236
237**Gen2 Graphs:**
238- Enable automatic recovery in graph settings
239- Configure snapshot intervals
240- Monitor recovery status
241
242**Gen1 Graphs:**
243- Implement manual error handling in operators
244- Use try-catch in script operators
245- Configure retry logic
246
247## Reference Files
248
249For detailed information, see:
250
251- `references/operators-reference.md` - Complete operator catalog (266 operators)
252- `references/abap-integration.md` - ABAP/S4HANA/BW integration with SAP Notes
253- `references/structured-data-operators.md` - Structured data processing
254- `references/dtl-functions.md` - Data Transformation Language (79 functions)
255- `references/ml-scenario-manager.md` - ML Scenario Manager, SDK, artifacts
256- `references/subengines.md` - Python, Node.js, C++ subengine development
257- `references/graphs-pipelines.md` - Graph execution, snapshots, recovery
258- `references/replication-flows.md` - Replication flows, cloud storage, Kafka
259- `references/data-workflow.md` - Data workflow operators, orchestration
260- `references/security-cdc.md` - Security, data protection, CDC methods
261- `references/additional-features.md` - Monitoring, cloud storage services, scenario templates, data types, Git terminal
262- `references/modeling-advanced.md` - Graph snippets, SAP cloud apps, configuration types, 141 graph templates
263
264## Templates
265
266Starter templates are available in `templates/`:
267
268- `templates/basic-graph.json` - Simple data processing graph
269- `templates/replication-flow.json` - Data replication pattern
270- `templates/ml-training-pipeline.json` - ML training workflow
271
272## Documentation Links
273
274**Primary Sources:**
275- GitHub Docs: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs)
276- SAP Help Portal: [https://help.sap.com/docs/SAP_DATA_INTELLIGENCE](https://help.sap.com/docs/SAP_DATA_INTELLIGENCE)
277- SAP Developer Center: [https://developers.sap.com/topics/data-intelligence.html](https://developers.sap.com/topics/data-intelligence.html)
278
279**Section-Specific:**
280- Modeling Guide: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/modelingguide](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/modelingguide)
281## Bundled Resources
282
283### Reference Documentation
284- `references/abap-integration.md` - ABAP system integration guide
285- `references/ml-scenario-manager.md` - Machine Learning scenario manager
286- `references/replication-flows.md` - Data replication flow configuration
287- `references/operators-reference.md` - Complete operators reference
288- `references/dtl-functions.md` - Data Transformation Language functions
289- `references/modeling-advanced.md` - Advanced modeling techniques
290- `references/structured-data-operators.md` - Structured data operators guide
291
292### Documentation Links
293- ABAP Integration: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/abapintegration](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/abapintegration)
294- Machine Learning: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/machinelearning](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/machinelearning)
295- Function Reference: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/functionreference](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/functionreference)
296- Repository Objects: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/repositoryobjects](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/repositoryobjects)
297
298## Version Information
299
300- **Last Updated**: 2025-11-27
301- **Evidence Status**: Stale docs-only guidance; source refresh and live tenant/runtime checks pending
302- **Documentation Source**: SAP-docs/sap-hana-cloud-data-intelligence (GitHub)