SAP HANA Cloud Data Intelligence Skill
Related Skills
- sap-hana-ml: Use for Python ML client workflows that run against SAP HANA
- sap-datasphere: Use for Datasphere modeling, replication, and analytics data warehousing
- sap-btp-connectivity: Use for cloud-to-on-premise connection and destination prerequisites
- sap-btp-integration-suite: Use for integration scenarios that should move out of Data Intelligence pipelines
This skill provides documentation-audited guidance for developing with SAP Data Intelligence Cloud, including pipeline creation, operator development, data integration, and machine learning scenarios. The last_verified date is intentionally stale until product lifecycle status and live graph/runtime behavior are checked against primary sources.
Table of Contents
When to Use This Skill
Use this skill when:
- Creating or modifying data processing graphs/pipelines
- Developing custom operators (Gen1 or Gen2)
- Integrating ABAP-based SAP systems (S/4HANA, BW)
- Building replication flows for data movement
- Developing ML scenarios with ML Scenario Manager
- Working with JupyterLab in Data Intelligence
- Using Data Transformation Language (DTL) functions
- Configuring subengines (Python, Node.js, C++)
- Working with structured data operators
Common Issues
| Issue |
First check |
| Graph fails after operator change |
Confirm all operators use the same generation and compatible subengine. |
| ABAP/S4HANA connection fails |
Verify Cloud Connector, destination, and credential configuration. |
| Python or Node operator behaves differently in runtime |
Check subengine version, package availability, and serialization boundaries. |
Core Concepts
Graphs (Pipelines)
Graphs are networks of operators connected via typed input/output ports for data transfer.
Two Generations:
- Gen1 Operators: Legacy operators, broad compatibility
- Gen2 Operators: Enhanced error recovery, state management, snapshots
Critical Rule: Graphs cannot mix Gen1 and Gen2 operators - choose one generation per graph.
Gen2 Advantages:
- Automatic error recovery with snapshots
- State management with periodic checkpoints
- Native multiplexing (one-to-many, many-to-one)
- Improved Python3 operator
Operators
Building blocks that process data within graphs. Each operator has:
- Ports: Typed input/output connections for data flow
- Configuration: Parameters that control behavior
- Runtime: Engine that executes the operator
Operator Categories:
- Messaging (Kafka, MQTT, NATS)
- Storage (Files, HDFS, S3, Azure, GCS)
- Database (HANA, SAP BW, SQL)
- Script (Python, JavaScript, R, Go)
- Data Processing (Transform, Anonymize, Validate)
- Machine Learning (TensorFlow, PyTorch, HANA ML)
- Integration (OData, REST, SAP CPI)
- Workflow (Pipeline, Data Workflow)
Subengines
Subengines enable operators to run on different runtimes within the same graph.
Supported Subengines:
- ABAP: For ABAP Pipeline Engine operators
- Python 3.9: For Python-based operators
- Node.js: For JavaScript-based operators
- C++: For high-performance native operators
Key Benefit: Connected operators on the same subengine run in a single OS process for optimal performance.
Trade-off: Cross-engine communication requires serialization/deserialization overhead.
Quick Start Patterns
Basic Graph Creation
1. Open SAP Data Intelligence Modeler
2. Create new graph
3. Add operators from repository
4. Connect operator ports (matching types)
5. Configure operator parameters
6. Validate graph
7. Execute and monitor
Replication Flow Pattern
1. Create replication flow in Modeler
2. Configure source connection (ABAP, HANA, etc.)
3. Configure target (HANA Cloud, S3, Kafka, etc.)
4. Add tasks with source objects
5. Define filters and mappings
6. Validate flow
7. Deploy to tenant repository
8. Run and monitor
Delivery Guarantees:
- Default: At-least-once (may have duplicates)
- With UPSERT to databases: Exactly-once
- For cloud storage: Use "Suppress Duplicates" option
ML Scenario Pattern
1. Open ML Scenario Manager from launchpad
2. Create new scenario
3. Add datasets (register data sources)
4. Create Jupyter notebooks for experiments
5. Build training pipelines
6. Track metrics with Metrics Explorer
7. Version scenario for reproducibility
8. Deploy model pipeline
Common Tasks
ABAP System Integration
For integrating ABAP-based SAP systems:
- Prerequisites: Configure Cloud Connector for on-premise systems
- Connection Setup: Create ABAP connection in Connection Management
- Metadata Access: Use Metadata Explorer for object discovery
- Data Sources: CDS Views, ODP (Operational Data Provisioning), Tables
Reference: See references/abap-integration.md for detailed setup.
Structured Data Processing
Use structured data operators for SQL-like transformations:
- Data Transform: Visual SQL editor for complex transformations
- Aggregation Node: GROUP BY with aggregation functions
- Join Node: INNER, LEFT, RIGHT, FULL joins
- Projection Node: Column selection and renaming
- Union Node: Combine multiple datasets
- Case Node: Conditional logic
Reference: See references/structured-data-operators.md for configuration.
Data Transformation Language
DTL provides SQL-like functions for data processing:
Function Categories:
- String: CONCAT, SUBSTRING, UPPER, LOWER, TRIM, REPLACE
- Numeric: ABS, CEIL, FLOOR, ROUND, MOD, POWER
- Date/Time: ADD_DAYS, MONTHS_BETWEEN, EXTRACT, CURRENT_UTCTIMESTAMP
- Conversion: TO_DATE, TO_STRING, TO_INTEGER, TO_DECIMAL
- Miscellaneous: CASE, COALESCE, IFNULL, NULLIF
Reference: See references/dtl-functions.md for complete reference.
Best Practices
Graph Design
- Choose Generation Early: Decide Gen1 vs Gen2 before building
- Minimize Cross-Engine Communication: Group operators by subengine
- Use Appropriate Port Types: Match data types for efficient transfer
- Enable Snapshots: For Gen2 graphs, enable auto-recovery
- Validate Before Execution: Always validate graphs
Operator Development
- Start with Built-in Operators: Use predefined operators first
- Extend When Needed: Create custom operators for specific needs
- Use Script Operators: For quick prototyping with Python/JS
- Version Your Operators: Track changes with operator versions
- Document Configuration: Describe all parameters
Replication Flows
- Plan Target Schema: Understand target structure requirements
- Use Filters: Reduce data volume with source filters
- Handle Duplicates: Configure for exactly-once when possible
- Monitor Execution: Track progress and errors
- Clean Up Artifacts: Remove source artifacts after completion
ML Scenarios
- Version Early: Create versions before major changes
- Track All Metrics: Use SDK for comprehensive tracking
- Use Notebooks for Exploration: JupyterLab for experimentation
- Productionize with Pipelines: Convert notebooks to pipelines
- Export/Import for Migration: Use ZIP export for transfers
Error Handling
Common Graph Errors
| Error |
Cause |
Solution |
| Port type mismatch |
Incompatible data types |
Use converter operator or matching types |
| Gen1/Gen2 mixing |
Combined operator generations |
Use single generation per graph |
| Resource exhaustion |
Insufficient memory/CPU |
Adjust resource requirements |
| Connection failure |
Network or credentials |
Verify connection settings |
| Validation errors |
Invalid configuration |
Review error messages, fix config |
Recovery Strategies
Gen2 Graphs:
- Enable automatic recovery in graph settings
- Configure snapshot intervals
- Monitor recovery status
Gen1 Graphs:
- Implement manual error handling in operators
- Use try-catch in script operators
- Configure retry logic
Reference Files
For detailed information, see:
references/operators-reference.md - Complete operator catalog (266 operators)references/abap-integration.md - ABAP/S4HANA/BW integration with SAP Notesreferences/structured-data-operators.md - Structured data processingreferences/dtl-functions.md - Data Transformation Language (79 functions)references/ml-scenario-manager.md - ML Scenario Manager, SDK, artifactsreferences/subengines.md - Python, Node.js, C++ subengine developmentreferences/graphs-pipelines.md - Graph execution, snapshots, recoveryreferences/replication-flows.md - Replication flows, cloud storage, Kafkareferences/data-workflow.md - Data workflow operators, orchestrationreferences/security-cdc.md - Security, data protection, CDC methodsreferences/additional-features.md - Monitoring, cloud storage services, scenario templates, data types, Git terminalreferences/modeling-advanced.md - Graph snippets, SAP cloud apps, configuration types, 141 graph templates
Templates
Starter templates are available in templates/:
templates/basic-graph.json - Simple data processing graphtemplates/replication-flow.json - Data replication patterntemplates/ml-training-pipeline.json - ML training workflow
Documentation Links
Primary Sources:
Section-Specific:
Bundled Resources
Reference Documentation
references/abap-integration.md - ABAP system integration guidereferences/ml-scenario-manager.md - Machine Learning scenario managerreferences/replication-flows.md - Data replication flow configurationreferences/operators-reference.md - Complete operators referencereferences/dtl-functions.md - Data Transformation Language functionsreferences/modeling-advanced.md - Advanced modeling techniquesreferences/structured-data-operators.md - Structured data operators guide
Documentation Links
Version Information
- Last Updated: 2025-11-27
- Evidence Status: Stale docs-only guidance; source refresh and live tenant/runtime checks pending
- Documentation Source: SAP-docs/sap-hana-cloud-data-intelligence (GitHub)
1---2name: sap-hana-cloud-data-intelligence3description: Develops data processing pipelines, integrations, and machine learning scenarios in SAP Data Intelligence Cloud. Use when building graphs/pipelines with operators, integrating ABAP/S4HANA systems, creating replication flows, developing ML scenarios with JupyterLab, or using Data Transformation Language functions. Covers Gen1/Gen2 operators, subengines (Python, Node.js, C++), structured data operators, and repository objects.4license: GPL-3.05---6
7# SAP HANA Cloud Data Intelligence Skill
8
9## Related Skills
10
11- **sap-hana-ml**: Use for Python ML client workflows that run against SAP HANA
12- **sap-datasphere**: Use for Datasphere modeling, replication, and analytics data warehousing
13- **sap-btp-connectivity**: Use for cloud-to-on-premise connection and destination prerequisites
14- **sap-btp-integration-suite**: Use for integration scenarios that should move out of Data Intelligence pipelines
15
16This skill provides documentation-audited guidance for developing with SAP Data Intelligence Cloud, including pipeline creation, operator development, data integration, and machine learning scenarios. The `last_verified` date is intentionally stale until product lifecycle status and live graph/runtime behavior are checked against primary sources.
17
18## Table of Contents
19- [When to Use This Skill](#when-to-use-this-skill)
20- [Core Concepts](#core-concepts)
21- [Quick Start Patterns](#quick-start-patterns)
22- [Common Tasks](#common-tasks)
23- [Bundled Resources](#bundled-resources)
24
25## When to Use This Skill
26
27Use this skill when:
28
29- Creating or modifying data processing graphs/pipelines
30- Developing custom operators (Gen1 or Gen2)
31- Integrating ABAP-based SAP systems (S/4HANA, BW)
32- Building replication flows for data movement
33- Developing ML scenarios with ML Scenario Manager
34- Working with JupyterLab in Data Intelligence
35- Using Data Transformation Language (DTL) functions
36- Configuring subengines (Python, Node.js, C++)
37- Working with structured data operators
38
39## Common Issues
40
41| Issue | First check |
42|-------|-------------|
43| Graph fails after operator change | Confirm all operators use the same generation and compatible subengine. |
44| ABAP/S4HANA connection fails | Verify Cloud Connector, destination, and credential configuration. |
45| Python or Node operator behaves differently in runtime | Check subengine version, package availability, and serialization boundaries. |
46
47## Core Concepts
48
49### Graphs (Pipelines)
50
51Graphs are networks of operators connected via typed input/output ports for data transfer.
52
53**Two Generations:**
54- **Gen1 Operators**: Legacy operators, broad compatibility
55- **Gen2 Operators**: Enhanced error recovery, state management, snapshots
56
57**Critical Rule**: Graphs cannot mix Gen1 and Gen2 operators - choose one generation per graph.
58
59**Gen2 Advantages:**
60- Automatic error recovery with snapshots
61- State management with periodic checkpoints
62- Native multiplexing (one-to-many, many-to-one)
63- Improved Python3 operator
64
65### Operators
66
67Building blocks that process data within graphs. Each operator has:
68- **Ports**: Typed input/output connections for data flow
69- **Configuration**: Parameters that control behavior
70- **Runtime**: Engine that executes the operator
71
72**Operator Categories:**
731. Messaging (Kafka, MQTT, NATS)
742. Storage (Files, HDFS, S3, Azure, GCS)
753. Database (HANA, SAP BW, SQL)
764. Script (Python, JavaScript, R, Go)
775. Data Processing (Transform, Anonymize, Validate)
786. Machine Learning (TensorFlow, PyTorch, HANA ML)
797. Integration (OData, REST, SAP CPI)
808. Workflow (Pipeline, Data Workflow)
81
82### Subengines
83
84Subengines enable operators to run on different runtimes within the same graph.
85
86**Supported Subengines:**
87- **ABAP**: For ABAP Pipeline Engine operators
88- **Python 3.9**: For Python-based operators
89- **Node.js**: For JavaScript-based operators
90- **C++**: For high-performance native operators
91
92**Key Benefit**: Connected operators on the same subengine run in a single OS process for optimal performance.
93
94**Trade-off**: Cross-engine communication requires serialization/deserialization overhead.
95
96## Quick Start Patterns
97
98### Basic Graph Creation
99
100```
1011. Open SAP Data Intelligence Modeler
1022. Create new graph
1033. Add operators from repository
1044. Connect operator ports (matching types)
1055. Configure operator parameters
1066. Validate graph
1077. Execute and monitor
108```
109
110### Replication Flow Pattern
111
112```
1131. Create replication flow in Modeler
1142. Configure source connection (ABAP, HANA, etc.)
1153. Configure target (HANA Cloud, S3, Kafka, etc.)
1164. Add tasks with source objects
1175. Define filters and mappings
1186. Validate flow
1197. Deploy to tenant repository
1208. Run and monitor
121```
122
123**Delivery Guarantees:**
124- Default: At-least-once (may have duplicates)
125- With UPSERT to databases: Exactly-once
126- For cloud storage: Use "Suppress Duplicates" option
127
128### ML Scenario Pattern
129
130```
1311. Open ML Scenario Manager from launchpad
1322. Create new scenario
1333. Add datasets (register data sources)
1344. Create Jupyter notebooks for experiments
1355. Build training pipelines
1366. Track metrics with Metrics Explorer
1377. Version scenario for reproducibility
1388. Deploy model pipeline
139```
140
141## Common Tasks
142
143### ABAP System Integration
144
145For integrating ABAP-based SAP systems:
146
1471. **Prerequisites**: Configure Cloud Connector for on-premise systems
1482. **Connection Setup**: Create ABAP connection in Connection Management
1493. **Metadata Access**: Use Metadata Explorer for object discovery
1504. **Data Sources**: CDS Views, ODP (Operational Data Provisioning), Tables
151
152**Reference**: See `references/abap-integration.md` for detailed setup.
153
154### Structured Data Processing
155
156Use structured data operators for SQL-like transformations:
157
158- **Data Transform**: Visual SQL editor for complex transformations
159- **Aggregation Node**: GROUP BY with aggregation functions
160- **Join Node**: INNER, LEFT, RIGHT, FULL joins
161- **Projection Node**: Column selection and renaming
162- **Union Node**: Combine multiple datasets
163- **Case Node**: Conditional logic
164
165**Reference**: See `references/structured-data-operators.md` for configuration.
166
167### Data Transformation Language
168
169DTL provides SQL-like functions for data processing:
170
171**Function Categories:**
172- String: CONCAT, SUBSTRING, UPPER, LOWER, TRIM, REPLACE
173- Numeric: ABS, CEIL, FLOOR, ROUND, MOD, POWER
174- Date/Time: ADD_DAYS, MONTHS_BETWEEN, EXTRACT, CURRENT_UTCTIMESTAMP
175- Conversion: TO_DATE, TO_STRING, TO_INTEGER, TO_DECIMAL
176- Miscellaneous: CASE, COALESCE, IFNULL, NULLIF
177
178**Reference**: See `references/dtl-functions.md` for complete reference.
179
180## Best Practices
181
182### Graph Design
183
1841. **Choose Generation Early**: Decide Gen1 vs Gen2 before building
1852. **Minimize Cross-Engine Communication**: Group operators by subengine
1863. **Use Appropriate Port Types**: Match data types for efficient transfer
1874. **Enable Snapshots**: For Gen2 graphs, enable auto-recovery
1885. **Validate Before Execution**: Always validate graphs
189
190### Operator Development
191
1921. **Start with Built-in Operators**: Use predefined operators first
1932. **Extend When Needed**: Create custom operators for specific needs
1943. **Use Script Operators**: For quick prototyping with Python/JS
1954. **Version Your Operators**: Track changes with operator versions
1965. **Document Configuration**: Describe all parameters
197
198### Replication Flows
199
2001. **Plan Target Schema**: Understand target structure requirements
2012. **Use Filters**: Reduce data volume with source filters
2023. **Handle Duplicates**: Configure for exactly-once when possible
2034. **Monitor Execution**: Track progress and errors
2045. **Clean Up Artifacts**: Remove source artifacts after completion
205
206### ML Scenarios
207
2081. **Version Early**: Create versions before major changes
2092. **Track All Metrics**: Use SDK for comprehensive tracking
2103. **Use Notebooks for Exploration**: JupyterLab for experimentation
2114. **Productionize with Pipelines**: Convert notebooks to pipelines
2125. **Export/Import for Migration**: Use ZIP export for transfers
213
214## Error Handling
215
216### Common Graph Errors
217
218| Error | Cause | Solution |
219|-------|-------|----------|
220| Port type mismatch | Incompatible data types | Use converter operator or matching types |
221| Gen1/Gen2 mixing | Combined operator generations | Use single generation per graph |
222| Resource exhaustion | Insufficient memory/CPU | Adjust resource requirements |
223| Connection failure | Network or credentials | Verify connection settings |
224| Validation errors | Invalid configuration | Review error messages, fix config |
225
226### Recovery Strategies
227
228**Gen2 Graphs:**
229- Enable automatic recovery in graph settings
230- Configure snapshot intervals
231- Monitor recovery status
232
233**Gen1 Graphs:**
234- Implement manual error handling in operators
235- Use try-catch in script operators
236- Configure retry logic
237
238## Reference Files
239
240For detailed information, see:
241
242- `references/operators-reference.md` - Complete operator catalog (266 operators)
243- `references/abap-integration.md` - ABAP/S4HANA/BW integration with SAP Notes
244- `references/structured-data-operators.md` - Structured data processing
245- `references/dtl-functions.md` - Data Transformation Language (79 functions)
246- `references/ml-scenario-manager.md` - ML Scenario Manager, SDK, artifacts
247- `references/subengines.md` - Python, Node.js, C++ subengine development
248- `references/graphs-pipelines.md` - Graph execution, snapshots, recovery
249- `references/replication-flows.md` - Replication flows, cloud storage, Kafka
250- `references/data-workflow.md` - Data workflow operators, orchestration
251- `references/security-cdc.md` - Security, data protection, CDC methods
252- `references/additional-features.md` - Monitoring, cloud storage services, scenario templates, data types, Git terminal
253- `references/modeling-advanced.md` - Graph snippets, SAP cloud apps, configuration types, 141 graph templates
254
255## Templates
256
257Starter templates are available in `templates/`:
258
259- `templates/basic-graph.json` - Simple data processing graph
260- `templates/replication-flow.json` - Data replication pattern
261- `templates/ml-training-pipeline.json` - ML training workflow
262
263## Documentation Links
264
265**Primary Sources:**
266- GitHub Docs: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs)
267- SAP Help Portal: [https://help.sap.com/docs/SAP_DATA_INTELLIGENCE](https://help.sap.com/docs/SAP_DATA_INTELLIGENCE)
268- SAP Developer Center: [https://developers.sap.com/topics/data-intelligence.html](https://developers.sap.com/topics/data-intelligence.html)
269
270**Section-Specific:**
271- Modeling Guide: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/modelingguide](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/modelingguide)
272## Bundled Resources
273
274### Reference Documentation
275- `references/abap-integration.md` - ABAP system integration guide
276- `references/ml-scenario-manager.md` - Machine Learning scenario manager
277- `references/replication-flows.md` - Data replication flow configuration
278- `references/operators-reference.md` - Complete operators reference
279- `references/dtl-functions.md` - Data Transformation Language functions
280- `references/modeling-advanced.md` - Advanced modeling techniques
281- `references/structured-data-operators.md` - Structured data operators guide
282
283### Documentation Links
284- ABAP Integration: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/abapintegration](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/abapintegration)
285- Machine Learning: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/machinelearning](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/machinelearning)
286- Function Reference: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/functionreference](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/functionreference)
287- Repository Objects: [https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/repositoryobjects](https://github.com/SAP-docs/sap-hana-cloud-data-intelligence/tree/main/docs/repositoryobjects)
288
289## Version Information
290
291- **Last Updated**: 2025-11-27
292- **Evidence Status**: Stale docs-only guidance; source refresh and live tenant/runtime checks pending
293- **Documentation Source**: SAP-docs/sap-hana-cloud-data-intelligence (GitHub)
secondsky