# Di Agent Knowledge Engine Datastage

> Q&A reference for the DataStage parallel engine — parallelism, partitioning theory, APT configuration files, concurrent job execution, restart/recovery, disk/resource tuning, dataset performance, flow optimization (partitioning/sorting/memory), and per-stage semantics. Use for conceptual engine questions and stage property lookups regardless of authoring tool.

- Skill: `ibm/di-agent-knowledge-engine-datastage` (Agent Skill, multi-file: 104 files)
- Install (CLI): `npx skillmds@latest add ibm/di-agent-knowledge-engine-datastage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ibm/di-agent-knowledge-engine-datastage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: IBM (https://skillmd.com/u/ibm)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ibm/di-agent-knowledge-engine-datastage

---


# DataStage Parallel Engine

## When to Use DataStage
- Batch ETL processing of large data volumes
- Parallel processing across multiple nodes
- Complex transformations with high throughput requirements
- Integration with enterprise databases and file systems
- Data warehouse loading and CDC operations

## Engine Characteristics
- **Parallel processing**: Divides data into partitions processed simultaneously
- **Pipeline parallelism**: Multiple stages process different data concurrently
- **Scalable**: Add nodes to increase throughput
- **High performance**: Optimized for large-scale data movement

## Key Concepts
- **Partitioning**: Data divided across processing nodes
- **Nodes**: Physical or logical processing units
- **Partitions**: Subsets of data processed independently
- **Configuration file**: Defines nodes and resources

## Performance Factors
- Job design (stage selection, partitioning, data flow)
- Configuration (node count, partition count, resources)
- Infrastructure (disk I/O, network bandwidth, CPU)

## References
- [Engine Details](DataStageEngineDetails.md)
- [Concurrent Job Execution](ConcurrentJobExecution.md)
- [Configuration Management](ConfigurationManagement.md)
- [Data Set Performance](DataSetPerformance.md)
- [Disk and Resource Optimization](DiskAndResourceOptimization.md)
- [Restart and Recovery](RestartAndRecovery.md)
- Flow optimization (partitioning, sorting, memory) → [optimization/overview.md](optimization/overview.md)
- Per-stage semantics, requirements, best practices, and properties → [stages/](stages/)
- [Transformer expressions](stages/TransformerStageFunctions/TransformerStageFunctionsOverview.md)