Workflow Orchestration

Workflow Development

Purpose

Use this skill to design, construct, and manage workflow definitions.

A workflow is an orchestration of data tasks, dependencies, schedules, and execution policies.

This skill is responsible for workflow orchestration and structure only.

It MUST NOT implement task logic.

Runtime Boundary

This skill manages workflow definitions and orchestration.

It does NOT manage workflow runtime instances after a workflow execution has already been created.

Important boundary:

• Starting / triggering / manually running a workflow belongs to workflow-orchestration
• Inspecting, diagnosing, rerunning, backfilling, or terminating an existing execution belongs to production-operations

The following responsibilities belong to production-operations once the execution already exists:

• Query workflow runs
• Query task runs
• View execution logs
• Analyze failures
• Retry failed runs
• Kill running instances
• Backfill existing schedules or executions
• SLA analysis
• Runtime performance analysis

When To Use

Use this skill when the user intent involves workflow orchestration decisions.

Workflow Lifecycle

• Create workflows
• Modify workflows
• Archive workflows
• Trigger workflow execution

Workflow Structure

• Add workflow nodes
• Remove workflow nodes
• Reorganize workflow structure
• Split workflows
• Merge workflows
• Configure dependencies

Scheduling & Policies

• Configure schedules
• Configure retries
• Configure concurrency
• Configure queue policies
• Configure timeout policies
• Configure failure strategies

Node Orchestration

• Select node types
• Configure node parameters
• Override node compute resources only when explicitly requested; compute resources are auto-selected at node creation
• Configure node execution strategies

Governance & Observability

• Configure workflow or task observability settings when needed
• Validate monitoring thresholds and notification targets
• Validate workflow safety
• Validate workflow governance requirements

When NOT To Use

Do NOT use this skill when the user intent is:

• Writing SQL logic → studio-development
• Writing Python logic → studio-development
• Notebook implementation → studio-development
• Creating datasource connections → connection-manage
• Designing business solutions → solution-design
• Managing existing workflow runtime instances → production-operations

Core Responsibilities

This skill owns the workflow orchestration layer.

Workflow Orchestration

• DAG construction
• DAG modification
• Node composition
• Dependency management

Workflow Lifecycle

• Create workflows
• Update workflows
• Trigger workflows

Scheduling & Policies

• Create schedules
• Modify schedules
• Enable schedules
• Disable schedules
• Validate schedules
• Schedule configuration
• Retry strategy configuration
• Concurrency configuration
• Timeout strategy configuration
• Failure policy configuration

Governance

• DAG structure review
• Cycle risk review
• Dependency consistency review
• Workflow compliance review

Node Orchestration

• Select node types
• Configure node relationships
• Configure node execution behavior

Supported Node Types

• SQL Task
• Python Task
• Notebook Task
• Offline Data Integration Task (data ingestion/access and data egress)
• Data Quality Task
• Nested Workflow Task

Available Commands

Workflow Lifecycle

Primary entrypoint:

wedatacli workflow

Responsibilities:

• create workflow
• query workflow
• update workflow metadata
• manage schedules, parameters, labels, resource groups, and observability settings
• trigger workflow execution
• export / diff / apply workflow specs

Alarm / monitor mutations are part of this skill, but detailed command choices and examples stay in references/usage/command_usage_guide.md. When reusing an existing notification channel, resolve the target first with wedatacli notification list|get.

Node Management

Workflow nodes are created and managed through:

wedatacli workflow task

Currently supported task types:

• sql
• python
• notebook
• ingestion
• quality
• nested-workflow

Supported task operations include:

• create / get / delete
• update description, parameters, resource group, and retry policy
• manage dependencies
• move canvas position
• manage task-level observability when needed
• discover task candidates; resource groups only need to be queried for explicit resource overrides

Dependencies are configured through wedatacli workflow task dependencies ....

Workflow dependencies are inspected through wedatacli workflow get.

Usage

Detailed command syntax, flags, and examples are available in:

references/usage/

References

Before designing workflows, consult references in the following order.

Layer 1 — Workflow Design Rules

Defines workflow orchestration principles and governance constraints.

• references/design/workflow_guide.md

Layer 2 — Node Specifications

Defines behavior and configuration rules for individual node types.

• references/node_specs/sql_node_design_guide.md
• references/node_specs/python_node_design_guide.md
• references/node_specs/notebook_node_design_guide.md
• references/node_specs/ingestion_node_design_guide.md
• references/node_specs/nested_workflow_node_design_guide.md

Layer 3 — Commands & Execution

Defines command syntax and execution behavior.

• references/usage/command_usage_guide.md

Execution Rules

1. Prefer existing workflow patterns before creating new workflow structures.
2. Review workflow DAG consistency before saving.
3. Use model reasoning to avoid circular dependencies.
4. Use model reasoning to ensure referenced nodes exist.
5. Validate scheduling configuration before execution.
6. Require workflow owner and description.
7. Reuse existing workflow definitions whenever possible.
8. Reuse existing task definitions whenever possible.
9. Keep workflow orchestration separate from task implementation.
10. For the same workflow_id, create workflow nodes sequentially one by one; do not run node create commands in parallel or batch them, because the platform may fail concurrent node creation for the same workflow.
11. Before changing workflow or task alarms/monitors, query the current workflow/task definition or the current alarm/monitor config first; when --alarm-group needs an existing channel, resolve the target with wedatacli notification list|get and verify the result again after mutation.
12. Before triggering the manual-run command wedatacli workflow run, show the target workflow and intended action, then obtain explicit user confirmation for that specific execution step.
13. When creating workflow nodes, do not select compute resources manually; wedatacli workflow task create auto-selects a default execution resource. Only pass --resource-group-id when the user explicitly requests a specific resource.

Success Criteria

A workflow is considered valid when:

• DAG dependency design has been reviewed by the model
• Nodes are correctly typed
• Dependencies are complete and reference intended upstream nodes
• Scheduling configuration is valid
• Workflow configuration is queryable
• Workflow can be triggered successfully
• Monitoring is configured when needed
• Alerts are configured when needed
• Failure strategies are configured when needed
• Retry strategies are configured when needed
• Governance requirements are satisfied