Apache Hop Development

Apache Hop Development

CRITICAL DIRECTIVE: PRE-FLIGHT GUARDRAIL (MANDATORY)

Before executing ANY tool call (edit, write, bash, mcp, etc.), you MUST evaluate the request against STANDARDS.md.

If the request violates ANY standard defined in STANDARDS.md:

1. FIRST TIME: You MUST halt and refuse. You are strictly forbidden from executing the tool on the first ask. Respond exactly like this: "This request conflicts with our Apache Hop standards:

   • Standard: [Name / section of standard from STANDARDS.md]
   • Why: [Why requested action violates standard / benefits of standard]

   Recommendation: [Provide a detailed, actionable alternative approach to achieve your goal without violating standards. E.g., explain how to split logic into sub-pipelines, restructure layout using standard S/U-curves, align to 50px grid, or replace scripting with native transforms.]

   If you have thoroughly considered this and still want to proceed, confirm and I will execute the original request."

2. SECOND TIME (Developer Insists): If the developer explicitly acknowledges the violation and insists on proceeding, you may execute the request.

Path Resolution Note: All reference files and directories listed below (such as STANDARDS.md or hop-user-manual/) reside inside the skill's own directory. Resolve paths to these reference files relative to the directory containing this SKILL.md file. Do not confuse with codebase files (pipelines, workflows being created/edited), which remain relative to the current workspace.

Quick start

To construct production-standard workflows and pipelines, first consult STANDARDS.md for structural layouts, and use the category guides below to find exact, tested XML structures of all 114 Hop transforms:

• Input & Output Transforms
• Data Transform & Formatting
• Lookups & Joins
• Scripting & Utility Transforms

Workflows

1. Generating or Editing Pipelines (.hpl)

• Check file complexity limits (<20 transforms). Push back if exceeded!
• Push back/question developer on any suggestion that violates STANDARDS.md.
• Follow 2D layout standards (S-Curve/U-Curve, 150px spacing, snap-to-grid).
• Implement mandatory <notepads> (Story, HowToTest, Author with & pi (AI Assistant)) with manual line breaks.
• Determine required transforms from category reference guides.
• Retrieve exact XML structure of needed transforms.
• Define pipeline skeleton: root <pipeline> element with <info> metadata.
• Define execution order in <order> with <hop> paths connecting transforms.
• Add <transform> blocks, modifying <name>, parameters, and <GUI> locations (xloc, yloc).

2. Generating or Editing Workflows (.hwf)

• Check action limits (<15 actions). Push back if exceeded!
• Implement mandatory <notepads> (Story, HowToTest, Author with & pi (AI Assistant)) with manual line breaks.
• Start with standard elements: root <workflow>, followed by <name>, and <parameters>.
• Set <run_configuration>local</run_configuration>.
• Add <actions> containing the action nodes (START, SUCCESS, PIPELINE, WORKFLOW, WRITE_TO_LOG).
• Connect actions using <hops> with source (<from>), destination (<to>), and evaluation rules (e.g., <evaluation>Y</evaluation>).

3. Standards Checklist

• Enforce STANDARDS.md: Never skip layout, notepad, parameterization, or clean-up steps.
• Prefer Native Transforms: Always use native Hop transforms instead of JavaScript/scripting transforms where possible. Only use scripting when native transforms cannot achieve required logic.
• Always use standard UTF-8 XML header: <?xml version="1.0" encoding="UTF-8"?>.
• Use variable referencing like ${PROJECT_HOME} rather than hardcoded/relative paths for filenames.
• Log incoming parameters at workflow startup using a WRITE_TO_LOG action.
• Ensure unique names for all transforms/actions within the same pipeline/workflow.
• Implement standard <notepads> blocks (#Story, #HowToTest, #Author with AI indicators) at yloc=0 per STANDARDS.md.
• Clean up stream with Select Values right before output. Avoid heavy inline SQL joins in table inputs; use Stream Lookup.

Advanced features

See detailed structure references:

• REFERENCE.md
• STANDARDS.md

Integration Test Reference Docs

The complete suite of original Apache Hop integration test cases (pipelines and workflows) is bundled directly inside this skill at:

• hop-transform-documentation/
• hop-http-documentation/ (HTTP/REST action and client tests)
• hop-database-documentation/ (Database input, output, join, and connection tests)
• hop-actions-documentation/ (Common workflow action tests like abort, copy, delete, move, simple evaluation)
• hop-parameters-documentation/ (Pipeline/workflow parameters and variables tests)

Official User Manuals

Official Apache Hop documentation guides for all transforms and actions are bundled as AsciiDoc files at:

• hop-user-manual/transforms/ (detailed explanation of all 230+ transforms and settings)
• hop-user-manual/actions/ (detailed explanation of all 75+ workflow actions and settings)

You can explore these files directly to see advanced, multi-transform orchestration patterns.