How do I install the Architecture Paradigm Pipeline skill?

Run npx skillmds@latest add majiayu000/architecture-paradigm-pipeline-4 in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Architecture Paradigm Pipeline safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Architecture Paradigm Pipeline?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Architecture Paradigm Pipeline free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Architecture Paradigm Pipeline?

majiayu000 (@majiayu000) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Architecture Paradigm Pipeline by majiayu000

Name: Architecture Paradigm Pipeline
Rating: 4.666666666666667 (3 reviews)
Author: majiayu000

Architecture Paradigm Pipeline

Compose processing stages using a pipes-and-filters model for ETL, media processing, or compiler-like workloads. Triggers: pipeline architecture, pipes and filters, ETL, data transformation, stream processing, CI/CD pipeline, media processing, batch processing Use when: data flows through fixed sequence of transformations, stages can be independently developed and tested, parallel processing of stages is beneficial DO NOT use when: selecting from multiple paradigms - use architecture-paradigms first. DO NOT use when: data flow isn't sequential or predictable. DO NOT use when: complex branching/merging logic dominates. Consult this skill when designing data pipelines or transformation workflows.

majiayu000 Updated Sep 22, 2026 567 repo stars

File contents

The Pipeline (Pipes and Filters) Paradigm

When to Employ This Paradigm

When data must flow through a fixed sequence of discrete transformations, such as in ETL jobs, streaming analytics, or CI/CD pipelines.
When reusing individual processing stages is needed, either independently or to scale bottleneck stages separately from others.
When failure isolation between stages is a critical requirement.

Adoption Steps

Define Filters: Design each stage (filter) to perform a single, well-defined transformation. Each filter must have a clear input and output data schema.
Connect via Pipes: Connect the filters using "pipes," which can be implemented as streams, message queues, or in-memory channels. validate these pipes support back-pressure and buffering.
Maintain Stateless Filters: Where possible, design filters to be stateless. Any required state should be persisted externally or managed at the boundaries of the pipeline.
Instrument Each Stage: Implement monitoring for each filter to track key metrics such as latency, throughput, and error rates.
Orchestrate Deployments: Design the deployment strategy to allow each stage to be scaled horizontally and upgraded independently.

Key Deliverables

An Architecture Decision Record (ADR) documenting the filters, the chosen pipe technology, the error-handling strategy, and the tools for replaying data.
A suite of contract tests for each filter, plus integration tests that cover representative end-to-end pipeline executions.
Observability dashboards that visualize stage-level Key Performance Indicators (KPIs).

Risks & Mitigations

Single-Stage Bottlenecks:
- Mitigation: Implement auto-scaling for individual filters. If a single filter remains a bottleneck, consider refactoring it into a more granular sub-pipeline.
Schema Drift Between Stages:
- Mitigation: Centralize schema definitions in a shared repository and enforce compatibility tests as part of the CI/CD process to prevent breaking changes.
Back-Pressure Failures:
- Mitigation: Conduct rigorous load testing to simulate high-volume scenarios. Validate that buffering, retry logic, and back-pressure mechanisms behave as expected under stress.

---name: architecture-paradigm-pipeline-4description: Compose processing stages using a pipes-and-filters model for ETL, media processing, or compiler-like workloads. Triggers: pipeline architecture, pipes and filters, ETL, data transformation, stream processing, CI/CD pipeline, media processing, batch processing Use when: data flows through fixed sequence of transformations, stages can be independently developed and tested, parallel processing of stages is beneficial DO NOT use when: selecting from multiple paradigms - use architecture-paradigms first. DO NOT use when: data flow isn't sequential or predictable. DO NOT use when: complex branching/merging logic dominates. Consult this skill when designing data pipelines or transformation workflows.---# The Pipeline (Pipes and Filters) Paradigm## When to Employ This Paradigm- When data must flow through a fixed sequence of discrete transformations, such as in ETL jobs, streaming analytics, or CI/CD pipelines.- When reusing individual processing stages is needed, either independently or to scale bottleneck stages separately from others.- When failure isolation between stages is a critical requirement.## Adoption Steps1. **Define Filters**: Design each stage (filter) to perform a single, well-defined transformation. Each filter must have a clear input and output data schema.2. **Connect via Pipes**: Connect the filters using "pipes," which can be implemented as streams, message queues, or in-memory channels. validate these pipes support back-pressure and buffering.3. **Maintain Stateless Filters**: Where possible, design filters to be stateless. Any required state should be persisted externally or managed at the boundaries of the pipeline.4. **Instrument Each Stage**: Implement monitoring for each filter to track key metrics such as latency, throughput, and error rates.5. **Orchestrate Deployments**: Design the deployment strategy to allow each stage to be scaled horizontally and upgraded independently.## Key Deliverables- An Architecture Decision Record (ADR) documenting the filters, the chosen pipe technology, the error-handling strategy, and the tools for replaying data.- A suite of contract tests for each filter, plus integration tests that cover representative end-to-end pipeline executions.- Observability dashboards that visualize stage-level Key Performance Indicators (KPIs).## Risks & Mitigations- **Single-Stage Bottlenecks**: - **Mitigation**: Implement auto-scaling for individual filters. If a single filter remains a bottleneck, consider refactoring it into a more granular sub-pipeline.- **Schema Drift Between Stages**: - **Mitigation**: Centralize schema definitions in a shared repository and enforce compatibility tests as part of the CI/CD process to prevent breaking changes.- **Back-Pressure Failures**: - **Mitigation**: Conduct rigorous load testing to simulate high-volume scenarios. Validate that buffering, retry logic, and back-pressure mechanisms behave as expected under stress.

Frequently asked questions