# Mixed R Python Pipeline

> This skill should be used when the user asks to "create a CLI tool from a Snakemake project", "make a pipeline deployable", "wrap a workflow as a pip package", "turn an analysis into a reusable tool", or "package a Snakemake workflow". Also use when the user has a working Snakemake + R/Python analysis project and wants to make it pip-installable with init, validate, update, clean, and info commands — even if they do not explicitly mention CLI packaging.

- Skill: `wolski/mixed-r-python-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add wolski/mixed-r-python-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/wolski/mixed-r-python-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: wolski (https://skillmd.com/u/wolski)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/wolski/mixed-r-python-pipeline

---


# Turn Existing Snakemake Projects into Deployable CLI Tools

Take an existing analysis project with a working Snakefile and src/ directory and wrap it into a pip-installable CLI tool.

## Starting Point

Assume a working project with this structure:
```
my-analysis-project/
├── Snakefile              ← Already exists and works
├── helpers.py             ← Already exists
├── src/                   ← Already exists
│   ├── analysis.Rmd
│   ├── process.R
│   └── integrate.py
├── config.yaml            ← Project-specific config
└── data/                  ← Project data
```

## Goal

The CLI tool deploys the workflow to new projects:
```bash
my-pipeline init /path/to/new/project
# → Discovers data, generates config, copies Snakefile + src/
```

## Reference Implementation

**ptm-pipeline** (if available at `/Users/wolski/projects/ptm-pipeline`):

Key files to study:
- `src/ptm_pipeline/cli.py` - Click commands
- `src/ptm_pipeline/discover.py` - Glob-based data discovery
- `src/ptm_pipeline/init.py` - Template copying logic
- `src/ptm_pipeline/config.py` - YAML generation
- `template/` - The original Snakefile + src/ moved here

## Transformation Steps

### Step 1: Create CLI Project Structure

```bash
mkdir my-pipeline && cd my-pipeline
uv init

mkdir -p src/my_pipeline template
```

### Step 2: Move Existing Workflow to template/

```bash
# From the existing working project:
cp Snakefile ../my-pipeline/template/
cp helpers.py ../my-pipeline/template/
cp -r src/ ../my-pipeline/template/src/
cp Makefile ../my-pipeline/template/  # if exists
```

### Step 3: Create CLI Modules

Create in `src/my_pipeline/`:

| Module | Purpose |
|--------|---------|
| `cli.py` | Click commands: init, validate, update, clean, info |
| `discover.py` | Auto-detect data folders with glob patterns |
| `init.py` | Copy template, generate config |
| `config.py` | YAML config generation |
| `validate.py` | Check files, commands, R packages |
| `clean.py` | Remove pipeline files |

### Step 4: Configure pyproject.toml

```toml
[project]
name = "my-pipeline"
version = "0.1.0"
dependencies = ["click>=8.0", "pyyaml>=6.0", "rich>=13.0"]

[project.scripts]
my-pipeline = "my_pipeline.cli:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

# CRITICAL: This makes template/ installable
[tool.hatch.build.targets.wheel.shared-data]
"template" = "share/my-pipeline/template"
```

## Key Patterns

### Template Path Resolution

```python
def get_template_path() -> Path:
    # 1. Development mode
    dev_path = Path(__file__).parent.parent.parent / "template"
    if dev_path.exists():
        return dev_path

    # 2. Installed mode
    installed_path = Path(sys.prefix) / "share" / "my-pipeline" / "template"
    if installed_path.exists():
        return installed_path

    raise FileNotFoundError("Could not locate template directory")
```

### Data Discovery

```python
def find_folders(directory: Path, patterns: List[str]) -> Dict[str, List[Path]]:
    results = {}
    for pattern in patterns:
        matches = sorted(directory.glob(pattern))
        if matches:
            key = pattern.replace("*", "").strip("_")
            results[key] = matches
    return results

# Define patterns for the project's data types:
DATA_PATTERNS = ["DEA_*", "results_*", "input_*"]
```

### Copy Template to Target

```python
def copy_template_files(target_dir: Path, template_path: Path):
    for item in ["Snakefile", "helpers.py", "Makefile"]:
        src = template_path / item
        if src.exists():
            shutil.copy2(src, target_dir / item)

    src_dir = template_path / "src"
    if src_dir.exists():
        target_src = target_dir / "src"
        if target_src.exists():
            shutil.rmtree(target_src)
        shutil.copytree(src_dir, target_src)
```

## Result

```
my-pipeline/                          # The CLI tool
├── pyproject.toml
├── src/my_pipeline/
│   ├── cli.py
│   ├── discover.py
│   ├── init.py
│   ├── config.py
│   ├── validate.py
│   └── clean.py
└── template/                         # The original workflow
    ├── Snakefile
    ├── helpers.py
    └── src/
        ├── analysis.Rmd
        └── ...

# Usage:
uv run my-pipeline init /path/to/new/project
uv run my-pipeline validate /path/to/new/project
cd /path/to/new/project && snakemake -s Snakefile -c1 all
```

## CLI Commands

| Command | Purpose |
|---------|---------|
| `init [DIR]` | Discover data, prompt settings, generate config, copy template |
| `validate [DIR]` | Check files, commands, R packages exist |
| `update [DIR]` | Re-copy template files, preserve config |
| `clean [DIR]` | Remove pipeline files, keep user data |
| `info [DIR]` | Show discovered data folders (debugging) |

## Improve this skill from use

After completing a task with this skill, reflect on whether its instructions or resources revealed a
gap, ambiguity, stale instruction, avoidable friction, or error. If concrete evidence surfaced, include
a brief `Skill feedback` note in the handoff or final response that names the affected file or section
and proposes the smallest useful correction. Do not invent feedback when no issue surfaced, and do not
edit the skill during an unrelated task without the user's authorization.