Nlp Research Repo Packaging

NLP Research Repo Package Installation

When reproducing an NLP research repo, align the environment to the repo's declared dependencies before installing anything. Most reproduction failures are not subtle bugs — they are environment drift. Two causes dominate:

1. Python version mismatch. Research code is usually validated against one interpreter (e.g., 3.11). A newer interpreter can change standard-library behavior, drop removed APIs, or ship wheels that pinned libraries were never built against.
2. Installing packages without following requirements.txt / environment.yml. Newer releases of torch, transformers, numpy, etc. routinely change APIs and numerical behavior. A repo that "ran fine for the authors" can silently produce different results — or fail to import — on latest.

The goal is not "install the newest, most secure stack." It is "recreate the stack the authors actually tested," then change it deliberately and minimally if you must.

When to Use

Use this skill when:

• Reproducing or running NLP research code from a repository, where matching the authors' results depends on matching their environment.
• Setting up a development environment for an NLP research project from scratch.
• Debugging dependency conflicts or Python version mismatches in an NLP project — these symptoms usually trace back to skipping the alignment step.
• About to install packages in a freshly cloned research repo. Doing the alignment check before the first pip install is far cheaper than unwinding a half-installed, mismatched environment afterward.

Trigger keywords: reproduce, NLP research repo, requirements.txt, environment.yml, dependency conflict, Python version mismatch, pip install, conda env create, torch, transformers, reproduction.

When NOT to Use

• General-purpose, non-research package installation. If you just need a library in your own project, the reproduction-grade ceremony here (snapshotting the interpreter, pinning to the authors' versions) is overhead with no payoff. Use normal packaging tooling instead.
• Installing directly without the alignment step. Skipping straight to pip install is what produces the mismatched environments this skill exists to prevent. Read the dependency files first so you install into the right interpreter.
• Assuming the system Python matches the repo. Containers and CI images ship whatever Python the base image happened to include; that is rarely the version the repo expects. Verify with python -VV rather than assuming.
• Reflexively chasing "latest stable" versions. For ordinary application code, newer is often better. For reproduction it is usually the opposite: the pins in requirements.txt are the contract that makes the results comparable. Upgrading "to be safe" is the single most common way to break a repo. Respect the pins by default.
• Blanket pip install --upgrade without a target version. An unbounded upgrade can pull a major release that changes APIs and invalidates the reproduction. If you genuinely need to patch one dependency (e.g., a known CVE in a transitive package), bump that dependency to a specific known-good version and then re-run the smoke test — don't upgrade everything at once.
• Treating deprecation warnings as noise. They often flag exactly the API the repo relies on. Read them; they tell you whether a version bump is safe or whether it will break the code you are trying to run.

Prerequisites

• Python installed and accessible on PATH (verify with python -VV).
• pip matching the target interpreter (verify with python -m pip --version).
• conda (Miniconda or Mamba) if the repo ships environment.yml — verify with conda --version.
• uv (optional but recommended) for fast, reproducible interpreter installation — install via curl -LsSf https://astral.sh/uv/install.sh | sh (Linux/macOS) or irm https://astral.sh/uv/install.ps1 | iex (PowerShell).
• Windows host (primary): PowerShell is the default shell. Use python -m pip (not bare pip) to ensure installs target the correct interpreter. On Windows, venv activation uses .\.venv\Scripts\Activate.ps1 (PowerShell) or .\.venv\Scripts\activate.bat (cmd). If execution policies block activation, run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser.
• Linux/WSL: Bash scripts below are verified for Debian/Ubuntu bases. On Windows, run them inside WSL or translate to PowerShell equivalents (provided where critical).

Procedure

Step 1 — Read the repo's dependency files (order matters)

Knowing which file is authoritative prevents you from installing a half-right environment:

1. Prefer environment.yml / environment.yaml. Conda environment files frequently pin Python itself, plus channels and non-pip (C/CUDA) dependencies that pip cannot install. That makes them the most complete description of the environment.
2. Otherwise use requirements.txt (pip-only dependencies).
3. If both exist, treat environment.yml as the base layer and requirements.txt as a supplement applied inside the created conda env — unless the README says otherwise. The README wins because authors sometimes document a non-obvious install order.

A typical environment.yml that pins Python:

name: nlp-research
channels:
  - conda-forge
  - pytorch
dependencies:
  - python=3.11
  - pip
  - pytorch=2.1.*
  - pip:
      - transformers==4.38.2
      - datasets==2.18.0

The - python=3.11 line is the one you most need to honor.

Step 2 — Snapshot the current environment before changing it

Write a snapshot file before you install or modify anything. This is a debuggable "before" picture: if the reproduction later misbehaves, this file tells you exactly which interpreter and which packages you started from.

HARD RULE: Always snapshot before any pip install or conda env create. No exceptions.

Capture, in order:

1. python -VV — full version banner (build, compiler). The Python version is the single most common root cause, so it leads.
2. python -m pip --version — confirms which pip maps to which interpreter (they can disagree).
3. python -m pip freeze — the exact pre-existing package set.

PowerShell (Windows host):

$logFile = "~\python_int.txt"
python -VV              | Out-File -FilePath $logFile -Encoding utf8
python -m pip --version | Out-File -FilePath $logFile -Encoding utf8 -Append
python -m pip freeze    | Out-File -FilePath $logFile -Encoding utf8 -Append

Bash (Linux/WSL):

set -euo pipefail
LOG_FILE="/root/python_int.txt"
mkdir -p "$(dirname "$LOG_FILE")"
python -VV              >  "$LOG_FILE"
python -m pip --version >> "$LOG_FILE"
python -m pip freeze    >> "$LOG_FILE"

Step 3 — Compare the repo's required Python against the running interpreter

Compare the repo's required Python major/minor against the running interpreter.

• If they match: install straight from the repo's dependency files (no ad-hoc upgrades), then run an import/smoke test.
• If they differ: build a matching interpreter in an isolated location first. HARD RULE: Never overwrite the system Python. OS tooling and other projects depend on it; replacing it in place can break the whole image. A fresh, isolated venv keeps the change contained.

You can detect mismatches programmatically before running any shell flow. The typed helper below parses environment.yml, compares only major/minor (research repos rarely need an exact patch), and returns a process exit code so a calling script can branch on it:

#!/usr/bin/env python3
"""Decide whether the running interpreter satisfies a repo's Python pin.

Exit codes:
    0  match, or no explicit pin found
    1  the dependency file could not be read/parsed
    2  wrong command-line usage
    3  mismatch -- build a matching interpreter before installing
"""
from __future__ import annotations

import re
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Final, Optional

_PYTHON_PIN: Final[re.Pattern[str]] = re.compile(
    r"^\s*-\s*python\s*(?P<op>>=|<=|==|~=|=|>|<)?\s*(?P<version>\d+(?:\.\d+){0,2})"
)


class DependencyFileError(RuntimeError):
    """Raised when a dependency file is missing, unreadable, or malformed."""


@dataclass(frozen=True)
class PythonRequirement:
    major: int
    minor: Optional[int]
    patch: Optional[int]
    operator: str
    source: Path
    raw_line: str

    def is_satisfied_by(self, version: tuple[int, int, int]) -> bool:
        running_major, running_minor, _patch = version
        if running_major != self.major:
            return False
        if self.minor is None:
            return True
        if self.operator in {"=", "=="}:
            return running_minor == self.minor
        if self.operator == ">=":
            return running_minor >= self.minor
        if self.operator == "<=":
            return running_minor <= self.minor
        if self.operator == ">":
            return running_minor > self.minor
        if self.operator == "<":
            return running_minor < self.minor
        if self.operator == "~=":
            return running_minor >= self.minor
        return running_minor == self.minor


def parse_python_requirement(env_file: Path) -> Optional[PythonRequirement]:
    if not env_file.exists():
        raise DependencyFileError(f"dependency file not found: {env_file}")
    if not env_file.is_file():
        raise DependencyFileError(f"expected a file but found a directory: {env_file}")
    try:
        text: str = env_file.read_text(encoding="utf-8")
    except OSError as exc:
        raise DependencyFileError(f"could not read {env_file}: {exc}") from exc

    for line in text.splitlines():
        match = _PYTHON_PIN.match(line)
        if match is None:
            continue
        parts: list[int] = [int(part) for part in match.group("version").split(".")]
        return PythonRequirement(
            major=parts[0],
            minor=parts[1] if len(parts) > 1 else None,
            patch=parts[2] if len(parts) > 2 else None,
            operator=match.group("op") or "=",
            source=env_file,
            raw_line=line.strip(),
        )
    return None


def current_python() -> tuple[int, int, int]:
    info = sys.version_info
    return (info.major, info.minor, info.micro)


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print(f"usage: {argv[0]} <path-to-environment.yml>", file=sys.stderr)
        return 2

    env_file: Path = Path(argv[1]).expanduser()
    try:
        requirement = parse_python_requirement(env_file)
    except DependencyFileError as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 1

    running = current_python()
    running_str = ".".join(str(part) for part in running)

    if requirement is None:
        print(
            f"No explicit Python pin in {env_file.name}; running {running_str}. "
            "Proceed, but treat import errors as a likely version problem."
        )
        return 0

    if requirement.is_satisfied_by(running):
        print(f"OK: Python {running_str} satisfies '{requirement.raw_line}'.")
        return 0

    minor = requirement.minor if requirement.minor is not None else "x"
    print(
        f"MISMATCH: repo wants python {requirement.operator}{requirement.major}.{minor} "
        f"(line: '{requirement.raw_line}') but you are running {running_str}. "
        "Build a matching interpreter before installing (see Step 4).",
        file=sys.stderr,
    )
    return 3


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Run it:

python check_python_pin.py environment.yml
# Exit 0 = OK, Exit 3 = mismatch (build matching interpreter)

Step 4 — Build a matching interpreter if needed (isolated, never system)

HARD RULE: Never overwrite the system Python. Always create an isolated venv.

Bash (Linux/WSL) with uv:

#!/usr/bin/env bash
set -euo pipefail

REQUIRED_PYTHON="3.12.5"
VENV_DIR="/opt/py312"

# Install uv only if missing (idempotent re-runs).
if ! command -v uv >/dev/null 2>&1; then
  apt-get update
  apt-get install -y --no-install-recommends curl ca-certificates
  rm -rf /var/lib/apt/lists/*
  curl -LsSf https://astral.sh/uv/install.sh | sh
  export PATH="/root/.local/bin:$PATH"
fi

uv python install "$REQUIRED_PYTHON"
uv venv --python "$REQUIRED_PYTHON" "$VENV_DIR"

PYBIN="$VENV_DIR/bin/python"
"$PYBIN" -VV
"$PYBIN" -m pip install -U pip setuptools wheel

PowerShell (Windows host) with uv:

$REQUIRED_PYTHON = "3.12.5"
$VENV_DIR = "~\venvs\py312"

# Install uv if missing.
if (-not (Get-Command uv -ErrorAction SilentlyContinue)) {
    irm https://astral.sh/uv/install.ps1 | iex
    $env:PATH = "$env:USERPROFILE\.local\bin;$env:PATH"
}

uv python install $REQUIRED_PYTHON
uv venv --python $REQUIRED_PYTHON $VENV_DIR

$PYBIN = "$VENV_DIR\Scripts\python.exe"
& $PYBIN -VV
& $PYBIN -m pip install -U pip setuptools wheel

Step 5 — Install from the repo's dependency files

HARD RULE: Install exactly what the repo pins. Do NOT add --upgrade. Those pins are the contract; replacing them with newer releases is how results silently diverge.

Path A: Repository with environment.yml (conda)

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/root/python_int.txt"
ENV_FILE="environment.yml"

if [[ ! -f "$ENV_FILE" ]]; then
  echo "error: $ENV_FILE not found in $(pwd); this repo may use requirements.txt instead." >&2
  exit 1
fi

if ! command -v conda >/dev/null 2>&1; then
  echo "error: conda not on PATH; install Miniconda/Mamba or use the requirements.txt path." >&2
  exit 1
fi

# Snapshot before changing anything.
mkdir -p "$(dirname "$LOG_FILE")"
python -VV              >  "$LOG_FILE"
python -m pip --version >> "$LOG_FILE"
python -m pip freeze    >> "$LOG_FILE"

# Read the declared env name from the file rather than hard-coding a guess.
ENV_NAME="$(grep -E '^[[:space:]]*name:' "$ENV_FILE" | head -n1 | awk '{print $2}' || true)"
if [[ -z "${ENV_NAME:-}" ]]; then
  echo "error: no 'name:' field in $ENV_FILE; add one or pass --name to 'conda env create'." >&2
  exit 1
fi

conda env create -f "$ENV_FILE"

# 'conda activate' needs the shell hook sourced in non-interactive scripts.
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "$ENV_NAME"

# Smoke test: importing the heavy libs surfaces version/ABI breakage right away.
python - <<'PY'
import importlib

for module_name in ("torch", "transformers"):
    try:
        module = importlib.import_module(module_name)
        version = getattr(module, "__version__", "unknown")
        print(f"{module_name} {version} OK")
    except Exception as exc:
        print(f"{module_name} import FAILED: {exc!r}")
PY

PowerShell equivalent for conda activation on Windows:

conda env create -f environment.yml
conda activate nlp-research  # Use the env name from environment.yml

# Smoke test
python -c "import torch; print(f'torch {torch.__version__} OK')"
python -c "import transformers; print(f'transformers {transformers.__version__} OK')"

Path B: Repository with requirements.txt only

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/root/python_int.txt"
REQ_FILE="requirements.txt"

if [[ ! -f "$REQ_FILE" ]]; then
  echo "error: $REQ_FILE not found in $(pwd)." >&2
  exit 1
fi

# Snapshot before installing.
mkdir -p "$(dirname "$LOG_FILE")"
python -VV              >  "$LOG_FILE"
python -m pip --version >> "$LOG_FILE"
python -m pip freeze    >> "$LOG_FILE"

# Install exactly what the repo pins; do NOT add --upgrade.
python -m pip install -r "$REQ_FILE"

PowerShell equivalent:

$logFile = "~\python_int.txt"
python -VV              | Out-File -FilePath $logFile -Encoding utf8
python -m pip --version | Out-File -FilePath $logFile -Encoding utf8 -Append
python -m pip freeze    | Out-File -FilePath $logFile -Encoding utf8 -Append

python -m pip install -r requirements.txt

Path C: Python version mismatch — build matching interpreter, then install

#!/usr/bin/env bash
set -euo pipefail

REQUIRED_PYTHON="3.11.8"
VENV_DIR="/opt/py311"
REQ_FILE="requirements.txt"

if [[ ! -f "$REQ_FILE" ]]; then
  echo "error: $REQ_FILE not found in $(pwd)." >&2
  exit 1
fi

if ! command -v uv >/dev/null 2>&1; then
  apt-get update
  apt-get install -y --no-install-recommends curl ca-certificates
  rm -rf /var/lib/apt/lists/*
  curl -LsSf https://astral.sh/uv/install.sh | sh
  export PATH="/root/.local/bin:$PATH"
fi

uv python install "$REQUIRED_PYTHON"
uv venv --python "$REQUIRED_PYTHON" "$VENV_DIR"

PYBIN="$VENV_DIR/bin/python"
"$PYBIN" -VV
"$PYBIN" -m pip install -U pip setuptools wheel
"$PYBIN" -m pip install -r "$REQ_FILE"

# Confirm the interpreter really is 3.11 before trusting any results.
"$PYBIN" - <<'PY'
import sys

major_minor = sys.version_info[:2]
assert major_minor == (3, 11), f"expected Python 3.11, got {'.'.join(map(str, major_minor))}"
print("interpreter OK:", sys.version.split()[0])
PY

Pitfalls

• Skipping the snapshot step. Without a "before" picture of the interpreter and packages, you cannot diagnose what drifted when reproduction fails. Always write the snapshot file before the first install.
• Overwriting the system Python. OS tooling and other projects depend on it. Always create an isolated venv or conda env. Never apt-get upgrade python3 or replace the system interpreter in place.
• Adding --upgrade to pip install -r requirements.txt. This is the single most common way to break a reproduction. The pins in requirements.txt are the contract. Install them exactly.
• Assuming pip maps to the same interpreter as python. They can disagree, especially on systems with multiple Python installations. Always use python -m pip (not bare pip) to guarantee installs target the correct interpreter.
• Ignoring environment.yml when both files exist. Conda env files pin Python itself plus C/CUDA dependencies that pip cannot install. If both exist, environment.yml is the base layer unless the README says otherwise.
• Hard-coding the conda env name. Read it from the name: field in environment.yml instead of guessing. A wrong env name means you install into a non-existent or wrong environment.
• Forgetting to source the conda shell hook in non-interactive scripts. conda activate does not work in bash scripts without source "$(conda info --base)/etc/profile.d/conda.sh" first. On Windows PowerShell, conda activate works if conda's hook is initialized (conda init powershell).
• Treating deprecation warnings as noise. They often flag exactly the API the repo relies on. Read them before deciding a version bump is safe.
• Blanket pip install --upgrade for a CVE fix. Bump only the affected dependency to a specific known-good version, then re-run the smoke test. Don't upgrade everything at once.
• Not smoke-testing heavy libraries after install. Importing torch and transformers immediately surfaces ABI/CUDA mismatches. If you skip this, the error surfaces deep inside a training run where it is much harder to diagnose.
• Windows execution policy blocking venv activation. Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser if PowerShell refuses to execute Activate.ps1.

Verification

Each check below maps to a failure mode this skill prevents. A failed check tells you what drifted:

1. Interpreter matches the repo.

   python -VV
   # or for isolated venv:
   "$PYBIN" -VV
   

   Output must report the major/minor the repo declares. This is the most common root cause — confirm it first.

2. Declared dependencies are present at the pinned versions.

   python -m pip freeze
   # or:
   conda list
   

   Output must show the versions from requirements.txt / environment.yml — not newer ones substituted by an accidental upgrade.

3. Heavy libraries import cleanly.

   python -c "import torch; print(f'torch {torch.__version__} OK')"
   python -c "import transformers; print(f'transformers {transformers.__version__} OK')"
   

   This catches ABI/CUDA mismatches before they surface mid-run.

4. No unresolved version conflicts.

   python -m pip check
   

   Must report no broken requirements. A conflict here means a dependency was bumped out of band.

5. A representative example or the test suite runs.

   python examples/train.py --dry-run
   # or:
   python -m pytest tests/ -x
   

   Executing one of the repo's example scripts (or its tests) is the real proof the environment is usable, not just installable.

6. Security patches were applied deliberately, not by blanket upgrade. If you bumped a dependency for a CVE, confirm you changed only that pin to a specific version and re-ran the smoke test.

7. The environment is isolated.

   which python
   # Should point to the venv or conda env, NOT /usr/bin/python or system Python
   

   On Windows:

   Get-Command python | Select-Object -ExpandProperty Source
   # Should point to the venv or conda env, NOT C:\Python3xx\python.exe
   

   Installs must land in the project's conda env or venv, not the system interpreter, so this repo can't disturb other projects (and vice versa).

Related Skills

• python-environment-setup — General Python environment configuration when reproduction guarantees are not the goal.
• conda-environment-management — Conda-specific operations (channels, solver, exporting an env).
• docker-containerization — Container-based isolation when you need the whole OS layer pinned, not just Python.
• dependency-resolution — Untangling conflicting transitive pins after the interpreter already matches.