# Linux Git Bash

> Write set -euo pipefail bash pipelines, parse FASTA/FASTQ/VCF/GTF/BED with grep/awk/sed and BAM with samtools, and run git workflows. Use when writing/debugging shell scripts or fixing git/BOM/CRLF issues.

- Skill: `pavel-kravchenko/linux-git-bash` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/linux-git-bash`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/linux-git-bash/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/linux-git-bash

---


# Linux, Git & Bash for Bioinformatics

## When to Use
- Processing FASTA/FASTQ/BAM/VCF files from the command line (grep/awk/sed/samtools)
- Writing robust batch pipeline scripts that loop over samples with error handling
- Setting up reproducible analysis projects with git (branches, `.gitignore`, undoing mistakes)
- Debugging silent bash failures, unquoted variables, or empty-glob loops
- Diagnosing garbled sequence data (BOM, Windows line endings, mixed encodings)

## Version Compatibility
- bash ≥ 4.4 (associative arrays, `${var/pat/repl}`); tested on bash 5.x
- git ≥ 2.30
- samtools ≥ 1.15
- Python ≥ 3.10 (for encoding-repair helpers; `chardet` optional)

## Prerequisites
- GNU coreutils (`grep`, `awk`, `sed`, `cut`, `sort`) and `samtools` on PATH
- A git identity configured (`git config user.name/user.email`)
- Comfort with shell variables/quoting; see `bio-sequence-io-*` skills for the file formats these commands operate on

**Goal:** Count/summarize records in FASTA, FASTQ, and VCF without loading them into memory.
**Approach:** Never `cat`/`grep` binary formats (BAM) directly — use `samtools`. For text formats, exploit fixed record structure (FASTQ = 4 lines/record) with `awk`/`grep -c`.

```bash
# Count bio-file records
grep -c "^>" proteins.fasta                    # FASTA sequences
zcat sample.fastq.gz | wc -l | awk '{print $1/4}'  # FASTQ reads
grep -v "^#" variants.vcf | wc -l              # VCF variants (skip header)

# VCF chromosome distribution
grep -v "^#" variants.vcf | cut -f1 | sort | uniq -c | sort -rn

# FASTQ -> FASTA
sed -n '1~4s/^@/>/p;2~4p' reads.fastq > reads.fasta

# Average read length
awk 'NR%4==2 {sum+=length($0); count++} END {print sum/count}' reads.fastq

# Extract gene names from GTF
awk -F'\t' '$3=="gene"' gencode.gtf \
  | grep -o 'gene_name "[^"]*"' \
  | sed 's/gene_name "//;s/"//' | sort -u

# BED feature lengths
awk -F'\t' '{print $0 "\t" $3-$2}' regions.bed

# Parallel FastQC across a directory
find data/ -name "*.fastq.gz" | xargs -P 4 -I {} fastqc {} -o results/qc/

# Paired-end R2 path from R1 path
r2="${r1/_R1/_R2}"; sample=$(basename "$r1" _R1.fastq.gz)
```

### samtools (never use cat/grep on BAM — it's binary)
```bash
samtools view aligned.bam | head -5              # View as SAM
samtools view -c -F 4 aligned.bam                # Count aligned reads
samtools index aligned.bam                       # Required before random access
samtools view aligned.bam chr17:7571720-7590868  # Region extract
samtools flagstat aligned.bam                    # Alignment statistics
samtools sort -o sorted.bam unsorted.bam         # Sort by coordinate
```

**Goal:** Write a batch pipeline script that fails loudly instead of silently producing garbage.
**Approach:** `set -euo pipefail` at the top, validate every input, log with timestamps, guard globs against zero matches, and clean up temp files with `trap ... EXIT`.

```bash
#!/bin/bash
set -euo pipefail

INPUT_DIR="${1:-}"
OUTPUT_DIR="${2:-results}"
LOGFILE="${OUTPUT_DIR}/pipeline.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOGFILE"; }
cleanup() { rm -f /tmp/pipeline_*.tmp 2>/dev/null || true; }
trap cleanup EXIT

[[ -z "$INPUT_DIR" ]] && { echo "Usage: $0 <input_dir> [output_dir]"; exit 1; }
[[ -d "$INPUT_DIR" ]] || { echo "ERROR: Not a directory: $INPUT_DIR"; exit 1; }
command -v samtools &>/dev/null || { log "ERROR: samtools not installed"; exit 1; }

mkdir -p "$OUTPUT_DIR"
log "Starting pipeline. Input: $INPUT_DIR"

count=0
for fastq in "${INPUT_DIR}"/*.fastq.gz; do
    [[ -f "$fastq" ]] || { log "No .fastq.gz files found"; exit 1; }  # guard empty glob
    sample=$(basename "$fastq" .fastq.gz)
    log "[$(( ++count ))] Processing: $sample"
    fastqc "$fastq" -o "$OUTPUT_DIR" -t 4
done
log "Done. Processed $count files."
```

```bash
# Sample sheet generator (paired-end): builds a TSV from *_R1/_R2 pairs
#!/bin/bash
set -euo pipefail
input_dir="${1:-.}"; output_file="${2:-sample_sheet.tsv}"
echo -e "sample_id\tR1_path\tR2_path" > "$output_file"
for r1 in "${input_dir}"/*_R1.fastq.gz; do
    [[ -f "$r1" ]] || { echo "No *_R1.fastq.gz files"; exit 1; }
    r2="${r1/_R1/_R2}"; sample=$(basename "$r1" _R1.fastq.gz)
    [[ -f "$r2" ]] || { echo "WARNING: Missing R2 for $sample"; continue; }
    echo -e "${sample}\t${r1}\t${r2}" >> "$output_file"
done
```

### Git Commands
| Command | Purpose |
|---------|---------|
| `git log --oneline --graph` | Visual history |
| `git diff --staged` | Staged vs last commit |
| `git restore file` | Discard working-dir changes |
| `git restore --staged file` | Unstage |
| `git reset --soft HEAD~1` | Undo commit, keep staged |
| `git revert <hash>` | Safe undo (new commit) |
| `git stash` / `git stash pop` | Shelve uncommitted changes |
| `git log -S "alpha"` | Find commits that changed a string |
| `git tag -a v1.0 -m "msg"` + `git push --tags` | Annotated release tag |

```text
# .gitignore for bioinformatics repos: keep large/binary/generated data out of git
*.fastq *.fastq.gz *.fq.gz
*.bam *.bam.bai *.sam *.cram
*.vcf *.vcf.gz *.bcf *.sra
*.fa *.fasta *.fa.fai *.dict
data/raw/ results/ *.log *.tmp
__pycache__/ *.pyc .ipynb_checkpoints/
.Rhistory .RData .DS_Store .vscode/ .idea/
```

**Goal:** Detect and repair mis-encoded or Windows-mangled sequence files before they corrupt a parser.
**Approach:** Try encodings in order of likelihood (`utf-8-sig` → `utf-8` → `latin-1`), normalize line endings, and strip characters outside the expected alphabet while reporting what was removed.

FASTQ Phred+33: `phred = ord(char) - 33`, `P_error = 10 ** (-phred / 10)`. Valid range: ASCII 33 (`!`) to 126 (`~`).

```python
import unicodedata

def read_text_file(filepath: str) -> str:
    """Read a text file, trying common bioinformatics encodings in priority order.

    1. utf-8-sig: modern standard, also strips a Windows-editor BOM if present.
    2. utf-8: standard, no BOM.
    3. latin-1: never fails (every byte is a valid code point) -- last resort,
       may silently produce wrong characters, so we warn when we fall back to it.
    """
    for encoding in ('utf-8-sig', 'utf-8', 'latin-1'):
        try:
            with open(filepath, encoding=encoding) as f:
                content = f.read()
            if encoding == 'latin-1':
                print(f"WARNING: Fell back to latin-1 for {filepath}; check for garbled chars")
            return content
        except UnicodeDecodeError:
            continue
    raise ValueError(f"Could not decode {filepath} with any known encoding")


def sanitize_sequence(seq: str, valid_chars: str = 'ATGCNatgcn') -> str:
    """Remove characters outside the valid alphabet, reporting what was stripped."""
    cleaned, removed = [], []
    for char in seq:
        if char in valid_chars:
            cleaned.append(char)
        elif char not in ('\n', '\r', ' ', '\t'):  # whitespace is expected, not an error
            removed.append(f"'{char}' ({unicodedata.name(char, f'U+{ord(char):04X}')})")
    if removed:
        print(f"WARNING: Removed {removed}")
    return ''.join(cleaned)
```

| Scenario | Solution |
|----------|---------|
| Windows file with BOM | `open(f, encoding='utf-8-sig')` |
| Windows line endings | `text.replace('\r\n', '\n')` |
| Unknown encoding | `chardet.detect(raw_bytes)` then try UTF-8 -> Latin-1 |
| Binary formats (BAM, gzip) | Always `'rb'` mode |

## Pitfalls

- **`set -euo pipefail` omitted**: silent failures cascade — pipelines produce garbage without error messages.
- **Unquoted variables**: `ls $file` breaks on spaces; always use `"$file"`.
- **`git add .` in large repos**: accidentally stages `.bam`/`.fastq.gz`; use `git add <specific files>` and set up `.gitignore` first.
- **Spaces around `=` in bash**: `var = "value"` is a syntax error; `var="value"` is correct.
- **`cat large.bam` or `grep pattern file.bam`**: BAM is binary — use `samtools view` instead.
- **`for f in *.fastq.gz` with no matches**: `$f` becomes the literal string `*.fastq.gz`; guard with `[[ -f "$f" ]]`.
- **`cleanup` trap failing**: use `|| true` so cleanup errors don't trigger `set -e` exit inside the trap.
- **Committing large data files**: GitHub rejects files >100 MB; configure `.gitignore` before the first commit.
- **`git reset --hard`**: permanently destroys uncommitted work; prefer `git restore` or `git reset --soft`.
- **Windows `\r\n` line endings in FASTA**: a trailing `\r` corrupts parsers; run `dos2unix` or normalize in Python.
- **Non-breaking space U+00A0 in sequences**: looks like a space, breaks parsers when copy-pasted from PDF/Word.

## See Also
- `bio-sequence-io-read-sequences` — parsing FASTA/FASTQ once files are clean
- `bio-alignment-files-sam-bam-basics` — samtools/BAM concepts referenced here
- `bio-variant-calling-vcf-basics` — VCF structure behind the grep/awk one-liners
- `bio-workflow-management-snakemake-workflows` — graduating ad-hoc bash loops to a real pipeline