# Foundations Linux Fundamentals

> Linux CLI basics: cp/mv/rm, grep/awk/find, pipes, chmod, gzip/tar, wget/scp/rsync, ps/kill. Use when writing a bash pipeline, inspecting FASTA/FASTQ/BAM/BED/VCF on a server, or filtering lines with grep/awk.

- Skill: `pavel-kravchenko/foundations-linux-fundamentals` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/foundations-linux-fundamentals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/foundations-linux-fundamentals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/foundations-linux-fundamentals

---


# Linux Fundamentals for Bioinformatics

## When to Use

- Writing or debugging a shell pipeline that processes FASTA/FASTQ/SAM/BAM/BED/VCF/GTF files.
- Navigating a remote server/cluster, moving/copying/deleting files, or setting up a project directory tree.
- Filtering, counting, or reformatting text files with `grep`/`awk`/`cut`/`sort`/`uniq` instead of writing a script.
- Downloading reference genomes/annotations (`wget`/`curl`) or syncing data to/from a remote host (`scp`/`rsync`).
- Diagnosing "permission denied", background job management, or disk/CPU/RAM usage on a shared machine.

## Version Compatibility

Applies to any modern Linux distro with GNU coreutils/bash ≥4 (Ubuntu 20.04+, CentOS/Rocky 8+, most HPC clusters). `awk` examples assume GNU awk (gawk) or POSIX awk — both work for the patterns below. macOS BSD variants of `sed`/`find` differ slightly (e.g., `sed -i ''` needs an explicit empty arg) but are noted where relevant.

## Prerequisites

- A terminal/SSH session; no packages to install (`gzip`, `grep`, `awk`, `find`, `wget`/`curl`, `rsync`, `tar` are on virtually every Linux box; install `rsync`/`htop` via `apt`/`yum` if missing).
- Concept: shells expand wildcards/redirection *before* the command runs (see Pitfalls).

## Navigation & File Operations

**Goal:** Move around the filesystem and manage bioinformatics project directories/files safely.
**Approach:** Use `pwd`/`cd`/`ls` to orient, `mkdir -p` with brace expansion to scaffold a project in one line, and `cp -r`/`mv`/`rm -rf` for file management (note `rm` has no undo).

```bash
# Orientation
pwd                 # where am I
ls -lah             # long + hidden + human sizes -- the daily-driver combo
cd -                # jump back to the previous directory

# Scaffold a standard RNA-seq project tree in one command
mkdir -p RNA_Seq/{00_raw,01_qc,02_trimmed,03_aligned,04_counts,scripts,logs}
touch RNA_Seq/scripts/run_pipeline.sh

# Copy, move, delete
cp -r results_dir/ results_backup/   # -r required for directories
mv unaligned.bam aligned.bam         # rename
rm -rf tmp_dir/                      # permanent -- no undo, never on / or ~

# Inspect files without decompressing
zcat sample.fastq.gz | head -8       # first 2 FASTQ reads (4 lines/read)
zgrep "PASS" variants.vcf.gz         # grep inside a .gz file
head -n 4 reads.fastq                # first FASTQ read
tail -f pipeline.log                 # follow a running log
wc -l reads.fastq                    # count lines
```

## Pipes, grep, awk, and find

**Goal:** Chain small commands into a pipeline to filter/count/transform genomic text files without writing a script.
**Approach:** Redirect with `>`/`>>`/`2>&1`, pipe stdout with `|`, filter lines with `grep`, do column math with `awk`, and locate files by name/size/time with `find`.

```bash
# Redirection operators
# >    overwrite stdout to a file      >>   append stdout to a file
# 2>   redirect stderr                 2>&1 merge stderr into stdout
# |    pipe one command's stdout into the next command's stdin

# Common bioinformatics pipelines
grep -c "^>" proteins.fasta                                   # count FASTA sequences
zcat sample.fastq.gz | wc -l | awk '{print $1/4, "reads"}'     # count reads in a gz FASTQ
grep -v "^#" variants.vcf | cut -f1 | sort | uniq -c           # variants per chromosome
cut -f1 regions.bed | sort | uniq -c | sort -rn                # regions per chromosome

# grep quick reference
grep -c "^>" genome.fa           # count sequences in FASTA
grep -v "^#" variants.vcf        # strip VCF header lines
grep -i -w "brca1" gencode.gtf   # case-insensitive, whole-word gene search
grep -B 1 "GAATTC" seqs.fasta    # show the header line above an EcoRI site match
grep -n "ERROR" pipeline.log     # show line numbers of matches

# awk on tab-delimited genomic formats (BED/VCF/GTF are tab-delimited)
awk -F'\t' '{print $1, $4, $5}' annotations.gtf                # print chrom, start, feature
awk '($3 - $2) > 1000' regions.bed                              # keep intervals longer than 1kb
awk '{sum += $4} END {print "Total reads:", sum}' counts.bed    # sum a count column
awk -F'\t' '{print $1 ":" $2 "-" $3}' regions.bed               # BED -> "chr:start-end" string

# find files by name / size / age, then act on them
find . -name "*.bam" -exec samtools index {} \;    # index every BAM under cwd
find . -size +1G                                    # files larger than 1 GB
find . -mtime -7 -name "*.log"                      # logs modified in the last 7 days
find . -name "*.fastq.gz" | xargs du -sh            # size of every gz FASTQ
```

## Wildcards, Permissions, and Compression

**Goal:** Match groups of files, set/read executable permissions, and compress/download data efficiently.
**Approach:** Let the shell expand globs before the command sees them; use octal `chmod` codes for scripts; prefer `gzip`/`zcat`/`zgrep` over manual decompression; use `wget -c`/`rsync -avz` for large transfers that may be interrupted.

```bash
# Wildcards (expanded by the shell, not the command)
ls *.fastq.gz          # all gzipped FASTQ files
ls sample_[123].bam    # only sample_1.bam, sample_2.bam, sample_3.bam
ls chr[0-9]*.fa        # chr1.fa ... chr9.fa (not chr10+)

# Permissions
chmod +x run_pipeline.sh   # make a script executable (most common use)
chmod 755 run_pipeline.sh  # rwxr-xr-x: owner all, group/others read+execute

# Compression -- the standard for sequencing data
gzip -k large_file.fastq          # compress but keep the original (-k)
tar -czvf archive.tar.gz results/ # create a compressed archive (c=create, z=gzip, v=verbose)
tar -xzvf archive.tar.gz          # extract it
tar -tvf archive.tar.gz           # list contents without extracting

# Download / sync reference data
wget -c https://ftp.ensembl.org/path/to/genome.fa.gz   # resumable download
curl -L -O https://example.org/annotation.gtf.gz       # follow redirects, keep filename
rsync -avz local_results/ user@cluster:~/remote_results/  # sync, only transfer changes
scp -r local_dir/ user@server:~/                          # copy a directory to a remote host
```

## Processes and Remote Sessions

**Goal:** Run long jobs in the background, monitor resource usage, and work on a remote server.
**Approach:** Append `&` to background a job, manage it with `jobs`/`fg`/`bg`/`kill`, and inspect system load before launching heavy alignments.

```bash
bwa mem ref.fa reads.fastq > aligned.sam &  # run in the background
jobs                                        # list background/suspended jobs
kill %1                                     # stop job 1 (or `kill -9 PID` to force)

ps aux | grep bowtie2   # find a specific running process
top -u "$(whoami)"      # interactive view of only your processes
free -h                 # RAM usage, human-readable
nproc                   # number of CPU cores available
df -h                   # disk space per filesystem
du -sh results/         # total size of a directory

ssh user@cluster.university.edu   # connect to a remote server
```

## Bioinformatics File Formats

| Format | Extension | Content |
|---|---|---|
| FASTA | `.fa`, `.fasta` | Sequences (genome, protein) |
| FASTQ | `.fq`, `.fastq.gz` | Reads + quality scores (4 lines/read) |
| SAM/BAM | `.sam`, `.bam` | Alignments (BAM = binary SAM) |
| BED | `.bed` | Genomic intervals (0-based, half-open) |
| VCF | `.vcf` | Variant calls (1-based) |
| GFF/GTF | `.gff`, `.gtf` | Gene annotations (1-based) |

## Vim Survival Guide

```text
vim filename    open file
i               insert mode (type text)
Esc             back to normal mode
:w              save
:q              quit
:wq             save and quit
:q!             quit without saving
/pattern        search forward
n / N           next / previous match
dd              delete line
u               undo
:%s/old/new/g   replace all occurrences in file
```

## Pitfalls

- **Spaces around `=` in Bash:** `var = value` is wrong; `var=value` is right — the space makes Bash treat `var` as a command name.
- **`rm` is permanent:** no trash bin. Double-check before `rm -rf`; never run `rm -rf /` or `rm -rf ~`.
- **Pipes discard stderr:** `cmd1 | cmd2` only passes stdout between commands. Add `2>&1` if you also need to capture/pipe error messages.
- **Wildcards expand before the command runs:** `rm *.fastq` — the shell does the glob expansion, not `rm`. If nothing matches, most shells error ("no such file") rather than silently doing nothing.
- **Relative vs. absolute paths:** cron jobs and cluster submission scripts often run from an unexpected working directory — always use absolute paths in scripts.
- **Compressed files:** use `zcat`/`zgrep`/`zless` on `.gz` files; decompressing a multi-GB FASTQ just to `grep` it wastes disk and time.
- **`find -exec` vs `xargs`:** `-exec cmd {} \;` runs the command once per file (slow for thousands of files); `find ... | xargs cmd` batches arguments and is much faster for bulk operations.

## See Also

- `bio-sequence-io-compressed-files` — working with gzipped FASTA/FASTQ in Python (BioPython/gzip).
- `bio-workflow-management-snakemake-workflows` — turning ad-hoc shell pipelines into reproducible workflows.
- `bio-genome-intervals-bed-file-basics` — BED coordinate semantics referenced above.
- `bio-alignment-files-sam-bam-basics` — SAM/BAM structure referenced above.