Foundations Linux Fundamentals

Linux Fundamentals for Bioinformatics

When to Use

• Writing or debugging a shell pipeline that processes FASTA/FASTQ/SAM/BAM/BED/VCF/GTF files.
• Navigating a remote server/cluster, moving/copying/deleting files, or setting up a project directory tree.
• Filtering, counting, or reformatting text files with grep/awk/cut/sort/uniq instead of writing a script.
• Downloading reference genomes/annotations (wget/curl) or syncing data to/from a remote host (scp/rsync).
• Diagnosing "permission denied", background job management, or disk/CPU/RAM usage on a shared machine.

Version Compatibility

Applies to any modern Linux distro with GNU coreutils/bash ≥4 (Ubuntu 20.04+, CentOS/Rocky 8+, most HPC clusters). awk examples assume GNU awk (gawk) or POSIX awk — both work for the patterns below. macOS BSD variants of sed/find differ slightly (e.g., sed -i '' needs an explicit empty arg) but are noted where relevant.

Prerequisites

• A terminal/SSH session; no packages to install (gzip, grep, awk, find, wget/curl, rsync, tar are on virtually every Linux box; install rsync/htop via apt/yum if missing).
• Concept: shells expand wildcards/redirection before the command runs (see Pitfalls).

Navigation & File Operations

Goal: Move around the filesystem and manage bioinformatics project directories/files safely. Approach: Use pwd/cd/ls to orient, mkdir -p with brace expansion to scaffold a project in one line, and cp -r/mv/rm -rf for file management (note rm has no undo).

# Orientation
pwd                 # where am I
ls -lah             # long + hidden + human sizes -- the daily-driver combo
cd -                # jump back to the previous directory

# Scaffold a standard RNA-seq project tree in one command
mkdir -p RNA_Seq/{00_raw,01_qc,02_trimmed,03_aligned,04_counts,scripts,logs}
touch RNA_Seq/scripts/run_pipeline.sh

# Copy, move, delete
cp -r results_dir/ results_backup/   # -r required for directories
mv unaligned.bam aligned.bam         # rename
rm -rf tmp_dir/                      # permanent -- no undo, never on / or ~

# Inspect files without decompressing
zcat sample.fastq.gz | head -8       # first 2 FASTQ reads (4 lines/read)
zgrep "PASS" variants.vcf.gz         # grep inside a .gz file
head -n 4 reads.fastq                # first FASTQ read
tail -f pipeline.log                 # follow a running log
wc -l reads.fastq                    # count lines

Pipes, grep, awk, and find

Goal: Chain small commands into a pipeline to filter/count/transform genomic text files without writing a script. Approach: Redirect with >/>>/2>&1, pipe stdout with |, filter lines with grep, do column math with awk, and locate files by name/size/time with find.

# Redirection operators
# >    overwrite stdout to a file      >>   append stdout to a file
# 2>   redirect stderr                 2>&1 merge stderr into stdout
# |    pipe one command's stdout into the next command's stdin

# Common bioinformatics pipelines
grep -c "^>" proteins.fasta                                   # count FASTA sequences
zcat sample.fastq.gz | wc -l | awk '{print $1/4, "reads"}'     # count reads in a gz FASTQ
grep -v "^#" variants.vcf | cut -f1 | sort | uniq -c           # variants per chromosome
cut -f1 regions.bed | sort | uniq -c | sort -rn                # regions per chromosome

# grep quick reference
grep -c "^>" genome.fa           # count sequences in FASTA
grep -v "^#" variants.vcf        # strip VCF header lines
grep -i -w "brca1" gencode.gtf   # case-insensitive, whole-word gene search
grep -B 1 "GAATTC" seqs.fasta    # show the header line above an EcoRI site match
grep -n "ERROR" pipeline.log     # show line numbers of matches

# awk on tab-delimited genomic formats (BED/VCF/GTF are tab-delimited)
awk -F'\t' '{print $1, $4, $5}' annotations.gtf                # print chrom, start, feature
awk '($3 - $2) > 1000' regions.bed                              # keep intervals longer than 1kb
awk '{sum += $4} END {print "Total reads:", sum}' counts.bed    # sum a count column
awk -F'\t' '{print $1 ":" $2 "-" $3}' regions.bed               # BED -> "chr:start-end" string

# find files by name / size / age, then act on them
find . -name "*.bam" -exec samtools index {} \;    # index every BAM under cwd
find . -size +1G                                    # files larger than 1 GB
find . -mtime -7 -name "*.log"                      # logs modified in the last 7 days
find . -name "*.fastq.gz" | xargs du -sh            # size of every gz FASTQ

Wildcards, Permissions, and Compression

Goal: Match groups of files, set/read executable permissions, and compress/download data efficiently. Approach: Let the shell expand globs before the command sees them; use octal chmod codes for scripts; prefer gzip/zcat/zgrep over manual decompression; use wget -c/rsync -avz for large transfers that may be interrupted.

# Wildcards (expanded by the shell, not the command)
ls *.fastq.gz          # all gzipped FASTQ files
ls sample_[123].bam    # only sample_1.bam, sample_2.bam, sample_3.bam
ls chr[0-9]*.fa        # chr1.fa ... chr9.fa (not chr10+)

# Permissions
chmod +x run_pipeline.sh   # make a script executable (most common use)
chmod 755 run_pipeline.sh  # rwxr-xr-x: owner all, group/others read+execute

# Compression -- the standard for sequencing data
gzip -k large_file.fastq          # compress but keep the original (-k)
tar -czvf archive.tar.gz results/ # create a compressed archive (c=create, z=gzip, v=verbose)
tar -xzvf archive.tar.gz          # extract it
tar -tvf archive.tar.gz           # list contents without extracting

# Download / sync reference data
wget -c https://ftp.ensembl.org/path/to/genome.fa.gz   # resumable download
curl -L -O https://example.org/annotation.gtf.gz       # follow redirects, keep filename
rsync -avz local_results/ user@cluster:~/remote_results/  # sync, only transfer changes
scp -r local_dir/ user@server:~/                          # copy a directory to a remote host

Processes and Remote Sessions

Goal: Run long jobs in the background, monitor resource usage, and work on a remote server. Approach: Append & to background a job, manage it with jobs/fg/bg/kill, and inspect system load before launching heavy alignments.

bwa mem ref.fa reads.fastq > aligned.sam &  # run in the background
jobs                                        # list background/suspended jobs
kill %1                                     # stop job 1 (or `kill -9 PID` to force)

ps aux | grep bowtie2   # find a specific running process
top -u "$(whoami)"      # interactive view of only your processes
free -h                 # RAM usage, human-readable
nproc                   # number of CPU cores available
df -h                   # disk space per filesystem
du -sh results/         # total size of a directory

ssh user@cluster.university.edu   # connect to a remote server

Bioinformatics File Formats

Vim Survival Guide

vim filename    open file
i               insert mode (type text)
Esc             back to normal mode
:w              save
:q              quit
:wq             save and quit
:q!             quit without saving
/pattern        search forward
n / N           next / previous match
dd              delete line
u               undo
:%s/old/new/g   replace all occurrences in file

Pitfalls

• Spaces around = in Bash: var = value is wrong; var=value is right — the space makes Bash treat var as a command name.
• rm is permanent: no trash bin. Double-check before rm -rf; never run rm -rf / or rm -rf ~.
• Pipes discard stderr: cmd1 | cmd2 only passes stdout between commands. Add 2>&1 if you also need to capture/pipe error messages.
• Wildcards expand before the command runs: rm *.fastq — the shell does the glob expansion, not rm. If nothing matches, most shells error ("no such file") rather than silently doing nothing.
• Relative vs. absolute paths: cron jobs and cluster submission scripts often run from an unexpected working directory — always use absolute paths in scripts.
• Compressed files: use zcat/zgrep/zless on .gz files; decompressing a multi-GB FASTQ just to grep it wastes disk and time.
• find -exec vs xargs: -exec cmd {} \; runs the command once per file (slow for thousands of files); find ... | xargs cmd batches arguments and is much faster for bulk operations.

See Also

• bio-sequence-io-compressed-files — working with gzipped FASTA/FASTQ in Python (BioPython/gzip).
• bio-workflow-management-snakemake-workflows — turning ad-hoc shell pipelines into reproducible workflows.
• bio-genome-intervals-bed-file-basics — BED coordinate semantics referenced above.
• bio-alignment-files-sam-bam-basics — SAM/BAM structure referenced above.