linux-git-bash

name	linux-git-bash
description	Write set -euo pipefail bash pipelines, parse FASTA/FASTQ/VCF/GTF/BED with grep/awk/sed and BAM with samtools, and run git workflows. Use when writing/debugging shell scripts or fixing git/BOM/CRLF issues.
tool_type	bash
primary_tool	bash

Linux, Git & Bash for Bioinformatics

When to Use

• Processing FASTA/FASTQ/BAM/VCF files from the command line (grep/awk/sed/samtools)
• Writing robust batch pipeline scripts that loop over samples with error handling
• Setting up reproducible analysis projects with git (branches, .gitignore, undoing mistakes)
• Debugging silent bash failures, unquoted variables, or empty-glob loops
• Diagnosing garbled sequence data (BOM, Windows line endings, mixed encodings)

Version Compatibility

• bash ≥ 4.4 (associative arrays, ${var/pat/repl}); tested on bash 5.x
• git ≥ 2.30
• samtools ≥ 1.15
• Python ≥ 3.10 (for encoding-repair helpers; chardet optional)

Prerequisites

• GNU coreutils (grep, awk, sed, cut, sort) and samtools on PATH
• A git identity configured (git config user.name/user.email)
• Comfort with shell variables/quoting; see bio-sequence-io-* skills for the file formats these commands operate on

Goal: Count/summarize records in FASTA, FASTQ, and VCF without loading them into memory. Approach: Never cat/grep binary formats (BAM) directly — use samtools. For text formats, exploit fixed record structure (FASTQ = 4 lines/record) with awk/grep -c.

# Count bio-file records
grep -c "^>" proteins.fasta                    # FASTA sequences
zcat sample.fastq.gz | wc -l | awk '{print $1/4}'  # FASTQ reads
grep -v "^#" variants.vcf | wc -l              # VCF variants (skip header)

# VCF chromosome distribution
grep -v "^#" variants.vcf | cut -f1 | sort | uniq -c | sort -rn

# FASTQ -> FASTA
sed -n '1~4s/^@/>/p;2~4p' reads.fastq > reads.fasta

# Average read length
awk 'NR%4==2 {sum+=length($0); count++} END {print sum/count}' reads.fastq

# Extract gene names from GTF
awk -F'\t' '$3=="gene"' gencode.gtf \
  | grep -o 'gene_name "[^"]*"' \
  | sed 's/gene_name "//;s/"//' | sort -u

# BED feature lengths
awk -F'\t' '{print $0 "\t" $3-$2}' regions.bed

# Parallel FastQC across a directory
find data/ -name "*.fastq.gz" | xargs -P 4 -I {} fastqc {} -o results/qc/

# Paired-end R2 path from R1 path
r2="${r1/_R1/_R2}"; sample=$(basename "$r1" _R1.fastq.gz)

samtools (never use cat/grep on BAM — it's binary)

samtools view aligned.bam | head -5              # View as SAM
samtools view -c -F 4 aligned.bam                # Count aligned reads
samtools index aligned.bam                       # Required before random access
samtools view aligned.bam chr17:7571720-7590868  # Region extract
samtools flagstat aligned.bam                    # Alignment statistics
samtools sort -o sorted.bam unsorted.bam         # Sort by coordinate

Goal: Write a batch pipeline script that fails loudly instead of silently producing garbage. Approach: set -euo pipefail at the top, validate every input, log with timestamps, guard globs against zero matches, and clean up temp files with trap ... EXIT.

#!/bin/bash
set -euo pipefail

INPUT_DIR="${1:-}"
OUTPUT_DIR="${2:-results}"
LOGFILE="${OUTPUT_DIR}/pipeline.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOGFILE"; }
cleanup() { rm -f /tmp/pipeline_*.tmp 2>/dev/null || true; }
trap cleanup EXIT

[[ -z "$INPUT_DIR" ]] && { echo "Usage: $0 <input_dir> [output_dir]"; exit 1; }
[[ -d "$INPUT_DIR" ]] || { echo "ERROR: Not a directory: $INPUT_DIR"; exit 1; }
command -v samtools &>/dev/null || { log "ERROR: samtools not installed"; exit 1; }

mkdir -p "$OUTPUT_DIR"
log "Starting pipeline. Input: $INPUT_DIR"

count=0
for fastq in "${INPUT_DIR}"/*.fastq.gz; do
    [[ -f "$fastq" ]] || { log "No .fastq.gz files found";  1; }  
    sample=$(  .fastq.gz)
     
    fastqc  -o  -t 4

 

# Sample sheet generator (paired-end): builds a TSV from *_R1/_R2 pairs
#!/bin/bash
set -euo pipefail
input_dir="${1:-.}"; output_file="${2:-sample_sheet.tsv}"
echo -e "sample_id\tR1_path\tR2_path" > "$output_file"
for r1 in "${input_dir}"/*_R1.fastq.gz; do
    [[ -f "$r1" ]] || { echo "No *_R1.fastq.gz files"; exit 1; }
    r2="${r1/_R1/_R2}"; sample=$(basename "$r1" _R1.fastq.gz)
    [[ -f "$r2" ]] || { echo "WARNING: Missing R2 for $sample"; continue; }
    echo -e "${sample}\t${r1}\t${r2}" >> "$output_file"
done

Git Commands

Command	Purpose
git log --oneline --graph	Visual history
git diff --staged	Staged vs last commit
git restore file	Discard working-dir changes
git restore --staged file	Unstage
git reset --soft HEAD~1	Undo commit, keep staged
git revert <hash>	Safe undo (new commit)
git stash / git stash pop	Shelve uncommitted changes
git log -S "alpha"	Find commits that changed a string
git tag -a v1.0 -m "msg" + git push --tags	Annotated release tag

# .gitignore for bioinformatics repos: keep large/binary/generated data out of git
*.fastq *.fastq.gz *.fq.gz
*.bam *.bam.bai *.sam *.cram
*.vcf *.vcf.gz *.bcf *.sra
*.fa *.fasta *.fa.fai *.dict
data/raw/ results/ *.log *.tmp
__pycache__/ *.pyc .ipynb_checkpoints/
.Rhistory .RData .DS_Store .vscode/ .idea/

Goal: Detect and repair mis-encoded or Windows-mangled sequence files before they corrupt a parser. Approach: Try encodings in order of likelihood (utf-8-sig → utf-8 → latin-1), normalize line endings, and strip characters outside the expected alphabet while reporting what was removed.

FASTQ Phred+33: phred = ord(char) - 33, P_error = 10 ** (-phred / 10). Valid range: ASCII 33 (!) to 126 (~).

import unicodedata

def read_text_file(filepath: str) -> str:
    """Read a text file, trying common bioinformatics encodings in priority order.

    1. utf-8-sig: modern standard, also strips a Windows-editor BOM if present.
    2. utf-8: standard, no BOM.
    3. latin-1: never fails (every byte is a valid code point) -- last resort,
       may silently produce wrong characters, so we warn when we fall back to it.
    """
    for encoding in ('utf-8-sig', 'utf-8', 'latin-1'):
        try:
            with open(filepath, encoding=encoding) as f:
                content = f.read()
            if encoding == 'latin-1':
                print(f"WARNING: Fell back to latin-1 for {filepath}; check for garbled chars")
            return content
        except UnicodeDecodeError:
            continue
    raise ValueError(f"Could not decode {filepath} with any known encoding")


def sanitize_sequence(seq: str, valid_chars: str = 'ATGCNatgcn') -> str:
    """Remove characters outside the valid alphabet, reporting what was stripped."""
    cleaned, removed = [], []
    for char in seq:
        if char in valid_chars:
            cleaned.append(char)
        elif char not in (, , , ):  
            removed.append()
     removed:
        ()
     .join(cleaned)

Scenario	Solution
Windows file with BOM	open(f, encoding='utf-8-sig')
Windows line endings	text.replace('\r\n', '\n')
Unknown encoding	chardet.detect(raw_bytes) then try UTF-8 -> Latin-1
Binary formats (BAM, gzip)	Always 'rb' mode

Pitfalls

• set -euo pipefail omitted: silent failures cascade — pipelines produce garbage without error messages.
• Unquoted variables: ls $file breaks on spaces; always use "$file".
• git add . in large repos: accidentally stages .bam/.fastq.gz; use git add <specific files> and set up .gitignore first.
• Spaces around = in bash: var = "value" is a syntax error; var="value" is correct.
• cat large.bam or grep pattern file.bam: BAM is binary — use samtools view instead.
• for f in *.fastq.gz with no matches: $f becomes the literal string *.fastq.gz; guard with [[ -f "$f" ]].
• cleanup trap failing: use || true so cleanup errors don't trigger set -e exit inside the trap.
• Committing large data files: GitHub rejects files >100 MB; configure .gitignore before the first commit.
• git reset --hard: permanently destroys uncommitted work; prefer git restore or git reset --soft.
• Windows \r\n line endings in FASTA: a trailing \r corrupts parsers; run dos2unix or normalize in Python.
• Non-breaking space U+00A0 in sequences: looks like a space, breaks parsers when copy-pasted from PDF/Word.

See Also

• bio-sequence-io-read-sequences — parsing FASTA/FASTQ once files are clean
• bio-alignment-files-sam-bam-basics — samtools/BAM concepts referenced here
• bio-variant-calling-vcf-basics — VCF structure behind the grep/awk one-liners
• bio-workflow-management-snakemake-workflows — graduating ad-hoc bash loops to a real pipeline