python-bio-file-operations

name	python-bio-file-operations
description	Read/write FASTA, FASTQ, CSV/TSV (BED), JSON, and pickle files in Python using open()/context managers, csv.DictReader/DictWriter, and streaming generators for large genomics files. Use when parsing a FASTA/FASTQ file, writing sequences back out with line wrapping, reading/writing gene expression CSV or BED/TSV files, loading GenBank ORIGIN sequence, saving Python objects with pickle, or handling files too large to fit in memory.
tool_type	python
primary_tool	Python

File Operations for Bioinformatics

When to Use

• Parsing a FASTA/FASTQ file into headers and sequences without pulling in BioPython.
• Writing sequences to FASTA with proper line wrapping (60/70/80 chars).
• Reading/writing gene expression tables (CSV) or genomic intervals (BED/TSV) with csv.DictReader/DictWriter.
• Streaming a multi-GB FASTA/FASTQ file line-by-line instead of loading it whole.
• Serializing intermediate analysis results (dicts with sets/numpy arrays) with pickle, or structured records with json.

Version Compatibility

Pure standard library — open(), csv, json, pickle, gzip — works unchanged on Python ≥3.8 (examples use f-strings and csv.DictReader, available since 3.6+).

Prerequisites

• No third-party packages required (stdlib only: csv, json, pickle, gzip).
• For heavier FASTA/FASTQ/SAM work, see bio-sequence-io-read-sequences and biopython — this skill covers hand-rolled parsers for when a dependency isn't warranted or you need full control over streaming.

Pitfalls

• strip() when reading lines: every line from a file has a trailing \n. Forgetting this causes sequences to contain invisible newlines that break string comparisons and length calculations.
• "w" mode destroys existing content. Use "a" to append, "x" to fail loudly if the file already exists.
• Binary mode for BAM/gzip/pickle: open(path, "rb")/"wb". Text mode tries to decode bytes as UTF-8 and corrupts binary data (or raises UnicodeDecodeError).
• f.read() and f.readlines() load the entire file into memory. For large FASTA/FASTQ, iterate with for line in f: or write a generator — memory stays constant regardless of file size.
• Forgetting the last record. Streaming FASTA/FASTQ parsers accumulate a sequence until they see the next header (or EOF) — the final record must be flushed explicitly after the loop ends.
• csv.writer/DictWriter on Windows/text mode: always open with newline='' when writing CSV, otherwise rows get doubled \r\n\r\n line endings.
• set objects are not JSON-serializable. json.dump({'x': {1,2}}) raises TypeError; use pickle for arbitrary Python objects (sets, numpy arrays, custom classes), json only for portable dict/list/str/number data.

Reading Methods

Method	Memory	Use when
f.read()	Loads entire file	Small file, need the whole string at once
f.readline()	One line	Need to peek at just the first line(s)
f.readlines()	Loads entire file (as list)	Small file, need random access to lines
for line in f:	One line at a time (preferred)	Large files, streaming/generator parsers

FASTA Parser (streaming)

Goal: turn a FASTA file into {header: sequence} without loading it as one giant string. Approach: iterate line-by-line, accumulate sequence chunks in a list (fast ''.join), flush on each new > header and again after the loop for the final record.

def read_fasta(filename):
    """Parse a FASTA file and return a dict mapping full headers to sequences.

    Args:
        filename: path to a FASTA file.

    Returns:
        dict of {header (without '>'): concatenated sequence string}.
    """
    sequences = {}
    current_header = None
    current_seq = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if current_header is not None:
                    sequences[current_header] = ''.join(current_seq)
                current_header = line[1:]
                current_seq = []
            else:
                current_seq.append(line)
        if current_header is not None:  # don't forget the last sequence!
            sequences[current_header] = ''.join(current_seq)
    return sequences

FASTA Writer (with line wrapping) and Generator Parser

Goal: write sequences back out with wrapped lines, and parse huge FASTA files with constant memory via a generator. Approach: write_fasta accepts a dict or a list of (header, seq) tuples; the generator yields one record at a time instead of building a dict, so callers can for header, seq in parse_fasta_generator(path): over a multi-GB file.

def write_fasta(sequences, filename, line_width=60):
    """Write sequences (dict or list of (header, seq) tuples) to FASTA, wrapped at line_width."""
    items = sequences.items() if isinstance(sequences, dict) else sequences
    with open(filename, 'w') as f:
        for header, seq in items:
            f.write(f">{header}\n")
            for i in range(0, len(seq), line_width):
                f.write(seq[i:i + line_width] + '\n')


def parse_fasta_generator(filename):
    """Yield (header, sequence) tuples one at a time — constant memory regardless of file size."""
    current_header = None
    current_seq = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if current_header is not None:
                    yield current_header, ''.join(current_seq)
                current_header = line[1:]
                current_seq = []
            :
                current_seq.append(line)
         current_header   :
             current_header, .join(current_seq)

CSV/TSV, JSON, and Pickle

Goal: read/write tabular gene-expression and BED-style data by column name, plus serialize structured or arbitrary Python results. Approach: csv.DictReader/DictWriter for row-as-dict access (delimiter='\t' for BED/TSV); json.dump/load for portable records; pickle.dump/load (binary mode) for objects json can't handle, like sets or numpy arrays.

import csv
import json
import pickle

# Read CSV with column-name access
with open('gene_expression.csv') as f:
    for row in csv.DictReader(f):
        print(row['gene_name'], float(row['expression_sample1']))

# Read TSV / BED files the same way
with open('genes.bed') as f:
    for row in csv.DictReader(f, delimiter='\t'):
        print(f"{row['gene']}: {row['chromosome']}:{row['start']}-{row['end']}")

# Write CSV with a header row (newline='' avoids blank rows on Windows)
with open('de_results.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=['gene', 'fold_change', 'p_value'])
    writer.writeheader()
    writer.writerows(results)  # list of dicts with those keys

# JSON round-trip for portable structured data
with open('gene.json', 'w') as f:
    json.dump(gene_annotation, f, indent=2)
 ()  f:
    data = json.load(f)


 (, )  f:
    pickle.dump(analysis_results, f)
 (, )  f:
    loaded = pickle.load(f)

See Also

• bio-sequence-io-read-sequences — BioPython SeqIO for FASTA/FASTQ/GenBank parsing when you don't want a hand-rolled parser.
• bio-sequence-io-compressed-files — same patterns over .gz-compressed inputs via gzip.open.
• python-bio-context-managers — the with statement mechanics these examples rely on.
• python-bio-generators — more on yield-based streaming parsers like parse_fasta_generator.