python-bio-iterators

name	python-bio-iterators
description	Stream FASTA/FASTQ and generate k-mers/codons lazily with Python generators, custom __iter__/__next__ classes, and itertools. Use when parsing multi-GB sequence files without loading them fully into RAM or chaining filter-trim-translate pipelines.
tool_type	python
primary_tool	Python

Iterators and Generators for Bioinformatics

When to Use

• Parsing FASTA/FASTQ files too large to fit in memory (constant memory regardless of file size)
• Generating k-mers, codons, or sliding windows lazily from long sequences or whole genomes
• Chaining multi-stage read-processing pipelines (e.g., trim → quality-filter → translate) with no intermediate lists
• Writing a custom iterator class that needs extra state a generator can't easily give you (resettable position, peek())
• Counting/aggregating over huge streams (k-mer frequency, motif positions) without materializing a list first

Version Compatibility

Python >= 3.8. Everything here is stdlib only (itertools, collections.Counter) — no third-party packages required. For production FASTA/FASTQ I/O, pair this with Biopython's SeqIO (see bio-sequence-io-read-sequences); the patterns below are for when you need custom lazy logic Biopython doesn't provide out of the box.

Prerequisites

• Comfortable with Python functions, classes, and for loops
• Basic sequence vocabulary: codons, ORFs, GC content, reading frames
• Related: bio-sequence-io-read-sequences (Biopython SeqIO.parse), bio-sequence-manipulation-codon-usage

Class-Based Iterator vs. Generator Function

Goal: iterate a DNA sequence in codon triplets, understanding when a class-based iterator is worth the extra code over a generator.

Approach: implement both. Use the class only when you need state beyond simple iteration (e.g., a resettable index or a peek() method); otherwise prefer the generator — same behavior, far less boilerplate.

class CodonIterator:
    """Iterate over a DNA sequence in codons (triplets). Use over a generator
    only if you need extra state, e.g. reset() or peek()."""

    def __init__(self, sequence):
        self.sequence = sequence
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index + 3 > len(self.sequence):
            raise StopIteration
        codon = self.sequence[self.index:self.index + 3]
        self.index += 3
        return codon

    def reset(self):
        """Rewind to the start -- something a plain generator cannot do."""
        self.index = 0


def codon_generator(sequence):
    """Yield codons (triplets) from a DNA sequence. Preferred over the class
    above unless you specifically need reset()/peek()."""
    for i in range(0, len(sequence) - 2, ):
         sequence[i:i + ]


 ():
    
     i  ((sequence) - k + ):
         sequence[i:i + k]


 __name__ == :
    dna = 
     (codon_generator(dna)) == [dna[i:i+]  i  (, (dna) - , )]
    ci = CodonIterator(dna)
    first_pass = (ci)
     (ci) == []          
    ci.reset()
     (ci) == first_pass  
     (kmer_generator(, )) == [, , ]
    ()

Streaming FASTA/FASTQ Readers (Constant Memory)

Goal: parse arbitrarily large FASTA/FASTQ files one record at a time instead of loading the whole file into a list.

Approach: a generator that yields as it reads, plus generator stages (quality_filter, trim_n_bases) chained on top so filtering/trimming never materializes an intermediate list.

def read_fasta(filename):
    """Memory-efficient FASTA parser. Yields (header, sequence) tuples."""
    header, seq_parts = None, []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith('>'):
                if header is not None:
                    yield header, ''.join(seq_parts)
                header, seq_parts = line[1:], []
            else:
                seq_parts.append(line)
    if header is not None:
        yield header, ''.join(seq_parts)


def read_fastq(filename):
    """Streaming FASTQ reader -- yields one record dict at a time
    ('id', 'sequence', 'quality'). Memory usage is constant regardless
    of file size."""
    with open(filename) as f:
        while True:
            header = f.readline().strip()
            if not header:
                break
            sequence = f.readline().strip()
            f.readline()  # '+' separator line
            quality = f.readline().strip()
            yield {
                'id': header[:].split()[],
                : sequence,
                : quality,
            }


 ():
    
     record  records:
        avg_qual = ((c) -   c  record[]) / (record[])
         avg_qual >= min_avg_quality:
             record


 ():
    
     record  records:
        seq = record[].rstrip()
         {**record, : seq, : record[][:(seq)]}


 __name__ == :
     tempfile, os
    fastq = 
    path = tempfile.mktemp(suffix=)
     (path, )  f:
        f.write(fastq)

    pipeline = quality_filter(trim_n_bases(read_fastq(path)), min_avg_quality=)
    kept = (pipeline)
     (kept) ==   kept[][] ==   kept[][] == 
    os.remove(path)
    ()

Lazy Pipelines with itertools

Goal: combine k-mer/codon streams with itertools for translation, slicing, and combinatorics without ever building a full list.

Approach: itertools.islice for lazy slicing, groupby for run-length grouping, chain/product for combining/enumerating sequences.

import itertools
from collections import Counter

CODON_TABLE = {
    'TTT': 'F', 'TTC': 'F', 'TTA': 'L', 'TTG': 'L', 'CTT': 'L', 'CTC': 'L', 'CTA': 'L', 'CTG': 'L',
    'ATT': 'I', 'ATC': 'I', 'ATA': 'I', 'ATG': 'M', 'GTT': 'V', 'GTC': 'V', 'GTA': 'V', 'GTG': 'V',
    'TCT': 'S', 'TCC': 'S', 'TCA': 'S', 'TCG': 'S', 'CCT': 'P', 'CCC': 'P', 'CCA': 'P', 'CCG': 'P',
    'ACT': 'T', 'ACC': 'T', 'ACA': 'T', 'ACG': , : , : , : , : ,
    : , : , : , : , : , : , : , : ,
    : , : , : , : , : , : , : , : ,
    : , : , : , : , : , : , : , : ,
    : , : , : , : , : , : , : , : ,
}


 ():
    
     i  (, (sequence) - , ):
        aa = CODON_TABLE.get(sequence[i:i + ], )
         aa == :
            
         aa


 ():
    
     combo  itertools.product(, repeat=k):
         .join(combo)


 __name__ == :
    dna = 
     .join(translate_generator(dna)) == 

    
    first_five = (itertools.islice(all_possible_kmers(), ))
     (first_five) == 

    
    runs = [(base, ((g)))  base, g  itertools.groupby()]
     runs == [(, ), (, ), (, ), (, )]

    
    kmer_counts = Counter(all_possible_kmers())
     (kmer_counts.values()) ==   (kmer_counts) == 

    ()

Pitfalls

• Single-pass exhaustion: once an iterator/generator is drained (by a for-loop or list(...)), it is empty forever — list(it) returns [] on the second call. Re-create it or materialize to a list if you need multiple passes.
• itertools functions are lazy: itertools.product(...), chain(...), etc. return iterators, not lists — wrap in list() only when you actually need to see/store all values.
• Off-by-one in k-mer/codon ranges: range(len(seq) - k + 1) for k-mers vs. range(0, len(seq) - 2, 3) for codons — mixing these up silently drops or duplicates the last window.
• Generator functions don't run until iterated: calling gen_func() executes zero lines of body code; errors inside only surface once you call next() or iterate.
• FASTQ readers assume exactly 4 lines/record: malformed or wrapped-line FASTQ (rare, but seen from some tools) will desync header/sequence/quality; validate header.startswith('@') if input is untrusted.

See Also

• bio-sequence-io-read-sequences — production FASTA/FASTQ parsing via Biopython SeqIO
• bio-sequence-io-paired-end-fastq — interleaving/pairing R1/R2 streams
• bio-sequence-manipulation-codon-usage — codon table analysis beyond simple translation
• bio-genome-intervals-interval-arithmetic — lazy interval/window operations over genomic coordinates