Read, write, and index compressed sequence files (gzip, bzip2, xz, BGZF) with Biopython and bgzip/samtools. Use when working with .gz, .bz2, or .bgz sequence files, when random access into a compressed FASTA/FASTQ is needed, or when SeqIO.index/faidx/tabix rejects a plain .gz. Covers the BGZF-vs-gzip seekability asymmetry, the 'rt'-not-'rb' handle trap, virtual offsets, and gzip-to-BGZF conversion.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Read, write, and index compressed sequence files (gzip, bzip2, xz, BGZF) with Biopython and bgzip/samtools. Use when working with .gz, .bz2, or .bgz sequence files, when random access into a compressed FASTA/FASTQ is needed, or when SeqIO.index/faidx/tabix rejects a plain .gz. Covers the BGZF-vs-gzip seekability asymmetry, the 'rt'-not-'rb' handle trap, virtual offsets, and gzip-to-BGZF conversion.
Before using code patterns, verify installed versions match. If versions differ:
Python: pip show <package> then help(module.function) to check signatures
CLI: <tool> --version then <tool> --help to confirm flags
If code throws ImportError, AttributeError, or TypeError, introspect the installed
package and adapt the example to match the actual API rather than retrying.
Compressed Files
Read, write, and randomly access gzip, bzip2, xz, and BGZF compressed sequence files.
"Read a compressed sequence file" -> Open a decompression handle in TEXT mode, then parse with the standard SeqIO interface.
gzip: gzip.open(path, 'rt') (Python stdlib)
bzip2: bz2.open(path, 'rt') (Python stdlib)
xz/LZMA: lzma.open(path, 'rt') (Python stdlib)
BGZF: bgzf.open(path, 'rt') (BioPython) - BGZF input ONLY
"Make a compressed file randomly accessible" -> Re-compress as BGZF, then index. Only BGZF supports SeqIO.index(), samtools faidx, and tabix on compressed data.
The Governing Principle: BGZF vs plain gzip is asymmetric
A BGZF (Blocked GNU Zip) file IS a valid gzip file - gunzip and zcat read it transparently. The reverse is FALSE: a plain .gz is NOT BGZF, so faidx/tabix/SeqIO.index reject it, and bgzf.open refuses to read it.
The reason is structural. BGZF is a series of concatenated gzip blocks, each <=64 KiB and independently decodable, so any record can be reached by seeking to its block. Plain gzip is one continuous DEFLATE stream with no block boundaries: reaching byte N means decompressing every byte before it (O(n)). Random access therefore REQUIRES BGZF; on a plain .gz, SeqIO.index() would re-decompress huge prefixes on every lookup, which is why Biopython forbids it outright (it raises rather than running slowly).
Consequences the agent must respect:
Reading sequentially: gzip, bzip2, xz, and BGZF all work via the matching *.open(path, 'rt') handle.
Random access / indexing: BGZF only. Convert plain gzip to BGZF first.
bgzf.open() reads BGZF input only. Pointing it at a plain .gz raises ValueError: A BGZF block should start with b'\x1f\x8b\x08\x04'.... To read plain gzip use gzip.open().
BAM and tabix-indexed files use BGZF natively; bzip2/xz are archive-only (no seekable index).
Required Imports
import gzip
import bz2
import lzma
from Bio import SeqIO
from Bio import bgzf
Reading Compressed Files
Goal: Parse sequence records from a compressed file without decompressing to disk.
Approach: Open a decompression handle in TEXT mode ('rt'), then pass the handle to SeqIO.parse(). The parser is format-agnostic about the underlying compression.
with gzip.open('reads.fastq.gz', 'rt') as handle:
for record in SeqIO.parse(handle, 'fastq'):
print(record.id, len(record.seq))
Swap gzip.open for bz2.open (.bz2), lzma.open (.xz), or bgzf.open (.bgz) - the parse loop is identical.
The 'rt' vs 'rb' trap
SeqIO.parse() in Python 3 needs a TEXT handle that yields str. A binary 'rb' handle yields bytes and raises TypeError: a bytes-like object is required (or a decode error). Always use 'rt' for reading and 'wt' for writing through SeqIO. The low-level SimpleFastaParser/FastqGeneralIterator also require text handles.
Writing Compressed Files
Goal: Save records straight to a compressed file with no intermediate plain copy.
Approach: Open a compression handle in TEXT mode ('wt'), then pass it to SeqIO.write().
with gzip.open('output.fasta.gz', 'wt') as handle:
SeqIO.write(records, handle, 'fasta')
For an indexable result write BGZF instead:
with bgzf.open('output.fasta.bgz', 'wt') as handle:
SeqIO.write(records, handle, 'fasta')
BgzfWriter.close() (and the with block exit) automatically appends the 28-byte empty-block EOF marker that htslib tools check for; let the context manager close the handle.
Random Access: index a BGZF file
Goal: Pull individual records by id from a large compressed file without a linear scan.
Approach: Compress as BGZF, then build a Biopython offset index. SeqIO.index() keeps virtual offsets in RAM; SeqIO.index_db() stores them in an on-disk SQLite index that persists across sessions and spans multiple files.
records = SeqIO.index('sequences.fasta.bgz', 'fasta')
target = records['gene_042'].seq
records.close()
# Persistent, multi-file, scales beyond RAM:
db = SeqIO.index_db('idx.sqlite', ['a.fasta.bgz', 'b.fasta.bgz'], 'fasta')
SeqIO.index() on a plain .gz raises ValueError: Gzipped files are not suitable for indexing, please use BGZF (blocked gzip format) instead. Convert first (below).
Convert plain gzip to BGZF
"Convert gzip to an indexable format" -> Decompress the gzip stream and re-compress it as BGZF.
CLI (fastest, htslib-native):
# Either decompress then bgzip in place...
gzip -d sequences.fasta.gz && bgzip sequences.fasta # -> sequences.fasta.gz (now BGZF)# ...or stream without touching disk:
zcat sequences.fasta.gz | bgzip -@ 4 > sequences.fasta.bgz
# Index a BGZF FASTA for region extraction:
samtools faidx sequences.fasta.bgz # writes BOTH .fai and .gzi
samtools faidx sequences.fasta.bgz gene_042:1-200
samtools faidx on a BGZF FASTA writes TWO index files: .fai (record offsets in uncompressed coordinates) AND .gzi (the compressed-to-uncompressed block map). Deleting .gzi breaks region extraction even though .fai survives. Note that bgzip keeps the .gz extension, so a .gz may be EITHER plain gzip or BGZF - check with bgzip -t file.gz (tests for a valid BGZF stream) rather than trusting the suffix.
Pure-Python equivalent (no external tools):
import gzip
from Bio import SeqIO, bgzf
with gzip.open('input.fasta.gz', 'rt') as in_handle:
with bgzf.open('output.fasta.bgz', 'wt') as out_handle:
SeqIO.write(SeqIO.parse(in_handle, 'fasta'), out_handle, 'fasta')
A virtual offset packs two coordinates into one 64-bit integer: voffset = coffset << 16 | uoffset, where coffset is the byte position of the block start in the compressed file (top 48 bits) and uoffset is the offset within that block's decompressed data (low 16 bits - 16 bits suffices because a block holds at most 64 KiB).
vo = bgzf.make_virtual_offset(100, 7) # 6553607
coffset, uoffset = bgzf.split_virtual_offset(vo) # (100, 7)with bgzf.open('sequences.fasta.bgz', 'rt') as handle:
handle.readline()
saved = handle.tell() # a VIRTUAL offset, not a byte position
handle.seek(saved) # jumps back to the same record
Critical caveat: virtual offsets may be COMPARED for ordering but NEVER SUBTRACTED to get a byte length - they live in two coordinate spaces (compressed position and within-block position), so vo2 - vo1 is meaningless. BgzfReader.tell() returns a virtual offset; seek() consumes one. Text mode forces latin1 and does no newline translation.
Compression Format Comparison
Format
Extension
Random access
Speed
Ratio
Stdlib handle
gzip
.gz
No (O(n) seek)
Fast
Good
gzip.open
BGZF
.bgz / .gz
Yes (block-seekable)
Fast, threadable
Good
bgzf.open (BioPython)
bzip2
.bz2
No
Slow
Better
bz2.open
xz / LZMA
.xz
No
Slowest
Best
lzma.open
When to Use Each Format
Use case
Format
Why
Sequential read/write, sharing
gzip
Universal, fast, every tool reads it
Need faidx/tabix/SeqIO.index
BGZF
Only seekable compressed format
BAM, tabix-indexed VCF/GFF/BED
BGZF
Required natively
Cold archive, max shrink, no random access
xz then bzip2
Highest ratios, slowest
Random access into an existing plain .gz without re-bgzipping
pyfastx
Adds a seek-point index over the gzip stream
pyfastx is the exception that gives random access into a PLAIN gzip FASTA/FASTQ: it builds a seek-point index (via zran from indexed_gzip) plus a SQLite .fxi/.fqi index alongside the file, a different strategy from faidx (which requires the stream itself to be BGZF). Use it when re-compressing a large gzipped genome to BGZF is not an option.
Common Errors
Symptom
Cause
Fix
TypeError: a bytes-like object is required
Handle opened 'rb' instead of 'rt'
Open compressed handles with 'rt'/'wt' for SeqIO
ValueError: Gzipped files are not suitable for indexing, please use BGZF...
SeqIO.index() on a plain .gz
Re-compress as BGZF (`zcat ...
ValueError: A BGZF block should start with b'\x1f\x8b\x08\x04'...
bgzf.open() pointed at a plain gzip file
Read plain gzip with gzip.open(); reserve bgzf.open for BGZF
[bgzf] file ... not BGZF / not compressed with bgzip (htslib)
faidx/tabix given a plain .gz
Convert to BGZF first
[faidx] Failed to read ... / could not load .gzi
.gzi deleted next to a BGZF FASTA
Re-run samtools faidx to regenerate .fai + .gzi
gzip.BadGzipFile / OSError: Not a gzipped file
File is not gzip (wrong suffix / corrupt)
Verify with bgzip -t or file; match handle to real format
UnicodeDecodeError
Non-UTF8 bytes in a text handle
gzip.open(path, 'rt', encoding='latin-1')
Related Skills
read-sequences - parse vs index vs index_db trade-offs for compressed handles
write-sequences - write records through a compression handle
batch-processing - stream many compressed files without loading them into RAM
filter-sequences - keep R1/R2 in sync when filtering gzipped paired reads
alignment-files/sam-bam-basics - BAM is BGZF natively; samtools manages the compression
References
Li H, Handsaker B, Wysoker A, et al. The Sequence Alignment/Map format and SAMtools. Bioinformatics. 2009;25(16):2078-2079. (Defines BGZF in the SAM/BAM specification.)
Bonfield JK. CRAM 3.1: advances in the CRAM file format. Bioinformatics. 2022;38(6):1497. (Per-column block compression beyond BGZF.)
Du L, Liu Q, Fan Z, et al. Pyfastx: a robust Python package for fast random access to sequences from plain and gzipped FASTA/Q files. Briefings in Bioinformatics. 2021;22(4):bbaa368.
