foundations-character-encodings

name	foundations-character-encodings
description	Fix UTF-8/Latin-1/CP1251 decode errors, strip BOM, repair mojibake, normalize CRLF endings, remove hidden Unicode in FASTA/GFF text. Use when hitting UnicodeDecodeError or garbled text from Windows/Excel exports.
tool_type	python
primary_tool	Python

Character Encodings and Binary Data in Bioinformatics

When to Use

• UnicodeDecodeError: 'utf-8' codec can't decode byte 0x... when opening a file downloaded from an external database or exported from Windows.
• FASTA/GFF/GenBank parsing gives sequences that are one character too long, or headers that start with an invisible character.
• Text copy-pasted from a paper or Word document (author names, organism names, Greek letters like α/β, µ, °) looks garbled after being read into Python.
• A gene/protein name with special characters (HLA-A*02:01, C/EBPα) needs to go into a URL query (UniProt, NCBI E-utilities).
• Need to detect or convert a file's encoding (e.g., CP1251 Cyrillic annotations) to UTF-8 before downstream tools choke on it.

Version Compatibility

Python ≥3.9 (all examples use only the standard library: codecs/str.encode/str.decode, unicodedata, urllib.parse). Optional: chardet ≥5.0 for encoding auto-detection.

Prerequisites

• Basic Python string/bytes distinction (str vs bytes, .encode()/.decode()).
• pip install chardet (optional, only needed for auto-detection).

From Bits to Bytes

Notation	Example	Meaning
Binary	01000001	8 bits = 1 byte
Decimal	65	Human-friendly
Hexadecimal	0x41	2 hex digits = 1 byte

for value in [0, 10, 32, 48, 65, 90, 97, 122, 127, 200, 255]:
    char = chr(value) if 32 <= value < 127 else '--'
    print(f"{value:7d}  {format(value, '08b'):>10}  0x{format(value, '02X')}  '{char}'")

Bioinformatics relevance: DNA (ATGCN), RNA (AUGC), and amino acid codes are pure ASCII (byte values 0-127) -- sequence data itself never has encoding issues. Problems arise in annotations, author names, organism names, and free-text fields, and in FASTQ quality strings (Phred+33 maps quality 0-93 to ASCII 33-126).

# FASTQ quality scores: Phred+33 maps quality 0-93 to ASCII 33-126
quality_string = "IIIIIIIIFFFFFFDDDDDBBBB"
for char in quality_string[:6]:
    phred = ord(char) - 33
    print(f"'{char}' -> Phred {phred} -> error rate {10 ** (-phred / 10):.6f}")

Core Operations

Goal: detect why a file fails to decode, repair mojibake, and normalize the text before parsing. Approach: try encodings in priority order (UTF-8 with BOM check first, Latin-1 last since it never raises), fix already-mis-decoded text by round-tripping through the wrong encoding, and strip invisible Unicode noise from biological sequences.

def read_text_file(filepath):
    """Read a text file, trying common bioinformatics encodings in order.

    Priority:
    1. utf-8-sig  (UTF-8, auto-strips a BOM if a Windows editor added one)
    2. utf-8      (modern standard, no BOM)
    3. latin-1    (never fails -- last-resort fallback, may give wrong characters)
    """
    for encoding in ('utf-8-sig', 'utf-8', 'latin-1'):
        try:
            with open(filepath, 'r', encoding=encoding) as f:
                content = f.read()
            if encoding == 'latin-1':
                print(f"WARNING: fell back to Latin-1 for {filepath}; "
                      "some characters may be wrong -- consider converting to UTF-8.")
            return content
        except UnicodeDecodeError:
            continue
    raise ValueError(f"Could not decode {filepath} with any known encoding")


def fix_mojibake(text, wrong_encoding='latin-1', right_encoding='utf-8'):
    """Repair text that was UTF-8 but got decoded as Latin-1 (classic mojibake).

    Round-trips through the wrong encoding's byte representation, then
    decodes those bytes with the correct encoding.
    """
    return text.encode(wrong_encoding).decode(right_encoding)


# Example: alpha-helix mis-decoded, then repaired
original = "α-helix structure"
utf8_bytes = original.encode('utf-8')
wrong = utf8_bytes.decode('latin-1')          # garbled, but does not raise
fixed = fix_mojibake(wrong)
 fixed == original

import unicodedata


def sanitize_sequence(seq, valid_chars='ATGCNatgcn'):
    """Strip characters that are not valid sequence letters or expected whitespace.

    Catches copy-paste artifacts like zero-width space (U+200B) and
    non-breaking space (U+00A0) that look identical to normal text but
    break equality checks, split(), and length calculations.
    """
    valid_set = set(valid_chars)
    cleaned, removed = [], []
    for char in seq:
        if char in valid_set:
            cleaned.append(char)
        elif char not in ('\n', '\r', ' ', '\t'):
            removed.append(f"'{char}' ({unicodedata.name(char, f'U+{ord(char):04X}')})")
    if removed:
        print(f"WARNING: removed {len(removed)} unexpected characters: {', '.join(removed)}")
    return ''.join(cleaned)


messy = "ATG​CGA TCGA\r\nGGG­TTT"
print(sanitize_sequence(messy))  # -> ATGCGATCGAGGGTTT

def parse_fasta_robust(raw_bytes):
    """Parse FASTA from raw bytes, handling BOM and any line-ending style."""
    text = raw_bytes.decode('utf-8-sig').replace('\r\n', '\n').replace('\r', '\n')
    sequences, current_header, current_seq = {}, None, []
    for line in text.strip().split('\n'):
        line = line.strip()
        if line.startswith('>'):
            if current_header is not None:
                sequences[current_header] = ''.join(current_seq)
            current_header, current_seq = line[1:], []
        elif line:
            current_seq.append(line)
    if current_header is not None:
        sequences[current_header] = ''.join(current_seq)
    return sequences


windows_fasta = b">gene1\r\nATGCGATCGA\r\nTTTAAAGGGC\r\n"
result = parse_fasta_robust(windows_fasta)
assert result['gene1'] == 'ATGCGATCGATTTAAAGGGC'

Goal: detect an unknown encoding, and percent-encode gene/protein names for database URLs. Approach: use chardet when the source encoding is unknown; use urllib.parse.quote/unquote for E-utilities/UniProt query strings containing *, /, +, or Greek letters.

from urllib.parse import quote, unquote

try:
    import chardet

    def detect_encoding(raw_bytes):
        """Return chardet's best-guess encoding and confidence for raw bytes."""
        result = chardet.detect(raw_bytes)
        return result['encoding'], result['confidence']

    guess, conf = detect_encoding("Анализ экспрессии генов".encode('cp1251'))
    print(f"detected {guess} (confidence {conf:.0%})")
except ImportError:
    print("chardet not installed: pip install chardet")

# HLA allele / transcription factor names in a UniProt/NCBI query URL
gene_queries = ["HLA-A*02:01", "C/EBPα", "Na+/K+ ATPase"]
base_url = "https://www.uniprot.org/uniprot/?query="
for gene in gene_queries:
    print(f"{gene:<15} -> {base_url}{quote(gene, safe='')}")

encoded_url = "https://www.ncbi.nlm.nih.gov/gene/?term=C%2FEBP%CE%B1"
print(unquote(encoded_url))  # -> .../gene/?term=C/EBPα

Pitfalls

• Latin-1 never fails -- but that does not mean it is right: Latin-1 decodes every byte 0-255 successfully, silently producing wrong characters (mojibake) when the file is actually UTF-8 with multi-byte sequences. Never use it as a first choice, only as a documented fallback.
• UTF-8 BOM confusion: Windows editors may prepend \xef\xbb\xbf. Decoding with plain 'utf-8' leaves an invisible U+FEFF at position 0, breaking line.startswith('>') FASTA header checks. Use 'utf-8-sig' to auto-strip it.
• Windows line endings in sequences: a trailing \r in a DNA line is invisible in most editors but breaks sequence length, regex matches, and split() results. Normalize with .replace('\r\n', '\n').replace('\r', '\n') before parsing.
• Non-breaking space vs. regular space:   looks identical to ' ' but is a different byte sequence -- split() and exact-match database lookups treat them differently.
• Coordinate systems: BED is 0-based half-open; VCF/GFF is 1-based inclusive -- mixing these causes off-by-one errors independent of encoding.

See Also

• python-bio-file-operations -- general file reading/writing patterns this builds on.
• python-bio-sequences -- working with Seq objects once FASTA text is decoded and clean.
• python-bio-strings -- string methods (.strip(), .split(), .encode()/.decode()) used throughout.
• bio-core-biological-databases -- building NCBI/UniProt query URLs with percent-encoded gene names.