bio-comparative-genomics-gene-tree-species-tree-reconciliation

## Version Compatibility Reference examples tested with: ALE 1.0+ (ssolo/ALE github), GeneRax 2.1.3+ (BenoitMorel/GeneRax), AleRax 1.2.0+ (BenoitMorel/AleRax; Morel 2024 Bioinformatics 40:btae162), Whale.jl 2.0+ (arzwa/Whale.jl), RANGER-DTL 2.0+ (Bansal lab; Bansal 2018 Bioinformatics 34:3214), NOTUNG 2.9.1.5+ (Stolzer 2012; Chen 2000), ecceTERA 1.2.5+, Treerecs 1.2+, IQ-TREE 2.3.6+, MrBayes 3.2.7+, BUSCO 5.7+, ete4 4.1.0+, BioPython 1.84+. Open Tree of Life and NCBI Taxonomy reference databases at 2024-Q3 minimum for species-tree-aware inference. Before using code patterns, verify installed versions match. If versions differ: - CLI: `ALEml_undated --help`, `ALEml --help` (dated), `generax --help`, `alerax --help` - Julia: `using Whale; Whale.WhaleProblem`; `]status` for package versions - Python: `pip show ete4`; `ete4 --help` If code throws `species tree mismatch`, `gene tree taxa not in species tree`, or `MPI process pool failure`, these reconciliation tools share strict label-consistency requirements: species labels must match exactly across the species tree and gene trees (case-sensitive, no whitespace), and gene IDs typically encode species via prefix (`species|gene_id` separator convention). Use `sed` / `awk` normalization scripts before reconciliation. # Gene Tree Species Tree Reconciliation **"Where did this gene family come from, and what events shaped its history?"** -> Reconcile gene trees against species trees under explicit probabilistic models of duplication (D), horizontal transfer (T), and loss (L). The reconciliation framework converts gene-tree-species-tree discordance into a quantitative history of evolutionary events. Modern probabilistic methods (ALE, GeneRax, AleRax) **distinguish gene-tree-error-driven discordance from biological discordance** by integrating over gene-tree uncertainty -- a critical advance over parsimony reconciliation (NOTUNG, RANGER) which treats input gene trees as fixed and inflates duplication/loss counts from gene-tree noise (Boussau 2013 Genome Res 23:323; Morel 2020 MBE 37:2763). - CLI: `ALEobserve` + `ALEml_undated` -- Bayesian DTL on a sample of gene trees (amalgamated likelihood) - CLI: `generax` -- ML reconciliation; refines gene trees jointly with reconciliation - CLI: `alerax` -- co-estimation of gene and species trees + DTL rates (Morel 2024) - Julia: `using Whale` -- Bayesian DL + WGD modeling - CLI: `ranger-dtl` -- parsimony DTL with cost weights - CLI: `notung` -- duplication-loss only (DL); user-friendly GUI; legacy ## Algorithmic Taxonomy | Tool | Approach | Events modeled | Inference | Strength | Fails when | |------|----------|----------------|-----------|----------|------------| | ALE undated (Szöllősi 2013 Syst Biol 62:901) | Amalgamated likelihood over gene-tree distribution; species-tree-aware | D, T, L | Bayesian | Posterior over D/T/L events at every species-tree branch; integrates over gene-tree uncertainty | Requires gene-tree posterior sample (>= 100 bootstrap/UFBoot trees); slow for many families | | ALE dated | Same as undated but uses time-calibrated species tree | D, T, L | Bayesian | Time-aware; better donor inference | Requires dated species tree (BEAST2 / RevBayes calibration) | | GeneRax (Morel 2020 MBE 37:2763) | ML reconciliation + joint gene-tree refinement | D, T, L | ML | Faster than ALE; refines noisy gene trees; species-tree-aware | Less uncertainty quantification than ALE | | AleRax (Morel 2024 Bioinformatics 40:btae162) | Co-estimation of gene tree, species tree, and DTL rates | D, T, L | Bayesian / ML hybrid | Gold standard 2024; corrects gene-tree-error feedback into species tree | Computationally heaviest; needs >= 20 species | | Whale.jl (Zwaenepoel & Van de Peer 2019 MBE 36:1384) | Bayesian DL + WGD via amalgamated likelihood | D, L, WGD | Bayesian (Turing.jl) | Native WGD modeling; modern Bayesian framework | Julia ecosystem dependency | | RANGER-DTL 2.0 (Bansal 2018 Bioinformatics 34:3214) | Parsimony DTL with user cost weights (D-cost, T-cost, L-cost) | D, T, L | Parsimony | Fast; deterministic; many gene families per minute | Cost weights are user choices; results sensitive to costs | | NOTUNG (Chen 2000 JCB 7:429; Stolzer 2012 Bioinformatics 28:i409) | Parsimony DL; HGT extension | D, L (optional T) | Parsimony | User-friendly GUI; widely used | DL-only by default; HGT extension less rigorous than ALE | | ecceTERA (Jacox 2016 Bioinformatics 32:2056) | DTL on input set of trees; sampled + unsampled ("dead") lineages | D, T, L | Parsimony / DP | Transfers from extinct/unsampled lineages; cost-sweep mode | No ILS model; less popular than ALE; smaller community | | Treerecs (Comte 2020 Bioinformatics 36:4822) | Gene-tree correction/rooting against a fixed species tree | D, L | ML | Refines gene trees by species-tree constraint | No HGT; eukaryote-focused | | DLCpar (Wu 2014 GR 24:475) | DLC parsimony for DL + coalescence (ILS) | D, L, C | Parsimony | Models ILS explicitly | No HGT; older | | GraphDTL (Tofigh 2011) | Graph algorithm for DTL | D, T, L | Parsimony | Fast on small instances | Less used today | | Phyldog (Boussau 2013 GR 23:323) | Joint species-tree-gene-tree DL with site-rate variation | D, L | ML | Joint inference; refines gene trees | Bacteria-unfriendly; eukaryote-only | Methodology evolves; verify the AleRax / ALE documentation before locking on a single approach. The probabilistic ALE / GeneRax / AleRax tools have largely superseded parsimony reconciliation for serious phylogenomic work; parsimony is fine for screening but not for publication-grade DTL inference. ## Decision Tree by Experimental Scenario | Scenario | Recommended approach | Why | |----------|------------------------|-----| | Bacterial / archaeal phylogenomics, 50-500 genomes | GeneRax (refinement) -> ALE undated (posterior) | Two-stage: GeneRax refines, ALE provides posterior | | Eukaryote DL inference, no HGT expected | NOTUNG (legacy) or Treerecs | DL is the dominant signal; HGT rare in animals | | Mixed prokaryote/eukaryote with HGT | ALE undated | Probabilistic D/T/L; ALE-rooting (Williams 2017) for deep questions | | Plant comparative genomics with WGD | Whale.jl | Native WGD modeling; Bayesian | | Need uncertainty quantification | ALE or AleRax | Posteriors on every branch; ML methods give point estimates only | | Need fastest possible per-family analysis | RANGER-DTL parsimony | Deterministic; multi-gene parallel | | Co-estimate species tree from many gene families | AleRax | Modern gold-standard; corrects gene-tree-error | | Root a deep species tree from DTL signal | ALE undated rooting (Williams 2017 method) | Root inference from D/T/L event distribution | | Detect ancient HGT in archaea / bacteria | ALE undated | Probabilistic T detection at each branch; donor inferred | | Identify ancestral gene family content | ALE; report origination events per branch | Posterior over presence/absence at internal nodes | | Test specific HGT hypothesis (e.g. plant -> nematode) | ALE on filtered OG set; manual gene tree inspection | Quantitative T posterior | | Distinguish HGT from differential gene loss | ALE event posteriors (T vs L on candidate branch) | Probabilistic ratio between alternatives | | WGD detection alongside DTL | Whale.jl explicit WGD modeling | Joint inference; replaces post hoc Ks plotting | | Refine noisy gene trees against species tree | GeneRax `--strategy SPR` | Species-tree-aware gene-tree refinement | | Gene family birth-death modeling | See [[gene-family-evolution]] (CAFE5) | Reconciliation is per-family; CAFE5 is across families | | Single gene of interest, single species | Manual gene-tree placement; ALE not needed | Reconciliation framework is genome-scale | ## Per-Tool Failure Modes ### Gene-tree-error feedback inflating duplications **Trigger:** Using GeneRax or ALE with poorly-supported gene trees (low bootstrap, short alignments). **Mechanism:** Noisy gene trees show spurious topology that, when reconciled, produces apparent duplications-followed-by-losses or transfers. The reconciliation framework cannot distinguish gene-tree noise from real DTL events; the output is biased toward more events (Boussau 2013). **Symptom:** D + T + L event counts exceed reasonable rates (e.g. > 5 events per gene per Myr in eukaryotes); per-branch event posteriors are diffuse; ALE convergence (in `_uTs` files) is slow. **Fix:** Use ALE (which integrates over gene-tree posterior sample) rather than GeneRax (which uses a single ML tree). For GeneRax users, provide UFBoot trees with high (`-B 1000`) bootstraps, and refine via `--strategy SPR`. AleRax (Morel 2024) co-estimates gene trees + species tree + DTL rates, addressing this feedback directly. ### Species labels and gene IDs mismatch **Trigger:** Different naming conventions across gene trees, species tree, and orthology files. **Mechanism:** Reconciliation tools require species labels in the species tree to match a defined prefix or suffix in each gene ID. Inconsistencies cause silent failures or partial reconciliation. **Symptom:** ALE fails with "taxon not in species tree"; or runs but produces zero reconciliation events; or reconciliation matrix is sparse. **Fix:** Strict normalization before reconciliation. ALE convention: gene IDs as `species|gene_id` (pipe separator); species labels match the species tree leaf names exactly. Pre-process all gene trees with: ```bash for tree in gene_trees/*.nwk; do sed -i 's/_gene_/|/g' "$tree" # adjust separator done ``` Verify with `nw_labels -I species_tree.nwk` vs `nw_labels -I gene_trees/OG0000001.nwk` -- both species sets must be subsets of the species tree leaves. ### Cost-weight sensitivity in parsimony reconciliation (RANGER) **Trigger:** Running RANGER-DTL with default costs (D=2, T=3, L=1). **Mechanism:** Parsimony reconciliation minimizes total cost; the inferred D / T / L event count depends linearly on these costs. Default costs are biased toward favoring duplication-loss explanations over transfer. **Symptom:** RANGER reports fewer transfers than ALE/GeneRax on the same data; sensitivity-analysis varies event counts dramatically with cost changes. **Fix:** Run RANGER with cost sensitivity sweep: D in {1, 2, 3, 4}, T in {1, 2, 3, 4, 5}, L = 1. Report consensus events appearing in all cost combinations. Or move to probabilistic ALE/GeneRax which infers rates from data, not user costs. ecceTERA also allows cost-sweep mode. ### Root sensitivity in ALE undated **Trigger:** ALE on a species tree with poorly-supported root. **Mechanism:** ALE undated treats the species tree root as fixed; event posteriors at deepest branches depend on the root location. Wrong root flips D vs T inference at deep nodes. **Symptom:** Running ALE under multiple candidate rootings (STRIDE, MAD, outgroup) produces qualitatively different event histories at deep branches. **Fix:** Run ALE under multiple rootings; report only robust events. For deep phylogenomic questions, use ALE-rooting (Williams 2017 PNAS 114:E4602): run ALE under all possible rootings, choose the root maximizing the joint likelihood. AleRax can co-estimate the root, removing this issue. ### WGD events misattributed as duplications **Trigger:** Reconciliation on a clade with known WGD (vertebrates 2R, fish 3R, salmonids Ss4R, plant lineages). **Mechanism:** Standard DTL models (ALE, GeneRax) treat WGD as a series of individual duplications; the posterior at the WGD branch is dominated by D events but the joint event is whole-genome. **Symptom:** D events at the WGD branch are 10-100x higher than other branches; many "duplications" cluster temporally. **Fix:** Use Whale.jl which natively models WGD as a single event with a flexible retention parameter (Zwaenepoel 2019 MBE 36:1384). Otherwise, post hoc identify clusters of synchronized duplications and label as WGD. ### Saturation at deep timescales **Trigger:** Reconciliation on extremely deep clades (>1 Gyr divergence in bacteria; >500 Myr in eukaryotes). **Mechanism:** Gene families have undergone many cycles of D, T, L; the observed pattern is consistent with many DTL histories. Parameter identifiability is lost. **Symptom:** ALE convergence requires hundreds of cycles; posterior on D/T/L rates is uninformative; branch event posteriors are diffuse. **Fix:** Restrict to subclades with more recent divergence for quantitative DTL claims; for deep questions, qualitative event-class identification only (e.g. "transfers occurred along this branch" without precise count). Williams 2017 PNAS 114:E4602 demonstrates how ALE on deep archaeal phylogeny still resolves event class. ### MPI parallelization failures in GeneRax **Trigger:** Running GeneRax on cluster with many gene families. **Mechanism:** GeneRax uses MPI to parallelize per-family analysis; misconfigured MPI environment (wrong `srun`/`mpirun`/`mpiexec`, wrong allocator) silently runs serial or hangs. **Symptom:** GeneRax progresses through few families per hour; cluster CPU usage shows only 1 core per node. **Fix:** Verify MPI: `mpirun -n 8 hostname` should show 8 different hostnames or threads. Use `--per-family-rates` and proper MPI launch: `mpirun -n $SLURM_NTASKS generax ...`. Reduce `--threads` to 1 per family (let MPI handle parallelism). AleRax uses the same convention. ### ILS misattributed as transfers **Trigger:** Reconciliation on rapidly radiated clade (incomplete lineage sorting expected). **Mechanism:** ILS produces gene-tree-species-tree discordance indistinguishable from HGT at short internodes. ALE / GeneRax cannot separate them. **Symptom:** Many "transfers" at short internal branches; transfer rate per branch correlates with branch length (more transfers at short branches). **Fix:** Use DLCpar, which models deep coalescence (ILS) jointly with duplication and loss; or pre-screen for ILS-likely loci via Dsuite ABBA-BABA (see [[introgression-detection]]); restrict ALE to gene families where ILS unlikely (long internodes). For phylogenomic-scale ILS, use an ASTRAL-Pro2 coalescent species tree as the fixed input to reconciliation. ### Multifurcations in the species tree **Trigger:** Using a species tree with polytomies (unresolved nodes). **Mechanism:** ALE / GeneRax / AleRax assume strictly bifurcating species trees; multifurcations break the inference. **Symptom:** Tool fails with "polytomy detected" or runs but produces nonsensical results at multifurcating nodes. **Fix:** Resolve polytomies before reconciliation via `ape::multi2di()` (random resolution) or with an outgroup-informed resolution (RAxML / ASTRAL-Pro2 on more data). Document the resolution. ## Quantitative Thresholds | Quantity | Threshold | Source / Rationale | |----------|-----------|-------------------| | ALE gene-tree sample size | >= 100 bootstrap or UFBoot trees per family | ALE documentation; below this, posterior poorly resolved | | Bacterial transfer rate (per gene per branch) | typically 0.001-0.05 in ALE inferences | Szöllősi 2013; varies clade | | Eukaryote duplication rate | typically 0.0001-0.005 | eukaryote-specific convention; calibrate per clade | | Branch-wise DTL event posterior | > 0.5 for "called" event | ssolo/ALE convention | | Minimum gene families for AleRax | >= 100 families; >= 20 species | Morel 2024 | | Minimum species for species-tree rooting via ALE | >= 30 species across the clade | Williams 2017 | | GeneRax `--strategy` choices | EVAL only (no refinement), SPR (refinement), HYBRID (random + SPR) | GeneRax docs | | RANGER cost weights default | D=2, T=3, L=1 (Bansal 2018) | Sensitivity sweep recommended | | Whale.jl MCMC burn-in | >= 1000 samples; convergence by ESS >= 200 | Whale.jl docs | | Reasonable runtime per family (ALE) | 1-30 minutes | Modern CPU; varies with gene-tree count | | Reasonable runtime per family (GeneRax) | 0.1-5 minutes | Modern CPU; SPR is slower than EVAL | | Maximum families per AleRax run | < 5000 (computational) | Above this, partition | | Species labels case-sensitivity | strict; case mismatch = silent failure | Universal convention | | Gene-ID separator | ALE / GeneRax accept a single-character separator via `separators="X"` (commonly `|` or `_`); Whale.jl uses a mapping file. Verify per-tool. | Tool-specific | | Transfer branch detection minimum support | branch posterior > 0.5 + AU test on alternative placement | Conservative publication-grade | | Distinguish T from L | T posterior - L posterior on candidate branch | Subtraction approach; ALE outputs both | ## ALE Standard Workflow **Goal:** Quantify D/T/L events per branch of a species tree, integrating gene-tree uncertainty. **Approach:** Build UFBoot gene trees per orthogroup -> `ALEobserve` to encode tree samples -> `ALEml_undated` for reconciliation -> aggregate per-branch event posteriors. ```bash # 1. Build UFBoot gene trees per orthogroup mkdir -p gene_trees for og in orthogroups/*.fa; do base=$(basename $og .fa) iqtree2 -s $og -m TEST -B 1000 -nt 2 --prefix gene_trees/$base done # 2. Encode for ALE for ufb in gene_trees/*.ufboot; do ALEobserve $ufb done # Produces gene_trees/*.ale files # 3. Reconcile against species tree mkdir -p reconciled for ale in gene_trees/*.ale; do ALEml_undated species_tree.nwk $ale \ separators="|" \ sample=100 \ output_format=newick mv ${ale%.*}_*.uml ${ale%.*}.uml mv ${ale%.*}_*.uTs ${ale%.*}.uTs done # 4. Aggregate branch-wise events python aggregate_ale_events.py reconciled/ > branch_events.tsv ``` ```python '''Aggregate ALE outputs per species-tree branch.''' import glob from collections import defaultdict import pandas as pd def parse_uts(path): '''ALE _uTs format: branch duplications transfers losses originations speciations.''' rows = [] with open(path) as fh: for ln in fh: if ln.startswith('#') or not ln.strip(): continue parts = ln.split() if len(parts) >= 5: rows.append({ 'branch_id': parts[0], 'duplications': float(parts[1]), 'transfers': float(parts[2]),