vdjdb-extract

name	vdjdb-extract
description	Extract TCR:pMHC specificity data from raw source files (papers, supplementary tables, XLS, PDF, 10X output, AIRR-format) and produce a VDJdb-formatted TSV chunk ready for /format and /proofread.

/extract — VDJdb Data Extraction Skill

Purpose

Extract T-cell receptor antigen-specificity records from arbitrary source files and write a single TSV in VDJdb chunk format. Every value written to the output must be verified back against the original source to prevent hallucinations. This skill feeds directly into /format and /proofread.

Invocation

/extract [path-to-folder-or-file]

The input may be a folder or individual file(s) containing any mix of:

• Supplementary Excel/CSV/TSV tables
• PDF manuscripts or supplementary PDFs
• Plain-text files (FASTA, exported tables)
• 10X Genomics filtered_contig_annotations.csv / clonotype files
• AIRR-format TSV files (productive, v_call, j_call, cdr3_aa columns)
• Adaptive Biotech ImmunoSEQ exports (TSV; gene columns use TCRB/TCRA prefix with zero-padded numbers like TCRBV06-05*01 — flag all gene names for conversion in /format)
• Jupyter notebooks or scripts used by the authors

If the user limits scope (e.g., "only beta chains", "skip MHC data"), respect that limit and note it in the extraction log.

Excel-specific pitfalls (apply when source is .xlsx/.xls)

1. Embedded sub-header rows: Submitters often repeat column headers mid-table to mark new donors or groups. These appear as rows where gene columns contain TCRα, TCRβ, TRAV, TRBV, CDR3α, CDR3β, CDR3 (literal text). Filter by checking both CDR3 and gene columns — some sub-header rows have blank CDR3 cells and TCRα/TCRβ only in gene columns; the simpler "check CDR3 column for header text" filter will miss them.

2. Allele suffix with functionality code: Some cells contain the gene name formatted as TRAV16*01 F (allele + space + IMGT functionality code). Standard regex \*\d+\s*$ fails because F follows the space. Use re.sub(r'\*.*$', '', v).strip() to strip everything from * onwards.

3. Excel formula artifacts: Cell merging or formula errors can produce values like TRAJ3+D107:D1082 (gene name + cell reference). Strip everything after + to recover the gene: val.split('+')[0].strip().

4. TRBJ/TRBD column swap: Submitters sometimes place TRBJ before TRBD in their table despite the column header saying the opposite. Always verify by gene name prefix (e.g., TRBJ2-7*01 starting with TRBJ → it is a J gene regardless of which column it's in). Apply swap correction when prefix contradicts column header.

5. Non-standard characters in CDR3: Excel auto-correct, copy-paste artefacts, or annotation notations can introduce characters like #, X, * in CDR3 fields. Exclude rows containing non-20-AA characters; log the exclusion.

6. Frequency as Excel formula: Cells like =I4/26*100 appear as literal strings if the workbook is loaded without data_only=True. Always use data_only=True in openpyxl to get cached computed values.

---

⚠️ Absolute Requirements (Non-Negotiable)

CDR3 sequences

• Must contain only standard amino acids: ARNDCQEGHILKMFPSTWYV
• Canonical form: starts with C, ends with F or W
• Minimum length: 4 residues

Handling non-standard CDR3s:

Case	Action
Contains non-20-AA character (X, B, #, *, etc.)	Exclude the row — log it; likely a data artefact
Does not start with C	Keep in chunks/ — flag in extraction log; VDJdb build marks it non-canonical automatically
Ends with residue other than F/W	Keep in chunks/ — flag in extraction log
Contains genuine modified/non-natural residues	Move to chunks_with_unconventional_aa/ after confirming with user

chunks_with_unconventional_aa/ is only for non-standard amino acids (beyond the 20 canonical). Non-canonical start/end residues stay in chunks/.

Epitope sequences

• Must be standard amino acids only
• If authors describe a chemical modification, a non-peptide antigen, or a long peptide pool: flag prominently in the log and ask before including

References — strict enforcement

Only these formats are acceptable in reference.id:

Format	Example	Notes
PMID:XXXXXXX	PMID:28975614	Strongly preferred; numeric only after colon
doi:10.XXXX/...	doi:10.1016/j.immuni.2023.01.001	Lowercase doi:, no URL prefix
Preprint URL	https://www.biorxiv.org/content/10.1101/2024.01.01.123456	Full URL
Unpublished	unpublished: Submitter Name YYYY-MM-DD	For submissions without a publication

NEVER invent or guess a PMID. If uncertain, leave blank and ask the user. DOI and preprint URLs must be quoted exactly from the source.

Hallucination prevention (mandatory for every extracted value)

After extracting any amino acid sequence, gene name, species name, MHC allele, or reference ID:

1. Run a grep or direct text search in the original source file to confirm the exact string is present
2. Log the verification result (found / not found / found with minor variant)
3. If the value cannot be confirmed in the source: mark as [UNVERIFIED] and do not include it without explicit user approval

---

Step-by-Step Workflow

Step 1 — Survey the source folder

1. List all files and identify types (PDF, XLS, TSV, CSV, FASTQ, etc.)
2. Note which files are likely to contain: TCR sequences, antigen/epitope data, MHC/HLA data, methods, references
3. Share the inventory with the user before proceeding if the folder contains more than 3 files or has an unclear structure

Step 2 — Build a cross-reference graph

Source data is often spread across multiple files. Build an explicit graph:

1. Identify all ID columns in each file: barcode, clone ID, sample ID, donor ID, clonotype ID, barcode, well ID, etc.
2. Determine which IDs appear in multiple files and can be used to join records
3. Perform the join; log the join keys used

Ambiguities (one-to-many links, missing join keys, contradictory values between files) must be logged immediately. Do not silently pick one option.

Step 3 — Extract TCR complex fields

For each record, extract the following. Leave blank if absent — never use a placeholder string (NA, N/A, null, nan, -, .).

VDJdb field	What to look for	Verification
cdr3.alpha	Alpha chain CDR3 amino acid sequence	grep in source
v.alpha	TRAV gene (IMGT style preferred; note if not IMGT)	grep in source
j.alpha	TRAJ gene	grep in source
cdr3.beta	Beta chain CDR3 amino acid sequence	grep in source
v.beta	TRBV gene	grep in source
d.beta	TRBD gene (often missing; leave blank)	grep if present
j.beta	TRBJ gene	grep in source
species	Organism (HomoSapiens, MusMusculus, RattusNorvegicus, MacacaMulatta)	confirm from text
mhc.a	First MHC chain (e.g., HLA-A*02:01, H-2Db)	grep in source
mhc.b	Second MHC chain (B2M for MHC-I; β-chain allele for MHC-II)	grep in source
mhc.class	MHCI or MHCII	confirm from context
antigen.epitope	Epitope amino acid sequence	grep in source
antigen.gene	Antigen gene name (e.g., pp65, NP, MART-1)	grep in source
antigen.species	Antigen origin (e.g., CMV, InfluenzaA, HomoSapiens)	confirm from text
reference.id	PMID, DOI, or preprint URL	strict format check + grep

At least one of cdr3.alpha or cdr3.beta must be non-blank per row. Both mhc.a and mhc.b must be filled if MHC data exists.

Cross-reference epitopes against patches/antigen_epitope_species_gene.dict — if the epitope is already known, use the dict's gene and species values.

Step 4 — Extract method fields

These fields directly affect the VDJdb confidence score (0–3) computed by py_src/ScoreFactory.py. Extract them carefully from the methods section.

Field	Recognised values	Notes
method.identification	tetramer-sort, dextramer-sort, pelimer-sort, pentamer-sort, antigen-loaded-targets, antigen-expressing-targets, beads, cultured-T-cells, limiting-dilution-cloning, tetramer-umi, cd8null-tetramer	Multiple values comma-separated, no spaces
method.frequency	X/X (e.g., 7/30), X%, or decimal fraction	Fraction format preferred
method.singlecell	yes if single-cell sequencing was used; blank otherwise
method.sequencing	sanger, rna-seq, amplicon-seq
method.verification	tetramer-stain, dextramer-stain, direct, restimulation, co-culture, antigen-loaded-targets, antigen-expressing-targets, beads

If the paper uses a method not in the above lists, do not force it into an existing category. Log it as a "Novel method" candidate for extending the VDJdb specification.

Score shortcut: If a PDB structure ID is available, record it in meta.structure.id — this grants score 3 automatically, bypassing all other scoring logic.

Step 5 — Extract metadata fields

Fill as many of the 12 meta columns as the source supports. Leave others blank.

Field	Description
meta.study.id	Internal study identifier used in the paper
meta.cell.subset	T cell subset (CD8+, CD4+CD25+, etc.)
meta.subset.frequency	Clone frequency within the cell subset
meta.subject.cohort	Donor cohort (healthy, HIV+, CMV-seroneg, etc.)
meta.subject.id	Donor/patient identifier
meta.replica.id	Replicate or timepoint identifier
meta.clone.id	T cell clone identifier or barcode
meta.epitope.id	Short epitope label from the paper (e.g., FL10)
meta.tissue	PBMC, spleen, TIL, TCL, etc.
meta.donor.MHC	Donor HLA typing if reported
meta.donor.MHC.method	HLA typing method if reported
meta.structure.id	PDB ID if a structure was solved for this complex
comment	Any important note not captured elsewhere (max 140 characters)

Step 6 — Assemble the output TSV

Canonical column order (must match exactly):

chunk.id
cdr3.alpha  v.alpha  j.alpha  cdr3.beta  v.beta  d.beta  j.beta
species  mhc.a  mhc.b  mhc.class  antigen.epitope  antigen.gene  antigen.species
reference.id
method.identification  method.frequency  method.singlecell  method.sequencing  method.verification
meta.study.id  meta.cell.subset  meta.subset.frequency  meta.subject.cohort  meta.subject.id
meta.replica.id  meta.clone.id  meta.epitope.id  meta.tissue
meta.donor.MHC  meta.donor.MHC.method  meta.structure.id
comment

Formatting rules:

• Tab-separated, UTF-8, Unix line endings (LF)
• chunk.id: sequential integers starting from 1
• Blank fields: truly empty (no quotes, no NA, no -)
• No trailing whitespace; no quoted fields (TSV, not CSV)
• comment is optional — include column only if at least one row has a comment

Step 7 — Write the extraction log

Write <output_basename>_extraction_log.txt containing:

1. Source inventory: all files found, their types, and assigned roles
2. Cross-reference graph: which ID columns were used to link which files, and join cardinality
3. Ambiguities: every unclear/ambiguous case and the decision made (or flagged for user)
4. Verification results: for each field, confirm or report failure to verify in source
5. Unverified values: any value included at user direction despite not being directly verifiable
6. Novel methods: identification/verification methods not in the current VDJdb vocabulary
7. Excluded records: rows dropped and the reason (failed canonical check, missing required fields, etc.)
8. Scope restrictions: any user-imposed limits on what was extracted

---

Validation Reference Files

File	Purpose
proofreading/imgt_alleles.tsv.gz	Primary V/D/J gene ID authority — check all extracted gene names here first
proofreading/imgt.md	IMGT nomenclature rules and gene structure explanation
proofreading/mhc_alleles.tsv.gz	Primary HLA allele authority — check all extracted human MHC alleles here
proofreading/mhc.md	MHC/HLA naming rules, class I vs II, non-human conventions
patches/IGM_nomenclature_table.tsv	Secondary V/D/J fallback (existing repo file)
patches/nomenclature.conversions	Old-style → IMGT gene name conversions
patches/antigen_epitope_species_gene.dict	Known epitope → antigen.gene/antigen.species mappings
py_src/ScoreFactory.py	Confidence score logic and method vocabulary
README.md	VDJdb column specification

---

Output

• Primary output: <PMID_xxxxxxx>_unformatted.txt — raw VDJdb TSV (gene names not yet IMGT-normalised)
• Log: <PMID_xxxxxxx>_extraction_log.txt — full provenance and ambiguity record

Suggest the output filename based on PMID if available (e.g., PMID_40713946_unformatted.txt), otherwise use the folder name or ask the user.

---

Next Steps

1. Run /format on the output to normalise gene names, species, and MHC alleles to IMGT standard
2. Run /proofread to validate against QC scripts in py_src/