| name | vdjdb-extract |
| description | Extract TCR:pMHC specificity data from raw source files (papers, supplementary tables, XLS, PDF, 10X output, AIRR-format) and produce a VDJdb-formatted TSV chunk ready for /format and /proofread. |
/extract — VDJdb Data Extraction Skill
Purpose
Extract T-cell receptor antigen-specificity records from arbitrary source files and write a single TSV in VDJdb chunk format. Every value written to the output must be verified back against the original source to prevent hallucinations. This skill feeds directly into /format and /proofread.
Invocation
/extract [path-to-folder-or-file]
The input may be a folder or individual file(s) containing any mix of:
- Supplementary Excel/CSV/TSV tables
- PDF manuscripts or supplementary PDFs
- Plain-text files (FASTA, exported tables)
- 10X Genomics
filtered_contig_annotations.csv / clonotype files
- AIRR-format TSV files (
productive, v_call, j_call, cdr3_aa columns)
- Adaptive Biotech ImmunoSEQ exports (TSV; gene columns use
TCRB/TCRA prefix with zero-padded numbers like TCRBV06-05*01 — flag all gene names for conversion in /format)
- Jupyter notebooks or scripts used by the authors
If the user limits scope (e.g., "only beta chains", "skip MHC data"), respect that limit and note it in the extraction log.
Excel-specific pitfalls (apply when source is .xlsx/.xls)
-
Embedded sub-header rows: Submitters often repeat column headers mid-table to mark new donors or groups. These appear as rows where gene columns contain TCRα, TCRβ, TRAV, TRBV, CDR3α, CDR3β, CDR3 (literal text). Filter by checking both CDR3 and gene columns — some sub-header rows have blank CDR3 cells and TCRα/TCRβ only in gene columns; the simpler "check CDR3 column for header text" filter will miss them.
-
Allele suffix with functionality code: Some cells contain the gene name formatted as TRAV16*01 F (allele + space + IMGT functionality code). Standard regex \*\d+\s*$ fails because F follows the space. Use re.sub(r'\*.*$', '', v).strip() to strip everything from * onwards.
-
Excel formula artifacts: Cell merging or formula errors can produce values like TRAJ3+D107:D1082 (gene name + cell reference). Strip everything after + to recover the gene: val.split('+')[0].strip().
-
TRBJ/TRBD column swap: Submitters sometimes place TRBJ before TRBD in their table despite the column header saying the opposite. Always verify by gene name prefix (e.g., TRBJ2-7*01 starting with TRBJ → it is a J gene regardless of which column it's in). Apply swap correction when prefix contradicts column header.
-
Non-standard characters in CDR3: Excel auto-correct, copy-paste artefacts, or annotation notations can introduce characters like #, X, * in CDR3 fields. Exclude rows containing non-20-AA characters; log the exclusion.
-
Frequency as Excel formula: Cells like =I4/26*100 appear as literal strings if the workbook is loaded without data_only=True. Always use data_only=True in openpyxl to get cached computed values.
⚠️ Absolute Requirements (Non-Negotiable)
CDR3 sequences
- Must contain only standard amino acids:
ARNDCQEGHILKMFPSTWYV
- Canonical form: starts with
C, ends with F or W
- Minimum length: 4 residues
Handling non-standard CDR3s:
| Case | Action |
|---|
Contains non-20-AA character (X, B, #, *, etc.) | Exclude the row — log it; likely a data artefact |
Does not start with C | Keep in chunks/ — flag in extraction log; VDJdb build marks it non-canonical automatically |
Ends with residue other than F/W | Keep in chunks/ — flag in extraction log |
| Contains genuine modified/non-natural residues | Move to chunks_with_unconventional_aa/ after confirming with user |
chunks_with_unconventional_aa/ is only for non-standard amino acids (beyond the 20 canonical). Non-canonical start/end residues stay in chunks/.
Epitope sequences
- Must be standard amino acids only
- If authors describe a chemical modification, a non-peptide antigen, or a long peptide pool: flag prominently in the log and ask before including
References — strict enforcement
Only these formats are acceptable in reference.id:
| Format | Example | Notes |
|---|
PMID:XXXXXXX | PMID:28975614 | Strongly preferred; numeric only after colon |
doi:10.XXXX/... | doi:10.1016/j.immuni.2023.01.001 | Lowercase doi:, no URL prefix |
| Preprint URL | https://www.biorxiv.org/content/10.1101/2024.01.01.123456 | Full URL |
| Unpublished | unpublished: Submitter Name YYYY-MM-DD | For submissions without a publication |
NEVER invent or guess a PMID. If uncertain, leave blank and ask the user. DOI and preprint URLs must be quoted exactly from the source.
Hallucination prevention (mandatory for every extracted value)
After extracting any amino acid sequence, gene name, species name, MHC allele, or reference ID:
- Run a grep or direct text search in the original source file to confirm the exact string is present
- Log the verification result (found / not found / found with minor variant)
- If the value cannot be confirmed in the source: mark as
[UNVERIFIED] and do not include it without explicit user approval
Step-by-Step Workflow
Step 1 — Survey the source folder
- List all files and identify types (PDF, XLS, TSV, CSV, FASTQ, etc.)
- Note which files are likely to contain: TCR sequences, antigen/epitope data, MHC/HLA data, methods, references
- Share the inventory with the user before proceeding if the folder contains more than 3 files or has an unclear structure
Step 2 — Build a cross-reference graph
Source data is often spread across multiple files. Build an explicit graph:
- Identify all ID columns in each file: barcode, clone ID, sample ID, donor ID, clonotype ID, barcode, well ID, etc.
- Determine which IDs appear in multiple files and can be used to join records
- Perform the join; log the join keys used
Ambiguities (one-to-many links, missing join keys, contradictory values between files) must be logged immediately. Do not silently pick one option.
Step 3 — Extract TCR complex fields
For each record, extract the following. Leave blank if absent — never use a placeholder string (NA, N/A, null, nan, -, .).
| VDJdb field | What to look for | Verification |
|---|
cdr3.alpha | Alpha chain CDR3 amino acid sequence | grep in source |
v.alpha | TRAV gene (IMGT style preferred; note if not IMGT) | grep in source |
j.alpha | TRAJ gene | grep in source |
cdr3.beta | Beta chain CDR3 amino acid sequence | grep in source |
v.beta | TRBV gene | grep in source |
d.beta | TRBD gene (often missing; leave blank) | grep if present |
j.beta | TRBJ gene | grep in source |
species | Organism (HomoSapiens, MusMusculus, RattusNorvegicus, MacacaMulatta) | confirm from text |
mhc.a | First MHC chain (e.g., HLA-A*02:01, H-2Db) | grep in source |
mhc.b | Second MHC chain (B2M for MHC-I; β-chain allele for MHC-II) | grep in source |
mhc.class | MHCI or MHCII | confirm from context |
antigen.epitope | Epitope amino acid sequence | grep in source |
antigen.gene | Antigen gene name (e.g., pp65, NP, MART-1) | grep in source |
antigen.species | Antigen origin (e.g., CMV, InfluenzaA, HomoSapiens) | confirm from text |
reference.id | PMID, DOI, or preprint URL | strict format check + grep |
At least one of cdr3.alpha or cdr3.beta must be non-blank per row.
Both mhc.a and mhc.b must be filled if MHC data exists.
Cross-reference epitopes against patches/antigen_epitope_species_gene.dict — if the epitope is already known, use the dict's gene and species values.
Step 4 — Extract method fields
These fields directly affect the VDJdb confidence score (0–3) computed by py_src/ScoreFactory.py. Extract them carefully from the methods section.
| Field | Recognised values | Notes |
|---|
method.identification | tetramer-sort, dextramer-sort, pelimer-sort, pentamer-sort, antigen-loaded-targets, antigen-expressing-targets, beads, cultured-T-cells, limiting-dilution-cloning, tetramer-umi, cd8null-tetramer | Multiple values comma-separated, no spaces |
method.frequency | X/X (e.g., 7/30), X%, or decimal fraction | Fraction format preferred |
method.singlecell | yes if single-cell sequencing was used; blank otherwise | |
method.sequencing | sanger, rna-seq, amplicon-seq | |
method.verification | tetramer-stain, dextramer-stain, direct, restimulation, co-culture, antigen-loaded-targets, antigen-expressing-targets, beads | |
If the paper uses a method not in the above lists, do not force it into an existing category. Log it as a "Novel method" candidate for extending the VDJdb specification.
Score shortcut: If a PDB structure ID is available, record it in meta.structure.id — this grants score 3 automatically, bypassing all other scoring logic.
Step 5 — Extract metadata fields
Fill as many of the 12 meta columns as the source supports. Leave others blank.
| Field | Description |
|---|
meta.study.id | Internal study identifier used in the paper |
meta.cell.subset | T cell subset (CD8+, CD4+CD25+, etc.) |
meta.subset.frequency | Clone frequency within the cell subset |
meta.subject.cohort | Donor cohort (healthy, HIV+, CMV-seroneg, etc.) |
meta.subject.id | Donor/patient identifier |
meta.replica.id | Replicate or timepoint identifier |
meta.clone.id | T cell clone identifier or barcode |
meta.epitope.id | Short epitope label from the paper (e.g., FL10) |
meta.tissue | PBMC, spleen, TIL, TCL, etc. |
meta.donor.MHC | Donor HLA typing if reported |
meta.donor.MHC.method | HLA typing method if reported |
meta.structure.id | PDB ID if a structure was solved for this complex |
comment | Any important note not captured elsewhere (max 140 characters) |
Step 6 — Assemble the output TSV
Canonical column order (must match exactly):
chunk.id
cdr3.alpha v.alpha j.alpha cdr3.beta v.beta d.beta j.beta
species mhc.a mhc.b mhc.class antigen.epitope antigen.gene antigen.species
reference.id
method.identification method.frequency method.singlecell method.sequencing method.verification
meta.study.id meta.cell.subset meta.subset.frequency meta.subject.cohort meta.subject.id
meta.replica.id meta.clone.id meta.epitope.id meta.tissue
meta.donor.MHC meta.donor.MHC.method meta.structure.id
comment
Formatting rules:
- Tab-separated, UTF-8, Unix line endings (LF)
chunk.id: sequential integers starting from 1- Blank fields: truly empty (no quotes, no
NA, no -)
- No trailing whitespace; no quoted fields (TSV, not CSV)
comment is optional — include column only if at least one row has a comment
Step 7 — Write the extraction log
Write <output_basename>_extraction_log.txt containing:
- Source inventory: all files found, their types, and assigned roles
- Cross-reference graph: which ID columns were used to link which files, and join cardinality
- Ambiguities: every unclear/ambiguous case and the decision made (or flagged for user)
- Verification results: for each field, confirm or report failure to verify in source
- Unverified values: any value included at user direction despite not being directly verifiable
- Novel methods: identification/verification methods not in the current VDJdb vocabulary
- Excluded records: rows dropped and the reason (failed canonical check, missing required fields, etc.)
- Scope restrictions: any user-imposed limits on what was extracted
Validation Reference Files
| File | Purpose |
|---|
proofreading/imgt_alleles.tsv.gz | Primary V/D/J gene ID authority — check all extracted gene names here first |
proofreading/imgt.md | IMGT nomenclature rules and gene structure explanation |
proofreading/mhc_alleles.tsv.gz | Primary HLA allele authority — check all extracted human MHC alleles here |
proofreading/mhc.md | MHC/HLA naming rules, class I vs II, non-human conventions |
patches/IGM_nomenclature_table.tsv | Secondary V/D/J fallback (existing repo file) |
patches/nomenclature.conversions | Old-style → IMGT gene name conversions |
patches/antigen_epitope_species_gene.dict | Known epitope → antigen.gene/antigen.species mappings |
py_src/ScoreFactory.py | Confidence score logic and method vocabulary |
README.md | VDJdb column specification |
Output
- Primary output:
<PMID_xxxxxxx>_unformatted.txt — raw VDJdb TSV (gene names not yet IMGT-normalised)
- Log:
<PMID_xxxxxxx>_extraction_log.txt — full provenance and ambiguity record
Suggest the output filename based on PMID if available (e.g., PMID_40713946_unformatted.txt), otherwise use the folder name or ask the user.
Next Steps
- Run
/format on the output to normalise gene names, species, and MHC alleles to IMGT standard
- Run
/proofread to validate against QC scripts in py_src/
