Transformers.js ONNX Pipelines
Transformers.js runs Hugging Face models on ONNX Runtime in-browser or in-Node. The high-level pipeline() is convenient until it isn't — most non-trivial models need direct tokenizer + model use because the pipeline's task wrapping makes assumptions that don't hold (e.g., the cross-encoder softmax-over-1 trap).
When to use
- Local embeddings without an API (cosine search, RAG, dedup).
- Cross-encoder reranking for retrieval cascades.
- Browser-side inference (PII redaction, classification) without a server roundtrip.
- Replacing OpenAI embedding calls with a quantized local model to control cost.
- Integrating with a Cloudflare Worker via Workers AI (different surface, but ONNX know-how transfers).
Core capabilities
Bi-encoder (sentence embeddings)
import { pipeline, env } from '@xenova/transformers';
env.allowLocalModels = false;
env.cacheDir = process.env.MODEL_CACHE
?? path.join(os.homedir(), '.cache', 'transformers-js');
const embed = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { quantized: true });
const out = await embed('the quick brown fox', { pooling: 'mean', normalize: true });
const vec: Float32Array = out.data;
pooling: 'mean' averages token embeddings; 'cls' uses [CLS]. normalize: true makes cosine == dot product:
function cosine(a: Float32Array, b: Float32Array) {
let dot = 0;
for (let i = 0; i < a.length; i++) dot += a[i] * b[i];
return dot;
}
Cross-encoder reranking — bypass the pipeline
The MS MARCO MiniLM rerankers publish a single regression head (num_labels=1). Transformers.js's text-classification pipeline applies softmax to the logits. Softmax over a single value collapses to 1.0 — every score becomes 1.0.
Skip the pipeline. Tokenize and forward manually:
import { AutoTokenizer, AutoModelForSequenceClassification } from '@xenova/transformers';
const modelId = 'Xenova/ms-marco-MiniLM-L-6-v2';
const [tokenizer, model] = await Promise.all([
AutoTokenizer.from_pretrained(modelId),
AutoModelForSequenceClassification.from_pretrained(modelId, { quantized: true }),
]);
async function rerank(query: string, candidates: string[]) {
const queries = new Array(candidates.length).fill(query);
const inputs = tokenizer(queries, {
text_pair: candidates,
padding: true, truncation: true, max_length: 512,
});
const outputs = await model(inputs);
const logits = outputs.logits;
const dims = logits.dims;
const numLabels = dims[1] ?? 1;
data = logits.;
candidates.( {
score = numLabels ===
? data[i]
: data[i * numLabels + ];
{ text, score };
}).( b. - a.);
}
The raw logit isn't bounded — calibration is the user's problem. For relative ranking it's fine.
Lazy loading + cache strategy
Models are 25-100MB quantized; download is the slow part. Cache aggressively:
let _embedderPromise: Promise<any> | null = null;
function getEmbedder() {
if (!_embedderPromise) {
_embedderPromise = pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { quantized: true });
}
return _embedderPromise;
}
Idempotent across concurrent calls — first call awaits the download, subsequent calls hit the in-memory model.
Cache directory matters in CI
env.cacheDir = process.env.MODEL_CACHE
?? path.join(os.homedir(), '.cache', 'transformers-js');
In GitHub Actions, point this at a workspace-local path so actions/cache can persist it:
env:
MODEL_CACHE: ${{ github.workspace }}/.cache/transformers-js
- uses: actions/cache@v4
with:
path: ${{ env.MODEL_CACHE }}
key: ${{ runner.os }}-tfjs-allmini-v1
Without this, every CI run re-downloads.
Building a corpus offline
For a static catalog (skill descriptions, doc chunks), embed once at build time and ship the vectors:
const embed = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', { quantized: true });
const items = loadCorpus();
const dim = 384;
const buf = new Float32Array(items.length * dim);
for (let i = 0; i < items.length; i++) {
const out = await embed(items[i].text, { pooling: 'mean', normalize: true });
buf.set(out.data, i * dim);
}
fs.writeFileSync('data/embeddings.bin', Buffer.from(buf.buffer));
fs.writeFileSync('data/embeddings.meta.json', JSON.stringify({
model: 'Xenova/all-MiniLM-L6-v2',
dim, count: items.length,
ids: items.map((x) => x.id),
}, null, 2));
Load at runtime with no parsing cost:
const meta = JSON.parse(fs.readFileSync('data/embeddings.meta.json', 'utf-8'));
const buf = fs.readFileSync('data/embeddings.bin');
const vectors = new Float32Array(buf.buffer, buf.byteOffset, buf.byteLength / 4);
Storing 384-dim Float32Array is 1.5KB per item — fine for tens of thousands.
Browser-side inference
import { pipeline, env } from '@xenova/transformers';
env.allowRemoteModels = true;
env.remoteHost = 'https://your-cdn.example.com';
const classifier = await pipeline(
'text-classification',
'Xenova/distilbert-base-uncased-finetuned-sst-2-english',
{ device: 'webgpu' },
);
Be honest about the download cost — show a progress bar.
Anti-patterns
Cross-encoder via text-classification pipeline
Symptom: Every candidate scores 1.0; ranking is degenerate.
Diagnosis: Pipeline softmaxes a single-logit head; softmax-over-1 = 1.
Fix: Use AutoTokenizer + AutoModelForSequenceClassification directly. Read outputs.logits.data as raw scores.
env.allowLocalModels = true in CI
Symptom: Model downloads succeed locally, fail in CI: "model not found".
Diagnosis: With allowLocalModels = true the SDK looks for files on disk first; in CI, no disk model = fail.
Fix: env.allowLocalModels = false (default). Combine with a CI cache for the download.
Importing the pipeline at module load
Symptom: Server startup blocks for 5s on first deploy.
Diagnosis: Top-level await pipeline(...) runs at import time.
Fix: Lazy-load inside a function. Cache the Promise so concurrent first calls share one load.
Forgetting normalize: true for cosine
Symptom: Scores look right but order is subtly off.
Diagnosis: Vectors aren't unit-normalized; raw dot products bias toward longer strings.
Fix: Always { pooling: 'mean', normalize: true }. Pre-normalize stored vectors.
Float32Array reuse across calls
Symptom: Stored vectors mysteriously change; cache looks corrupted.
Diagnosis: Some pipeline implementations return a view into a reused tensor buffer; the next call overwrites.
Fix: Copy on capture: new Float32Array(out.data).
Reranker on too-long documents
Symptom: RangeError: Tokenized inputs exceeded 512 tokens.
Diagnosis: Cross-encoders truncate at 512 tokens jointly across (query, candidate). Long candidates lose context.
Fix: Truncate candidate text to ~400 tokens. Description + name only; skip body.
Quality gates
Deterministic Audit
Before wiring (or reviewing) a Transformers.js integration, write the decisions as a JSON
plan matching schemas/transformers-js-onnx-pipelines-plan.schema.json and run the
deterministic auditor:
node scripts/transformers_js_onnx_pipelines_audit.mjs --input examples/sample-input.json
auditTransformersJsOnnxPipelines(plan) (in scripts/transformers_js_onnx_pipelines_audit.mjs)
turns this skill's anti-patterns and Quality Gates into machine-checkable rules over
structured fields: a cross-encoder routed through the text-classification pipeline
instead of raw logits (the softmax-over-1 trap), un-normalized bi-encoder vectors fed to
cosine, allowLocalModels: true in CI, a missing CI model cache, an uncached first-load
Promise, persisted vectors that were never copied out of the reused tensor buffer,
unpinned model versions, and reranker inputs past the 512-token joint limit. It returns
{ pass, score, findings, recommendations }. examples/sample-input.json is a correctly
bypassed cross-encoder rerank plan (pass: true). Changes are tracked in CHANGELOG.md.
NOT for
- Python
transformers — different SDK; don't transfer assumptions.
- TensorFlow.js — different runtime, different model formats.
- ONNX Runtime Web directly without Transformers.js — lower-level; use only if you need custom ops.
- Workers AI / Vertex AI — managed inference; use the platform SDK.
- Model training — Transformers.js is inference-only.
