Langfuse Migration Deep Dive
Current State
!npm list langfuse @langfuse/client 2>/dev/null | head -5 || echo 'No langfuse packages'
Overview
Comprehensive guide for complex migrations: cloud-to-self-hosted, LangSmith-to-Langfuse, cross-instance data migration, and zero-downtime dual-write patterns.
Prerequisites
- Understanding of source and target Langfuse instances
- API keys for both source and target
- Git branch for migration work
- Rollback plan documented
Migration Scenarios
| Scenario | Complexity | Downtime | Data Loss Risk |
|---|
| Cloud to Cloud (different project) | Low | None | None |
| Cloud to Self-hosted | Medium | Minutes | Low |
| Self-hosted to Cloud | Medium | Minutes | Low |
| LangSmith to Langfuse | High | Hours | Medium |
| SDK v3 to v4+ (no data migration) | Low | None | None |
Instructions
Step 1: Export Data from Source Instance
import { LangfuseClient } from "@langfuse/client";
import { writeFileSync, mkdirSync } from "fs";
const source = new LangfuseClient({
publicKey: process.env.SOURCE_LANGFUSE_PUBLIC_KEY,
secretKey: process.env.SOURCE_LANGFUSE_SECRET_KEY,
baseUrl: process.env.SOURCE_LANGFUSE_BASE_URL,
});
async function exportAll(outputDir: string) {
mkdirSync(outputDir, { recursive: true });
let page = 1;
let allTraces: any[] = [];
let hasMore = true;
console.log("Exporting traces...");
while (hasMore) {
const result = await source.api.traces.list({ limit: 100, page });
allTraces.push(...result.data);
hasMore = result.data.length === 100;
page++;
await new Promise((r) => setTimeout(r, 200));
}
writeFileSync(`${outputDir}/traces.json`, JSON.stringify(allTraces, null, 2));
console.log(` Exported ${allTraces.length} traces`);
page = 1;
let allScores: any[] = [];
hasMore = true;
console.log("Exporting scores...");
while (hasMore) {
const result = await source.api.scores.list({ limit: 100, page });
allScores.push(...result.data);
hasMore = result.data.length === 100;
page++;
await new Promise((r) => setTimeout(r, 200));
}
writeFileSync(`${outputDir}/scores.json`, JSON.stringify(allScores, null, 2));
console.log(` Exported ${allScores.length} scores`);
console.log("Exporting prompts...");
const prompts = await source.api.prompts.list({ limit: 100 });
writeFileSync(`${outputDir}/prompts.json`, JSON.stringify(prompts.data, null, 2));
console.log(` Exported ${prompts.data.length} prompts`);
console.log("Exporting datasets...");
const datasets = await source.api.datasets.list({ limit: 100 });
const fullDatasets = [];
for (const ds of datasets.data) {
const items = await source.api.datasetItems.list({ datasetName: ds.name, limit: 1000 });
fullDatasets.push({ ...ds, items: items.data });
await new Promise((r) => setTimeout(r, 200));
}
writeFileSync(`${outputDir}/datasets.json`, JSON.stringify(fullDatasets, null, 2));
console.log(` Exported ${fullDatasets.length} datasets`);
}
exportAll("./migration-export");
Step 2: Import Data to Target Instance
import { LangfuseClient } from "@langfuse/client";
import { readFileSync } from "fs";
const target = new LangfuseClient({
publicKey: process.env.TARGET_LANGFUSE_PUBLIC_KEY,
secretKey: process.env.TARGET_LANGFUSE_SECRET_KEY,
baseUrl: process.env.TARGET_LANGFUSE_BASE_URL,
});
async function importAll(inputDir: string) {
console.log("Importing prompts...");
const prompts = JSON.parse(readFileSync(`${inputDir}/prompts.json`, "utf-8"));
for (const prompt of prompts) {
await target.api.prompts.create({
name: prompt.name,
prompt: prompt.prompt,
type: prompt.type,
config: prompt.,
: prompt.,
});
.();
( (r, ));
}
.();
datasets = .((, ));
( ds datasets) {
target...({
: ds.,
: ds.,
: { ...ds., : },
});
( item ds. || []) {
target...({
: ds.,
: item.,
: item.,
: item.,
});
( (r, ));
}
.();
}
.();
.();
.();
}
();
Step 3: Dual-Write for Zero-Downtime Migration
Write traces to both instances during transition:
import { LangfuseSpanProcessor } from "@langfuse/otel";
import { NodeSDK } from "@opentelemetry/sdk-node";
const sourceProcessor = new LangfuseSpanProcessor({
publicKey: process.env.SOURCE_LANGFUSE_PUBLIC_KEY,
secretKey: process.env.SOURCE_LANGFUSE_SECRET_KEY,
baseUrl: process.env.SOURCE_LANGFUSE_BASE_URL,
});
const targetProcessor = new LangfuseSpanProcessor({
publicKey: process.env.TARGET_LANGFUSE_PUBLIC_KEY,
secretKey: process.env.TARGET_LANGFUSE_SECRET_KEY,
baseUrl: process.env.TARGET_LANGFUSE_BASE_URL,
});
const sdk = new NodeSDK({
spanProcessors: [sourceProcessor, targetProcessor],
});
sdk.start();
Step 4: Validate Migration
import { LangfuseClient } from "@langfuse/client";
const source = new LangfuseClient({
publicKey: process.env.SOURCE_LANGFUSE_PUBLIC_KEY,
secretKey: process.env.SOURCE_LANGFUSE_SECRET_KEY,
baseUrl: process.env.SOURCE_LANGFUSE_BASE_URL,
});
const target = new LangfuseClient({
publicKey: process.env.TARGET_LANGFUSE_PUBLIC_KEY,
secretKey: process.env.TARGET_LANGFUSE_SECRET_KEY,
baseUrl: process.env.TARGET_LANGFUSE_BASE_URL,
});
async function validate() {
const sourcePrompts = await source.api.prompts.list({ limit: 100 });
const targetPrompts = await target.api.prompts.list({ limit: 100 });
console.log(`Prompts: source=, target=`);
sourceDatasets = source...({ : });
targetDatasets = target...({ : });
.();
since = (.() - ).();
sourceTraces = source...({ : since, : });
targetTraces = target...({ : since, : });
.();
variance = .(sourceTraces.. - targetTraces..) / .(sourceTraces.., );
.();
}
();
Step 5: Cutover and Cleanup
const sdk = new NodeSDK({
spanProcessors: [targetProcessor],
});
Rollback Plan
set -euo pipefail
export LANGFUSE_PUBLIC_KEY="pk-lf-source-..."
export LANGFUSE_SECRET_KEY="sk-lf-source-..."
export LANGFUSE_BASE_URL="https://source.langfuse.com"
Error Handling
| Issue | Cause | Solution |
|---|
| Export timeout | Too much data | Paginate with smaller page sizes |
| Import duplicates | Re-running import | Use idempotent creates with unique names |
| Dual-write divergence | One instance failing | Monitor both, alert on variance > 5% |
| Missing prompts | Not exported | Export prompts before datasets |
Resources
