Cohere Migration Deep Dive
Overview
Comprehensive guide for migrating to Cohere from OpenAI, Anthropic, or other LLM providers, including embedding re-vectorization, prompt adaptation, and gradual traffic shifting.
Prerequisites
- Current LLM integration documented
- Cohere API key and SDK installed
- Feature flag infrastructure
- Rollback strategy
Migration Types
| From | Complexity | Duration | Key Challenge |
|---|
| OpenAI → Cohere | Medium | 1-2 weeks | Prompt adaptation, embedding migration |
| Anthropic → Cohere | Medium | 1-2 weeks | Message format, tool definitions |
| Custom/OSS → Cohere | Low | Days | SDK integration |
| Embedding migration | High | 2-4 weeks | Re-vectorize entire corpus |
Instructions
Step 1: OpenAI to Cohere Chat Migration
import OpenAI from 'openai';
const openai = new OpenAI();
const response = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [
{ role: 'system', content: 'You are helpful.' },
{ role: 'user', content: 'Hello' },
],
max_tokens: 500,
temperature: 0.7,
});
const text = response.choices[0].message.content;
import { CohereClientV2 } from 'cohere-ai';
const cohere = new CohereClientV2();
const response = await cohere.chat({
model: 'command-a-03-2025',
messages: [
{ role: 'system', content: 'You are helpful.' },
{ role: 'user', content: 'Hello' },
],
maxTokens: 500,
temperature: 0.7,
});
const text = response.message?.content?.[0]?.text;
Step 2: Embedding Migration
async function migrateEmbeddings(
documents: Array<{ id: string; text: string }>,
batchSize = 96
) {
const cohere = new CohereClientV2();
let processed = 0;
for (let i = 0; i < documents.length; i += batchSize) {
const batch = documents.slice(i, i + batchSize);
const response = await cohere.embed({
model: 'embed-v4.0',
texts: batch.map(d => d.text),
inputType: 'search_document',
embeddingTypes: ['float'],
});
for (let j = 0; j < batch.length; j++) {
vectorDB.({
: ,
: batch[j].,
: response..[j],
: { : batch[j]. },
});
}
processed += batch.;
.();
}
}
Step 3: Tool Use Migration
const openaiTools = [{
type: 'function',
function: {
name: 'get_weather',
description: 'Get weather',
parameters: {
type: 'object',
properties: { city: { type: 'string' } },
required: ['city'],
},
},
}];
const cohereTools = [{
type: 'function',
function: {
name: 'get_weather',
description: 'Get weather',
parameters: {
type: 'object',
properties: { city: { type: 'string' } },
required: ['city'],
},
},
}];
Step 4: Streaming Migration
const openaiStream = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [...],
stream: true,
});
for await (const chunk of openaiStream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}
const cohereStream = await cohere.chatStream({
model: 'command-a-03-2025',
messages: [...],
});
for await (const event of cohereStream) {
if (event.type === 'content-delta') {
process.stdout.write(event.delta?.message?.content?.text ?? '');
}
}
Step 5: Adapter Pattern for Gradual Migration
interface LLMAdapter {
chat(message: string, options?: { system?: string; maxTokens?: number }): Promise<string>;
embed(texts: string[]): Promise<number[][]>;
rerank(query: string, docs: string[], topN?: number): Promise<Array<{ index: number; score: number }>>;
}
class CohereAdapter implements LLMAdapter {
private client = new CohereClientV2();
async chat(message: string, options?: { system?: string; maxTokens?: number }): Promise<string> {
const messages: any[] = [];
if (options?.system) messages.push({ : , : options. });
messages.({ : , : message });
response = ..({
: ,
messages,
: options?.,
});
response.?.?.[]?. ?? ;
}
(: []): <[][]> {
response = ..({
: ,
texts,
: ,
: [],
});
response..;
}
(: , : [], topN = ): <<{ : ; : }>> {
response = ..({
: ,
query,
: docs,
topN,
});
response..( ({ : r., : r. }));
}
}
{
}
(): {
coherePercentage = ();
(.() * < coherePercentage) {
();
}
();
}
Step 6: Validation and Comparison
async function compareOutputs(message: string): Promise<{
openai: string;
cohere: string;
latencyMs: { openai: number; cohere: number };
}> {
const startOpenAI = Date.now();
const openaiResult = await openaiAdapter.chat(message);
const openaiLatency = Date.now() - startOpenAI;
const startCohere = Date.now();
const cohereResult = await cohereAdapter.chat(message);
const cohereLatency = Date.now() - startCohere;
return {
openai: openaiResult,
cohere: cohereResult,
latencyMs: { openai: openaiLatency, cohere: cohereLatency },
};
}
const testQueries = ['Summarize this text', 'Translate to French', 'Extract key points'];
for (const q of testQueries) {
const result = (q);
.();
.();
.();
}
Cohere-Unique Features (Not in OpenAI)
| Feature | Cohere | OpenAI |
|---|
| Built-in Rerank | cohere.rerank() | Not available |
| RAG with citations | documents param + citations | Manual implementation |
| Connectors (data sources) | connectors param | Not available |
| Classify endpoint | cohere.classify() | Not available |
| Safety modes | safetyMode param | Moderation API (separate) |
Rollback Plan
curl -X POST https://flagservice/flags/cohere_migration_pct -d '{"value": 0}'
Output
- Adapter layer abstracting LLM provider
- Embedding migration with batch processing
- A/B comparison for output quality validation
- Feature-flag controlled traffic shifting
- Rollback via feature flag (instant, no deploy)
Error Handling
| Issue | Cause | Solution |
|---|
| Embedding dimension mismatch | Mixed providers in same DB | Separate collections per provider |
| Response shape different | Provider-specific format | Use adapter pattern |
| Higher latency on Cohere | Different model size | Try command-r7b for speed |
| Quality difference | Different model strengths | Tune system prompts per provider |
Resources
Next Steps
For Cohere-specific architecture patterns, see cohere-reference-architecture.
