Skip to main content الرئيسية المنشئون comeonoliver skillshub cohere-performance-tuning
cohere-performance-tuning Optimize Cohere API performance with caching, batching, model selection, and streaming.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Cohere Chat, Embed, and Rerank.
Trigger with phrases like "cohere performance", "optimize cohere",
"cohere latency", "cohere caching", "cohere slow", "cohere batch".
الانتقال إلى التثبيت سوق المهارات اكتشف واستكشف مهارات الذكاء الاصطناعي التي بناها المجتمع.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
عرض تفاصيل Prompt يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/ComeOnOliver/skillshub --skill cohere-performance-tuningيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
تحميل Zip جاري التحميل... المزيد من هذا المستودع Review product and feature risk before an AI coding agent starts implementation.
Use Xquik for X data and confirmation-gated X actions: tweet search, user lookup, follower export, media download, monitors, webhooks, MCP, and SDK workflows.
Canton Network open-source ecosystem guide covering DAML SDK, Canton runtime, and Splice applications. Use when working with Canton Network, DAML smart contracts, or building decentralized applications.
المهن ذات الصلة SOC
استنادا إلى تصنيف SOC المهني
name cohere-performance-tuning description Optimize Cohere API performance with caching, batching, model selection, and streaming.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Cohere Chat, Embed, and Rerank.
Trigger with phrases like "cohere performance", "optimize cohere",
"cohere latency", "cohere caching", "cohere slow", "cohere batch".
allowed-tools Read, Write, Edit version 1.0.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","ai","nlp","cohere"] compatible-with claude-code
Cohere Performance Tuning
Overview
Optimize Cohere API v2 performance through model selection, embedding batches, rerank pipelines, caching, and streaming for time-to-first-token.
Prerequisites
cohere-ai SDK installedUnderstanding of Cohere endpoints (Chat, Embed, Rerank)
Redis or in-memory cache (optional)
Latency Benchmarks (Typical)
Operation Model P50 P95 Chat (short) command-r7b-12-2024500ms 1.5s Chat (short) command-a-03-2025800ms 2.5s Chat (stream TTFT) command-a-03-2025200ms 600ms Embed (96 texts) embed-v4.0150ms 400ms Rerank (100 docs) rerank-v3.5100ms 300ms Classify (96 inputs) embed-english-v3.0200ms 500ms
Instructions
Strategy 1: Model Selection by Latency Budget
function selectModel (latencyBudgetMs : number string {
if (latencyBudgetMs < 1000 ) return 'command-r7b-12-2024' ;
if (latencyBudgetMs < 3000 ) return 'command-r-08-2024' ;
return 'command-a-03-2025' ;
}
await
chat
model
selectModel
1500
messages
role
'user'
content
maxTokens
200
Strategy 2: Streaming for Time-to-First-Token
async function streamForUI (message : string Promise <string > {
const stream = await cohere.chatStream ({
model : 'command-a-03-2025' ,
messages : [{ role : 'user' , content : message }],
});
let fullText = '' ;
for await (const event of stream) {
if (event.type === 'content-delta' ) {
const text = event.delta ?.message ?.content ?.text ?? '' ;
fullText += text;
}
}
return fullText;
}
Strategy 3: Batch Embeddings (96 per Call)
for (const text of texts) {
await cohere.embed ({ model : 'embed-v4.0' , texts : [text], ... });
}
async function batchEmbed (texts : string []Promise <number [][]> {
const BATCH = 96 ;
const results : number [][] = [];
const batches = [];
for (let i = 0 ; i < texts.length ; i += BATCH ) {
batches.push (texts.slice (i, i + BATCH ));
}
const responses = await Promise .all (
batches.map (batch =>embed ({
model : 'embed-v4.0' ,
texts : batch,
inputType : 'search_document' ,
embeddingTypes : ['float' ],
})
)
);
for (const resp of responses) {
results.push (...resp.embeddings .float );
}
return results;
}
Strategy 4: Compressed Embeddings
const response = await cohere.embed ({
model : 'embed-v4.0' ,
texts : documents,
inputType : 'search_document' ,
embeddingTypes : ['int8' ],
});
const storageVectors = response.embeddings .int8 ;
Strategy 5: Rerank as a Pre-filter
async function efficientSearch (query : string , corpus : string []const reranked = await cohere.rerank ({
model : 'rerank-v3.5' ,
query,
documents : corpus,
topN : 5 ,
});
const topDocs = reranked.results .map (r =>text : corpus[r.index ],
score : r.relevanceScore ,
}));
return topDocs;
}
Strategy 6: Embedding Cache import { LRUCache } from 'lru-cache' ;
import crypto from 'crypto' ;
const embedCache = new LRUCache <string , number []>({
max : 10_000 ,
ttl : 24 * 60 * 60 * 1000 ,
});
function hashText (text : string string {
return crypto.createHash ('sha256' ).update (text).digest ('hex' ).slice (0 , 16 );
}
async function cachedEmbed (texts : string []Promise <number [][]> {
const results : number [][] = new Array (texts.length );
const uncached : { index : number ; text : string }[] = [];
for (let i = 0 ; i < texts.length ; i++) {
const key = hashText (texts[i]);
const cached = embedCache.get (key);
if (cached) {
results[i] = cached;
} else {
uncached.push ({ index : i, text : texts[i] });
}
}
if (uncached.length > 0 ) {
const vectors = await batchEmbed (uncached.map (u =>text ));
for (let j = 0 ; j < uncached.length ; j++) {
results[uncached[j].index ] = vectors[j];
embedCache.set (hashText (uncached[j].text ), vectors[j]);
}
}
return results;
}
Strategy 7: Response Caching for Chat import { LRUCache } from 'lru-cache' ;
const chatCache = new LRUCache <string , string >({
max : 1000 ,
ttl : 5 * 60 * 1000 ,
});
async function cachedChat (message : string , system ?: string Promise <string > {
const key = `${system ?? '' } :${message} ` ;
const cached = chatCache.get (key);
if (cached) return cached;
const response = await cohere.chat ({
model : 'command-a-03-2025' ,
messages : [
...(system ? [{ role : 'system' as const , content : system }] : []),
{ role : 'user' as const , content : message },
],
temperature : 0 ,
});
const text = response.message ?.content ?.[0 ]?.text ?? '' ;
chatCache.set (key, text);
return text;
}
Performance Monitoring async function timedCohereCall<T>(
endpoint : string ,
fn : () => Promise <T>
): Promise <T> {
const start = performance.now ();
try {
const result = await fn ();
const ms = performance.now () - start;
console .log (`[cohere] ${endpoint} : ${ms.toFixed(0 )} ms` );
return result;
} catch (err) {
const ms = performance.now () - start;
console .error (`[cohere] ${endpoint} FAILED: ${ms.toFixed(0 )} ms` , err);
throw err;
}
}
Output
Model selection by latency budget
Streaming for sub-200ms TTFT
Batch embedding (96x fewer API calls)
Compressed embeddings (75-97% storage savings)
Cache layer for deterministic queries
Rerank as fast pre-filter
Error Handling Issue Cause Solution Chat > 5s Long output + slow model Use streaming, reduce maxTokens Embed timeout Too many texts Batch to 96 per call Cache stale Long TTL Reduce TTL for volatile data High costs No caching Cache embeddings (deterministic)
Resources
Next Steps For cost optimization, see cohere-cost-tuning.
