Skip to main content ホーム クリエイター comeonoliver skillshub cohere-performance-tuning
cohere-performance-tuning Optimize Cohere API performance with caching, batching, model selection, and streaming.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Cohere Chat, Embed, and Rerank.
Trigger with phrases like "cohere performance", "optimize cohere",
"cohere latency", "cohere caching", "cohere slow", "cohere batch".
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/ComeOnOliver/skillshub --skill cohere-performance-tuningコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills Review product and feature risk before an AI coding agent starts implementation.
Use Xquik for X data and confirmation-gated X actions: tweet search, user lookup, follower export, media download, monitors, webhooks, MCP, and SDK workflows.
Canton Network open-source ecosystem guide covering DAML SDK, Canton runtime, and Splice applications. Use when working with Canton Network, DAML smart contracts, or building decentralized applications.
name cohere-performance-tuning description Optimize Cohere API performance with caching, batching, model selection, and streaming.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Cohere Chat, Embed, and Rerank.
Trigger with phrases like "cohere performance", "optimize cohere",
"cohere latency", "cohere caching", "cohere slow", "cohere batch".
allowed-tools Read, Write, Edit version 1.0.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","ai","nlp","cohere"] compatible-with claude-code
Cohere Performance Tuning
Overview
Optimize Cohere API v2 performance through model selection, embedding batches, rerank pipelines, caching, and streaming for time-to-first-token.
Prerequisites
cohere-ai SDK installedUnderstanding of Cohere endpoints (Chat, Embed, Rerank)
Redis or in-memory cache (optional)
Latency Benchmarks (Typical)
Operation Model P50 P95 Chat (short) command-r7b-12-2024500ms 1.5s Chat (short) command-a-03-2025800ms 2.5s Chat (stream TTFT) command-a-03-2025200ms 600ms Embed (96 texts) embed-v4.0150ms 400ms Rerank (100 docs) rerank-v3.5100ms 300ms Classify (96 inputs) embed-english-v3.0200ms 500ms
Instructions
Strategy 1: Model Selection by Latency Budget
function selectModel (latencyBudgetMs : number string {
if (latencyBudgetMs < 1000 ) return 'command-r7b-12-2024' ;
if (latencyBudgetMs < 3000 ) return 'command-r-08-2024' ;
return 'command-a-03-2025' ;
}
await
chat
model
selectModel
1500
messages
role
'user'
content
maxTokens
200
Strategy 2: Streaming for Time-to-First-Token
async function streamForUI (message : string Promise <string > {
const stream = await cohere.chatStream ({
model : 'command-a-03-2025' ,
messages : [{ role : 'user' , content : message }],
});
let fullText = '' ;
for await (const event of stream) {
if (event.type === 'content-delta' ) {
const text = event.delta ?.message ?.content ?.text ?? '' ;
fullText += text;
}
}
return fullText;
}
Strategy 3: Batch Embeddings (96 per Call)
for (const text of texts) {
await cohere.embed ({ model : 'embed-v4.0' , texts : [text], ... });
}
async function batchEmbed (texts : string []Promise <number [][]> {
const BATCH = 96 ;
const results : number [][] = [];
const batches = [];
for (let i = 0 ; i < texts.length ; i += BATCH ) {
batches.push (texts.slice (i, i + BATCH ));
}
const responses = await Promise .all (
batches.map (batch =>embed ({
model : 'embed-v4.0' ,
texts : batch,
inputType : 'search_document' ,
embeddingTypes : ['float' ],
})
)
);
for (const resp of responses) {
results.push (...resp.embeddings .float );
}
return results;
}
Strategy 4: Compressed Embeddings
const response = await cohere.embed ({
model : 'embed-v4.0' ,
texts : documents,
inputType : 'search_document' ,
embeddingTypes : ['int8' ],
});
const storageVectors = response.embeddings .int8 ;
Strategy 5: Rerank as a Pre-filter
async function efficientSearch (query : string , corpus : string []const reranked = await cohere.rerank ({
model : 'rerank-v3.5' ,
query,
documents : corpus,
topN : 5 ,
});
const topDocs = reranked.results .map (r =>text : corpus[r.index ],
score : r.relevanceScore ,
}));
return topDocs;
}
Strategy 6: Embedding Cache import { LRUCache } from 'lru-cache' ;
import crypto from 'crypto' ;
const embedCache = new LRUCache <string , number []>({
max : 10_000 ,
ttl : 24 * 60 * 60 * 1000 ,
});
function hashText (text : string string {
return crypto.createHash ('sha256' ).update (text).digest ('hex' ).slice (0 , 16 );
}
async function cachedEmbed (texts : string []Promise <number [][]> {
const results : number [][] = new Array (texts.length );
const uncached : { index : number ; text : string }[] = [];
for (let i = 0 ; i < texts.length ; i++) {
const key = hashText (texts[i]);
const cached = embedCache.get (key);
if (cached) {
results[i] = cached;
} else {
uncached.push ({ index : i, text : texts[i] });
}
}
if (uncached.length > 0 ) {
const vectors = await batchEmbed (uncached.map (u =>text ));
for (let j = 0 ; j < uncached.length ; j++) {
results[uncached[j].index ] = vectors[j];
embedCache.set (hashText (uncached[j].text ), vectors[j]);
}
}
return results;
}
Strategy 7: Response Caching for Chat import { LRUCache } from 'lru-cache' ;
const chatCache = new LRUCache <string , string >({
max : 1000 ,
ttl : 5 * 60 * 1000 ,
});
async function cachedChat (message : string , system ?: string Promise <string > {
const key = `${system ?? '' } :${message} ` ;
const cached = chatCache.get (key);
if (cached) return cached;
const response = await cohere.chat ({
model : 'command-a-03-2025' ,
messages : [
...(system ? [{ role : 'system' as const , content : system }] : []),
{ role : 'user' as const , content : message },
],
temperature : 0 ,
});
const text = response.message ?.content ?.[0 ]?.text ?? '' ;
chatCache.set (key, text);
return text;
}
Performance Monitoring async function timedCohereCall<T>(
endpoint : string ,
fn : () => Promise <T>
): Promise <T> {
const start = performance.now ();
try {
const result = await fn ();
const ms = performance.now () - start;
console .log (`[cohere] ${endpoint} : ${ms.toFixed(0 )} ms` );
return result;
} catch (err) {
const ms = performance.now () - start;
console .error (`[cohere] ${endpoint} FAILED: ${ms.toFixed(0 )} ms` , err);
throw err;
}
}
Output
Model selection by latency budget
Streaming for sub-200ms TTFT
Batch embedding (96x fewer API calls)
Compressed embeddings (75-97% storage savings)
Cache layer for deterministic queries
Rerank as fast pre-filter
Error Handling Issue Cause Solution Chat > 5s Long output + slow model Use streaming, reduce maxTokens Embed timeout Too many texts Batch to 96 per call Cache stale Long TTL Reduce TTL for volatile data High costs No caching Cache embeddings (deterministic)
Resources
Next Steps For cost optimization, see cohere-cost-tuning.
