Skip to main content
cohere-performance-tuning Optimize Cohere API performance with caching, batching, model selection, and streaming.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Cohere Chat, Embed, and Rerank.
Trigger with phrases like "cohere performance", "optimize cohere",
"cohere latency", "cohere caching", "cohere slow", "cohere batch".
跳到安装 Skills Marketplace 发现并探索由社区构建的 Agent Skills
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/jeremylongshore/claude-code-plugins-plus-skills --skill cohere-performance-tuning命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
下载 Zip 下载中... 同仓库更多 Skills Implement user sign-up and sign-in flows with Clerk.
Use when building authentication UI, customizing sign-in experience,
or implementing OAuth social login.
Trigger with phrases like "clerk sign-in", "clerk sign-up",
"clerk login flow", "clerk OAuth", "clerk social login".
Implement session management and middleware with Clerk.
Use when managing user sessions, configuring route protection,
or implementing token refresh and custom JWT templates.
Trigger with phrases like "clerk session", "clerk middleware",
"clerk route protection", "clerk token", "clerk JWT".
Configure enterprise SSO, role-based access control, and organization management.
Use when implementing SSO integration, configuring role-based permissions,
or setting up organization-level controls.
Trigger with phrases like "clerk SSO", "clerk RBAC",
"clerk enterprise", "clerk roles", "clerk permissions", "clerk organizations".
jeremylongshore
jeremylongshore/claude-code-plugins-plus-skills
打开 GitHub 仓库 name cohere-performance-tuning description Optimize Cohere API performance with caching, batching, model selection, and streaming.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Cohere Chat, Embed, and Rerank.
Trigger with phrases like "cohere performance", "optimize cohere",
"cohere latency", "cohere caching", "cohere slow", "cohere batch".
allowed-tools Read, Write, Edit version 1.5.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","ai","nlp","cohere"] compatibility Designed for Claude Code
Cohere Performance Tuning
Overview
Optimize Cohere API v2 performance through model selection, embedding batches, rerank pipelines, caching, and streaming for time-to-first-token.
Prerequisites
cohere-ai SDK installedUnderstanding of Cohere endpoints (Chat, Embed, Rerank)
Redis or in-memory cache (optional)
Latency Benchmarks (Typical)
Operation Model P50 P95 Chat (short) command-r7b-12-2024500ms 1.5s Chat (short) command-a-03-2025800ms 2.5s Chat (stream TTFT) command-a-03-2025200ms 600ms Embed (96 texts) embed-v4.0150ms 400ms Rerank (100 docs) rerank-v3.5100ms 300ms Classify (96 inputs) embed-english-v3.0200ms 500ms
Instructions
Strategy 1: Model Selection by Latency Budget
function selectModel (latencyBudgetMs : number string {
if (latencyBudgetMs < 1000 ) return 'command-r7b-12-2024' ;
if (latencyBudgetMs < 3000 ) return 'command-r-08-2024' ;
return 'command-a-03-2025' ;
}
await
chat
model
selectModel
1500
messages
role
'user'
content
maxTokens
200
Strategy 2: Streaming for Time-to-First-Token
async function streamForUI (message : string Promise <string > {
const stream = await cohere.chatStream ({
model : 'command-a-03-2025' ,
messages : [{ role : 'user' , content : message }],
});
let fullText = '' ;
for await (const event of stream) {
if (event.type === 'content-delta' ) {
const text = event.delta ?.message ?.content ?.text ?? '' ;
fullText += text;
}
}
return fullText;
}
Strategy 3: Batch Embeddings (96 per Call)
for (const text of texts) {
await cohere.embed ({ model : 'embed-v4.0' , texts : [text], ... });
}
async function batchEmbed (texts : string []Promise <number [][]> {
const BATCH = 96 ;
const results : number [][] = [];
const batches = [];
for (let i = 0 ; i < texts.length ; i += BATCH ) {
batches.push (texts.slice (i, i + BATCH ));
}
const responses = await Promise .all (
batches.map (batch =>embed ({
model : 'embed-v4.0' ,
texts : batch,
inputType : 'search_document' ,
embeddingTypes : ['float' ],
})
)
);
for (const resp of responses) {
results.push (...resp.embeddings .float );
}
return results;
}
Strategy 4: Compressed Embeddings
const response = await cohere.embed ({
model : 'embed-v4.0' ,
texts : documents,
inputType : 'search_document' ,
embeddingTypes : ['int8' ],
});
const storageVectors = response.embeddings .int8 ;
Strategy 5: Rerank as a Pre-filter
async function efficientSearch (query : string , corpus : string []const reranked = await cohere.rerank ({
model : 'rerank-v3.5' ,
query,
documents : corpus,
topN : 5 ,
});
const topDocs = reranked.results .map (r =>text : corpus[r.index ],
score : r.relevanceScore ,
}));
return topDocs;
}
Strategy 6: Embedding Cache import { LRUCache } from 'lru-cache' ;
import crypto from 'crypto' ;
const embedCache = new LRUCache <string , number []>({
max : 10_000 ,
ttl : 24 * 60 * 60 * 1000 ,
});
function hashText (text : string string {
return crypto.createHash ('sha256' ).update (text).digest ('hex' ).slice (0 , 16 );
}
async function cachedEmbed (texts : string []Promise <number [][]> {
const results : number [][] = new Array (texts.length );
const uncached : { index : number ; text : string }[] = [];
for (let i = 0 ; i < texts.length ; i++) {
const key = hashText (texts[i]);
const cached = embedCache.get (key);
if (cached) {
results[i] = cached;
} else {
uncached.push ({ index : i, text : texts[i] });
}
}
if (uncached.length > 0 ) {
const vectors = await batchEmbed (uncached.map (u =>text ));
for (let j = 0 ; j < uncached.length ; j++) {
results[uncached[j].index ] = vectors[j];
embedCache.set (hashText (uncached[j].text ), vectors[j]);
}
}
return results;
}
Strategy 7: Response Caching for Chat import { LRUCache } from 'lru-cache' ;
const chatCache = new LRUCache <string , string >({
max : 1000 ,
ttl : 5 * 60 * 1000 ,
});
async function cachedChat (message : string , system ?: string Promise <string > {
const key = `${system ?? '' } :${message} ` ;
const cached = chatCache.get (key);
if (cached) return cached;
const response = await cohere.chat ({
model : 'command-a-03-2025' ,
messages : [
...(system ? [{ role : 'system' as const , content : system }] : []),
{ role : 'user' as const , content : message },
],
temperature : 0 ,
});
const text = response.message ?.content ?.[0 ]?.text ?? '' ;
chatCache.set (key, text);
return text;
}
Performance Monitoring async function timedCohereCall<T>(
endpoint : string ,
fn : () => Promise <T>
): Promise <T> {
const start = performance.now ();
try {
const result = await fn ();
const ms = performance.now () - start;
console .log (`[cohere] ${endpoint} : ${ms.toFixed(0 )} ms` );
return result;
} catch (err) {
const ms = performance.now () - start;
console .error (`[cohere] ${endpoint} FAILED: ${ms.toFixed(0 )} ms` , err);
throw err;
}
}
Output
Model selection by latency budget
Streaming for sub-200ms TTFT
Batch embedding (96x fewer API calls)
Compressed embeddings (75-97% storage savings)
Cache layer for deterministic queries
Rerank as fast pre-filter
Error Handling Issue Cause Solution Chat > 5s Long output + slow model Use streaming, reduce maxTokens Embed timeout Too many texts Batch to 96 per call Cache stale Long TTL Reduce TTL for volatile data High costs No caching Cache embeddings (deterministic)
Resources
Next Steps For cost optimization, see cohere-cost-tuning.
