メニュー
Add voice message transcription to Gandalf using OpenAI's Whisper API. Automatically transcribes WhatsApp voice notes so the agent can read and respond to them.
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/sasha-computer/2026 --skill add-voice-transcriptionコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Add Gmail integration to Gandalf. Can be configured as a tool (agent reads/sends emails when requested from WhatsApp) or as a full channel (emails can trigger the agent, schedule tasks, and receive replies). Guides through GCP OAuth setup and implements the integration.
Add new capabilities or modify Gandalf behavior. Use when user wants to add channels (Telegram, Slack, email input), add integrations, modify the router, or make any other customizations. This is an interactive skill that asks questions to understand what the user wants.
Debug agent execution issues. Use when things aren't working, authentication problems, or to understand the subprocess system. Covers logs, environment variables, and common issues.
SKILL.md を表示中
SOC 職業分類に基づく
| name | add-voice-transcription |
| description | Add voice message transcription to Gandalf using OpenAI's Whisper API. Automatically transcribes WhatsApp voice notes so the agent can read and respond to them. |
This skill adds automatic voice message transcription using OpenAI's Whisper API. When users send voice notes in WhatsApp, they'll be transcribed and the agent can read and respond to the content.
UX Note: When asking the user questions, prefer using the AskUserQuestion tool instead of just outputting text. This integrates with Claude's built-in question/answer system for a better experience.
USER ACTION REQUIRED
Use the AskUserQuestion tool to present this:
You'll need an OpenAI API key for Whisper transcription.
Get one at: https://platform.openai.com/api-keys
Cost:
$0.006 per minute of audio ($0.003 per typical 30-second voice note)Once you have your API key, we'll configure it securely.
Wait for user to confirm they have an API key before continuing.
Read package.json and add the openai package to dependencies:
"dependencies": {
...existing dependencies...
"openai": "^4.77.0"
}
Then install it:
bun install
Create a configuration file for transcription settings (without the API key):
Write to .transcription.config.json:
{
"provider": "openai",
"openai": {
"apiKey": "",
"model": "whisper-1"
},
"enabled": true,
"fallbackMessage": "[Voice Message - transcription unavailable]"
}
Add this file to .gitignore to prevent committing API keys:
echo ".transcription.config.json" >> .gitignore
Use the AskUserQuestion tool to confirm:
I've created
.transcription.config.jsonin the project root. You'll need to add your OpenAI API key to it manually:
- Open
.transcription.config.json- Replace the empty
"apiKey": ""with your key:"apiKey": "sk-proj-..."- Save the file
Let me know when you've added it.
Wait for user confirmation.
Create src/transcription.ts:
import { downloadMediaMessage } from '@whiskeysockets/baileys';
import { WAMessage, WASocket } from '@whiskeysockets/baileys';
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import { dirname } from 'path';
// Get __dirname equivalent in ES modules
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// Configuration interface
interface TranscriptionConfig {
provider: string;
openai?: {
apiKey: string;
model: string;
};
enabled: boolean;
fallbackMessage: string;
}
// Load configuration
function loadConfig(): TranscriptionConfig {
const configPath = path.join(__dirname, '../.transcription.config.json');
try {
const configData = fs.readFileSync(configPath, 'utf-8');
return JSON.parse(configData);
} catch (err) {
console.error('Failed to load transcription config:', err);
return {
provider: 'openai',
enabled: false,
fallbackMessage: '[Voice Message - transcription unavailable]'
};
}
}
// Transcribe audio using OpenAI Whisper API
async function transcribeWithOpenAI(audioBuffer: Buffer, config: TranscriptionConfig): Promise<string | null> {
if (!config.openai?.apiKey || config.openai.apiKey === '') {
console.warn('OpenAI API key not configured');
return null;
}
try {
// Dynamic import of openai
const openaiModule = await import('openai');
const OpenAI = openaiModule.default;
const toFile = openaiModule.toFile;
const openai = new OpenAI({
apiKey: config.openai.apiKey
});
// Use OpenAI's toFile helper to create a proper file upload
const file = await toFile(audioBuffer, 'voice.ogg', {
type: 'audio/ogg'
});
// Call Whisper API
const transcription = await openai.audio.transcriptions.create({
file: file,
model: config.openai.model || 'whisper-1',
response_format: 'text'
});
// Type assertion needed: OpenAI SDK types response_format='text' as Transcription object,
// but it actually returns a plain string when response_format is 'text'
return transcription as unknown as string;
} catch (err) {
console.error('OpenAI transcription failed:', err);
return null;
}
}
// Main transcription function
export async function transcribeAudioMessage(
msg: WAMessage,
sock: WASocket
): Promise<string | null> {
const config = loadConfig();
// Check if transcription is enabled
if (!config.enabled) {
console.log('Transcription disabled in config');
return config.fallbackMessage;
}
try {
// Download the audio message
const buffer = await downloadMediaMessage(
msg,
'buffer',
{},
{
logger: console as any,
reuploadRequest: sock.updateMediaMessage
}
) as Buffer;
if (!buffer || buffer.length === 0) {
console.error('Failed to download audio message');
return config.fallbackMessage;
}
console.log(`Downloaded audio message: ${buffer.length} bytes`);
// Transcribe based on provider
let transcript: string | null = null;
switch (config.provider) {
case 'openai':
transcript = await transcribeWithOpenAI(buffer, config);
break;
default:
console.error(`Unknown transcription provider: ${config.provider}`);
return config.fallbackMessage;
}
if (!transcript) {
return config.fallbackMessage;
}
return transcript.trim();
} catch (err) {
console.error('Transcription error:', err);
return config.fallbackMessage;
}
}
// Helper to check if a message is a voice note
export function isVoiceMessage(msg: WAMessage): boolean {
return msg.message?.audioMessage?.ptt === true;
}
Read src/db.ts and find the storeMessage function. Update its signature and implementation to accept transcribed content:
Change the function signature from:
export function storeMessage(msg: proto.IWebMessageInfo, chatJid: string, isFromMe: boolean, pushName?: string): void
To:
export function storeMessage(msg: proto.IWebMessageInfo, chatJid: string, isFromMe: boolean, pushName?: string, transcribedContent?: string): void
Update the content extraction to use transcribed content if provided:
const content = transcribedContent ||
msg.message?.conversation ||
msg.message?.extendedTextMessage?.text ||
msg.message?.imageMessage?.caption ||
msg.message?.videoMessage?.caption ||
(msg.message?.audioMessage?.ptt ? '[Voice Message]' : '') ||
'';
Note: Voice messages are transcribed for all messages in registered groups. This is because:
Read src/index.ts and find the sock.ev.on('messages.upsert', ...) event handler.
Change the callback from synchronous to async:
sock.ev.on('messages.upsert', async ({ messages }) => {
Inside the loop where messages are stored, add voice message detection and transcription:
// Only store full message content for registered groups
if (registeredGroups[chatJid]) {
// Check if this is a voice message
if (msg.message.audioMessage?.ptt) {
try {
// Import transcription module
const { transcribeAudioMessage } = await import('./transcription.js');
const transcript = await transcribeAudioMessage(msg, sock);
if (transcript) {
// Store with transcribed content
storeMessage(msg, chatJid, msg.key.fromMe || false, msg.pushName || undefined, `[Voice: ${transcript}]`);
logger.info({ chatJid, length: transcript.length }, 'Transcribed voice message');
} else {
// Store with fallback message
storeMessage(msg, chatJid, msg.key.fromMe || false, msg.pushName || undefined, '[Voice Message - transcription unavailable]');
}
} catch (err) {
logger.error({ err }, 'Voice transcription error');
storeMessage(msg, chatJid, msg.key.fromMe || false, msg.pushName || undefined, '[Voice Message - transcription failed]');
}
} else {
// Regular message, store normally
storeMessage(msg, chatJid, msg.key.fromMe || false, msg.pushName || undefined);
}
}
Run these commands to ensure everything compiles:
bun install
bun run build
Restart the service to load the new transcription code:
# If using launchd (macOS):
launchctl kickstart -k gui/$(id -u)/com.gandalf
# Or if running manually:
# Stop the current process and restart with:
bun run start
Verify it started:
sleep 2 && launchctl list | grep gandalf
# or check logs:
tail -f logs/gandalf.log
Tell the user:
Voice transcription is ready! Test it by:
- Open WhatsApp on your phone
- Go to a registered group chat
- Send a voice note using the microphone button
- The agent should receive the transcribed text and respond
In the database and agent context, voice messages appear as:
[Voice: <transcribed text here>]
Watch for transcription in the logs:
tail -f logs/gandalf.log | grep -i "voice\|transcri"
To temporarily disable without removing code, edit .transcription.config.json:
{
"enabled": false
}
Customize what's stored when transcription fails:
{
"fallbackMessage": "[🎤 Voice note - transcription unavailable]"
}
The architecture supports multiple providers. To add Groq, Deepgram, or local Whisper:
.transcription.config.jsonsrc/transcription.ts (similar to transcribeWithOpenAI)Check logs for specific errors:
tail -100 logs/gandalf.log | grep -i transcription
Common causes:
audioMessage.ptt is true in the message object__dirname is not defined)The fix is already included in the implementation above using:
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
.transcription.config.json file contains your API key and should NOT be committed to version control.gitignore by this skillMonitor usage in your OpenAI dashboard: https://platform.openai.com/usage
Tips to control costs:
"enabled": falseTo remove the feature:
Remove from package.json:
bun remove openai
Delete src/transcription.ts
Revert changes in src/index.ts:
Revert changes in src/db.ts:
transcribedContent parameter from storeMessageDelete .transcription.config.json
Rebuild:
bun run build
launchctl kickstart -k gui/$(id -u)/com.gandalf
Potential additions:
whisper.cpp or faster-whisper for offline transcription