Skip to main content
inbox-processing-expert Expert guidance for building and maintaining the Para Obsidian inbox processing system - a security-hardened automation framework for processing PDFs and attachments with AI-powered metadata extraction. Use when building inbox processors, implementing security patterns (TOCTOU, command injection prevention, atomic writes), designing interactive CLIs with suggestion workflows, integrating LLM detection, implementing idempotency with SHA256 registries, or working with the para-obsidian inbox codebase. Covers engine/interface separation, suggestion-based architecture, confidence scoring, error taxonomy, structured logging, and testing patterns. Useful when user mentions inbox automation, PDF processing, document classification, security-hardened file processing, or interactive CLI design.
الانتقال إلى التثبيت سوق المهارات اكتشف واستكشف مهارات الذكاء الاصطناعي التي بناها المجتمع.
المهن ذات الصلة SOC
استنادا إلى تصنيف SOC المهني
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
عرض تفاصيل Prompt يتجاوز الأمر المباشر Prompt المخصّص للمراجعة. افحص المصدر قبل تشغيله.
npx skills add https://github.com/nathanvale/side-quest-marketplace-old --skill inbox-processing-expertيبقى الأمر في سطر واحد. مرّر أفقيًا لمراجعته كاملًا قبل النسخ.
تفضّل نسخة محلية؟ نزّل الملفات المتاحة حاليًا لدى SkillsMP.
تحميل Zip جاري التحميل... المزيد من هذا المستودع Web scraping, site crawling, search, structured data extraction, and AI-powered research with Firecrawl CLI. Use when you need full page content as markdown, JS-rendered pages, anti-bot bypass, crawling entire documentation sites, extracting structured data with schemas, or deep web research. Prefer WebFetch for quick questions about a known URL. Prefer WebSearch for finding links without full content.
Build production-grade MCP (Model Context Protocol) servers with observability, correlation ID tracing, and dual logging. Use when creating new MCP servers, adding tools to existing servers, implementing file logging, debugging MCP issues, wrapping CLI tools with spawnSyncCollect, or following Side Quest marketplace patterns. Covers @side-quest/core/mcp declarative API, @side-quest/core/spawn CLI wrapper patterns, Zod schemas, Bun runtime, and 9 gold standard patterns validated across Kit plugin (18 tools). Includes error handling, response format switching, MCP annotations, and graceful degradation.
Unified inbox processor - handles ALL content types (clippings, transcriptions, VTT files, attachments) with parallel subagents and single-table review. Routes to appropriate creator based on proposed_template.
name inbox-processing-expert description Expert guidance for building and maintaining the Para Obsidian inbox processing system - a security-hardened automation framework for processing PDFs and attachments with AI-powered metadata extraction. Use when building inbox processors, implementing security patterns (TOCTOU, command injection prevention, atomic writes), designing interactive CLIs with suggestion workflows, integrating LLM detection, implementing idempotency with SHA256 registries, or working with the para-obsidian inbox codebase. Covers engine/interface separation, suggestion-based architecture, confidence scoring, error taxonomy, structured logging, and testing patterns. Useful when user mentions inbox automation, PDF processing, document classification, security-hardened file processing, or interactive CLI design. allowed-tools Read, Grep, Glob
Inbox Processing Expert
Build security-hardened inbox automation with AI-powered metadata extraction following the Para Obsidian inbox processing framework.
Quick Navigation
Architecture Overview
Engine/Interface Separation Core principle: Engine logic is UI-agnostic. Same core powers CLI, web app, or API.
const engine = createInboxEngine ({ vaultPath : "/path/to/vault" });
const suggestions = await engine.scan ();
const updated = await engine.editWithPrompt ("abc123" , "put in Health area instead" );
const results = await engine.execute (["abc123" , "def456" ]);
const report = engine.generateReport (suggestions);
UI can be replaced without touching core logic
Easy to test (engine is pure logic, no console.log or process.exit)
Multiple interfaces (CLI, web, CI/CD) share same engine
Suggestion-Based Architecture Never mutate state directly. All operations return suggestions that require human approval.
interface InboxSuggestion {
id : string ;
source : string ;
processor : "attachments" | "notes" | "images" ;
confidence : "high" | "medium" | "low" ;
action : "create-note" | "move" | "rename" | "link" | "skip" ;
suggestedNoteType ?: string ;
suggestedTitle ?: string ;
suggestedDestination ?: string ;
suggestedArea ?: string ;
suggestedProject ?: string ;
extractedFields ?: Record <string , unknown >;
suggestedAttachmentName ?: string ;
attachmentLink ?: string ;
reason : string ;
}
Key insight: Suggestions are immutable. editWithPrompt() returns a NEW suggestion.
Security Patterns
P0 Critical Protections
1. Command Injection Prevention Always use array args, never string interpolation.
await $`pdftotext ${filePath} -` ;
const proc = Bun .spawn (["pdftotext" , filePath, "-" ]);
2. TOCTOU (Time-of-Check-Time-of-Use) Mitigation Check file before AND after operations to detect tampering.
import { stat } from "@side-quest/core/fs" ;
const preStats = await stat (filePath);
const text = await extractPdfText (filePath, cid);
const postStats = await stat (filePath);
if (postStats.mtimeMs !== preStats.mtimeMs ) {
throw createInboxError ("EXT_PDF_TOCTOU" , { cid, source : filePath });
}
Use case: Prevent file swapping during multi-step operations.
3. Atomic Registry Writes Write to temp file, then atomically rename.
import { rename } from "@side-quest/core/fs" ;
await Bun .write (tempPath, JSON .stringify (registry));
await rename (tempPath, registryPath);
Why: Prevents corrupt registry if process crashes mid-write.
4. File Locking Acquire lock before concurrent operations.
await acquireLock (lockPath);
try {
} finally {
releaseLock (lockPath);
}
Use case: Multiple processes accessing same registry.
5. Process Lifecycle Management Kill child processes on timeout to prevent zombies.
const timeout = setTimeout (() => {
proc.kill ();
reject (new Error ("Timeout" ));
}, 30000 );
try {
await proc.exited ;
clearTimeout (timeout);
} catch (error) {
clearTimeout (timeout);
throw error;
}
6. Prompt Injection Sanitization Strip control characters from user input.
function sanitizePrompt (input : string string {
return input.replace (/[\x00-\x1F\x7F]/g , "" );
}
const userPrompt = sanitizePrompt (rawInput);
7. Rollback on Failure Delete orphaned resources if operation fails.
try {
await createNote (notePath, content);
await moveFile (source, dest);
} catch (error) {
if (pathExistsSync (notePath)) {
unlinkSync (notePath);
}
throw error;
}
Core Concepts
Confidence Scoring Level Criteria HIGH Heuristics AND AI agree + target location exists + template available MEDIUM AI detects type but filename/content ambiguous LOW AI uncertain, content unclear, extraction failed
let confidence : "high" | "medium" | "low" = llmResult.confidence > 0.8 ? "high" : "medium" ;
if (filenameHint !== llmType) {
confidence = confidence === "high" ? "medium" : "low" ;
}
if (!pathExistsSync (targetFolder)) {
confidence = "low" ;
}
if (!templateExists (suggestedNoteType)) {
confidence = confidence === "high" ? "medium" : "low" ;
}
Idempotency with SHA256 Registry Use content hashing to prevent duplicate processing.
import { hashFile, createRegistry } from "./registry" ;
const registry = createRegistry (vaultPath);
await registry.load ();
const hash = await hashFile (filePath);
if (registry.isProcessed (hash)) {
console .log ("Already processed - skipping" );
return ;
}
registry.markProcessed ({
sourceHash : hash,
sourcePath : filePath,
processedAt : new Date ().toISOString (),
createdNote : notePath,
});
await registry.save ();
Filename changes don't break idempotency
Safe to re-run on same files
Registry tracks what was created from each source
Converters Architecture Extensible document type detection via converter configuration.
The converters module provides a pluggable architecture for detecting document types:
import type { InboxConverter } from "./converters/types" ;
const invoiceConverter : InboxConverter = {
id : "invoice" ,
displayName : "Invoice" ,
enabled : true ,
priority : 90 ,
heuristics : {
filenamePatterns : [
{ pattern : "invoice|rechnung|factura" , weight : 0.9 },
{ pattern : "receipt|bill" , weight : 0.7 },
],
contentMarkers : [
{ pattern : "total|amount due|subtotal" , weight : 0.8 },
{ pattern : "invoice number|inv[.#]" , weight : 0.9 },
],
threshold : 0.3 ,
},
fields : [
{ name : "provider" , type : "string" , description : "Company name" , required : true },
{ name : "amount" , type : "currency" , description : "Total amount" , required : true },
{ name : "date" , type : "date" , description : "Invoice date" , required : true },
{ name : "invoiceNumber" , type : "string" , description : "Invoice #" , required : false },
],
extraction : {
promptHint : "Extract invoice details including provider, amount, and date." ,
keyFields : ["provider" , "amount" ],
},
template : {
name : "Invoice" ,
fieldMappings : {
provider : "Provider" ,
amount : "Amount" ,
date : "Date" ,
invoiceNumber : "Invoice Number" ,
},
},
scoring : {
heuristicWeight : 0.3 ,
llmWeight : 0.7 ,
highThreshold : 0.85 ,
mediumThreshold : 0.6 ,
},
};
Heuristics first: Quick filename/content pattern matching (0ms)LLM second: AI-powered extraction only for matched files (~1-3s)Field-driven: Each converter defines extraction fields and template mappingsPriority-based: Higher priority converters are checked firstExtensible: Add new document types by creating converters
Performance Characteristics Operation Typical Time Notes Scan (10 PDFs) ~15-30s Depends on LLM latency (3 concurrent) PDF extraction ~500-2000ms Per file, depends on size LLM detection ~1-3s Per file (haiku model) Execute (10 items) ~2-5s File I/O bound (10 concurrent) Registry load ~10-50ms Depends on size (1000 items = ~50ms)
Concurrency Limits import pLimit from "p-limit" ;
const pdfLimit = pLimit (5 );
const llmLimit = pLimit (3 );
const ioLimit = pLimit (10 );
Prevent API rate limit errors
Avoid OOM from too many parallel operations
Balance throughput vs. resource usage
Interactive CLI
Command Loop Pattern Display → Parse → Execute → Update display
while (true ) {
console .log (formatSuggestionsTable (suggestions));
console .log ("\nCommands:" );
console .log (" a - Approve all HIGH confidence" );
console .log (" e<N> - Edit suggestion with prompt" );
console .log (" <N>,<M> - Execute specific suggestions" );
console .log (" q - Quit" );
const cmd = await getUserInput ();
if (cmd === 'a' ) {
const highIds = suggestions
.filter (s =>confidence === "high" )
.map (s =>id );
const results = await engine.execute (highIds);
suggestions = suggestions.filter (s =>includes (s.id ));
}
else if (cmd.match (/^e(\d+)/ )) {
const index = parseInt (cmd.slice (1 ));
const suggestion = suggestions[index];
const prompt = await getUserInput ("Custom instructions: " );
const updated = await engine.editWithPrompt (suggestion.id , sanitizePrompt (prompt));
suggestions[index] = updated;
}
else if (cmd === 'q' ) {
break ;
}
}
Stable ID-based lookups (not array indices)
Sanitize all user input
Update display after each operation
Clear command structure
Related: See Bun CLI skill for argument parsing and output formatting patterns.
Formatted Output Use tables for suggestion display:
function formatSuggestionsTable (suggestions : InboxSuggestion []string {
const rows = suggestions.map ((s, i ) => [
i.toString (),
s.confidence ,
s.action ,
s.suggestedTitle || s.source ,
s.reason .slice (0 , 50 ) + "..." ,
]);
return table ([
["#" , "Confidence" , "Action" , "Title" , "Reason" ],
...rows,
]);
}
Scannable at a glance
Clear column alignment
Truncated text for readability
Error Handling
Error Taxonomy (23 Codes) Category Example Codes Recoverable? dependency DEP_PDFTOTEXT_MISSING, DEP_LLM_UNAVAILABLENo extraction EXT_PDF_CORRUPT, EXT_PDF_EMPTY, EXT_PDF_TOO_LARGENo detection DET_TYPE_UNKNOWN, DET_FIELDS_INCOMPLETENo validation VAL_AREA_NOT_FOUND, VAL_TEMPLATE_MISSINGNo execution EXE_NOTE_CREATE_FAILED, EXE_ATTACHMENT_MOVE_FAILEDNo registry REG_READ_FAILED, REG_WRITE_FAILED, REG_CORRUPTYes user USR_INVALID_COMMAND, USR_EDIT_PROMPT_EMPTYYes
Error Factory Pattern interface InboxError extends Error {
code : string ;
category : string ;
recoverable : boolean ;
context : Record <string , unknown >;
}
function createInboxError (
code : string ,
context : Record <string , unknown >,
): InboxError {
const error = new Error (ERROR_MESSAGES [code]) as InboxError ;
error.code = code;
error.category = code.split ("_" )[0 ].toLowerCase ();
error.recoverable = RECOVERABLE_ERRORS .includes (code);
error.context = context;
return error;
}
if (!pathExistsSync (pdfPath)) {
throw createInboxError ("EXT_PDF_NOT_FOUND" , {
cid,
source : pdfPath
});
}
Structured error handling
Correlation IDs for debugging
User-facing messages separate from codes
Recoverable vs. fatal distinction
Logging & Observability
Structured Logging Every log includes correlation ID.
import { inboxLogger, pdfLogger, llmLogger, executeLogger } from "./logger" ;
const cid = crypto.randomUUID ().slice (0 , 8 );
inboxLogger.info `Scan started items=${count} ${cid} ` ;
pdfLogger.debug `Extracting ${filePath} ${cid} ` ;
llmLogger.info `Detection complete type=${type } confidence=${conf} ${cid} ` ;
executeLogger.info `Note created path=${notePath} ${cid} ` ;
Log location: ~/.claude/logs/para-obsidian.jsonl
Key Metrics Metric Purpose scan.duration_msOverall scan performance pdf.extraction_duration_mspdftotext latency llm.call_duration_msLLM API latency llm.calls_per_scanCost tracking execute.success_rateReliability
grep "abc12345" ~/.claude/logs/para-obsidian.jsonl
jq 'select(.llm.call_duration_ms) | .llm.call_duration_ms' \
~/.claude/logs/para-obsidian.jsonl | \
awk '{sum+=$1; count++} END {print sum/count}'
Testing Strategy
Coverage (246 Tests)
Registry (28 tests) - Atomic writes, locking, validation, idempotencyPDF Processor - Extraction, heuristics, TOCTOU, timeout handlingEngine - Scan, execute, edit, rollback on failureCLI Adapter - Command parsing, display, prompt sanitizationErrors - All 23 error codes, recovery strategiesLogging - Correlation IDs, subsystem loggers
Testing Patterns
Security Testing test ("prevents command injection in PDF extraction" , async () => {
const maliciousPath = "/tmp/file.pdf; rm -rf /" ;
await expect (extractPdfText (maliciousPath, "cid" ))
.rejects .toThrow ("EXT_PDF_NOT_FOUND" );
});
TOCTOU Testing test ("detects file tampering during extraction" , async () => {
const filePath = await createTempFile ("test.pdf" );
const originalStat = stat;
vi.spyOn (fs, "stat" ).mockImplementation (async (path) => {
const result = await originalStat (path);
result.mtimeMs += 1000 ;
return result;
});
await expect (extractPdfText (filePath, "cid" ))
.rejects .toThrow ("EXT_PDF_TOCTOU" );
});
Idempotency Testing test ("doesn't reprocess same file twice" , async () => {
const engine = createInboxEngine ({ vaultPath });
const suggestions1 = await engine.scan ();
expect (suggestions1).toHaveLength (1 );
await engine.execute ([suggestions1[0 ].id ]);
const suggestions2 = await engine.scan ();
expect (suggestions2).toHaveLength (0 );
});
Common Questions
When should I use HIGH vs MEDIUM vs LOW confidence? HIGH confidence requires all of:
LLM detection confidence > 0.8
Filename heuristics match LLM type
Target destination folder exists
Required template is available
LLM is confident but heuristics disagree
Target location exists but some ambiguity
Most fields extracted successfully
LLM confidence < 0.5
Target location doesn't exist
Template missing
Extraction failed or incomplete
How do I handle files that fail processing? Use the error taxonomy to determine if recoverable:
try {
await processPDF (file);
} catch (error) {
if (error.recoverable ) {
logger.warn `Recoverable error: ${error.code} ${cid} ` ;
} else {
logger.error `Fatal error: ${error.code} ${cid} ` ;
throw error;
}
}
Should I process files in CI/CD or interactively?
Review suggestions before executing
Edit with custom prompts
Handle MEDIUM/LOW confidence items
Auto-execute HIGH confidence only
Queue MEDIUM/LOW for manual review
Generate report for human oversight
How do I debug slow LLM calls? Check logs for correlation ID:
grep "abc12345" ~/.claude/logs/para-obsidian.jsonl | grep "llm.call_duration_ms"
jq 'select(.llm.call_duration_ms) | .llm.call_duration_ms' \
~/.claude/logs/para-obsidian.jsonl | \
awk '{sum+=$1; count++} END {print sum/count " ms"}'
Use faster model (haiku vs sonnet)
Reduce concurrency limit (less rate limiting)
Cache common vault context (areas, projects)
How do I prevent duplicate processing after renaming files? The registry uses SHA256 content hashing, not filenames:
const hash = await sha256File ("invoice-new.pdf" );
if (registry.isProcessed (hash)) {
console .log ("Already processed (content match)" );
}
Filename changes don't affect idempotency.
What happens if a file changes during processing (TOCTOU)? Pre- and post-checks detect tampering:
const preStats = await stat (filePath);
const text = await extractPdfText (filePath);
const postStats = await stat (filePath);
if (postStats.mtimeMs !== preStats.mtimeMs ) {
throw createInboxError ("EXT_PDF_TOCTOU" , { cid, source : filePath });
}
If file modified during processing, operation fails safely.
Related Skills
Bun CLI Development
Argument parsing patterns (--flag value, --flag=value, --flag)
Dual output formatting (markdown + JSON)
Error handling with exit codes
Subcommand dispatch
Usage text structure
const { command, flags, positional } = parseArgs (process.argv .slice (2 ));
if (command === "process" ) {
const format = parseOutputFormat (flags.format );
const dryRun = flags["dry-run" ] === true ;
console .log (formatOutput (result, format));
}
Bun FS Helpers
Command-injection-safe file operations
TOCTOU protection with stat()
Atomic file updates (temp + rename)
SHA256 hashing for idempotency
Pure Bun-native APIs (no node:fs)
Example from inbox engine:
import {
pathExistsSync,
readTextFileSync,
writeTextFileSync,
rename,
sha256File,
stat,
} from "@side-quest/core/fs" ;
const tempPath = `${targetPath} .tmp` ;
writeTextFileSync (tempPath, newContent);
await rename (tempPath, targetPath);
const hash = await sha256File (sourceFile);
if (registry.isProcessed (hash)) return ;
const preStat = await stat (filePath);
const postStat = await stat (filePath);
if (postStat.mtimeMs !== preStat.mtimeMs ) throw error;
Common Patterns
Engine Factory function createInboxEngine (options : { vaultPath: string }InboxEngine {
let cachedSuggestions : InboxSuggestion [] = [];
return {
async scan (await scanInbox (options.vaultPath );
return cachedSuggestions;
},
async editWithPrompt (id : string , prompt : string const suggestion = cachedSuggestions.find (s =>id === id);
if (!suggestion) throw error;
const updated = await llmEditSuggestion (suggestion, prompt);
cachedSuggestions = cachedSuggestions.map (s =>id === id ? updated : s
);
return updated;
},
async execute (ids : string []const toExecute = cachedSuggestions.filter (s =>includes (s.id ));
const results = await Promise .all (
toExecute.map (s =>executeSuggestion (s))
);
cachedSuggestions = cachedSuggestions.filter (
s =>includes (s.id )
);
return results;
},
generateReport (suggestions : InboxSuggestion []return formatMarkdownReport (suggestions);
},
};
}
LLM Integration import { buildInboxPrompt, parseDetectionResponse } from "./llm-detection" ;
import { callLLM } from "@sidequest/marketplace-core/llm" ;
async function detectDocumentType (
content : string ,
filename : string ,
vaultContext : { areas: string []; projects: string [] },
): Promise <DocumentTypeResult > {
const prompt = buildInboxPrompt ({
content,
filename,
vaultContext,
});
const response = await callLLM (prompt, {
model : "haiku" ,
temperature : 0.3 ,
});
return parseDetectionResponse (response);
}
Quick Reference
File Structure src/inbox/
├── types.ts # Core types (InboxSuggestion, InboxEngine)
├── engine.ts # Engine factory (scan/execute/edit/report)
├── registry.ts # Idempotency tracking (SHA256, locking)
├── pdf-processor.ts # PDF extraction + heuristics
├── llm-detection.ts # AI type detection + field extraction
├── cli-adapter.ts # Interactive terminal UI
├── cli.ts # Interactive CLI entry point
├── errors.ts # Error taxonomy (23 codes)
├── logger.ts # Structured logging with correlation IDs
├── unique-path.ts # Path collision detection and resolution
├── converters/ # Document type detection configuration
│ ├── types.ts # InboxConverter interface definitions
│ ├── defaults.ts # Default converters (invoice, booking)
│ ├── loader.ts # Converter loading and merging
│ └── index.ts # Module exports
└── [*.test.ts] # 246 comprehensive tests (10 files)
Key Dependencies
p-limit - Controlled concurrencynanospinner - Progress indicators for CLI@side-quest/core/fs - Atomic write utilities (ensureDirSync, moveFile, readTextFileSync)@sidequest/core/glob - File globbing utilities (globFilesSync)pdftotext - External CLI (brew install poppler)crypto.subtle - SHA256 hashing (Bun native)
Checklist: Building an Inbox Processor
