senior-prompt-engineer

# Senior Prompt Engineer Prompt engineering patterns, LLM evaluation frameworks, and agentic system design. Provides static (deterministic) analysis tools to optimize prompts, evaluate RAG retrieval and generation quality, and validate/visualize agent workflows — plus deep reference libraries of prompt patterns, evaluation metrics, and agent architectures. ## Core Capabilities - **Prompt optimization** — token counting and cost estimation, clarity/structure scoring, ambiguity and redundancy detection, and generation of optimized prompt versions. - **Few-shot & structured output design** — extract/manage few-shot examples, design diverse example sets (simple/edge/complex/negative), and enforce reliable JSON/XML schema outputs. - **RAG evaluation** — context relevance, answer faithfulness, groundedness (ROUGE-L), and retrieval metrics (Precision@K, MRR, NDCG) over pre-retrieved contexts. - **Agentic system design** — validate agent configs, visualize flows (ASCII/Mermaid), estimate token cost per run, and apply ReAct / Plan-Execute / Tool-Use / multi-agent patterns. - **Pattern library** — 10 prompt patterns, evaluation frameworks (A/B testing, benchmarks, human eval), and agent architectures with pseudocode. ## When to Use - Optimizing an existing prompt's performance or reducing token costs. - Designing prompt templates, few-shot examples, or structured-output workflows. - Evaluating LLM outputs or RAG retrieval/generation quality. - Building or validating agentic systems and tool-calling workflows. ## Tools | Tool | Purpose | Command | |------|---------|---------| | `prompt_optimizer.py` | Analyze/optimize prompts: tokens, clarity, structure, few-shot extraction | `python scripts/prompt_optimizer.py prompt.txt --analyze` | | `rag_evaluator.py` | Evaluate RAG context relevance, faithfulness, retrieval metrics | `python scripts/rag_evaluator.py --contexts ctx.json --questions q.json` | | `agent_orchestrator.py` | Validate, visualize, and cost-estimate agent configs | `python scripts/agent_orchestrator.py agent.yaml --validate` | ## References Load the reference that matches the task — keep this file lean and pull detail on demand: - **[references/tools-and-workflows.md](references/tools-and-workflows.md)** — full tool usage with sample outputs, the prompt-optimization / few-shot / structured-output workflows, common-patterns and command quick references, troubleshooting table, success criteria, and complete per-script parameter/output-format reference. Read when running any tool or executing a workflow. - **[references/prompt_engineering_patterns.md](references/prompt_engineering_patterns.md)** — 10 prompt patterns (zero/few-shot, CoT, role, structured output, self-consistency, ReAct, tree-of-thoughts, RAG) with example inputs and expected outputs. Read when choosing or applying a prompt technique. - **[references/llm_evaluation_frameworks.md](references/llm_evaluation_frameworks.md)** — evaluation metrics, text-generation and RAG-specific scoring, human-eval frameworks, A/B testing, benchmark datasets, and pipeline design. Read when measuring quality or comparing prompts. - **[references/agentic_system_design.md](references/agentic_system_design.md)** — agent architectures (ReAct, Plan-and-Execute, Tool Use, multi-agent, memory/state) and design patterns with pseudocode. Read when building agents or tool-calling systems. ## Scope & Limitations **This skill covers:** - Static prompt analysis: token counting, clarity scoring, structure detection, and optimization suggestions - RAG evaluation: context relevance, answer faithfulness, groundedness, and retrieval metrics (Precision@K, ROUGE-L, MRR, NDCG) - Agent workflow design: configuration validation, ASCII/Mermaid visualization, and token cost estimation - Few-shot example extraction and management from existing prompts **This skill does NOT cover:** - Live LLM calls or runtime prompt testing --- all analysis is static/deterministic (see `senior-ml-engineer` for LLM integration) - Vector database setup or embedding generation --- RAG evaluator scores pre-retrieved contexts only (see `senior-data-engineer` for pipeline orchestration) - Fine-tuning, RLHF, or model training workflows (see `senior-ml-engineer` for model deployment) - Production monitoring, A/B test execution, or real-time drift detection (see `senior-data-scientist` for experiment design) ## Integration Points | Skill | Integration | Data Flow | |-------|-------------|-----------| | `senior-ml-engineer` | LLM integration and model deployment | Optimized prompts from this skill feed into the prompt templates of its LLM integration layer | | `senior-data-scientist` | A/B test design for prompt experiments | `experiment_designer.py` defines test parameters; this skill provides the prompt variants to compare | | `senior-data-engineer` | RAG pipeline orchestration | `pipeline_orchestrator.py` builds the retrieval pipeline; this skill evaluates its output quality | | `senior-fullstack` | End-to-end application scaffolding | Fullstack apps consume agent configs validated by `agent_orchestrator.py` | | `senior-security` | Prompt injection and adversarial input review | Security analysis covers the attack surface; this skill ensures prompts include defensive constraints | | `senior-qa` | Quality assurance for AI-powered features | QA test suites validate that optimized prompts produce consistent outputs in production |