| name | prompt_design_tuning_best_practice |
| description | Collaboratively design, evaluate, iterate on, and recommend a final launch candidate for a target prompt under the principle of “human-gated, agent-executed” workflow. |
Prompt Design & Tuning Best Practices
The goal of this Skill is not to casually “chat about prompts,” but to turn prompt tuning into an executable, reviewable, and cost-controlled engineering workflow.
The Agent handles most of the execution work.
Humans are responsible only for validating direction, approving high-cost loops, and signing off on the final launch candidate.
When to Use
Use this Skill when the user needs to:
- design or optimize a target prompt from scratch
- design a separate evaluation / judge prompt
- compare the performance of multiple models on an evaluation set
- work with an existing API curl, SDK integration, or request protocol
- run controlled prompt iterations under a limited budget
- turn prompt tuning into a reusable workflow instead of a one-off chat exercise
Working Modes
1. Design-Only Mode
Use this mode when:
- there is no runnable environment yet
- no evaluation resources are available yet
- real model calls cannot be executed for now
In this mode, the Agent should produce:
- task definition
- target prompt draft
- judge prompt draft
- evaluation plan
- script skeletons
- manual execution guidance
2. Execution Mode
Use this mode when:
- a runnable environment already exists
- the model invocation method has been provided
- the evaluation set, resource limits, and candidate models have been provided
In this mode, the Agent should continue with:
- batch generation
- automatic evaluation
- result analysis
- prompt iteration
- final candidate recommendation
Core Principles
The following rules are non-negotiable by default:
-
The target prompt and the judge prompt must be separated.
Do not silently modify both in the same comparison round and then mix their gains together.
-
Before large-scale evaluation, the task definition (task spec) must be frozen first.
-
Every round of prompt optimization must have a clear optimization hypothesis.
No random “this sentence feels off, let’s tweak it” behavior.
-
An experiment log must be maintained, including at least:
- version number
- summary of changes in the current round
- optimization hypothesis
- evaluation results
- cost information
- conclusion
-
Any high-cost evaluation loop must be approved by a human beforehand.
-
The final launch candidate must be reviewed by a human.
A high machine-evaluation score does not automatically mean it is ready for launch.
-
If the input information is incomplete, low-risk assumptions may be made, but they must be stated explicitly.
Recommended Inputs to Collect
The Agent should gather or infer the following whenever possible:
- business goal
- user scenario
- input format
- output format
- hard constraints
- unacceptable errors
- success criteria
- online acceptance threshold
- evaluation set
- candidate models
- invocation method (curl / SDK / API)
- resource limits (TPM, RPM, timeout, budget, retry cap)
Target Deliverables
By default, the workflow should aim to produce the following:
docs/task_spec.mdprompts/production_prompt_v{n}.mdprompts/judge_prompt_v{n}.mddocs/eval_plan.mdscripts/run_generation.pyscripts/run_judge.pyreports/iteration_{n}_summary.mdreports/final_recommendation.mdreports/experiment_log.md
Human Gates
By default, human confirmation is required only at the following key checkpoints:
Gate A — Freeze the Task Definition
Confirm:
- whether the task is understood correctly
- whether the success criteria are reasonable
- whether the constraints are complete
Gate B — Confirm the Direction of the Target Prompt
Confirm:
- whether the target prompt is directionally correct
- whether it is ready to enter evaluation
Gate C — Confirm the Direction of the Judge Prompt
Confirm:
- whether the evaluation standard is fair
- whether the judge is evaluating what actually matters
Gate D — Approve a High-Cost Iteration Loop
Confirm:
- model list
- TPM budget
- number of iteration rounds
- whether it is worth spending more resources
Gate E — Final Review
Confirm:
- whether the current best version can serve as a launch candidate
- whether to continue optimizing
- whether to stop
Unless the user explicitly asks for finer-grained control, do not interrupt too frequently in the middle.
Execution Flow
Phase 0 — Task Definition (Task Spec)
Before writing any prompt, first establish a clear task definition.
The task definition should include at least:
- problem description
- input format
- output format
- business goal
- user goal
- constraints
- explicitly forbidden outputs
- positive and negative examples
- success metrics
- unresolved issues
- current assumptions
If the user’s description is incomplete, do not stall.
Fill in reasonable assumptions first, then present them for confirmation.
After this, proceed to Gate A.
Phase 1 — Generate the First Draft of the Target Prompt
Based on the task definition, produce the first draft of the target prompt.
Requirements:
- instructions must be clear
- constraints must be explicit
- output structure must be stable
- ambiguity should be minimized
- controllability should be prioritized over fluffy “stylistic” wording
- examples should be included only when they are truly helpful
Also output:
- key design rationale
- predicted risk points
- likely failure scenarios
- what to pay close attention to in the first evaluation round
After this, proceed to Gate B.
Phase 2 — Generate the First Draft of the Judge Prompt
Design an independent Judge / Eval Prompt.
Requirements:
- evaluate the task outcome, not whether the prompt itself reads nicely
- score across separate dimensions, then aggregate
- include hard-fail categories
- output must be structured JSON
- minimize bias caused by stylistic model preferences
- explicitly handle the following cases:
- partially correct outputs
- format errors
- misunderstanding of the task
- unsafe or policy-violating content
- reasonable uncertainty caused by incomplete task information
Also output:
- scoring dimensions
- weight design
- hard-fail conditions
- Judge output JSON schema
- Judge blind spots
After this, proceed to Gate C.
Phase 3 — Design the Evaluation Plan
Before running large-scale evaluations, define the evaluation plan clearly.
The plan should include at least:
- source and size of the evaluation set
- sample slicing strategy (easy / medium / hard / edge cases)
- online acceptance threshold
- primary metrics
- secondary diagnostic metrics
- tie-breaking rules
- maximum number of iteration rounds
- total budget limit
- early stopping conditions
Default loop policy:
- by default, run at most 2 high-cost optimization rounds
- stop if the gains are marginal and the failure types have not improved in substance
- if the Judge itself looks unreliable, fix the Judge first instead of continuing to modify the target prompt
Phase 4 — Write the Generation Script
If executable conditions are available, the Agent should write a batch generation script.
The script should support, as much as possible:
- jsonl / csv / excel input
- multiple models
- resume from checkpoint
- retries and backoff
- logging
- strict input-output order preservation
- TPM / RPM rate limiting
- structured outputs for downstream evaluation
TPM Handling Principles
Do not crudely translate TPM directly into high concurrency.
Preferred approach:
- estimate token consumption per request
- use token-bucket or time-window rate limiting
- use conservative concurrency when RPM and latency are unknown
- prioritize stability before speed
Phase 5 — Batch Generate Model Outputs
Run the full evaluation set across all specified models and prompt versions.
At minimum, record:
- model name
- prompt version
- input sample ID
- raw output
- token usage (if available)
- latency
- retry count
- request failure information
- truncation / parsing failures
If generation failures occur frequently:
- first separate infrastructure issues from prompt issues
- do not conclude that the prompt is bad before ruling out quota, network, rate-limiting, or protocol problems
Phase 6 — Run Automatic Evaluation
Use the Judge Prompt to evaluate generated outputs in batch.
Requirements:
- Judge output must be structured JSON
- raw judge outputs must be traceable
- compute overall scores and slice-level metrics
- automatically identify major failure clusters
- distinguish format errors from content errors
- if the Judge is noisy, state that explicitly instead of pretending the results are reliable
Phase 7 — Analyze and Optimize
A new prompt iteration is allowed only when there is a clear optimization hypothesis.
Each round must include:
- summarize the previous round’s results
- identify the major failure clusters
- propose the optimization hypothesis for this round
- modify only the most necessary prompt sections
- provide a version-diff summary
- predict what should improve and what may regress
Do not run another round for no reason.
If the next round will consume meaningful resources, go to Gate D first.
Phase 8 — Final Recommendation
Once a version reaches a sufficiently strong level, the Agent should produce a final review package.
It should include at least:
- final target prompt
- final judge prompt
- recommended model
- overall metrics
- slice-level metrics
- major remaining failure types
- cost / latency notes
- whether launch is recommended
- what should be monitored after launch
After this, proceed to Gate E.
Default Outputs at Each Gate
Gate A Output
- task definition document
- current assumptions
- missing information
- recommended acceptance criteria
Gate B Output
- target prompt v1
- design rationale
- expected risks
Gate C Output
- judge prompt v1
- scoring rubric
- JSON schema
- Judge blind spots
Gate D Output
- current result comparison
- failure analysis
- prompt change summary
- next-round optimization hypothesis
- estimated resource consumption
Gate E Output
- final candidate
- why it is the current best version
- where it may still fail
- recommendation to launch / continue optimizing / stop
Default Analysis Templates
Experiment Log Fields
Each experiment round should record at least:
- iteration
- production_prompt_version
- judge_prompt_version
- model
- dataset_version
- hypothesis
- change_summary
- aggregate_score
- slice_scores
- dominant_failures
- cost
- verdict
Suggested Failure Taxonomy
The Agent should try to classify failures into one of the following:
- task misunderstanding
- missing constraints
- extraction error
- reasoning error
- incomplete coverage
- unsafe / policy-violating output
- format / schema error
- verbose / redundant output
- hallucinated details
- mismatch between Judge and actual task goal
- infrastructure failure
Explicitly Forbidden Anti-Patterns
Do not do the following:
- modify both the target prompt and the judge prompt in the same round without saying so
- look only at aggregate score and ignore the failure distribution
- overfit to a tiny evaluation set without warning about the risk
- use machine evaluation as a substitute for final human review
- loop endlessly because the score moved slightly
- rewrite the whole prompt when only one part is broken
- hide critical assumptions
- declare success without showing hard examples
Default Behavior When the Skill Is Triggered
When this Skill is triggered, the Agent should follow this order:
- build or refresh the task definition
- determine which phase the workflow is currently in
- prioritize filling missing artifacts before rewriting existing ones
- prefer incremental optimization over full rewrites
- request confirmation only at the defined human gates
- after each major step, output a concise decision memo including:
- what changed
- why it changed
- which metrics improved
- what major issues remain
- whether another round is worth it
Example Trigger Phrases
The following requests are suitable triggers for this Skill:
- “Help me automate this prompt tuning workflow”
- “Write the target prompt first, then the judge prompt, then design the evaluation”
- “Use the evaluation set and several models to find the current best prompt”
- “Run 1 to 2 prompt iteration rounds under a controlled budget”
- “Turn this prompt tuning process into a reusable agent skill”
- “Let the agent drive the process, and keep humans only at key checkpoints”
