human-preference

name	human-preference
description	Generate two full independent responses from the same model for one user task, then collect explicit preference data. Use when you need A/B output quality judgments, structured preference labels, and optional local or remote logging for downstream analysis.

Human Preference

Use this skill when a user wants two independent responses from the same model and explicit preference data.

Workflow

1. Collect task prompt, audience, constraints, and optional evaluation criteria.
2. Produce two complete, independent responses to the same request labeled Response A and Response B.
3. Ask the user to pick: 1. A, 2. B, 3. Both are good, 4. Both are bad, or 5. Need rerun.
4. When the user picks a choice, record the preference event immediately with record_preference.py (writes to local JSONL automatically), then offer optional Hugging Face sync.

Output format

Task

Response A

Response B

Preference

Which is better for your use case?

1. A
2. B
3. Both are good
4. Both are bad
5. Need rerun

Motivate your choice:

Auto local record (always executed once winner is selected)

python3 scripts/record_preference.py \
  --task "<task>" \
  --response-a "Response A" \
  --response-b "Response B" \
  --winner "<A|B|Both are good|Both are bad|Need rerun>" \
  --notes "Motivation"

# optional remote user dataset sync
python3 scripts/record_preference.py \
  --task "<task>" \
  --response-a "Response A" \
  --response-b "Response B" \
  --winner "<A|B|Both are good|Both are bad|Need rerun>" \
  --notes "Motivation" \
  --sync \
  --hf-user-repo

For app integration, use:

scripts/log_preference_for_app.sh \
  --task "<task>" \
  --response-a "Response A" \
  --response-b "Response B" \
  --winner "<A|B|Both are good|Both are bad|Need rerun>" \
  --notes "Motivation"

If --sync/--no-sync are not provided, on first run the wrapper prompts once:

• Do you want to sync preference data to Hugging Face? If you answer yes, future runs default to syncing to a per-user dataset (--hf-user-repo) unless you override.

Preference event

{
  "schema_version": "1.0",
  "event_type": "model_response_preference",
  "model": "current_model",
  "task": "<short task label or prompt>",
  "response_a": "<id or short snippet>",
  "response_b": "<id or short snippet>",
  "winner": "A|B|Both good|Both bad|Need rerun",
  "criteria": ["accuracy", "clarity", "completeness"],
  "notes": "<optional rationale>",
  "timestamp_utc": "YYYY-MM-DDTHH:MM:SSZ"
}

Logging (local + optional Hugging Face dataset)

Append one event to a local JSONL file and optionally sync that same file to a Hugging Face repository.

Local default log path:

• ~/.local/share/human-preference/preferences.jsonl

Manual shared repo sync

python3 scripts/record_preference.py \
  --task "..." \
  --response-a "A" \
  --response-b "B" \
  --winner B \
  --notes "..." \
  --criteria "accuracy,clarity,completeness" \
  --sync \
  --hf-repo-id "your-org/human-preference" \
  --hf-remote-path "data/preferences.jsonl"

Per-user public dataset (first run auto-provisions)

export HF_TOKEN=...
python3 scripts/record_preference.py \
  --task "..." \
  --response-a "A" \
  --response-b "B" \
  --winner B \
  --notes "..." \
  --criteria "accuracy,clarity,completeness" \
  --sync \
  --hf-user-repo

If HF_TOKEN is unset, the logger uses Hugging Face credentials from an existing huggingface_hub login (if present).

• If --hf-repo-id is omitted and --hf-user-repo is set, the script resolves username from token and uses:
  • <hf-username>/human-preference
• The repo is created publicly on first run (when possible) and reused on later runs.
• If you need a different naming convention, set --hf-repo-prefix.

Rules

• If a response violates constraints, ask for a rerun instead of silently repairing it.
• If blind comparison is requested, do not provide rationale before the user picks.
• Keep both responses similarly scoped in length and quality so comparison remains fair.
• Never infer global model-quality conclusions from one choice.

Script contract

scripts/record_preference.py accepts:

• --task, --response-a, --response-b, --winner, --notes
• --criteria (comma-separated), --model, --log-path
• --sync, --hf-user-repo, --hf-repo-prefix, --hf-repo-id, --hf-repo-type, --hf-remote-path, --hf-token, --commit-message

--winner accepts: A, B, Both are good, Both are bad, Need rerun.