메뉴
Rank candidate prompts by computing LLM perplexity of the full conversation conditioned on each prompt, selecting the lowest-perplexity candidate as the best match
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Memory-efficient Keras generator that streams sharded CSV files in chunks, renders strokes to images on-the-fly, and yields batches for training on datasets too large for memory
Stack 1D convolutions for local feature extraction before bidirectional LSTMs to classify variable-length stroke sequences into hundreds of doodle categories
Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files
Normalize raw stroke coordinates to 0-255 range, resample at uniform arc-length spacing, then apply Ramer-Douglas-Peucker simplification
Render stroke sequences to grayscale images with temporal intensity encoding where earlier strokes are brighter and later strokes fade to encode drawing order
Score each face in a video as anomalous by computing L2 distance from the embedding centroid of all faces, then convert to probability via logistic function
| name | llm-perplexity-prompt-ranking |
| description | Rank candidate prompts by computing LLM perplexity of the full conversation conditioned on each prompt, selecting the lowest-perplexity candidate as the best match |
When selecting among multiple candidate prompts (e.g., which instruction produced a given output), format each candidate into a full conversation template with the input/output text, compute the perplexity of each formatted sequence, and rank by lowest perplexity. The LLM assigns lower perplexity to sequences that are more "natural" given its training — the prompt that actually produced the output will typically score lowest.
import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, AutoTokenizer
def rank_prompts(candidates, context_template, model, tokenizer):
loss_fn = nn.CrossEntropyLoss(reduction='none')
scores = []
texts = [context_template.format(prompt=p) for p in candidates]
inputs = tokenizer(texts, return_tensors="pt", padding=True,
truncation=True, add_special_tokens=False).to(model.device)
with torch.no_grad():
logits = model(**inputs).logits
labels = inputs["input_ids"].clone()
labels[~inputs["attention_mask"].bool()] = -100
shift_logits = logits[:, :-1].contiguous()
shift_labels = labels[:, 1:].contiguous()
for i in range(len(candidates)):
loss = loss_fn(shift_logits[i], shift_labels[i])
valid = (shift_labels[i] != -100).sum()
scores.append((loss.sum() / valid).item())
ranked = sorted(zip(candidates, scores), key=lambda x: x[1])
return ranked[0][0], ranked
best_prompt, all_ranked = rank_prompts(
["Rewrite formally", "Make it rhyme", "Simplify"],
"Instruction: {prompt}\nInput: ...\nOutput: ...",
model, tokenizer)