메뉴
Evaluates text fidelity and typography accuracy in AI-generated images by measuring spelling, case sensitivity, repetition, and structural inconsistencies against reference prompts. Use when the user has predictions and gold and needs to compute ABHINAW Score.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Open-weights chemistry reasoning model + verifiable reward functions from FutureHouse's ether0 (arXiv 2506.17238). Use to score model-generated chemistry outputs (SMILES validity, molecular completion, synthesis reasoning) against ground truth, or to run the open-weights ether0 model itself for chemistry reasoning. Also useful for visualizing molecules and reactions from SMILES.
Multilingual reading comprehension across text and speech modalities. It probes a model's ability to understand spoken or written passages in 39 languages and answer multiple-choice questions based on them. Use when the user wants to benchmark on 2M-Belebele, or asks about evaluating this task. Reports accuracy.
Evaluates the capability of neural radiance field models to perform real-time, high-fidelity novel view synthesis on large-scale indoor scenes using 360° panoramic imagery. It probes the trade-off between rendering quality, computational efficiency, and geometric awareness in complex, unbounded indoor environments. Use when the user wants to benchmark on 360Roam Dataset, or asks about evaluating this task. Reports PSNR.
Evaluates a unified vision-language model's ability to ground natural language instructions to 3D objects and 2D regions, as well as answer 3D visual questions. It probes spatial reasoning, cross-modal alignment, and robustness to different 3D input representations (mesh-sampled vs. sensor RGB-D point clouds). Use when the user wants to benchmark on SR3D, NR3D, ScanRefer, RefCOCO, RefCOCO+, RefCOCOg, ScanQA, SQA3D, or asks about evaluating this task. Reports top-1 accuracy (Acc@25/50/75).
Evaluates 3D point cloud networks on visual object affordance understanding. It probes the model's ability to predict point-wise probabilistic scores for 18 affordance classes given full, partial, or rotated 3D shapes. Use when the user wants to benchmark on 3D AffordanceNet, or asks about evaluating this task. Reports mAP.
Evaluates the accuracy and realism of 3D face reconstruction and generation from single 2D images. Probes shape reconstruction fidelity against ground truth meshes, texture identity preservation across novel poses, and the diversity of synthesized 3D faces. Use when the user wants to benchmark on NoW Benchmark, REALY 3D Benchmark, or asks about evaluating this task. Reports Per-vertex error (mm).
| name | abhinaw-score |
| description | Evaluates text fidelity and typography accuracy in AI-generated images by measuring spelling, case sensitivity, repetition, and structural inconsistencies against reference prompts. Use when the user has predictions and gold and needs to compute ABHINAW Score. |
| metadata | {"skill_kind":"metric","source_arxiv":2409.11874,"bibtex_key":"jagtap2024abhinaw","confidence":"medium"} |
ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images — Jagtap et al. (2024) (arXiv:2409.11874, 2024)
Evaluates text fidelity and typography accuracy in AI-generated images by measuring spelling, case sensitivity, repetition, and structural inconsistencies against reference prompts.
ABHINAW Score (primary) — range: [0, 1]
Input: AI-generated images paired with their corresponding reference text prompts.
Output: A normalized accuracy measure (ABHINAW Score) per image, averaged across 100 GPT-based OCR runs per image.
scores = []
for img in images:
img_runs = []
for _ in range(10):
gen_text = custom_gpt_ocr(img)
ref_vec = embed(lowercase(ref_text))
gen_vec = embed(lowercase(gen_text))
sim = cosine_similarity(ref_vec, gen_vec)
if sim > 0.9:
sim = allow_rearrangements(sim)
sim = apply_brevity_adjustment(sim, gen_text, ref_text)
img_runs.append(sim)
scores.append(mean(img_runs))
return mean(scores)
In the next step, we have developed a mathematical framework where finally we are calculating the ABHINAW Score here are the main highlights of the framework, the stored text were converted to lower case (reason for incorporating the lower case is already discussed in section[3.1]). Then the generated text were converted to vectors and we calculate the cosine similarity for all the generated text (see section[3.2]). If the cosine similarity is greater than 0.9 then only the text rearrangements are permitted in the scoring. In the final step, the text redundancy is taken into account by introduction of brevity adjustment (See section[4.3]). The scores were then averaged to provide a normalized accuracy measure.
@misc{jagtap2024abhinaw,
title={ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images},
author={Jagtap et al. (2024)},
year={2024},
note={arXiv:2409.11874}
}