メニュー
Evaluate semantic consistency between AI-generated clinical notes and expert gold standards using BERTScore and COMET
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/majiayu000/claude-skill-registry --skill semantic-consistency-auditorコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SKILL.md を表示中
| name | semantic-consistency-auditor |
| description | Evaluate semantic consistency between AI-generated clinical notes and expert gold standards using BERTScore and COMET |
| version | 1.0.0 |
| category | Research |
| tags | [] |
| author | AIPOCH |
| license | MIT |
| status | Draft |
| risk_level | Medium |
| skill_type | Tool/Script |
| owner | AIPOCH |
| reviewer | |
| last_updated | 2026-02-06 |
ID: 212
Name: semantic-consistency-auditor
Description: Introduces BERTScore and COMET algorithms to evaluate the semantic consistency between AI-generated clinical notes and expert gold standards from the "semantic entailment" level.
Semantic Consistency Auditor is a medical AI evaluation tool used to assess the semantic consistency between AI-generated clinical notes and expert-written gold standards from a semantic level. This tool is not limited to traditional string matching or bag-of-words models, but uses deep learning models to understand semantic entailment relationships, capable of identifying expressions with different wording but similar meaning.
BERTScore uses pre-trained BERT model contextual embeddings to calculate similarity between candidate text and reference text:
COMET is a neural network-based evaluation metric originally used for machine translation evaluation, applicable to semantic entailment tasks:
# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate # Linux/Mac
# Or venv\Scripts\activate # Windows
# Install dependencies
pip install bertscore comet-ml transformers torch
Configure in ~/.openclaw/skills/semantic-consistency-auditor/config.yaml:
# BERTScore Configuration
bertscore:
model: "microsoft/deberta-xlarge-mnli" # Or "bert-base-chinese" for Chinese
lang: "zh"
# Evaluate single case pair
python scripts/main.py \
--ai-generated "Patient presented with fever for 3 days, highest temperature 39°C, accompanied by cough." \
--gold-standard "Patient chief complaint of fever for 3 days, highest temperature 39°C, accompanied by cough symptoms." \
--output results.json
# Batch evaluation from JSON file
python scripts/main.py \
--input-file batch_cases.json \
--output results.json \
--format detailed
# Use specific model
python scripts/main.py \
--ai-generated "..." \
--gold-standard "..." \
--bert-model "bert-base-chinese" \
--comet-model "Unbabel/wmt20-comet-da"
from semantic_consistency_auditor import SemanticConsistencyAuditor
# Initialize evaluator
auditor = SemanticConsistencyAuditor(
bert_model="microsoft/deberta-xlarge-mnli",
comet_model="Unbabel/wmt22-comet-da",
lang="zh"
)
# Evaluate single case
result = auditor.evaluate(
ai_text="Patient presented with fever for 3 days...",
gold_text="Patient chief complaint of fever for 3 days..."
)
print(f"BERTScore F1: {result['bertscore']['f1']:.4f}")
print(f"COMET Score: {result['comet']['score']:.4f}")
print(f"Consistency: {result['consistency']:.4f}")
print(f"Passed: {result['passed']}")
# Batch evaluation
results = auditor.evaluate_batch([
{"ai": "...", "gold": "..."},
{"ai": "...", "gold": "..."}
])
Pass text directly through --ai-generated and --gold-standard parameters.
[
{
"case_id": "CASE001",
"ai_generated": "Patient presented with fever for 3 days, highest temperature 39°C, accompanied by cough.",
"gold_standard": "Patient chief complaint of fever for 3 days, highest temperature 39°C, accompanied by cough symptoms.",
"metadata": {
"department": "Respiratory",
"disease_type": "Upper respiratory infection"
}
},
{
"case_id": "CASE002",
"ai_generated": "...",
"gold_standard": "..."
}
]
{
"overall": {
"total_cases": 100,
"passed_cases": 85,
"pass_rate": 0.85,
"avg_bertscore_f1": 0.8923,
"avg_comet_score": 0.8234,
"avg_consistency": 0.8579
},
"thresholds": {
"bertscore_f1": 0.85,
"comet_score": 0.75,
"semantic_consistency": 0.80
}
}
{
"cases": [
{
"case_id": "CASE001",
"ai_generated": "Patient presented with fever for 3 days...",
"gold_standard": "Patient chief complaint of fever for 3 days...",
"metrics": {
"bertscore": {
"precision": 0.9123,
"recall": 0.8934,
"f1": 0.9028
},
"comet": {
"score": 0.8234,
"system_score": 0.8156
},
"semantic_consistency": 0.8631
},
"passed": true,
"details": {
"semantic_gaps": [],
"matched_concepts": ["fever for 3 days", "temperature 39°C", "cough"]
}
}
],
"summary": { ... }
}
| Score Range | Grade | Description |
|---|---|---|
| 0.90 - 1.00 | Excellent | Highly consistent semantics, almost no difference |
| 0.80 - 0.89 | Good | Basically consistent semantics, minor differences exist |
| 0.70 - 0.79 | Pass | Generally consistent semantics, partial omissions or deviations exist |
| 0.60 - 0.69 | Needs Improvement | Obvious semantic differences exist |
| < 0.60 | Fail | Semantic inconsistency, major omissions or errors exist |
Default pass conditions (configurable):
| Error Code | Description | Handling Suggestion |
|---|---|---|
| E001 | Input text is empty | Check input data integrity |
| E002 | Model loading failed | Check network connection or local model path |
| E003 | Language not supported | Use supported lang parameter |
| E004 | Insufficient GPU memory | Switch to CPU mode or reduce batch_size |
| E005 | JSON parsing error | Check input file format |
| Risk Indicator | Assessment | Level |
|---|---|---|
| Code Execution | Python/R scripts executed locally | Medium |
| Network Access | No external API calls | Low |
| File System Access | Read input files, write output files | Medium |
| Instruction Tampering | Standard prompt guidelines | Low |
| Data Exposure | Output files saved to workspace | Low |
# Python dependencies
pip install -r requirements.txt
SOC 職業分類に基づく
