semantic-consistency-auditor

name	semantic-consistency-auditor
description	Evaluate semantic consistency between AI-generated clinical notes and expert gold standards using BERTScore and COMET
version	1.0.0
category	Research
tags	[]
author	AIPOCH
license	MIT
status	Draft
risk_level	Medium
skill_type	Tool/Script
owner	AIPOCH
reviewer
last_updated	2026-02-06

Skill: Semantic Consistency Auditor

ID: 212
Name: semantic-consistency-auditor
Description: Introduces BERTScore and COMET algorithms to evaluate the semantic consistency between AI-generated clinical notes and expert gold standards from the "semantic entailment" level.

Overview

Semantic Consistency Auditor is a medical AI evaluation tool used to assess the semantic consistency between AI-generated clinical notes and expert-written gold standards from a semantic level. This tool is not limited to traditional string matching or bag-of-words models, but uses deep learning models to understand semantic entailment relationships, capable of identifying expressions with different wording but similar meaning.

Algorithms

1. BERTScore

BERTScore uses pre-trained BERT model contextual embeddings to calculate similarity between candidate text and reference text:

• Precision: How much semantics in the candidate text is covered by the reference text
• Recall: How much semantics in the reference text is covered by the candidate text
• F1 Score: Harmonic mean of Precision and Recall

2. COMET (Cross-lingual Optimized Metric for Evaluation of Translation)

COMET is a neural network-based evaluation metric originally used for machine translation evaluation, applicable to semantic entailment tasks:

• Uses XLM-RoBERTa encoder to capture deep semantics
• Outputs semantic consistency scores between 0-1
• Gives high scores to semantically equivalent but differently expressed text

Installation

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Linux/Mac
# Or venv\Scripts\activate  # Windows

# Install dependencies
pip install bertscore comet-ml transformers torch

Configuration

Configure in ~/.openclaw/skills/semantic-consistency-auditor/config.yaml:

# BERTScore Configuration
bertscore:
  model: "microsoft/deberta-xlarge-mnli"  # Or "bert-base-chinese" for Chinese
  lang: "zh"