bias-detection

# Bias Detection Detects **demographic bias** in data quality by running all 16 LYDUS quality metrics on patient subgroups (Sex, Race, Age) and comparing scores. The **GDI (Group Disparity Index)** quantifies how much any group deviates from the overall mean for each metric. ## When to Use This Skill - After QUIQ conversion, to assess whether data quality differs across demographic groups - To identify metrics where one group is systematically under-served or over-represented - To generate an equity report for LYDUS quality management assessment - Before clinical ML model training, to detect data quality biases that may propagate into model bias ## SQL Support **Not applicable.** Orchestrates Python implementations of all 16 sub-metrics plus Claude CLI calls. ## Pipeline Overview ``` 1. LLM identifies Sex / Race / BirthDate columns in QUIQ 2. Build per-patient demographics pivot (one row per Patient_id) 3. Compute Age_Group from BirthDate + Event_date 4. For each group in {Sex, Race, Age_Group}: Run all 16 quality metrics on the group-filtered QUIQ subset 5. Build results table: rows = metrics, cols = group labels + Mean + GDI ``` ## 16 Sub-Metrics | Metric | API Key | Skill | |--------|:-------:|-------| | Range Validity | ✗ | range-validity | | Date Validity | optional | date-validity | | Format Validity | optional | format-validity | | Sequence Validity | Claude CLI | sequence-validity | | Completeness | ✗ | completeness | | Logical Accuracy | Claude CLI | logical-accuracy | | Cross-Sectional Consistency | Claude CLI | cross-sectional-consistency | | Time Series Consistency | ✗ | time-series-consistency | | Class Diversity | ✗ | class-diversity | | Instance Diversity | ✗ | instance-diversity | | Fidelity | ✗ | fidelity | | Preciseness | ✗ | preciseness | | Accuracy / Precision / Recall / F1 / AUROC | ✗ | classification | | Note Fidelity | Claude CLI | note-fidelity | | Note Accuracy | Claude CLI | note-accuracy | | Vocabulary Diversity | ✗ | vocabulary-diversity | | Sentence Diversity | ✗ | sentence-diversity | ## GDI (Group Disparity Index) `GDI = max(|group_score − mean_score|)` across all groups for each metric. - GDI = 0 → all groups have identical scores (no bias) - High GDI → large disparity between groups for that metric ## Demographic Grouping | Group | Column | Detection | Age Bins | |-------|--------|-----------|----------| | Sex | LLM-identified | biological sex variable | — | | Race | LLM-identified | race variable | — | | Age | BirthDate + Event_date | date of birth variable | 0-9, 10-19, …, 80+ | Rows with null Sex/Race/Age_Group are dropped before group iteration. ## Output | File | Description | |------|-------------| | `bias_sex_summary.csv` | Metrics × sex groups + Mean + GDI | | `bias_race_summary.csv` | Metrics × race groups + Mean + GDI | | `bias_age_summary.csv` | Metrics × age groups + Mean + GDI | | Sub-metric plots | SHAP plots, AUROC plots, boxplots (saved to save_path) | ## How to Run ```python import pandas as pd from scripts.bias_detection import get_bias_detection quiq = pd.read_csv("/path/to/quiq.csv") via = pd.read_csv("/path/to/via.csv") config = { 'save_path': '/path/to/output', 'operation_type_manual': False, 'target_variable': '', 'automatic_num': 5, 'recommend_num': 5, } df_sex, df_race, df_age = get_bias_detection(quiq, via, config) print(df_sex) ``` ### As a script with config ```yaml # config.yaml quiq_path: /path/to/quiq.csv via_path: /path/to/via.csv save_path: /path/to/output operation_type_manual: false target_variable: "" automatic_num: 5 recommend_num: 5 ``` ```bash python scripts/bias_detection.py --config config.yaml ``` ## Critical Notes 1. **All 16 sub-metrics run per group** — each metric is wrapped in try/except; failures return `np.nan` without stopping the run. Expect long runtimes for large datasets or many groups. 2. **Skill dependency imports** — `bias_detection.py` dynamically adds sibling skills' `scripts/` directories to `sys.path` at runtime. All 17 sibling skills must be present under the same skills root (`~/.claude/skills/` or `src/m4/skills/lydus/`). 3. **원본 코드 버그 수정**: - `model_ver` / `api_key` 제거: `_llm_ask_column`이 Claude CLI subprocess로 대체됨 - `get_time_series_consistency(config['save_path'], quiq)` 인자 순서 반전 → `(quiq, save_path)` 로 수정 - `get_code_validity` 반환값 3개 언팩 → 2개(`validation_df, error_summary`)로 수정 - `get_unstructured_fidelity` / `get_unstructured_accuracy` → M4 스킬의 `get_note_fidelity` / `get_note_accuracy` 로 교체 - `config['save_path'] = -1` 하드코딩 → 실제 save_path 사용 4. **GDI 계산** — `df.sub(df['Mean'], axis=0).abs().max(axis=1)`: Mean 열을 먼저 계산 후 그룹 열들과의 최대 절댓값 편차. Mean 열 자체는 GDI 계산에서 제외. 5. **Age bins** — `pd.cut(..., right=False)`: 0-9 includes 0 and excludes 10. 80+ = [80, 120). 6. **Dependencies** — all 16 sub-metric skill dependencies (see each skill's SKILL.md). LLM 호출은 Claude CLI via subprocess — API 키 불필요. ## References - LYDUS 품질관리 프로그램 활용 가이드라인 (비공개 내부 문서) - Original Python implementation: LYDUS_Bias_Detection.py (이성민 작성)