auto-dsm-evaluation-framework

name	auto-dsm-evaluation-framework
description	Black-box evaluation framework for assessing LLM-generated Design Structure Matrices (DSMs) from structured technical documentation. Integrates structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss' κ) into a Composite Quality Score (Q). Provides transparent benchmarking methodology for auditing Auto-DSM pipelines in MBSE workflows.
category	systems-engineering
tags	["mbse","dsm","llm-evaluation","design-structure-matrix","systems-engineering","model-based"]
source	arxiv:2607.05985
date	2026-07-07T00:00:00.000Z

Auto-DSM Evaluation Framework: Black-Box Assessment of LLM-Based DSM Generation

Paper

• Title: Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation
• Authors: Niels Potters, Theo Hofman
• arXiv: 2607.05985
• Date: 2026-07-07
• Categories: cs.AI, cs.AR, cs.CE, eess.SY

Problem

LLMs are increasingly used to auto-generate Design Structure Matrices (DSMs) from technical documentation, but:

• Auto-DSM pipelines are closed-source and opaque
• No standardized methodology exists for evaluating LLM-generated DSMs
• LLMs are sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation
• Hallucination and abstention failures need systematic characterization

Evaluation Framework

Three-Perspective Approach

Perspective	Metrics	Purpose
Single-Run	Completeness, Correctness, Coupling Density	Structural quality of one DSM
Multi-Run	Selective Accuracy, Abstention Coverage	Consistency across runs
Stability	Entropy, Fleiss' κ	Reproducibility and agreement

1. Structural Metrics (Single-Run)

Completeness: Fraction of GT-DSM dependencies recovered by GEN-DSM

Completeness = |GEN-DSM ∩ GT-DSM| / |GT-DSM|

Correctness: Fraction of GEN-DSM dependencies that match GT-DSM

Correctness = |GEN-DSM ∩ GT-DSM| / |GEN-DSM|

Coupling Density: Ratio of actual dependencies to possible dependencies

Density = |DSM| / (n × (n-1))

where n = number of system elements.

2. Classification Metrics (Multi-Run)

Selective Accuracy: Accuracy when the LLM does NOT abstain

SA = Correct Predictions / (Total Predictions - Abstentions)

Abstention Coverage: Fraction of cases where the LLM correctly abstains

AC = Correct Abstentions / Total Abstentions

3. Stability Measures

Entropy: Measures variability across multiple runs

H = -Σ p_ij × log(p_ij)

where p_ij is the empirical probability of a dependency between elements i and j.

Fleiss' κ: Inter-rater agreement across multiple LLM runs

κ = (P̄ - P̄e) / (1 - P̄e)

where P̄ is observed agreement and P̄e is expected agreement by chance.

Composite Quality Score (Q)

Synthesizes all metrics into a single score:

Q = w1·Completeness + w2·Correctness + w3·Selective Accuracy
  + w4·(1 - Entropy/Entropy_max) + w5·Fleiss_κ

Weights can be tuned based on application priorities (e.g., safety-critical vs. exploratory).

Experimental Design

Controlled Variables

1. Phrasing variations: Same system described with different terminology
2. Parameter-dataset alignment: Matching LLM temperature/top_p to task difficulty
3. System complexity: Abstract vs. real-world (refrigerator) decompositions

Key Findings

• LLMs produce structurally plausible DSMs under well-structured inputs
• High reproducibility achievable with clear dependency definitions
• Systematic failure modes:
  • Ambiguous dependency definitions → hallucination
  • Inconsistent terminology → missed dependencies
  • Poor prompt formulation → excessive abstention or over-confident errors
• Performance degrades significantly with system complexity increase

Implementation Pattern

import numpy as np
from collections import Counter

class DSMEvaluator:
    """Black-box evaluation framework for LLM-generated DSMs."""
    
    def __init__(self, weights=None):
        self.weights = weights or {
            'completeness': 0.25,
            'correctness': 0.25,
            'selective_accuracy': 0.20,
            'stability': 0.15,
            'agreement': 0.15
        }
    
    def _to_binary_matrix(self, dsm, elements):
        """Convert DSM dict to binary adjacency matrix."""
        n = len(elements)
        idx = {e: i for i, e in enumerate(elements)}
        mat = np.zeros((n, n))
        for (src, tgt), val in dsm.items():
            if src in idx and tgt in idx:
                mat[idx[src], idx[tgt]] = 1
        return mat
    
    def completeness(self, gen_dsm, gt_dsm):
        """Fraction of GT dependencies recovered."""
        gen_set = set(gen_dsm.keys())
        gt_set = set(gt_dsm.keys())
        intersection = gen_set & gt_set
         (intersection) / ((gt_set), )
    
     ():
        
        gen_set = (gen_dsm.keys())
        gt_set = (gt_dsm.keys())
        intersection = gen_set & gt_set
         (intersection) / ((gen_set), )
    
     ():
        
        n_possible = n_elements * (n_elements - )
         (dsm) / (n_possible, )
    
     ():
        
        non_abstained = [(p, g)  p, g, a  (predictions, ground_truth, abstentions)   a]
          non_abstained:
             
        correct = (  p, g  non_abstained  p == g)
         correct / (non_abstained)
    
     ():
        
        
        
          (abstentions):
               
         (abstentions) / (abstentions)  
    
     ():
        
        n = (elements)
        prob_matrix = np.zeros((n, n))
         dsm  dsm_runs:
            prob_matrix += ._to_binary_matrix(dsm, elements)
        prob_matrix /= ((dsm_runs), )
        
        
        probs = prob_matrix.flatten()
        probs = probs[probs > ]  
        entropy = -np.(probs * np.log2(probs + ))
        max_entropy = np.log2(n * n)
         entropy / (max_entropy, )  
    
     ():
        
        n_elements = (elements)
        n_raters = (dsm_runs)
        
        
        ratings = np.zeros((n_elements * (n_elements - ), n_raters))
         r, dsm  (dsm_runs):
            mat = ._to_binary_matrix(dsm, elements)
            idx = 
             i  (n_elements):
                 j  (n_elements):
                     i != j:
                        ratings[idx, r] = mat[i, j]
                        idx += 
        
        
        n_items, n_raters = ratings.shape
        p_j = np.(ratings, axis=) / (n_items * n_raters)
        p_bar_e = np.(p_j * ( - p_j))
        
        p_i = np.(ratings * ( - ratings), axis=) / (n_raters * (n_raters - ))
        p_bar =  - np.mean(p_i)
        
         p_bar_e == :
             
         (p_bar - p_bar_e) / ( - p_bar_e)
    
     ():
        
        metrics = {
            : .completeness(gen_dsm, gt_dsm),
            : .correctness(gen_dsm, gt_dsm),
        }
        
         predictions     abstentions   :
            metrics[] = .selective_accuracy(
                predictions, ground_truth, abstentions)
        
        metrics[] =  - .stability_entropy(dsm_runs, elements)
        metrics[] = (, .fleiss_kappa(dsm_runs, elements))
        
        q = (.weights.get(k, ) * metrics.get(k, )  k  .weights)
         q, metrics

Key Insights for Systems Engineering

1. Structured inputs are critical: LLMs perform well on DSM generation only when documentation is clear and dependency definitions are consistent
2. Multi-run evaluation is essential: Single-run metrics hide hallucination patterns; stability measures reveal systematic failure modes
3. Abstention is a feature, not a bug: Proper abstention when information is ambiguous is preferable to confident hallucination
4. Composite scoring enables trade-offs: Different MBSE contexts prioritize different aspects (safety → correctness, exploration → completeness)
5. Black-box auditing bridges the gap: This framework enables systematic evaluation without requiring access to Auto-DSM internals

Activation Keywords

DSM generation, design structure matrix, auto-dsm, MBSE evaluation, LLM systems engineering, model-based decomposition, system architecture evaluation, dependency matrix, systems engineering LLM