Evaluate LLM coding capabilities across three dimensions—problem analysis, code implementation, and test validation—to identify specific reasoning gaps and improve model robustness through ensemble approaches.
Evaluate LLM coding capabilities across three dimensions—problem analysis, code implementation, and test validation—to identify specific reasoning gaps and improve model robustness through ensemble approaches.
Coding Triangle: Decomposing LLM Code Understanding into Three Dimensions
Large language models excel at individual coding tasks, but understanding how they break down problems remains opaque. Standard benchmarks measure only final correctness (pass/fail), hiding whether failures stem from misunderstanding the problem, poor implementation, or inadequate validation. A model might correctly understand a problem but produce buggy code, or implement code correctly yet fail to generate meaningful test cases.
The Coding Triangle framework decomposes coding ability into three interconnected dimensions—Editorial (problem analysis), Code (implementation), and Cases (test generation)—revealing alignment and misalignment between capabilities. This granular view exposes that LLMs make predictable, correlated errors across dimensions and that model ensembles can overcome individual model limitations by increasing solution diversity.
Core Concept
Rather than treating code generation as monolithic, the Coding Triangle treats it as three coupled competencies: (1) Editorial—understanding the problem statement, constraints, and approach, (2) Code—translating approach into correct implementation, and (3) Cases—generating test cases that validate solutions. Models can succeed or fail at each dimension independently. A model might solve a problem correctly but fail to write tests that catch edge cases. Or it might generate test cases that are actually correct solutions but fail to implement efficiently.
The key insight is that self-consistency emerges at the dimension level: if a model confidently solves a problem one way, it tends to make similar mistakes across all three dimensions. This predictability enables targeted improvement—rather than attempting generic "better reasoning," practitioners can identify which dimension causes failure and apply targeted interventions.
Architecture Overview
Editorial Dimension: Parse problem statement into constraints, identify input/output specifications, outline solution approach, explain key insights
Cases Dimension: Generate diverse test cases covering edge cases, stress scenarios, and normal inputs; validate test effectiveness
Evaluation Framework: Assess each dimension independently, then analyze relationships—e.g., does Editorial correctness predict Code correctness?
Ensemble Strategy: Combine multiple models' outputs by voting or diversity selection to increase overall solution coverage
Analysis Tools: Measure dimension-level correctness, correlation matrices between dimensions, error clustering analysis, distribution comparison to human solutions
Implementation
The following implements the Coding Triangle evaluation framework for assessing LLM coding abilities across dimensions.
Step 1: Problem Analysis (Editorial Dimension)
This code extracts problem understanding from model outputs and validates completeness against gold standards.
import json
from typing importDict, List, Tuplefrom dataclasses import dataclass
@dataclassclassEditorialAnalysis:
problem_statement: str
constraints: List[str]
input_format: str
output_format: str
approach: str
key_insights: List[str]
classEditorialEvaluator:
def__init__(self):
self.required_components = [
"constraints",
"input_format",
"output_format",
"approach",
"key_insights"
]
defextract_editorial(self, model_output: str) -> EditorialAnalysis:
"""Parse model's problem analysis from text output."""
lines = model_output.split("\n")
analysis = EditorialAnalysis(
problem_statement=model_output[:200],
constraints=[],
input_format="",
output_format="",
approach="",
key_insights=[]
)
current_section = Nonefor line in lines:
if"constraint"in line.lower():
current_section = "constraints"elif"input"in line.lower():
current_section = "input_format"elif"output"in line.lower():
current_section = "output_format"elif"approach"in line.lower() or"algorithm"in line.lower():
current_section = "approach"elif"insight"in line.lower():
current_section = "key_insights"elif current_section and line.strip():
if current_section == "constraints":
analysis.constraints.append(line.strip())
elif current_section == "key_insights":
analysis.key_insights.append(line.strip())
return analysis
defscore_editorial(self, analysis: EditorialAnalysis, gold_standard: Dict) -> float:
"""Score completeness of problem analysis."""
score = 0.0
total = len(self.required_components)
if analysis.constraints andany(
gold_constraint instr(analysis.constraints)
for gold_constraint in gold_standard.get("constraints", [])
):
score += 1if analysis.input_format and gold_standard.get("input_format"):
score += 0.5iflen(analysis.input_format) > 20else0if analysis.output_format and gold_standard.get("output_format"):
score += 0.5iflen(analysis.output_format) > 20else0if analysis.approach and gold_standard.get("expected_approach"):
score += 1if analysis.key_insights:
score += len(analysis.key_insights) * 0.2returnmin(score / total, 1.0)
Step 2: Code Implementation Dimension
This evaluates whether the LLM-generated code correctly solves the problem.
from typing importCallableimport subprocess
import tempfile
classCodeEvaluator:
def__init__(self, language: str = "python"):
self.language = language
defextract_code(self, model_output: str) -> str:
"""Extract executable code from model output."""
lines = model_output.split("\n")
code_lines = []
in_code_block = Falsefor line in lines:
if"```"in line:
in_code_block = not in_code_block
elif in_code_block:
code_lines.append(line)
return"\n".join(code_lines)
deftest_code(self, code: str, test_cases: List[Tuple]) -> Dict:
"""Execute code against test cases and report results."""
results = {
"passed": 0,
"failed": 0,
"errors": [],
"outputs": []
}
with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
f.write(code)
f.flush()
temp_path = f.name
for test_input, expected_output in test_cases:
try:
result = subprocess.run(
["python", temp_path],
input=str(test_input),
capture_output=True,
text=True,
timeout=5
)
actual_output = result.stdout.strip()
expected_output_str = str(expected_output).strip()
if actual_output == expected_output_str:
results["passed"] += 1else:
results["failed"] += 1
results["outputs"].append({
"input": test_input,
"expected": expected_output_str,
"actual": actual_output
})
except subprocess.TimeoutExpired:
results["failed"] += 1
results["errors"].append(f"Timeout on input: {test_input}")
except Exception as e:
results["failed"] += 1
results["errors"].append(str(e))
import os
os.unlink(temp_path)
return results
defscore_code(self, test_results: Dict) -> float:
"""Score code based on test passage rate."""
total = test_results["passed"] + test_results["failed"]
return test_results["passed"] / total if total > 0else0.0
Step 3: Test Case Generation (Cases Dimension)
This evaluates the diversity and effectiveness of test cases generated by the model.
classCasesEvaluator:
def__init__(self):
self.case_categories = [
"edge_case",
"boundary_case",
"normal_case",
"stress_case"
]
defextract_test_cases(self, model_output: str) -> List[Tuple]:
"""Extract test cases from model output."""import re
test_cases = []
lines = model_output.split("\n")
for i, line inenumerate(lines):
if"test"in line.lower() or"case"in line.lower():
# Look for input/output patternsif i + 1 < len(lines):
match = re.search(r"(\[.*?\]|{.*?})", lines[i + 1])
ifmatch:
test_cases.append(lines[i + 1])
return test_cases
defcategorize_test_case(self, test_case: str, problem_constraints: Dict) -> str:
"""Classify test case as edge, boundary, normal, or stress."""ifany(val == 0or val == 1for val in test_case ifisinstance(val, int)):
return"boundary_case"ifany(str(val) in test_case for val in problem_constraints.get("limits", [])):
return"edge_case"# Rough heuristic: large inputs are stress casesiflen(str(test_case)) > 100:
return"stress_case"return"normal_case"defscore_test_coverage(self, test_cases: List[str], problem_constraints: Dict) -> float:
"""Score diversity of test case coverage."""ifnot test_cases:
return0.0
categories = [self.categorize_test_case(tc, problem_constraints) for tc in test_cases]
unique_categories = len(set(categories))
# Higher diversity score
coverage_score = unique_categories / len(self.case_categories)
returnmin(coverage_score + len(test_cases) * 0.05, 1.0)
Step 4: Integrated Triangle Evaluation and Ensemble
This combines all three dimensions and demonstrates ensemble improvement through diversity.
from dataclasses import dataclass
from typing importListimport numpy as np
@dataclassclassTriangleDimensions:
editorial_score: float
code_score: float
cases_score: float @propertydefoverall_score(self) -> float:
return np.mean([self.editorial_score, self.code_score, self.cases_score])
classCodingTriangleEvaluator:
def__init__(self):
self.editorial_eval = EditorialEvaluator()
self.code_eval = CodeEvaluator()
self.cases_eval = CasesEvaluator()
defevaluate_response(self, model_output: str, problem: Dict, test_cases: List[Tuple]) -> TriangleDimensions:
"""Evaluate LLM response across all three dimensions."""# Editorial dimension
editorial = self.editorial_eval.extract_editorial(model_output)
editorial_score = self.editorial_eval.score_editorial(editorial, problem)
# Code dimension
code = self.code_eval.extract_code(model_output)
test_results = self.code_eval.test_code(code, test_cases)
code_score = self.code_eval.score_code(test_results)
# Cases dimension
generated_cases = self.cases_eval.extract_test_cases(model_output)
cases_score = self.cases_eval.score_test_coverage(generated_cases, problem)
return TriangleDimensions(
editorial_score=editorial_score,
code_score=code_score,
cases_score=cases_score
)
defensemble_evaluation(self, model_outputs: List[str], problem: Dict, test_cases: List[Tuple]) -> Dict:
"""Evaluate multiple models and combine results via diversity voting."""
all_scores = []
all_codes = []
for output in model_outputs:
scores = self.evaluate_response(output, problem, test_cases)
all_scores.append(scores)
all_codes.append(self.code_eval.extract_code(output))
# Select most diverse solution set
unique_codes = list(set(all_codes))
best_code = None
best_test_rate = 0for code in unique_codes:
results = self.code_eval.test_code(code, test_cases)
test_rate = self.code_eval.score_code(results)
if test_rate > best_test_rate:
best_test_rate = test_rate
best_code = code
avg_editorial = np.mean([s.editorial_score for s in all_scores])
avg_cases = np.mean([s.cases_score for s in all_scores])
return {
"ensemble_code_score": best_test_rate,
"avg_editorial_score": avg_editorial,
"avg_cases_score": avg_cases,
"solution_diversity": len(unique_codes) / len(all_codes),
"best_solution": best_code
}
Practical Guidance
Hyperparameters and Configuration
Parameter
Recommended Value
Range
Notes
Ensemble Size
3-5 models
1-10
Diminishing diversity returns beyond 5
Test Case Diversity Threshold
0.6-0.8
0.4-1.0
Higher threshold ensures coverage of edge/stress cases
Editorial Completeness Threshold
0.7
0.5-1.0
Minimum required components for problem understanding
Timeout per Test Case
5 seconds
1-30
Catches infinite loops without excessive waiting
Code Extraction Confidence
0.8
0.5-1.0
Minimum confidence required to test extracted code
When to Use
Competitive programming evaluation pipelines where deep analysis of model capabilities is needed
Identifying which reasoning dimensions cause LLM failures on coding tasks
Building self-improving code systems that target specific dimensional weaknesses
Creating diverse solution ensembles for more robust code generation
Benchmarking LLM progress on fine-grained coding abilities rather than just pass rates
When NOT to Use
Not suitable when only final code correctness matters (use simpler pass/fail metrics)
Avoid for production code generation systems without extensive validation safeguards
Not recommended for real-time code systems with strict latency requirements (evaluation is slow)
Do not use as a replacement for human code review in production systems
Common Pitfalls
Ignoring distribution shift: Model-generated solutions differ systematically from human solutions. Measure this gap rather than assuming equivalence.
Overweighting ensemble diversity: More models don't automatically improve results. Measure actual improvement and stop at diminishing returns.
Poor code extraction: Fragile regex-based extraction fails on valid code formatting variations. Use robust parsing libraries.
Incomplete test coverage: Generated test cases often miss subtle edge cases. Always include domain expert-designed test cases alongside model-generated ones.
Treating dimensions as independent: Editorial, Code, and Cases dimensions interact. A poor understanding propagates to code quality. Address root problems, not symptoms.
