메뉴
Map changelog entries and release notes to affected test cases, ensuring every user-facing change has corresponding test coverage verification.
기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.
설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.
npx skills add https://github.com/PramodDutta/qaskills --skill changelog-test-mapper명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.
로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.
Use when publishing SEO blog articles to qaskills.sh, e.g. "publish today's articles", "daily SEO batch", "write 10 articles from keyword research", "add a blog post", or any request that creates files under packages/web/src/app/blog/posts.
Generate test data from the schemas you already have. Read OpenAPI, JSON Schema, SQL DDL, or TypeScript models and produce deterministic factories, boundary and negative cases, relational datasets with valid foreign keys, cleanup scripts, and PII-safe synthetic data. Production records never leave the machine.
Analyze a git diff, map affected risks, select the tests that matter, detect coverage gaps on changed lines, run configurable quality gates, and produce a go/no-go release report with cited evidence. Recommends only; never merges or deploys.
SOC 직업 분류 기준
SKILL.md 표시 중
| name | Changelog Test Mapper |
| description | Map changelog entries and release notes to affected test cases, ensuring every user-facing change has corresponding test coverage verification. |
| version | 1.0.0 |
| author | Pramod |
| license | MIT |
| tags | ["changelog","release-notes","test-mapping","traceability","impact-analysis","coverage"] |
| testingTypes | ["code-quality"] |
| frameworks | [] |
| languages | ["typescript","javascript"] |
| domains | ["devops"] |
| agents | ["claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp"] |
Every user-facing change in a software release should have corresponding test coverage that verifies the change works as intended. In practice, this mapping between changelog entries and test cases is rarely maintained, leading to releases where features ship without adequate verification and bug fixes lack regression tests. The Changelog Test Mapper bridges this gap by programmatically analyzing changelog entries, release notes, and commit histories, then mapping them to existing test cases and identifying coverage gaps. This skill covers the full pipeline: parsing changelogs in various formats, extracting change metadata, building a bidirectional mapping between changes and tests, scoring coverage completeness, and integrating the analysis into release workflows to prevent untested changes from reaching production.
The fundamental premise of changelog-test mapping is that if a change is important enough to appear in the changelog, it is important enough to have a test. Bug fixes need regression tests. New features need functional tests. Performance improvements need benchmark tests. Changes without tests are unverified promises.
The mapping must work in both directions. Given a changelog entry, you should be able to find all related tests. Given a test, you should be able to find which changelog entries it covers. This bidirectional traceability enables both forward analysis ("is this release well-tested?") and backward analysis ("what does this test protect?").
Manually maintaining a spreadsheet of change-to-test mappings does not scale. The mapping should be derived automatically from commit messages, PR descriptions, issue tracker links, and file-level change analysis. Manual annotations should supplement, not replace, automated discovery.
Not all changelog entries need the same level of test coverage. A breaking API change requires more comprehensive testing than a documentation fix. The coverage scoring model must account for change severity, affected surface area, and historical defect rates.
The mapping analysis should run as part of the release process, not as an afterthought. A release checklist that includes "all changelog entries have mapped tests" prevents untested changes from shipping.
changelog-test-mapper/
src/
parsers/
changelog-parser.ts
conventional-commits-parser.ts
github-releases-parser.ts
keep-a-changelog-parser.ts
analyzers/
change-classifier.ts
test-finder.ts
coverage-mapper.ts
gap-detector.ts
mappers/
file-change-mapper.ts
semantic-mapper.ts
annotation-mapper.ts
reporters/
coverage-reporter.ts
gap-reporter.ts
release-readiness-reporter.ts
config/
mapper-config.ts
index.ts
scripts/
map-release.ts
check-coverage.ts
generate-report.ts
tests/
parsers/
analyzers/
mappers/
package.json
tsconfig.json
Changelogs come in many formats. The parser supports Keep a Changelog, conventional commits, and GitHub release formats:
// src/parsers/changelog-parser.ts
export interface ChangelogEntry {
id: string;
type: 'added' | 'changed' | 'deprecated' | 'removed' | 'fixed' | 'security' | 'performance';
description: string;
version: string;
date: string;
scope?: string;
breakingChange: boolean;
issueRefs: string[];
prRefs: string[];
commitShas: string[];
affectedFiles: string[];
severity: 'critical' | 'major' | 'minor' | 'patch';
}
export interface ParsedChangelog {
versions: Array<{
version: string;
date: string;
entries: ChangelogEntry[];
}>;
}
export function parseKeepAChangelog(content: string): ParsedChangelog {
const versions: ParsedChangelog['versions'] = [];
const versionRegex = /^## \[([^\]]+)\](?: - (\d{4}-\d{2}-\d{2}))?/gm;
const typeRegex = /^### (Added|Changed|Deprecated|Removed|Fixed|Security)/gm;
const entryRegex = /^- (.+)$/gm;
const sections = content.split(/^## /gm).filter(Boolean);
for (const section of sections) {
const versionMatch = section.match(/^\[([^\]]+)\](?: - (\d{4}-\d{2}-\d{2}))?/);
if (!versionMatch) continue;
const version = versionMatch[1];
const date = versionMatch[2] || '';
const entries: ChangelogEntry[] = [];
const typeSections = section.split(/^### /gm).filter(Boolean);
for (const typeSection of typeSections.slice(1)) {
const typeMatch = typeSection.match(/^(Added|Changed|Deprecated|Removed|Fixed|Security)/);
if (!typeMatch) continue;
const type = typeMatch[1].toLowerCase() as ChangelogEntry['type'];
const lines = typeSection.split('\n').filter((l) => l.startsWith('- '));
for (const line of lines) {
const description = line.replace(/^- /, '').trim();
const issueRefs = extractIssueRefs(description);
const prRefs = extractPRRefs(description);
const breakingChange = /breaking|BREAKING/.test(description);
entries.push({
id: generateEntryId(version, type, description),
type,
description,
version,
date,
breakingChange,
issueRefs,
prRefs,
commitShas: [],
affectedFiles: [],
severity: classifySeverity(type, breakingChange),
});
}
}
versions.push({ version, date, entries });
}
return { versions };
}
function extractIssueRefs(text: string): string[] {
const matches = text.match(/#(\d+)/g) || [];
return matches.map((m) => m.replace('#', ''));
}
function extractPRRefs(text: string): string[] {
const matches = text.match(/\bPR[: ]?#?(\d+)/gi) || [];
return matches.map((m) => m.replace(/\bPR[: ]?#?/i, ''));
}
function generateEntryId(version: string, type: string, description: string): string {
const hash = description
.toLowerCase()
.replace(/[^a-z0-9]/g, '')
.substring(0, 20);
return `${version}-${type}-${hash}`;
}
function classifySeverity(
type: ChangelogEntry['type'],
breakingChange: boolean
): ChangelogEntry['severity'] {
if (breakingChange) return 'critical';
if (type === 'security') return 'critical';
if (type === 'fixed') return 'major';
if (type === 'removed' || type === 'deprecated') return 'major';
if (type === 'changed') return 'minor';
return 'patch';
}
// src/parsers/conventional-commits-parser.ts
import type { ChangelogEntry } from './changelog-parser';
interface ConventionalCommit {
sha: string;
type: string;
scope?: string;
description: string;
body?: string;
breaking: boolean;
issueRefs: string[];
files: string[];
}
export function parseConventionalCommits(
commits: ConventionalCommit[],
version: string
): ChangelogEntry[] {
const typeMapping: Record<string, ChangelogEntry['type']> = {
feat: 'added',
fix: 'fixed',
perf: 'performance',
security: 'security',
refactor: 'changed',
deprecate: 'deprecated',
};
return commits
.filter((commit) => typeMapping[commit.type])
.map((commit) => ({
id: `${version}-${commit.sha.substring(0, 8)}`,
type: typeMapping[commit.type] || 'changed',
description: commit.description,
version,
date: new Date().toISOString().split('T')[0],
scope: commit.scope,
breakingChange: commit.breaking,
issueRefs: commit.issueRefs,
prRefs: [],
commitShas: [commit.sha],
affectedFiles: commit.files,
severity: classifyCommitSeverity(commit),
}));
}
function classifyCommitSeverity(commit: ConventionalCommit): ChangelogEntry['severity'] {
if (commit.breaking) return 'critical';
if (commit.type === 'security') return 'critical';
if (commit.type === 'fix') return 'major';
if (commit.type === 'feat') return 'minor';
return 'patch';
}
The file-based mapper connects changelog entries to tests by analyzing which source files changed and finding tests that import or reference those files:
// src/mappers/file-change-mapper.ts
import { execSync } from 'child_process';
import { readFileSync, existsSync } from 'fs';
import { join, basename, dirname } from 'path';
interface TestMapping {
changeEntryId: string;
changeDescription: string;
mappedTests: Array<{
testFile: string;
testName: string;
confidence: 'high' | 'medium' | 'low';
reason: string;
}>;
unmappedFiles: string[];
coverageScore: number;
}
export function mapChangesToTests(
affectedFiles: string[],
testFiles: string[],
changeEntryId: string,
changeDescription: string
): TestMapping {
const mappedTests: TestMapping['mappedTests'] = [];
const mappedSourceFiles = new Set<string>();
const unmappedFiles: string[] = [];
for (const sourceFile of affectedFiles) {
// Skip test files themselves
if (isTestFile(sourceFile)) continue;
let hasMapping = false;
// Strategy 1: Co-located test files
const colocatedTest = findColocatedTest(sourceFile, testFiles);
if (colocatedTest) {
mappedTests.push({
testFile: colocatedTest,
testName: `Tests for ${basename(sourceFile)}`,
confidence: 'high',
reason: 'Co-located test file',
});
mappedSourceFiles.add(sourceFile);
hasMapping = true;
}
// Strategy 2: Convention-based test discovery
const conventionTests = findConventionBasedTests(sourceFile, testFiles);
for (const testFile of conventionTests) {
if (!mappedTests.some((m) => m.testFile === testFile)) {
mappedTests.push({
testFile,
testName: `Tests for ${basename(sourceFile)}`,
confidence: 'medium',
reason: 'Naming convention match',
});
mappedSourceFiles.add(sourceFile);
hasMapping = true;
}
}
// Strategy 3: Import analysis
const importTests = findTestsByImport(sourceFile, testFiles);
for (const testFile of importTests) {
if (!mappedTests.some((m) => m.testFile === testFile)) {
mappedTests.push({
testFile,
testName: `Imports ${basename(sourceFile)}`,
confidence: 'medium',
reason: 'Import dependency',
});
mappedSourceFiles.add(sourceFile);
hasMapping = true;
}
}
if (!hasMapping) {
unmappedFiles.push(sourceFile);
}
}
const totalSourceFiles = affectedFiles.filter((f) => !isTestFile(f)).length;
const coverageScore =
totalSourceFiles > 0 ? mappedSourceFiles.size / totalSourceFiles : 1;
return {
changeEntryId,
changeDescription,
mappedTests,
unmappedFiles,
coverageScore,
};
}
function isTestFile(filePath: string): boolean {
const name = basename(filePath);
return (
name.includes('.test.') ||
name.includes('.spec.') ||
name.includes('__tests__') ||
filePath.includes('__tests__/')
);
}
function findColocatedTest(sourceFile: string, testFiles: string[]): string | null {
const dir = dirname(sourceFile);
const name = basename(sourceFile).replace(/\.(ts|tsx|js|jsx)$/, '');
const patterns = [
join(dir, `${name}.test.ts`),
join(dir, `${name}.test.tsx`),
join(dir, `${name}.spec.ts`),
join(dir, `${name}.spec.tsx`),
join(dir, '__tests__', `${name}.test.ts`),
join(dir, '__tests__', `${name}.test.tsx`),
];
for (const pattern of patterns) {
if (testFiles.includes(pattern)) {
return pattern;
}
}
return null;
}
function findConventionBasedTests(sourceFile: string, testFiles: string[]): string[] {
const name = basename(sourceFile).replace(/\.(ts|tsx|js|jsx)$/, '');
const matches: string[] = [];
for (const testFile of testFiles) {
const testName = basename(testFile).replace(/\.(test|spec)\.(ts|tsx|js|jsx)$/, '');
if (testName === name) {
matches.push(testFile);
}
}
return matches;
}
function findTestsByImport(sourceFile: string, testFiles: string[]): string[] {
const matches: string[] = [];
const sourceBaseName = basename(sourceFile).replace(/\.(ts|tsx|js|jsx)$/, '');
for (const testFile of testFiles) {
if (!existsSync(testFile)) continue;
try {
const content = readFileSync(testFile, 'utf-8');
if (
content.includes(`from './${sourceBaseName}'`) ||
content.includes(`from '../${sourceBaseName}'`) ||
content.includes(`require('./${sourceBaseName}')`) ||
content.includes(`/${sourceBaseName}'`)
) {
matches.push(testFile);
}
} catch {
// Skip files that cannot be read
}
}
return matches;
}
The semantic mapper uses keyword and pattern analysis to connect changelog descriptions to test names:
// src/mappers/semantic-mapper.ts
interface SemanticMatch {
testFile: string;
testName: string;
confidence: number;
matchedTerms: string[];
}
export function findSemanticMatches(
changeDescription: string,
testIndex: Array<{ file: string; name: string; description: string }>
): SemanticMatch[] {
const changeTerms = extractKeyTerms(changeDescription);
const matches: SemanticMatch[] = [];
for (const test of testIndex) {
const testTerms = extractKeyTerms(`${test.name} ${test.description}`);
const matchedTerms = changeTerms.filter((term) =>
testTerms.some(
(testTerm) =>
testTerm.includes(term) || term.includes(testTerm) || levenshteinSimilarity(term, testTerm) > 0.8
)
);
if (matchedTerms.length > 0) {
const confidence = matchedTerms.length / Math.max(changeTerms.length, 1);
matches.push({
testFile: test.file,
testName: test.name,
confidence: Math.min(1, confidence),
matchedTerms,
});
}
}
return matches.sort((a, b) => b.confidence - a.confidence);
}
function extractKeyTerms(text: string): string[] {
const stopWords = new Set([
'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could',
'should', 'may', 'might', 'can', 'shall', 'to', 'of', 'in', 'for',
'on', 'with', 'at', 'by', 'from', 'as', 'into', 'through', 'during',
'before', 'after', 'above', 'below', 'between', 'and', 'but', 'or',
'not', 'no', 'nor', 'so', 'yet', 'both', 'each', 'all', 'any',
'few', 'more', 'most', 'other', 'some', 'such', 'than', 'too',
'very', 'just', 'about', 'when', 'where', 'how', 'what', 'which',
'who', 'whom', 'this', 'that', 'these', 'those', 'it', 'its',
'add', 'update', 'fix', 'remove', 'change', 'new', 'now',
]);
return text
.toLowerCase()
.replace(/[^a-z0-9\s-]/g, ' ')
.split(/[\s-]+/)
.filter((word) => word.length > 2 && !stopWords.has(word));
}
function levenshteinSimilarity(a: string, b: string): number {
const maxLen = Math.max(a.length, b.length);
if (maxLen === 0) return 1;
const matrix: number[][] = Array(a.length + 1)
.fill(null)
.map(() => Array(b.length + 1).fill(0));
for (let i = 0; i <= a.length; i++) matrix[i][0] = i;
for (let j = 0; j <= b.length; j++) matrix[0][j] = j;
for (let i = 1; i <= a.length; i++) {
for (let j = 1; j <= b.length; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
matrix[i][j] = Math.min(
matrix[i - 1][j] + 1,
matrix[i][j - 1] + 1,
matrix[i - 1][j - 1] + cost
);
}
}
return 1 - matrix[a.length][b.length] / maxLen;
}
// src/analyzers/gap-detector.ts
import type { ChangelogEntry } from '../parsers/changelog-parser';
interface CoverageGap {
entry: ChangelogEntry;
gapType: 'no-tests' | 'low-confidence' | 'partial-coverage' | 'missing-regression';
riskLevel: 'critical' | 'high' | 'medium' | 'low';
recommendation: string;
}
interface ReleaseReadiness {
version: string;
totalEntries: number;
coveredEntries: number;
coveragePercentage: number;
gaps: CoverageGap[];
readyForRelease: boolean;
riskScore: number;
blockers: string[];
warnings: string[];
}
export function analyzeReleaseReadiness(
entries: ChangelogEntry[],
mappings: Map<string, { coverageScore: number; mappedTests: unknown[] }>,
thresholds: { minCoverage: number; criticalRequireTests: boolean }
): ReleaseReadiness {
const gaps: CoverageGap[] = [];
const blockers: string[] = [];
const warnings: string[] = [];
let coveredEntries = 0;
for (const entry of entries) {
const mapping = mappings.get(entry.id);
const coverageScore = mapping?.coverageScore || 0;
const hasTests = mapping && mapping.mappedTests.length > 0;
if (coverageScore >= 0.8) {
coveredEntries++;
continue;
}
let gapType: CoverageGap['gapType'];
let recommendation: string;
if (!hasTests) {
gapType = 'no-tests';
recommendation = `Add test coverage for: ${entry.description}`;
} else if (coverageScore < 0.3) {
gapType = 'low-confidence';
recommendation = `Improve test mapping confidence for: ${entry.description}. Current tests may not adequately cover this change.`;
} else if (entry.type === 'fixed' && coverageScore < 0.8) {
gapType = 'missing-regression';
recommendation = `Add specific regression test for bug fix: ${entry.description}`;
} else {
gapType = 'partial-coverage';
recommendation = `Increase test coverage for: ${entry.description}. Currently at ${(coverageScore * 100).toFixed(0)}%.`;
}
const riskLevel = assessRisk(entry, coverageScore);
gaps.push({ entry, gapType, riskLevel, recommendation });
if (riskLevel === 'critical') {
blockers.push(`${entry.type.toUpperCase()}: ${entry.description} - no adequate test coverage`);
} else if (riskLevel === 'high') {
warnings.push(`${entry.type.toUpperCase()}: ${entry.description} - insufficient test coverage`);
}
}
const coveragePercentage =
entries.length > 0 ? (coveredEntries / entries.length) * 100 : 100;
const criticalUntested = gaps.some(
(g) => g.riskLevel === 'critical' && thresholds.criticalRequireTests
);
const readyForRelease =
coveragePercentage >= thresholds.minCoverage && !criticalUntested;
const riskScore = calculateRiskScore(gaps, entries.length);
return {
version: entries[0]?.version || 'unknown',
totalEntries: entries.length,
coveredEntries,
coveragePercentage: Math.round(coveragePercentage * 10) / 10,
gaps,
readyForRelease,
riskScore,
blockers,
warnings,
};
}
function assessRisk(
entry: ChangelogEntry,
coverageScore: number
): CoverageGap['riskLevel'] {
if (entry.severity === 'critical' && coverageScore < 0.5) return 'critical';
if (entry.breakingChange && coverageScore < 0.8) return 'critical';
if (entry.type === 'security' && coverageScore < 0.8) return 'critical';
if (entry.type === 'fixed' && coverageScore < 0.3) return 'high';
if (entry.severity === 'major' && coverageScore < 0.5) return 'high';
if (coverageScore < 0.3) return 'medium';
return 'low';
}
function calculateRiskScore(gaps: CoverageGap[], totalEntries: number): number {
if (totalEntries === 0) return 0;
const riskWeights = { critical: 4, high: 2, medium: 1, low: 0.5 };
const totalRisk = gaps.reduce((sum, gap) => sum + riskWeights[gap.riskLevel], 0);
const maxRisk = totalEntries * riskWeights.critical;
return Math.min(1, totalRisk / maxRisk);
}
// src/reporters/release-readiness-reporter.ts
import type { ReleaseReadiness } from '../analyzers/gap-detector';
export function generateReleaseReport(readiness: ReleaseReadiness): string {
const lines: string[] = [];
const status = readiness.readyForRelease ? 'READY' : 'NOT READY';
lines.push(`# Release Readiness Report: v${readiness.version}`);
lines.push('');
lines.push(`**Status:** ${status}`);
lines.push(`**Coverage:** ${readiness.coveragePercentage}% of changelog entries have mapped tests`);
lines.push(`**Risk Score:** ${(readiness.riskScore * 100).toFixed(1)}%`);
lines.push(`**Entries:** ${readiness.coveredEntries}/${readiness.totalEntries} covered`);
lines.push('');
if (readiness.blockers.length > 0) {
lines.push('## Blockers');
lines.push('');
for (const blocker of readiness.blockers) {
lines.push(`- ${blocker}`);
}
lines.push('');
}
if (readiness.warnings.length > 0) {
lines.push('## Warnings');
lines.push('');
for (const warning of readiness.warnings) {
lines.push(`- ${warning}`);
}
lines.push('');
}
if (readiness.gaps.length > 0) {
lines.push('## Coverage Gaps');
lines.push('');
lines.push('| Change | Type | Risk | Gap Type | Recommendation |');
lines.push('|--------|------|------|----------|----------------|');
for (const gap of readiness.gaps) {
const desc =
gap.entry.description.length > 50
? gap.entry.description.substring(0, 50) + '...'
: gap.entry.description;
lines.push(
`| ${desc} | ${gap.entry.type} | ${gap.riskLevel} | ${gap.gapType} | ${gap.recommendation.substring(0, 60)}... |`
);
}
}
return lines.join('\n');
}
name: Release Readiness Check
on:
pull_request:
branches: [main]
paths:
- 'CHANGELOG.md'
jobs:
check-readiness:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install dependencies
run: npm ci
- name: Run changelog test mapping
run: tsx scripts/map-release.ts
env:
MIN_COVERAGE: '80'
REQUIRE_CRITICAL_TESTS: 'true'
- name: Comment PR with readiness report
if: always()
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
if (fs.existsSync('release-readiness-report.md')) {
const report = fs.readFileSync('release-readiness-report.md', 'utf-8');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: report,
});
}
// scripts/map-release.ts
import { readFileSync, writeFileSync } from 'fs';
import { execSync } from 'child_process';
import { parseKeepAChangelog } from '../src/parsers/changelog-parser';
import { mapChangesToTests } from '../src/mappers/file-change-mapper';
import { findSemanticMatches } from '../src/mappers/semantic-mapper';
import { analyzeReleaseReadiness } from '../src/analyzers/gap-detector';
import { generateReleaseReport } from '../src/reporters/release-readiness-reporter';
import { join } from 'path';
async function main(): Promise<void> {
const changelogContent = readFileSync('CHANGELOG.md', 'utf-8');
const changelog = parseKeepAChangelog(changelogContent);
if (changelog.versions.length === 0) {
console.log('No versions found in changelog');
process.exit(0);
}
const latestVersion = changelog.versions[0];
console.log(
`Analyzing release v${latestVersion.version} (${latestVersion.entries.length} entries)`
);
// Discover all test files in the project
const testFilesOutput = execSync(
'find . -name "*.test.ts" -o -name "*.spec.ts" -o -name "*.test.tsx" -o -name "*.spec.tsx" | grep -v node_modules',
{ encoding: 'utf-8' }
);
const testFiles = testFilesOutput.trim().split('\n').filter(Boolean);
// Map each changelog entry to tests
const mappings = new Map<string, { coverageScore: number; mappedTests: unknown[] }>();
for (const entry of latestVersion.entries) {
// Get affected files from git if commit SHAs are available
let affectedFiles = entry.affectedFiles;
if (affectedFiles.length === 0 && entry.commitShas.length > 0) {
for (const sha of entry.commitShas) {
try {
const files = execSync(`git diff-tree --no-commit-id --name-only -r ${sha}`, {
encoding: 'utf-8',
});
affectedFiles.push(...files.trim().split('\n').filter(Boolean));
} catch {
// Commit may not exist in current history
}
}
}
const fileMapping = mapChangesToTests(
affectedFiles,
testFiles,
entry.id,
entry.description
);
mappings.set(entry.id, {
coverageScore: fileMapping.coverageScore,
mappedTests: fileMapping.mappedTests,
});
}
const minCoverage = parseInt(process.env.MIN_COVERAGE || '80', 10);
const requireCriticalTests = process.env.REQUIRE_CRITICAL_TESTS === 'true';
const readiness = analyzeReleaseReadiness(latestVersion.entries, mappings, {
minCoverage,
criticalRequireTests: requireCriticalTests,
});
const report = generateReleaseReport(readiness);
writeFileSync('release-readiness-report.md', report, 'utf-8');
console.log(`Coverage: ${readiness.coveragePercentage}%`);
console.log(`Gaps: ${readiness.gaps.length}`);
console.log(`Blockers: ${readiness.blockers.length}`);
console.log(`Ready: ${readiness.readyForRelease}`);
if (!readiness.readyForRelease) {
console.error('Release is not ready. See report for details.');
process.exit(1);
}
}
main().catch((error) => {
console.error('Release mapping failed:', error);
process.exit(1);
});
Enable developers to explicitly link tests to changelog entries and issues:
// src/mappers/annotation-mapper.ts
interface TestAnnotation {
testFile: string;
testName: string;
covers: string[]; // Issue IDs: ["PROJ-123", "PROJ-456"]
verifies: string[]; // Changelog entry descriptions or IDs
regression: string; // Bug ID this test prevents regression for
}
export function parseTestAnnotations(testFileContent: string): TestAnnotation[] {
const annotations: TestAnnotation[] = [];
// Match @covers, @verifies, @regression JSDoc-style annotations
const testBlockRegex = /(?:\/\*\*[\s\S]*?\*\/\s*)?(?:it|test)\s*\(\s*['"`]([^'"`]+)['"`]/g;
const coversRegex = /@covers\s+(.+)/g;
const verifiesRegex = /@verifies\s+(.+)/g;
const regressionRegex = /@regression\s+(.+)/g;
let match;
while ((match = testBlockRegex.exec(testFileContent)) !== null) {
const testName = match[1];
const precedingBlock = testFileContent.substring(
Math.max(0, match.index - 500),
match.index
);
const covers: string[] = [];
const verifies: string[] = [];
let regression = '';
let annotationMatch;
while ((annotationMatch = coversRegex.exec(precedingBlock)) !== null) {
covers.push(...annotationMatch[1].split(',').map((s) => s.trim()));
}
while ((annotationMatch = verifiesRegex.exec(precedingBlock)) !== null) {
verifies.push(...annotationMatch[1].split(',').map((s) => s.trim()));
}
if ((annotationMatch = regressionRegex.exec(precedingBlock)) !== null) {
regression = annotationMatch[1].trim();
}
if (covers.length > 0 || verifies.length > 0 || regression) {
annotations.push({
testFile: '',
testName,
covers,
verifies,
regression,
});
}
}
return annotations;
}
Usage in test files:
/**
* @covers PROJ-123
* @verifies User can reset password via email link
* @regression BUG-456
*/
test('password reset flow sends email and allows reset', async ({ page }) => {
await page.goto('/forgot-password');
await page.getByLabel('Email').fill('test@example.com');
await page.getByRole('button', { name: 'Send Reset Link' }).click();
await expect(page.getByText('Reset link sent')).toBeVisible();
});
Parse changelogs automatically on every PR that modifies CHANGELOG.md. This ensures the mapping analysis runs before release decisions are made, not after.
Use conventional commits to generate structured changelog data. Conventional commits provide machine-parseable change metadata (type, scope, breaking changes) that dramatically improves mapping accuracy.
Require explicit test annotations for breaking changes. Breaking changes have the highest risk of user impact. Require developers to annotate at least one test with @verifies for every breaking change entry.
Combine multiple mapping strategies. No single mapping strategy catches everything. Use file-based mapping, import analysis, and semantic matching together, then merge results with confidence scores.
Set release gates based on coverage thresholds. Block releases when changelog-to-test coverage falls below a configurable threshold. Start at 50% and increase as the team matures.
Track coverage trends across releases. Compare mapping coverage between releases to identify whether the team is improving or declining in test discipline.
Generate reports as PR comments. Posting the readiness report directly on the PR makes coverage gaps visible during code review, when they can still be addressed.
Maintain a test index for semantic matching. Periodically build an index of all test names and descriptions to enable efficient semantic matching against changelog entries.
Differentiate between types of coverage. A bug fix needs a specific regression test. A new feature needs functional tests. A performance improvement needs a benchmark. The mapping should verify the right type of test exists.
Use issue tracker references as primary mapping keys. When both changelog entries and tests reference the same Jira ticket or GitHub issue, the mapping confidence is highest. Encourage developers to include issue references in both places.
Audit the mapping regularly. Periodically review false positives and false negatives in the mapping to improve parser accuracy and mapping strategies.
Treating changelog-test mapping as optional documentation. If the mapping is not enforced in CI, it will be ignored. Treat it as a release gate, not a nice-to-have report.
Relying solely on file-based mapping. File-based mapping misses behavioral connections. A change to a shared utility function may affect many features, but only tests that directly import the utility will appear in file-based mapping.
Mapping quantity over quality. Having 10 low-confidence test mappings for a changelog entry is not the same as having 1 high-confidence mapping. Weight confidence scores when calculating coverage.
Ignoring security changelog entries. Security changes are the highest-risk entries in any changelog. Never skip test mapping for security fixes and vulnerability patches.
Running the analysis only at release time. By the time the release is ready, it is too late to write missing tests without delaying the release. Run the analysis continuously on PRs.
Hard-coding changelog formats. Support multiple changelog formats from the start. Teams migrate between formats, and the tool should adapt without requiring rewriting.
Parser finds no entries in the changelog. Check the exact format of your CHANGELOG.md. The Keep a Changelog parser expects specific heading formats: ## [version] - date for versions and ### Added for types. Verify that your file matches the expected format exactly.
File-based mapping produces zero matches. Check that your test files follow naming conventions that the mapper recognizes. If tests are in a __tests__ directory with different names than source files, the convention-based strategy will miss them. Consider adding import analysis or explicit annotations.
Semantic matching produces too many false positives. Tighten the confidence threshold. If the default threshold of 0.3 produces noise, increase it to 0.5 or 0.6. Also review and expand the stop words list to filter common words that create spurious matches.
Coverage score seems artificially high. The score may be inflated if many changelog entries have no affected files listed. Ensure that commit SHAs or file paths are populated for each entry so the mapper has data to work with.
The release gate blocks a valid release. If the mapping legitimately cannot find tests for a low-risk change (e.g., documentation update), add an exclusion mechanism. Allow entries to be marked as @no-test-needed with a justification.