Optimize CI test pipelines through intelligent test splitting, parallelization, caching strategies, and selective test execution based on code changes.
Optimize CI test pipelines through intelligent test splitting, parallelization, caching strategies, and selective test execution based on code changes.
Slow CI pipelines are one of the most common productivity killers in software teams. A test suite that takes 30 minutes to run means developers context-switch away from their work, batch multiple changes into fewer PRs to avoid waiting, and eventually start skipping CI altogether. This skill addresses CI pipeline performance through four complementary strategies: intelligent test splitting across parallel workers, selective test execution based on code changes, aggressive caching of dependencies and build artifacts, and pipeline architecture that minimizes total wall-clock time. The techniques apply to any CI system but use GitHub Actions as the primary example, with patterns that transfer to GitLab CI, CircleCI, Jenkins, and other platforms.
Core Principles
1. Wall-Clock Time Is the Only Metric That Matters
CPU time, billable minutes, and total test count are secondary metrics. The developer waits for wall-clock time. A pipeline that uses 60 CPU-minutes across 10 parallel workers in 6 minutes is vastly preferable to a pipeline that uses 20 CPU-minutes sequentially in 20 minutes. Optimize for the duration the developer experiences.
2. Never Run Tests You Do Not Need
If a PR changes only documentation files, running the full test suite is waste. If a PR modifies only the frontend, running backend integration tests is waste. Selective test execution identifies the minimum set of tests needed to validate a specific change, with a safety net that defaults to running everything when the analysis is uncertain.
3. Cache Everything That Does Not Change
Dependencies, build artifacts, Docker layers, and browser binaries are the same across most CI runs. Downloading and building them from scratch on every run is unnecessary. Aggressive caching can eliminate minutes from every pipeline execution.
4. Split Tests by Duration, Not by File Count
Naive test splitting distributes files evenly across workers. But if one file contains a 5-minute test and another contains a 5-second test, the distribution is severely unbalanced. Intelligent splitting uses historical duration data to distribute tests so that all workers finish at approximately the same time.
5. Fail Fast, Report Completely
Run the fastest tests first. If unit tests catch a bug in 30 seconds, there is no reason to wait 10 minutes for e2e tests to also catch it. Structure the pipeline so the cheapest checks (lint, type-check) run first to provide the fastest possible feedback, but still collect complete results from all shards for thorough reporting.
The scripts/ directory contains automation tools for test splitting, change detection, cache management, and pipeline analysis. The config/ directory holds the mapping from file changes to test groups, cache key definitions, and test group specifications.
Test Splitting Strategies
Duration-Based Test Splitting
Split tests across parallel workers based on historical execution times so that each worker runs for approximately the same duration:
// scripts/split-tests.tsimport { readFileSync, writeFileSync, existsSync } from'fs';
import { execSync } from'child_process';
interfaceTestTiming {
file: string;
duration: number;
lastRun: string;
}
interfaceSplitResult {
shardIndex: number;
files: string[];
estimatedDuration: number;
}
exportfunctionsplitTestsByDuration(testFiles: string[],
shardCount: number,
timingsFile: string = 'test-timings.json'): SplitResult[] {
// Load historical timingsconst timings = loadTimings(timingsFile);
// Create a list of files with their estimated durationconstfileTimings: Array<{ file: string; duration: number }> = testFiles.map((file) => ({
file,
duration: timings.get(file) || estimateDefaultDuration(file),
}));
// Sort by duration descending (greedy algorithm: assign largest jobs first)
fileTimings.sort((a, b) => b.duration - a.duration);
// Initialize shardsconstshards: SplitResult[] = Array.from({ length: shardCount }, (_, i) => ({
shardIndex: i,
files: [],
estimatedDuration: 0,
}));
// Greedy assignment: always assign to the shard with the least total durationfor (const fileTiming of fileTimings) {
const lightest = shards.reduce((min, shard) =>
shard.estimatedDuration < min.estimatedDuration ? shard : min
);
lightest.files.push(fileTiming.file);
lightest.estimatedDuration += fileTiming.duration;
}
return shards;
}
functionloadTimings(timingsFile: string): Map<string, number> {
const timings = newMap<string, number>();
if (!existsSync(timingsFile)) {
return timings;
}
try {
constdata: TestTiming[] = JSON.parse(readFileSync(timingsFile, 'utf-8'));
for (const entry of data) {
timings.set(entry.file, entry.duration);
}
} catch {
console.warn(`Failed to parse timings file: ${timingsFile}`);
}
return timings;
}
functionestimateDefaultDuration(file: string): number {
// Heuristic estimates based on test type when no historical data existsif (file.includes('.e2e.') || file.includes('e2e/')) return30000;
if (file.includes('.integration.') || file.includes('integration/')) return10000;
if (file.includes('.spec.')) return5000;
return3000; // Default estimate for unit tests
}
// CLI entry point for use in GitHub Actionsfunctionmain(): void {
const shardIndex = parseInt(process.env.SHARD_INDEX || '0', 10);
const shardCount = parseInt(process.env.SHARD_COUNT || '1', 10);
// Discover all test filesconst testFilesOutput = execSync(
'find . -name "*.test.ts" -o -name "*.spec.ts" | grep -v node_modules',
{ encoding: 'utf-8' }
);
const testFiles = testFilesOutput.trim().split('\n').filter(Boolean);
const shards = splitTestsByDuration(testFiles, shardCount);
const myShard = shards[shardIndex];
if (!myShard) {
console.error(`Invalid shard index ${shardIndex} for ${shardCount} shards`);
process.exit(1);
}
console.log(
`Shard ${shardIndex + 1}/${shardCount}: ${myShard.files.length} files, ~${(myShard.estimatedDuration / 1000).toFixed(1)}s`
);
// Write shard files list for the test runner to consumewriteFileSync('shard-files.txt', myShard.files.join('\n'), 'utf-8');
}
main();
Collecting Test Timings
After each CI run, collect and store test execution timings for future split optimization:
// scripts/timing-collector.tsimport { readFileSync, writeFileSync, existsSync } from'fs';
interfaceJestResult {
testResults: Array<{
testFilePath: string;
perfStats: {
runtime: number;
};
}>;
}
interfacePlaywrightResult {
suites: Array<{
file: string;
specs: Array<{
tests: Array<{
results: Array<{
duration: number;
}>;
}>;
}>;
}>;
}
interfaceTestTiming {
file: string;
duration: number;
lastRun: string;
}
exportfunctioncollectJestTimings(resultsFile: string): TestTiming[] {
constresults: JestResult = JSON.parse(readFileSync(resultsFile, 'utf-8'));
return results.testResults.map((test) => ({
file: test.testFilePath.replace(process.cwd() + '/', ''),
duration: test.perfStats.runtime,
lastRun: newDate().toISOString(),
}));
}
exportfunctioncollectPlaywrightTimings(resultsFile: string): TestTiming[] {
constresults: PlaywrightResult = JSON.parse(readFileSync(resultsFile, 'utf-8'));
consttimings: TestTiming[] = [];
for (const suite of results.suites) {
let totalDuration = 0;
for (const spec of suite.specs) {
for (const test of spec.tests) {
for (const result of test.results) {
totalDuration += result.duration;
}
}
}
timings.push({
file: suite.file,
duration: totalDuration,
lastRun: newDate().toISOString(),
});
}
return timings;
}
exportfunctionmergeTimings(existing: TestTiming[],
latest: TestTiming[]
): TestTiming[] {
const map = newMap<string, TestTiming>();
for (const timing of existing) {
map.set(timing.file, timing);
}
// Merge with exponential moving average to smooth out outliersfor (const timing of latest) {
const prev = map.get(timing.file);
if (prev) {
// EMA with alpha = 0.3 gives 70% weight to history, 30% to new dataconst smoothedDuration = prev.duration * 0.7 + timing.duration * 0.3;
map.set(timing.file, {
file: timing.file,
duration: Math.round(smoothedDuration),
lastRun: timing.lastRun,
});
} else {
map.set(timing.file, timing);
}
}
returnArray.from(map.values());
}
functionmain(): void {
const timingsFile = 'test-timings.json';
constexisting: TestTiming[] = existsSync(timingsFile)
? JSON.parse(readFileSync(timingsFile, 'utf-8'))
: [];
letlatest: TestTiming[] = [];
if (existsSync('jest-results.json')) {
latest = collectJestTimings('jest-results.json');
} elseif (existsSync('playwright-results.json')) {
latest = collectPlaywrightTimings('playwright-results.json');
}
if (latest.length > 0) {
const merged = mergeTimings(existing, latest);
writeFileSync(timingsFile, JSON.stringify(merged, null, 2), 'utf-8');
console.log(`Updated timings for ${latest.length} test files (${merged.length} total)`);
} else {
console.warn('No test results found to collect timings from');
}
}
main();
Selective Test Execution
Change Detection Engine
Detect which files changed and determine which test groups need to run:
Pre-populate caches on a schedule so the first PR of the day gets warm caches:
# .github/workflows/cache-warmup.ymlname:CacheWarmupon:schedule:-cron:'0 6 * * 1-5'# Weekdays at 6 AM UTCworkflow_dispatch:jobs:warmup:runs-on:ubuntu-lateststeps:-uses:actions/checkout@v4-uses:pnpm/action-setup@v2with:version:9-uses:actions/setup-node@v4with:node-version:20-name:Installdependenciesrun:pnpminstall--frozen-lockfile-name:Cachenode_modulesuses:actions/cache/save@v4with:path:|
node_modules
~/.pnpm-store
key:pnpm-${{runner.os}}-${{hashFiles('pnpm-lock.yaml')}}-name:InstallPlaywrightbrowsersrun:npxplaywrightinstall--with-depschromium-name:CachePlaywrightuses:actions/cache/save@v4with:path:~/.cache/ms-playwrightkey:playwright-${{runner.os}}-${{hashFiles('pnpm-lock.yaml')}}-name:Buildapplicationrun:pnpmbuild-name:Cachebuildartifactsuses:actions/cache/save@v4with:path:|
.next/cache
dist
.turbo
key:build-${{runner.os}}-${{hashFiles('tsconfig.json')}}-${{github.sha}}
Pipeline Analysis and Monitoring
Pipeline Duration Analyzer
Track and analyze pipeline performance over time to identify regressions and optimization opportunities:
// scripts/pipeline-analyzer.tsinterfacePipelineRun {
runId: string;
branch: string;
timestamp: string;
totalDuration: number;
jobs: Array<{
name: string;
duration: number;
status: 'success' | 'failure' | 'cancelled';
steps: Array<{
name: string;
duration: number;
}>;
}>;
}
interfacePipelineAnalysis {
averageDuration: number;
medianDuration: number;
p95Duration: number;
slowestJobs: Array<{ name: string; avgDuration: number }>;
slowestSteps: Array<{ jobName: string; stepName: string; avgDuration: number }>;
cacheSavings: number;
recommendations: string[];
}
exportfunctionanalyzePipeline(runs: PipelineRun[]): PipelineAnalysis {
if (runs.length === 0) {
return {
averageDuration: 0,
medianDuration: 0,
p95Duration: 0,
slowestJobs: [],
slowestSteps: [],
cacheSavings: 0,
recommendations: [],
};
}
const durations = runs.map((r) => r.totalDuration).sort((a, b) => a - b);
const average = durations.reduce((a, b) => a + b, 0) / durations.length;
const median = durations[Math.floor(durations.length / 2)];
const p95 = durations[Math.floor(durations.length * 0.95)];
// Aggregate job durations across runsconst jobDurations = newMap<string, number[]>();
const stepDurations = newMap<string, number[]>();
for (const run of runs) {
for (const job of run.jobs) {
if (!jobDurations.has(job.name)) {
jobDurations.set(job.name, []);
}
jobDurations.get(job.name)!.push(job.duration);
for (const step of job.steps) {
const key = `${job.name}::${step.name}`;
if (!stepDurations.has(key)) {
stepDurations.set(key, []);
}
stepDurations.get(key)!.push(step.duration);
}
}
}
const slowestJobs = Array.from(jobDurations.entries())
.map(([name, times]) => ({
name,
avgDuration: times.reduce((a, b) => a + b, 0) / times.length,
}))
.sort((a, b) => b.avgDuration - a.avgDuration)
.slice(0, 5);
const slowestSteps = Array.from(stepDurations.entries())
.map(([key, times]) => {
const [jobName, stepName] = key.split('::');
return {
jobName,
stepName,
avgDuration: times.reduce((a, b) => a + b, 0) / times.length,
};
})
.sort((a, b) => b.avgDuration - a.avgDuration)
.slice(0, 10);
const recommendations = generateOptimizationRecommendations(
average,
slowestJobs,
slowestSteps
);
return {
averageDuration: Math.round(average),
medianDuration: Math.round(median),
p95Duration: Math.round(p95),
slowestJobs,
slowestSteps,
cacheSavings: estimateCacheSavings(stepDurations),
recommendations,
};
}
functiongenerateOptimizationRecommendations(avgDuration: number,
slowestJobs: Array<{ name: string; avgDuration: number }>,
slowestSteps: Array<{ jobName: string; stepName: string; avgDuration: number }>
): string[] {
constrecommendations: string[] = [];
if (avgDuration > 600000) {
recommendations.push(
'Pipeline average exceeds 10 minutes. Consider increasing parallelization or enabling selective test execution.'
);
}
for (const job of slowestJobs) {
if (job.avgDuration > 300000) {
recommendations.push(
`Job "${job.name}" averages ${(job.avgDuration / 60000).toFixed(1)}min. Consider splitting into shards.`
);
}
}
for (const step of slowestSteps) {
if (step.stepName.toLowerCase().includes('install') && step.avgDuration > 60000) {
recommendations.push(
`Step "${step.stepName}" in "${step.jobName}" takes ${(step.avgDuration / 1000).toFixed(0)}s. Verify dependency caching is working correctly.`
);
}
if (step.stepName.toLowerCase().includes('build') && step.avgDuration > 120000) {
recommendations.push(
`Build step in "${step.jobName}" takes ${(step.avgDuration / 1000).toFixed(0)}s. Consider incremental builds or build caching with Turborepo.`
);
}
}
return recommendations;
}
functionestimateCacheSavings(stepDurations: Map<string, number[]>): number {
let savings = 0;
for (const [key, times] of stepDurations) {
if (key.toLowerCase().includes('install') || key.toLowerCase().includes('cache')) {
if (times.length > 1) {
const first = times[0];
const subsequent = times.slice(1);
const avgSubsequent =
subsequent.reduce((a, b) => a + b, 0) / subsequent.length;
if (first > avgSubsequent * 2) {
savings += (first - avgSubsequent) * subsequent.length;
}
}
}
}
returnMath.round(savings);
}
Critical Path Optimization
Understanding the Pipeline Dependency Graph
The critical path is the longest sequential chain of jobs in your pipeline. Parallelizing jobs off the critical path does not reduce total duration. Focus optimization on the critical path:
// scripts/critical-path.tsinterfaceJobNode {
name: string;
estimatedDuration: number;
dependencies: string[];
canParallelize: boolean;
shardable: boolean;
}
exportfunctionfindCriticalPath(jobs: JobNode[]): {
path: JobNode[];
duration: number;
parallelizableSavings: number;
} {
const jobMap = newMap(jobs.map((j) => [j.name, j]));
const memo = newMap<string, number>();
functionlongestPath(jobName: string): number {
if (memo.has(jobName)) return memo.get(jobName)!;
const job = jobMap.get(jobName);
if (!job) return0;
let maxDepDuration = 0;
for (const dep of job.dependencies) {
maxDepDuration = Math.max(maxDepDuration, longestPath(dep));
}
const total = maxDepDuration + job.estimatedDuration;
memo.set(jobName, total);
return total;
}
// Find the endpoint of the critical pathlet maxDuration = 0;
let criticalEndJob = '';
for (const job of jobs) {
const duration = longestPath(job.name);
if (duration > maxDuration) {
maxDuration = duration;
criticalEndJob = job.name;
}
}
// Reconstruct the critical path by walking backwardsconstpath: JobNode[] = [];
let current = criticalEndJob;
while (current) {
const job = jobMap.get(current);
if (!job) break;
path.unshift(job);
let nextJob = '';
let nextDuration = 0;
for (const dep of job.dependencies) {
const depDuration = memo.get(dep) || 0;
if (depDuration > nextDuration) {
nextDuration = depDuration;
nextJob = dep;
}
}
current = nextJob;
}
const totalSequentialDuration = jobs.reduce(
(sum, job) => sum + job.estimatedDuration,
0
);
const criticalPathDuration = path.reduce(
(sum, job) => sum + job.estimatedDuration,
0
);
return {
path,
duration: criticalPathDuration,
parallelizableSavings: totalSequentialDuration - criticalPathDuration,
};
}
Best Practices
Measure before optimizing. Profile your current pipeline to identify actual bottlenecks. The slowest step may not be what you expect. Use pipeline analytics to find the real performance killers before implementing any optimizations.
Use concurrency controls to cancel redundant runs. When a developer pushes multiple commits in quick succession, cancel the in-progress run for the previous commit. The GitHub Actions concurrency key with cancel-in-progress: true handles this automatically.
Cache aggressively but validate cache correctness. Cache node_modules, build artifacts, Playwright browsers, and Docker layers. But always verify that stale caches do not cause false-positive test results. Include lock file hashes in cache keys to invalidate on dependency changes.
Split E2E tests across more shards than unit tests. E2E tests are typically 10-50x slower than unit tests. A suite that benefits from 2 unit test shards may need 4-8 E2E test shards for balanced execution times.
Run lint and type-check before tests. These checks are fast (typically under 60 seconds) and catch a large class of issues. Running them before expensive test suites provides faster initial feedback when they fail.
Use fail-fast: false for sharded jobs. When tests are split across shards, a failure in shard 1 should not cancel shard 3. You want complete results from all shards to see the full scope of failures.
Collect and store test timing data persistently. Duration-based test splitting requires historical data to be effective. Collect timings after every CI run and cache them for future runs. Use exponential moving averages to smooth outliers.
Implement selective test execution with a safety net. When change detection cannot determine which tests to run (e.g., for config file changes), default to running all tests. A false positive (running unnecessary tests) is always safer than a false negative (missing a regression).
Pre-warm caches on a schedule. Run a scheduled workflow that installs dependencies and populates caches before the workday begins. This ensures the first PR of the day gets a cache hit instead of a cold start.
Monitor pipeline duration trends over time. Track P50 and P95 pipeline durations weekly. A gradual increase of 30 seconds per week adds up to 25 minutes over a year. Catch regressions early with automated alerting on duration spikes.
Use service containers for integration test dependencies. Instead of installing PostgreSQL or Redis in your workflow steps, use GitHub Actions service containers. They start in parallel with job setup and are ready when your tests need them.
Keep Docker images minimal for CI runners. If your CI uses custom Docker images, minimize their size. Every megabyte of image that needs to be pulled adds to every pipeline run. Use multi-stage builds and slim base images.
Anti-Patterns to Avoid
Running all tests on every PR regardless of changes. A documentation-only PR does not need E2E tests. A frontend-only change does not need backend integration tests. Implement change detection to skip unnecessary work and save both time and compute resources.
Using sequential job execution when parallelism is possible. If unit tests and E2E tests have no dependency on each other, run them in parallel. Restructure your pipeline so that only actual data dependencies create sequential chains.
Caching too broadly or too narrowly. Caching node_modules without including the lock file hash in the cache key means stale dependencies persist. Caching only node_modules but not Playwright browsers means downloading 100+ MB of browser binaries on every run.
Splitting tests by file count instead of duration. Ten test files may take 10 seconds or 10 minutes depending on their content. Always use duration-based splitting when timing data is available to ensure balanced shard execution times.
Ignoring the critical path. Parallelizing jobs that are already off the critical path does not reduce total pipeline duration. Identify and focus optimization on the jobs that form the longest sequential chain.
Not canceling redundant pipeline runs. Without concurrency controls, every push creates a new pipeline run. Three pushes in 5 minutes create three full runs, wasting resources and delaying results for other developers waiting in the queue.
Hard-coding shard counts. A static 4-shard configuration may be wasteful for small PRs and insufficient for full test runs on main. Dynamically determine shard counts based on the scope of changes detected.
Skipping tests to speed up the pipeline. Disabling tests is not optimization; it is risk accumulation. Optimize the tests themselves (reduce flakiness, parallelize, cache) rather than removing them from the pipeline.
Debugging Tips
Cache misses on every run. Verify that the cache key includes only deterministic inputs. Environment variables, timestamps, and random values in cache keys cause every run to miss. Use hashFiles() on lock files and configuration files for consistent keys.
Sharded tests produce inconsistent results. Test isolation issues become visible under sharding because tests run in different orders and different processes. Identify tests that depend on shared state (global variables, database records, file system artifacts) and fix them to be truly independent.
Selective test execution misses a regression. Review your change-to-test mapping configuration. Common gaps include: shared utility files that are not mapped to all their consumers, configuration changes that should trigger full runs, and transitive dependencies that are not tracked in the change map.
Pipeline duration increases after adding caching. This can happen when cache save and restore steps take longer than the operation they are caching. Very small caches (under 10 MB) may not be worth the overhead. Profile the cache steps themselves to verify they provide net savings.
Test timings file grows unbounded. Prune entries for test files that no longer exist. When files are renamed or deleted, their timing entries remain in the file indefinitely. Add a cleanup step that removes entries for files not present in the current codebase.
GitHub Actions job matrix generates too many combinations. Matrix strategies with multiple dimensions (OS x Node version x shard) can produce dozens of jobs. Use include and exclude to limit combinations to only those that provide unique value.
Jobs fail with "No space left on device." CI runners have limited disk space. Large caches, Docker images, and build artifacts can exhaust the available storage. Add cleanup steps between resource-intensive operations, use slimmer base images, and prune unnecessary artifacts before running tests.
