| name | parallel-chunked-download |
| description | Splitting one file into N byte-range requests issued in parallel over a bare HTTP client, each chunk to its own temp file, merged in order, with progress reported through a channel-backed flow — plus when ranges are actually safe, how big a chunk should be, and why a failure retries one chunk rather than the file. Use when a large download is slower than the link allows, when a download restarts from zero after a hiccup, or when a progress bar sticks just short of full. |
N ranges, one file
Three decisions in order: may I range this, should I, and then the ranges themselves.
fun download(url: String, target: Path, maxRetries: Int = 3, parallel: Int = 4): Flow<Progress> =
channelFlow {
with(bareDownloadClient()) {
val supportsRange = try {
val probe = head(url) { header("Range", "bytes=0-0") }
probe.headers["Content-Range"] != null || probe.status.value in 200..299
} catch (e: Exception) { false }
val fileSize = try {
head(url).headers[HttpHeaders.ContentLength]?.toLong() ?: 0L
} catch (e: Exception) { 0L }
if (supportsRange && fileSize > CHUNK_SIZE * 2) {
parallelDownload(url, target, fileSize, parallel, maxRetries, )
} else {
singleThreadedDownload(url, target, maxRetries, )
}
}
}
Each chunk is an independent job with its own attempt counter, writing to its own temp file:
val chunkSize = fileSize / parallel
coroutineScope {
val jobs = (0 until parallel).map { index ->
val startByte = index * chunkSize
val endByte = if (index == parallel - 1) fileSize - 1L else (index + 1L) * chunkSize - 1L
launch {
var attempt = 0; var success = false
while (attempt <= maxRetries && !success) {
try {
if (attempt > 0) delay(500L * (1L shl (attempt - 1)).coerceAtMost(10_000L))
prepareRequest { url(url); header("Range", "bytes=$startByte-$endByte") }
.execute { res ->
if (res.status.value != 206) throw IllegalStateException("not 206")
}
success = true
} catch (e: Exception) {
attempt++
if (attempt > maxRetries) throw e
}
}
}
}
jobs.forEach { it.join() }
}
Traps
A 2xx on the probe does not mean the server honoured the range — it usually means it ignored
it. A server that supports ranges answers Range: bytes=0-0 with 206 and a Content-Range
header; one that does not answers 200 with the entire file. Accepting any 2xx therefore reads
"ignored my range" as "supports ranges", and the mistake is only caught per-chunk, after the loop has
burned its retries on a server that was never going to comply. Gate on 206 or on Content-Range.
Keep the per-chunk 206 check anyway. Probe and real requests can disagree behind a cache or a
redirect, and without the check a server replying 200 to a ranged request hands each of the N jobs
the whole file — every job writes it in full and the merge produces a file N times too long, with
no error anywhere.
One chunk failing must retry that chunk, not the file. That is the entire reason to keep each
range in its own temp file: at 90% of a large download a dropped connection costs one chunk's worth
of bytes. The fallback path — a single request into the destination file — has the opposite property,
and its retry deletes the partial file and starts over: correct there, disastrous here.
A chunk retry must rewrite from the chunk's first byte. Reopen the temp file truncating, never
appending. If a first attempt wrote half a chunk and the retry appends the whole thing on top, that
chunk is one and a half chunks long — and the merge is a blind concatenation, so everything after it
shifts and the file is corrupt with no failed request anywhere. Inspect the sink call; watching a
download succeed proves nothing.
Integer division loses the remainder, so the last chunk is special. fileSize / parallel
truncates; the final range must end at fileSize - 1, not parallel * chunkSize - 1. Get it wrong
and trailing bytes go missing — a decode error at the very end of a media or archive file, long after
the download reported success.
A progress send whose result is discarded is not a delivery. trySend reports whether the value
was accepted; on a full buffer it is not. Dropping a progress tick is harmless — dropping the
terminal one is not, and a slow collector then misses the only signal that the download finished
while the file sits complete on disk. Express completion through the flow's own completion, and keep
the channel for progress alone.
Chunk-granularity progress moves in N steps and does not reach the end. Reporting
completedChunks * chunkSize / fileSize jumps in quarters and, because chunkSize was truncated,
tops out just below 1.0 — the "sticks at 99%" symptom, usually patched by rounding instead of fixed.
Count bytes across chunks if the bar matters, and make the counter atomic or two chunks finishing
together count once.
Parallelism has a floor and a ceiling, both worth a constant. Below the threshold the two probe
round trips cost more than the concurrency saves — and the gate is twice a fixed floor constant,
not twice the per-chunk size the code derives from the file. Above it, more connections stop helping
once the link is saturated and start hurting on mobile networks and hosts that throttle per connection.
The failure path owes you two things: a clean temp directory and the throwable. Clean the
directory in the catch as well as after the merge, or a cancelled download leaves chunks behind on
every attempt. And do not capture the exception into a variable, emit a terminal "finished, progress
0", then never read the variable again — the collector cannot distinguish a failure from a zero-byte
success, and the log is empty because the exception went nowhere. Note also what this design does
not buy: the temp directory is created and removed inside one call, so nothing resumes across a
process restart — resume means keying it to the URL and re-probing, a different feature.
Verifying it
grep -rn 'header("Range"' --include='*.kt' . | grep -v '/build/'
grep -rn 'PartialContent\|!= 206\|== 206' --include='*.kt' . | grep -v '/build/'
grep -rn 'trySend(' --include='*.kt' . | grep -v '/build/'
grep -rn 'trySend(' --include='*.kt' . | grep -v '/build/' | grep -cE 'isSuccess|isFailure|getOrNull'
The first two go together: expect one Range hit for the probe and one for the chunk request, plus
at least one 206 comparison. A probe with no matching 206 check anywhere is the false-positive trap
above, unguarded. The third lists every progress send; the fourth counts how many inspect the result
— 0 against a non-empty third list means no send is checked anywhere, so find the terminal one in
that list and confirm completion does not depend on it.
For the chunk-boundary trap the cheap test is arithmetic, not a download: pick a file size not
divisible by the chunk count, compute every startByte/endByte pair, assert they tile
0..fileSize-1 with no gap and no overlap. The bare client this runs on is described in
ktor-kmp-client-architecture — attaching a verbose logging plugin here is that skill's first trap.
