| name | media-upload-pipelines |
| description | Design resilient mobile media upload pipelines with signed URLs, resumable uploads (tus, S3 multipart), and post-upload processing. Use when implementing or reviewing user-generated media ingestion. |
Media Upload Pipelines
Instructions
Mobile uploads happen on flaky radios, at full battery cost, on large files. Never stream user media through your API pods. Offload to object storage via signed URLs and process asynchronously.
1. Architecture
app -> POST /v1/uploads (API pod -- tiny)
<- { upload_url, object_key, upload_id }
app -> PUT/PATCH upload_url (object storage directly)
<- 200 / 201
app -> POST /v1/uploads/{id}/complete
-> triggers worker
-> validate, scan, transcode
-> emit "media.ready" event
- The API issues a short-lived signed URL.
- The client uploads directly to S3 / GCS / R2 / Azure Blob.
- A post-upload worker (triggered by object-event or by the
/complete call) does the heavy lifting.
2. Signed URLs
- Short TTL (15 min typical, 1 h maximum).
- Scoped to a single object key and content-type.
- For small files (< 5 MB), a single PUT is enough.
func SignUpload(ctx context.Context, key, contentType string) (string, error) {
ps := s3.NewPresignClient(s3client)
out, err := ps.PresignPutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String("uploads"),
Key: aws.String(key),
ContentType: aws.String(contentType),
}, s3.WithPresignExpires(15*time.Minute))
if err != nil { return "", err }
return out.URL, nil
}
3. Resumable Uploads
For files > 5 MB, single PUTs lose too much on failure. Use a resumable protocol:
- tus.io -- open protocol, works across object stores with a tus gateway; best client-library support (tus-java-client, TUSKit).
- S3 Multipart -- native on S3-compatible stores; client uploads parts, then sends a completion.
- GCS resumable -- simple session URL; PATCH with
Content-Range.
tus upload (Kotlin, tus-android-client):
val upload = TusUpload(file).apply { metadata = mapOf("filename" to file.name) }
val uploader = TusUploader(tusClient, upload, upload.size)
while (uploader.uploadChunk() > -1) {
progress.onProgress(uploader.offset, upload.size)
}
uploader.finish()
S3 multipart (Swift):
let init = try await s3.createMultipartUpload(bucket: bucket, key: key)
var parts: [CompletedPart] = []
for (i, chunk) in chunks.enumerated() {
let etag = try await s3.uploadPart(bucket: bucket, key: key,
uploadId: init.uploadId!, partNumber: i + 1, body: chunk)
parts.append(.init(partNumber: i + 1, eTag: etag))
}
try await s3.completeMultipartUpload(bucket: bucket, key: key,
uploadId: init.uploadId!, parts: parts)
4. Retry Semantics
- Single PUT: retry from zero on transient failure.
- Resumable: on failure, query the server for the current offset (
HEAD for tus, ListParts for S3) and continue.
- Network change (Wi-Fi -> cellular): pause, confirm offset, resume. Do not restart blindly.
- Exponential backoff on 5xx and 429. Give up after ~5 attempts with a user-visible error.
5. Validation and Scanning
A post-upload worker must:
- Check file size and content-type match the signed URL parameters (prevent bypass).
- Sniff the actual file (magic bytes), not trust
Content-Type from the client.
- Run a virus/malware scan for user-uploaded binaries (ClamAV, vendor service).
- Reject and delete the object on failure; emit a
media.rejected event.
6. Derivatives
For images: generate thumbnails, WebP/AVIF variants (see image-resizing-service).
For video: produce HLS/DASH renditions (see video-streaming).
For audio: transcode to a common codec, normalize loudness if needed.
Derivatives go into a separate "processed" bucket with deterministic keys:
uploads/<user_id>/<upload_id>/original.<ext>
media/<media_id>/thumb_256.webp
media/<media_id>/hls/master.m3u8
7. Quotas and Abuse
- Per-user quota (daily and total), enforced before signing.
- Per-IP rate limits on
/uploads to deter farming.
- Deny-list of repeated-offender hashes (perceptual hashes for media).
- Scan for CSAM / policy violations asynchronously; emit moderation events to review tooling.
8. API Endpoints
POST /v1/uploads
{ "content_type": "image/jpeg", "size": 2800000, "purpose": "avatar" }
-> { "upload_id": "up_01HX7...", "upload_url": "https://...", "expires_at": "...", "method": "PUT" }
POST /v1/uploads/up_01HX7.../complete
{ "etag": "abc" } // or multipart parts list
-> 202 { "media_id": "med_01HX7...", "status": "processing" }
GET /v1/media/med_01HX7...
-> { "id": "...", "status": "ready", "url": "https://cdn...", "variants": { "thumb_256": "..." } }
9. Client Consumption
- Show progress tied to uploaded bytes, not naive "pending" / "done".
- Upload in the background using platform-appropriate APIs:
URLSession background configuration on iOS; WorkManager with expedited constraints on Android.
- Pause on low battery / metered network if user setting allows.
- Persist the
upload_id so the client can resume across app restarts.
10. Observability
Track per stage: signing latency, upload duration, upload failure reasons (network vs 4xx vs 5xx), processing queue lag, end-to-end latency (tap-to-ready).
Checklist
