file-upload-object-storage

name	file-upload-object-storage
description	Implements secure file/image/video upload to object storage via short-lived presigned URLs or POST policies, with content-type + size validation, magic-byte verification, non-guessable tenant-scoped key namespacing, multipart/resumable transfer, private buckets with signed-URL access, and post-upload scan/transcode + lifecycle cleanup.
when_to_use	User is adding file/image/video upload, generating presigned/direct upload URLs, handling large/resumable/multipart transfers, validating uploads, controlling object access, or serving via signed URLs/CDN. Distinct from auth-jwt-session (who the caller is — this consumes that identity) and secrets-management (storing the bucket credentials themselves).

When to Use

Reach for this skill when bytes flow from a client into a bucket and back out under access control:

• "Let users upload an avatar / document / video to object storage (S3, GCS, R2, Azure Blob)"
• "Generate a presigned URL / POST policy so the browser uploads direct-to-bucket"
• "Handle large/resumable uploads, multipart with retry"
• "Validate uploaded files — block executables, cap size, check the real type"
• "Keep these files private and serve them with a time-limited signed URL / behind a CDN"
• "Scan/resize/transcode after upload; clean up orphaned objects when a record is deleted"

NOT this skill:

• Deciding who may request an upload URL or read an object → auth-jwt-session (this skill consumes the authenticated identity; it does not establish it)
• Where the bucket's own access keys / service-account JSON live → secrets-management
• Broad storage-class/egress cost modeling across services → cloud-cost-optimize (this covers only per-object lifecycle/tiering for uploads)
• Front-end image perf (responsive srcset, lazy-load, format) → optimize-core-web-vitals
• Validating the other form fields around the file → build-form-validation
• App-response caching to cut load → caching-strategy (CDN here is for object delivery, not API responses)

Steps

1. Default: client uploads direct-to-bucket via a short-lived presigned credential — never stream bytes through your app server. Proxying uploads burns app memory/bandwidth and adds a hop. Your server only mints a credential and records metadata. Pick the credential type:

   Mechanism	Use when	Constrains
   Presigned POST policy (S3 create_presigned_post, R2 same)	Browser/HTML form, single file, want server-enforced size + type	Content-Type, content-length-range, exact key or key prefix, expiry
   Presigned PUT URL	Simple programmatic single-shot PUT	content-type + expiry only (no size cap pre-upload)
   Multipart presigned (create_multipart_upload + per-part upload_part URLs)	File > ~100 MB, flaky networks, resumable	per-part, parallel, retryable
   GCS resumable session (x-goog-resumable:start) / Azure Block Blob (PutBlock+PutBlockList)	GCS/Azure large or resumable	session URL + range

   Default for web uploads = presigned POST policy because it is the only one that enforces a size ceiling before bytes land. Expiry: 5 minutes (Expires=300). Generate per-upload, single-use.

2. Validate on the boundary — twice. Never trust client-supplied MIME or extension. A .jpg can be a polyglot HTML/JS or a zip bomb. Enforce in two places:

   • In the policy (pre-upload, hard ceiling): size via content-length-range and a Content-Type allowlist condition. Example POST policy conditions:

     fields = {"Content-Type": content_type, "x-amz-meta-owner": str(user_id)}
     conditions = [
         {"bucket": BUCKET},
         ["starts-with", "$key", f"tenants/{tenant_id}/uploads/"],
         {: content_type},          
         [, ,  *  * ],  
     ]
     post = s3.generate_presigned_post(BUCKET, key, Fields=fields,
                                       Conditions=conditions, ExpiresIn=)
     

     S3 rejects with / if the upload violates the policy — the server never sees the oversized body.

Common Errors

• Proxying upload bytes through the app server. OOMs on large files, doubles bandwidth. Mint a presigned credential; let the client PUT/POST straight to the bucket.
• Trusting Content-Type/extension from the client. Spoofable; enables stored-XSS via a .jpg that's really HTML served inline. Verify magic bytes server-side and serve user files with Content-Disposition: attachment + X-Content-Type-Options: nosniff.
• Putting the user's filename in the key. Invites path traversal and enumeration. Key on a server UUID; keep the display name in the DB.
• Public bucket / public ACL "just to make it work." Leaks every object and lets anyone overwrite. Block public access; use signed URLs. A ?-less object URL that loads in incognito is a finding.
• Presigned URL with hours/days expiry. A leaked long-lived URL is a permanent backdoor. Cap at minutes; mint per request.
• No content-length-range in the POST policy. Client uploads a 5 GB file and you pay for it. Always set a size ceiling in the policy, not just a client-side JS check.
• Authz only at URL-mint time, never re-checked. Object IDs leak in logs/referers; a stale signed URL outlives the user's permission. Keep expiry short and re-authorize on every mint.
• Forgetting AbortIncompleteMultipartUpload. Failed multipart uploads accrue invisible, billable parts indefinitely. Add the 1-day abort rule on day one.
• Marking ready before the scan/transcode finishes. Serves unscanned malware or a half-written object. Flip to ready only from the post-upload handler.
• Deleting the DB row but not the object (or vice-versa). Orphans cost money and leak data; missing objects 404 live records. Enqueue an idempotent object delete on row delete + run a reconcile sweep.
• CORS not configured on the bucket. Browser direct-PUT/POST fails preflight. Set AllowedMethods (PUT POST), AllowedOrigins (your exact origins, not *), and expose ETag for multipart.
• Same bucket for public assets and private originals. One misconfig exposes everything. Split public-asset and private-data buckets with different policies.

Verify

1. Direct-to-bucket works: client obtains a presigned POST/PUT and uploads with no bytes touching the app server (confirm app logs show only the mint call, not the body).
2. Size ceiling enforced server-side: upload max+1 bytes → bucket rejects (403 EntityTooLarge/policy violation); the body never reaches you. A client that disables the JS size check still cannot exceed the policy.
3. Type spoof blocked: upload an HTML/EXE file renamed .png with Content-Type: image/png → passes the policy but the magic-byte check deletes it and sets status=rejected; it is never served ready.
4. Key is non-guessable + traversal-proof: the stored key is a server UUID under tenants/{id}/...; a key containing ../, a leading /, or the raw filename is rejected/never produced.
5. Privacy: the raw object URL (no signature) returns 403 in an incognito session; a freshly signed URL returns 200; the same URL after Expires returns 403.
6. Cross-tenant denied: user A's signed-URL request for user B's key fails the authz check at mint time (no URL issued), not just at read time.
7. Resumable: kill the network mid-multipart, resume → only missing parts re-upload, complete succeeds, final object bytes match the source checksum (s3 cp then sha256sum).
8. Orphan hygiene: an abandoned multipart upload is gone after the abort window; a tmp/ object expires per its 1-day rule; deleting a record removes its object + renditions (verify via aws s3 ls/reconcile sweep shows no drift).
9. Post-upload state machine: a fresh upload is pending, becomes ready only after scan+derive complete, and a malicious/corrupt file ends rejected with the object deleted.

Done = uploads go direct-to-bucket under a ≤5-min single-use credential with a server-enforced size cap and magic-byte type check; objects are private, tenant-scoped, non-guessable, and readable only via short-lived signed URLs after an ownership check; large uploads resume and abandoned parts auto-abort; and every record delete (plus a reconcile sweep) leaves zero orphaned objects.