+91 98726 60544 hello@mitstech.co Mon–Sat · 09:00–18:30 IST

File uploads: doing it properly at scale

Cloud By Mits Engineering Team 2 min read
File uploads: doing it properly at scale

The obvious way to accept a file is to post it to your application server, which validates it and writes it to storage. It works, it is easy to reason about, and it becomes a problem the moment files are large or users are numerous - because every upload occupies a request thread for its entire duration, and a hundred people uploading video on a slow connection can exhaust your server while it does no computation at all.

The fix is to keep the bytes away from your servers entirely. The client asks your API for permission to upload; your API returns a pre-signed URL scoped to one object, one method and a short expiry; the browser uploads directly to object storage; then it tells your API the upload is done and you process it. Your server handles two small JSON requests instead of a large file, and the upload capacity becomes the storage provider's problem rather than yours.

Validate after upload rather than trusting anything the client claimed. The filename extension means nothing, and the content-type header means less - both are attacker-controlled. Read the file's magic bytes to determine what it actually is, check that against what you allow, and enforce a size limit in the pre-signed URL policy so an oversized file is rejected by storage rather than after you have paid to receive it. Never serve a user-uploaded file from your own domain without careful headers, because an uploaded HTML file served from your origin is a stored cross-site scripting vulnerability.

For large files, use multipart or resumable upload rather than one long stream. A single upload that fails at ninety per cent on a mobile connection has to restart from zero, which on a poor network can mean it never succeeds at all. Multipart splits it into chunks that retry independently and can resume, which is the difference between a feature that works in a Bengaluru office and one that works for a user on the move.

Process asynchronously. Thumbnail generation, virus scanning, transcoding, text extraction - none of it belongs in the request. Store the file, record its state as pending, return immediately, and let a worker do the work and update the state. The user gets a fast response and a progress indicator instead of a spinner that occasionally times out.

Two things worth deciding early because they are awkward later. Storage lifecycle: files accumulate forever unless something removes them, so decide what happens to an upload whose parent record is deleted, and set a policy for incomplete multipart uploads, which otherwise sit invisible in your bucket accruing charges for years. And access control: a file in a public bucket is public to anyone with the URL, and URLs leak - if the content is private, serve it through short-lived signed download URLs instead.

Need help with this? Explore our Cloud Solutions & Migration services. Learn more Back to all news

Keep reading

More on Cloud