Frontend · Node.js

Building a Full-Stack Large File Uploader with Chunking, Instant Upload, and Resume

By 谢小飞 · Aug 10, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Handling multi-gigabyte uploads reliably is a recurring pain point in web applications, and off-the-shelf libraries often obscure the control flow needed for production features like pause, resume, and deduplication. This implementation exposes the exact decisions — chunk sizing, hash sampling tradeoffs, memory-efficient merging, and concurrency limits — that turn a brittle upload into a resilient pipeline.

Summary

A complete walkthrough builds a large file upload system from the ground up, starting with the browser primitives Blob, File, and ArrayBuffer. The frontend slices files into 5MB chunks, computes an MD5 fingerprint with SparkMD5 (including a sampling shortcut for files over 100MB), and controls concurrency with p-limit. The Express backend receives chunks into memory via Multer, writes them to hash-named directories, and merges them sequentially using a write stream to keep memory constant regardless of file size. On top of this pipeline, two optimizations are layered in: an instant-upload check that skips transmission when the server already holds the file, and a resumable-upload mechanism that queries which chunks are missing and retransmits only those, using AbortController to cancel in-flight requests on pause.

Takeaways

— File.slice() creates lightweight references to byte ranges without duplicating data, so chunking a 5GB file does not multiply memory usage.

— A 5MB chunk size balances network resilience, concurrency overhead, and server connection pressure for most deployments.

— SparkMD5 supports incremental hashing: feed it one chunk at a time to compute an MD5 without loading the entire file into memory.

— For files larger than 100MB, sampling roughly 10 evenly distributed chunks plus the file size produces a fingerprint that is fast to compute and collision-resistant enough for deduplication.

— Multer's memoryStorage keeps chunk data in req.file.buffer, allowing the route handler to decide the disk path after reading the fileHash from req.body.

— Merging chunks with a write stream keeps memory constant; only the current chunk's buffer is held at any moment.

— Instant upload works by checking whether a file with the same hash already exists on disk before any chunk is sent.

— Resumable upload queries the server for a list of already-persisted chunk indices and uploads only the missing ones.

— AbortController per chunk enables true pause: all in-flight requests are cancelled, and the upload can restart from the last saved state.

Conclusions

Hash sampling is a pragmatic tradeoff that many tutorials skip; the 8-sample ceiling plus file-size salt is a concrete, repeatable heuristic that keeps the fingerprint fast even for multi-GB files.

The design choice to avoid a database and rely solely on filesystem naming conventions (hash-prefixed filenames, chunk-indexed directories) keeps the implementation self-contained and easy to reason about, though it limits multi-server scale.

Placing the merge logic's instant-upload fallback inside the merge endpoint itself is a clever concurrency safeguard: if two users race to upload the same file, the second merge call detects the already-written file and cleans up without corrupting data.

Concepts & terms

Blob

An immutable, raw-data container in the browser. It represents a file-like object and provides methods like slice(), text(), and arrayBuffer() for reading or splitting binary data.

File

A subclass of Blob that adds file-system metadata (name, lastModified). Every file selected by a user via an <input> is a File object and inherits all Blob methods.

ArrayBuffer

A fixed-length raw binary buffer in JavaScript. It is the byte-addressable representation needed by hash libraries like SparkMD5, and can be obtained from a Blob via blob.arrayBuffer().

Incremental Hashing (SparkMD5)

A hashing technique that feeds data to the algorithm in pieces rather than all at once. SparkMD5.append() ingests each chunk's ArrayBuffer, and spark.end() finalizes the digest, keeping memory usage low.

Hash Sampling

A shortcut for large files where only a subset of chunks (e.g., first, last, and several evenly spaced middle chunks) plus the file size are hashed. It trades a tiny risk of collision for dramatically faster fingerprint computation.

Multer memoryStorage

A Multer storage engine that holds uploaded files in memory (req.file.buffer) instead of writing them to disk immediately. This allows the application to decide the final disk path after inspecting request body fields.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗