Uploading a large file? Network drops, browser crashes, users leave. Don't worry, this article covers it all.

Ahem, don't scroll away when you see "large file upload." I know this topic sounds a bit "cliché" — chunked uploads, resumable uploads, everyone has heard of them countless times. But I'm not here to read you a PPT today. You see, in the past, when we talked about large file uploads, we either found a ready-made wheel online to use directly, or copied a few pieces of code and tweaked them. As a result, when we encountered a real problem, we were stumped: How to determine the chunk size? How to calculate progress? How does the server merge them? What if it gets interrupted midway?

Today, based on my practical experience, I'll talk about how to implement a large file upload feature from scratch. The core is the word "chunking," but we won't just talk about concepts. I'll guide you through the code step by step, ensuring you can use it in your projects right after reading.

Why is large file upload a "stubborn problem"?

Usually, when calling an API to upload an avatar or a document, a file of three to five MB is the maximum. Using FormData directly works fine:

const formData = new FormData();
formData.append('file', file);
await axios.post('/api/upload', formData);

But once you encounter a large file — say, a 500MB video or a 2GB design source file — a series of problems arise:

Partners who frequently upload short videos should have encountered such problems. However, the solution is quite simple: Split the large file into small chunks, upload them one by one, and finally merge them on the server.

It's like sending a package — if you need to ship a large piece of furniture, you certainly wouldn't move it whole. Instead, you disassemble it into parts, pack them, and reassemble them upon arrival. Chunked upload follows the same logic:

Large File (500MB)
    ↓
Split into 10 chunks (50MB each)
    ↓
Upload chunks one by one (1MB, 2MB...)
    ↓
Server merges chunks in order
    ↓
Restore to complete file

This approach has three benefits:

1. Each chunk request is short, less likely to time out.
2. If a chunk fails, only that one needs to be re-uploaded, no need to start over.
3. Multiple chunks can be uploaded concurrently, doubling the speed.

Source Code Address

Prerequisite Knowledge: Understanding File, Blob, and ArrayBuffer

Before diving into the chunking code, we need to clarify a few file-related concepts in the browser. Otherwise, you'll be confused when you see APIs like slice and ArrayBuffer in the code later.

Blob — Binary Large Object

Blob stands for Binary Large Object. MDN's definition is:

The Blob object represents a blob, which is a file-like object of immutable, raw data; they can be read as text or binary data, or converted into a ReadableStream so its methods can be used for processing the data.

Simply put, a Blob is a container holding raw data. You can think of it as a read-only "stand-in" for a file — it's like a file, but not exactly a file. Create a Blob object using the Blob() constructor:

// Create a text Blob
const textBlob = new Blob(['Hello, World!'], { type: 'text/plain' });

// Create a JSON Blob
const jsonBlob = new Blob(
  [JSON.stringify({ name: 'Large File Upload', version: '1.0' }, null, 2)],
  { type: 'application/json' }
);

// Create a Blob from multiple data fragments
const multiBlob = new Blob(['First part of data', 'Second part of data'], { type: 'text/plain' });

Blob has only two properties, both read-only:

Property	Type	Description
size	number	The size of the Blob in bytes
type	string	The MIME type of the Blob
stream	ReadableStream	A readable stream of the Blob
bytes	Promise	Asynchronously reads the Blob's content as a Uint8Array

const blob = new Blob(['Hello'], { type: 'text/plain' });
console.log(blob.size); // 5
console.log(blob.type); // "text/plain"

The methods provided by Blob are mainly for reading and slicing:

Method	Return Value	Description
slice	Blob object	Extracts a sub-Blob from a specified range
text	Promise	Asynchronously reads the Blob's text content
arrayBuffer	Promise	Asynchronously reads the Blob's ArrayBuffer content

// Chunking — the core operation of large file uploads
const largeBlob = new Blob(['A'.repeat(1024 * 1024)]); // 1MB of data
const chunkSize = 256 * 1024; // 256KB per chunk
const firstChunk = largeBlob.slice(0, chunkSize);

console.log(firstChunk.size); // 262144

// Asynchronously read as text
const text = await blob.text();
console.log(text); // "Hello"

// Asynchronously read as ArrayBuffer
const buffer = await blob.arrayBuffer();
console.log(buffer.byteLength); // 5

File — The Concrete Embodiment of a User File

File is Blob's "own son" — it inherits from Blob and extends it with metadata related to the user's file system. MDN's definition of File is:

The File interface provides information about files and allows JavaScript in a web page to access their content.

File objects are a specific kind of Blob, and can be used in any context that a Blob can.

File objects on a web page typically come from two sources:

1. After a user selects a file via <input type="file">, obtained from the returned FileList.
2. During a drag-and-drop operation, obtained from the DataTransfer object.

Although we rarely manually construct a File in actual development (the browser usually creates it for us), understanding its construction helps:

const file = new File(['File content'], 'example.txt', {
  type: 'text/plain',
  lastModified: Date.now()
});

File inherits all properties of Blob (size, type) and adds the following:

Property	Type	Description
name	string	The file name
lastModified	number	The last modification time in milliseconds
lastModifiedDate	Date	The last modification time as a Date object

// Get the File object after a user selects a file
const input = document.querySelector('input[type="file"]');
input.addEventListener('change', (e) => {
  const file = e.target.files[0];
  console.log(file.name);        // "profile.jpg"
  console.log(file.size);        // 245760 (inherited from Blob)
  console.log(file.type);        // "image/jpeg" (inherited from Blob)
  console.log(file.lastModified); // 1698768000000
});

File has no unique instance methods of its own; all methods are inherited from Blob — meaning File can use all of Blob's methods: slice(), arrayBuffer(), text(), stream(), etc.

ArrayBuffer

If Blob is a "file container holding data," then ArrayBuffer is a chunk of raw, fixed-length memory. MDN's definition of ArrayBuffer is:

The ArrayBuffer object is used to represent a generic raw binary data buffer. It is an array of bytes, often referred to in other languages as a "byte array."

Its constructor is as follows:

new ArrayBuffer(length, options)

• length: The size of the buffer to create (in bytes).
• options (optional): maxByteLength, specifying the maximum size the buffer can be resized to.

    // Create an 8-byte ArrayBuffer
    const buffer = new ArrayBuffer(8);
    console.log(buffer.byteLength); // 8

    // Create a resizable ArrayBuffer (max 16 bytes)
    const resizableBuffer = new ArrayBuffer(8, { maxByteLength: 16 });
    resizableBuffer.resize(12); // Resize to 12 bytes
    console.log(resizableBuffer.byteLength); // 12

The Relationship Between the Three

Now let's connect the three concepts. There is a clear "lineage" among them:

File is Blob's own son — File inherits everything from Blob. You can use a File directly as a Blob. The only core difference is: File carries a file name and modification time, while Blob does not. In other words, every file a user selects via <input type="file"> is essentially a "Blob with a name."

Blob and ArrayBuffer have a "partnership" relationship, not an inheritance one. A Blob can be converted to an ArrayBuffer, and an ArrayBuffer can be wrapped into a Blob. They can "transform" into each other.

In large file uploads, they each play their own roles. The main flow is as follows:

1. When a user selects a file, you get a File. It tells you the file name, size, and last modification time — this information needs to be displayed to the user.
2. When chunking, you call the File's slice() method, which returns a Blob. Each chunk Blob is stuffed into FormData and sent to the backend.
3. When calculating the file hash (MD5), you use ArrayBuffer. Hash calculation libraries (like spark-md5) need to process data byte by byte, but a Blob itself is a "black box" — you cannot directly manipulate its bytes. At this point, you need to convert the Blob into an ArrayBuffer for the hash library to read.
4. Instant upload relies on the hash value: The frontend calculates the MD5, sends it to the backend for a check. If it exists, it's an instant pass.
5. Resumable upload relies on the hash value + uploaded chunk records: Before resuming an upload, the frontend first asks the backend with the hash value "which chunks have already been uploaded," and then only transmits the missing ones.

To summarize the relationship in one sentence:

• File is the "original" the user gives you, with a name and properties. It's the starting point.
• Blob is the "fragment" cut from the File, only caring about data, not the name. Used for transmission and storage.
• ArrayBuffer is the "innards" of the Blob, allowing you to directly manipulate bytes. Used for calculation and verification.

Once you understand the relationship and conversion between these three, the subsequent implementation of chunked upload will fall into place naturally.

Frontend Implementation: Writing Chunked Upload with Vue3 + TypeScript

After laying the theoretical groundwork, this section gets our hands dirty with actual code. The frontend is the "command center" for large file uploads — it's responsible for splitting the file, scheduling chunks, tracking progress, and handling exceptions. I'll use Vue3's Composition API with TypeScript to implement the entire upload logic.

File Chunking — Divide and Conquer

Chunking is the first and most fundamental step in large file uploads. Understanding the core logic of chunking makes the subsequent flow smooth. There's no standard answer for chunk size; it's usually recommended to be between 1MB ~ 10MB. The choice depends on several factors:

Factor	Smaller Chunks	Larger Chunks
Network	Less impact from single failure, easy recovery	Greater impact from single failure, high retransmission cost
Concurrency	More concurrent requests possible, but higher request overhead	Fewer concurrent requests, lower request overhead
Server Load	Frequent requests, higher CPU load	Fewer requests, but single connection occupies longer
Memory Usage	Lower memory usage, but requires more requests	Higher memory usage, but fewer requests

In practice, 5MB is a relatively balanced choice, and many cloud storage services default to this value. You can also dynamically adjust based on the network environment — for unstable mobile networks, set it smaller (like 2MB); for intranets or high-speed broadband, set it larger (like 10MB).

The chunking function utilizes the File object's slice method, returning a new Blob object:

/**
 * Split a file into chunks
 * @param file The original file
 * @param chunkSize Chunk size in bytes, default 5MB
 */
export function splitFile(
  file: File,
  chunkSize: number = 5 * 1024 * 1024,
): Blob[] {
  const chunks: Blob[] = [];
  let cur = 0;

  while (cur < file.size) {
    const chunk = file.slice(cur, cur + chunkSize);
    chunks.push(chunk);
    cur += chunkSize;
  }

  return chunks;
}

It's important to note that the slice function performs a shallow copy. It doesn't truly duplicate the data; it just creates a new reference pointing to a segment of the original file's data — so even if you cut 100 chunks, it won't consume 100 times the memory.

File Hash Calculation

In the previous section, we completed file chunking, but one crucial step remains: generating an "ID card" for the file. This "ID card" is the file's hash value (usually calculated using the MD5 algorithm). In large file uploads, the hash value has three core uses:

• Instant Upload: Before uploading, send the hash value to the backend. The backend checks — if this file has already been uploaded by someone else, it returns success directly, saving the entire upload process.
• Resumable Upload: Use the hash value to ask the backend "how much of this file have I already uploaded?" The backend returns a list of uploaded chunks, and the frontend only uploads the missing ones.
• Integrity Check: After the upload is complete, the backend can compare the file hash to confirm the data wasn't corrupted during transmission.

The most commonly used library for frontend MD5 calculation is SparkMD5. Its core advantage is support for incremental computation — you can feed it file chunks one by one, it calculates as it goes, and finally spits out a complete MD5 value. This method has extremely low memory usage, making it perfect for large files. The core code is as follows:

/**
 * Calculate file hash (used to identify file uniqueness, enabling instant/resumable uploads)
 * Uses the spark-md5 library
 */
export async function calcFileHash(chunks: Blob[]): Promise<string> {
  const fileReader = new FileReader();
  const spark = new SparkMD5.ArrayBuffer();

  let currentChunk = 0;

  function loadNext() {
    fileReader.readAsArrayBuffer(chunks[currentChunk]);
  }

  return new Promise((resolve, reject) => {
    // Return empty content hash for an empty array
    if (chunks.length === 0) {
      resolve('');
      return;
    }

    fileReader.onload = (e) => {
      spark.append(e.target?.result as ArrayBuffer);
      currentChunk++;

      if (currentChunk === chunks.length) {
        resolve(spark.end());
      } else {
        loadNext();
      }
    };

    fileReader.onerror = () => {
      resolve("");
    };

    loadNext();
  });
}

Here, we pass in the Blob array of file chunks from the previous section. Since SparkMD5 calculates asynchronously, we only read one chunk into memory at a time. When all chunks have been calculated, we return a Promise<string>, whose value is the file's MD5 hash.

Besides using FileReader, we can also use the Blob.arrayBuffer() function to directly convert the file to an ArrayBuffer, then use SparkMD5 to calculate the hash, making the code cleaner and avoiding event callbacks.

/**
 * New file hash calculation (implemented based on Blob.arrayBuffer)
 * Uses native Blob.arrayBuffer() instead of FileReader, making code cleaner without event callbacks
 */
export async function calcFileHashNew(chunks: Blob[]): Promise<string> {
  // Return empty content hash for an empty array
  if (chunks.length === 0) {
    return '';
  }

  const spark = new SparkMD5.ArrayBuffer();

  for (const chunk of chunks) {
    // Convert each chunk to ArrayBuffer and append to the md5 calculator
    const buffer = await chunk.arrayBuffer();
    spark.append(buffer);
  }
  return spark.end();
}

Advanced: File Hash Calculation via Sampling

In the previous section, we discussed how to use SparkMD5 incremental calculation to compute the hash of large files. While this solution is already good, I must be honest — when files reach several GB or even tens of GB, even incremental calculation still takes a considerable amount of time.

I encountered such a scenario during development: a user tried to upload a 5GB virtual machine image, and the MD5 calculation ran for nearly a minute. During that minute, the user stared at a motionless progress bar before the actual chunked transfer process began. This experience was hardly elegant.

Thus, an idea emerged to find a balance between "accuracy" and "calculation speed": instead of calculating the full hash, generate a sufficiently reliable "file fingerprint" through sampling. This is the solution discussed in this section.

async function calcFileHashSampling(
  chunks: Blob[],
  fileSize: number,
): Promise<string> {
  // Return empty hash for empty file or empty chunks array
  if (chunks.length === 0) {
    return '';
  }

  // For small files, calculate the full hash directly to avoid precision loss from sampling
  if (chunks.length <= 20) {
    return calcFileHashNew(chunks);
  }
}

First, if the number of chunks is only 20 (which, at 5MB per chunk, means files within 100MB), sampling doesn't make much sense — a full calculation doesn't take much time either. Why introduce extra logic and risk just to save a fraction of a second? Just go straight to the full hash path, which is accurate and worry-free.

// Set of sample indices: always includes the first and last chunks
const sampleIndices = new Set<number>([0, totalChunks - 1]);

The file header often contains metadata (like EXIF info for images, file headers for documents), and the file tail usually contains checksum information or end markers. These two positions have the highest "information entropy" and the strongest distinctiveness, so they are prioritized for inclusion in the sampling range.

// Number of middle sample points: increases with total chunks, up to a maximum of 8
const middleSampleCount = Math.min(8, Math.max(1, Math.floor(totalChunks / 10)));

Next, there's a smooth sampling strategy: total chunks divided by 10, with a lower limit of 1 and an upper limit of 8. That means:

• 50 chunks → sample 5 middle points
• 100 chunks → sample 8 middle points (upper limit reached)
• 1000 chunks → still only sample 8 middle points

The intention behind this design is: for very large files, the number of sample points won't grow indefinitely, keeping the computation time strictly within an acceptable range. The upper limit of 8 is an empirical value — 8 sample points plus the first and last 2 make 10 chunks total. Even if each chunk is 5MB, the amount of data read for one sampling hash is only 50MB. Coupled with SparkMD5's calculation time, the user experience is nearly instantaneous.

const step = totalChunks / (middleSampleCount + 1);
for (let i = 1; i <= middleSampleCount; i++) {
  const index = Math.floor(i * step);
  sampleIndices.add(index);
}

Imagine you have a book, and you need to determine if it's a specific edition by reading a few pages. You wouldn't just read the first 10 pages, nor would you flip through randomly. Instead, you'd select evenly — the beginning, slightly before the middle, the exact middle, slightly after the middle, and the end. The step in the code does exactly this, distributing the sample points as evenly as possible throughout the file.

const sizeText = new TextEncoder().encode(String(fileSize));
spark.append(sizeBuffer);

This is a key trick to prevent hash collisions. For example: File A and File B are completely different, but if they happen to have identical content exactly at those 10 sampled chunks (though the probability is extremely low), these two files would be mistakenly judged as identical. Including the file size as part of the hash input adds an extra layer of "insurance" to this fingerprint — even if the sample points collide, different file sizes will result in different final hashes.

// Sort by index to ensure stable hash calculation order
const sortedIndices = Array.from(sampleIndices).sort((a, b) => a - b);

for (const index of sortedIndices) {
  const buffer = await chunks[index].arrayBuffer();
  spark.append(buffer);
}

return spark.end();

Finally, sort the sampleIndices to ensure a stable calculation order.

Chunked Upload

Now that we have the chunking function and the file hash calculation function, the next step is to "dismember" the file and send the pieces out one by one. First, we need an Upload component to select the file:

<template>
  <n-upload
    ref="uploadRef"
    @change="handleFileChange"
  >
    <n-button>
      Select File
    </n-button>
  </n-upload>
</template>
<script setup lang="ts">
const currentFile = ref<File | null>(null);

const handleFileChange = (options: { file: UploadFileInfo }) => {
  if (options.file.file) {
    currentFile.value = options.file.file;
  }
};
</script>

The file is stored in the currentFile variable. We use a button on the page to start the upload:

<template>
  <n-button type="primary" @click="startUpload">
    Start Upload
  </n-button>
</template>
<script setup lang="ts">
// Single chunk size is 5MB
const CHUNK_SIZE = 1024 * 1024 * 5;

const startUpload = async () => {
  if (!currentFile.value) return;
  // File splitting
  const chunks = splitFile(currentFile.value, CHUNK_SIZE);
  // Calculate file hash
  const fileHash = await calcFileHashNew(chunks);
  // todo upload chunks
}
</script>

We get the chunks array and the file hash value. The next step is to upload the chunks.

const startUpload = async () => {
  const total = chunks.length;
  let completed = 0;
  
  // Create an array of tasks, each returning a Promise
  const uploadTasks = chunks.map((chunk, index) => {
    return async () => {
      await uploadChunk(chunk, index, total, fileHash);
      // Single chunk completed, update progress
      completed++;
      uploadProgress.value = Math.round((completed * 100) / total);
    };
  });
  
  // Execute all tasks in parallel
  await Promise.all(uploadTasks.map((task) => task()));
}

But if we open the console, if the number of chunks is too high (hundreds), the browser will initiate a large number of requests simultaneously, potentially causing browser resource strain or server-side rate limiting. At this point, use the p-limit library to control the number of concurrent requests and avoid excessive concurrency.

import pLimit from "p-limit";
const startUpload = async () => {
  const limit = pLimit(5);

  // Create tasks with concurrency control, each task executes immediately but is controlled by the limit
  const tasks = chunks.map((chunk, index) =>
    limit(async () => {
      await uploadChunk(chunk, index, total, fileHash);
      completed++;
      uploadProgress.value = Math.round((completed * 100) / total);
    }),
  );

  // Wait for all tasks to complete
  await Promise.all(tasks);
}

This way, our file uploads 5 chunks at a time, avoiding browser resource strain or server-side rate limiting.

Backend Implementation: Express Receives and Merges Chunks

In the previous section, we completed the frontend work — the file is cut into small pieces and sent to the frontend one by one. Now it's the backend's turn to shine: it needs to receive these chunks, store them temporarily, and piece them back into a complete file once all chunks have arrived.

Before formally implementing the code, let's design the directory structure for uploaded files on the backend:

Project Root/
└── uploads/                    # Root directory for all uploaded files
    ├── chunks/                 # Chunk staging area
    │   ├── {fileHash1}/        # Directory per file hash
    │   │   ├── chunk-0
    │   │   ├── chunk-1
    │   │   └── chunk-2
    │   └── {fileHash2}/
    │       ├── chunk-0
    │       └── chunk-1
    └── {fileHash}-{filename}   # Merged complete file

　　The uploads directory is where all our uploaded files are saved, so it needs to be added to the .gitignore file to prevent it from being committed to version control. Under the chunks directory, subdirectories are created based on the file hash value, and each subdirectory stores all chunks of that file.

When all chunks of a file have been uploaded, the backend needs to merge these chunks and reassemble them back into a complete file in the uploads directory. The filename will be {fileHash}-{filename}, and the chunk directory for that file under the chunks directory will be deleted.

Why not store the filename and file hash in a database? In this article, we only focus on file upload and merging, not file storage and management. Therefore, we won't design database tables to store this information, nor will we consider the scenario where the same file is uploaded after being renamed.

Receiving Chunks

Receiving chunks is essentially the same as receiving a regular file. We use the multer library to implement this. However, Multer's default diskStorage writes files directly to disk. The problem is: the directory for storing chunks depends on fileHash, which is in req.body. When using diskStorage, the file is written to disk before we get a chance to access req.body, making it impossible to decide which directory to save to at write time.

Therefore, the code uses memoryStorage:

const upload = multer({
  storage: multer.memoryStorage(),
  limits: {
    fileSize: 10 * 1024 * 1024, // Max 10MB per chunk
  },
});

memoryStorage keeps the file data in memory (req.file.buffer) instead of writing it directly to disk. This way, after we get the fileHash in the middleware, we can manually write the buffer to the target directory — the flow is entirely under our control.

uploadChunkMiddleware is the core of the entire chunk receiving process. Let's break it down:

export const uploadChunkMiddleware = (req, res, next) => {
  upload.single('chunk')(req, res, (err) => {
    if (err) {
      return next(err);
    }

    const { index, fileHash } = req.body;
    if (!req.file || !fileHash || index === undefined) {
      // If fields are missing, pass through to the subsequent route handler to handle the error
      return next();
    }

    const targetDir = getChunksDir(fileHash);
    ensureDir(targetDir);

    const targetPath = path.join(targetDir, `chunk-${index}`);
    // Write the chunk from memory to the target directory
    fs.writeFileSync(targetPath, req.file.buffer);

    next();
  });
};

After receiving the chunk data, the naming rule is chunk-${index}, which includes the index number. This way, when merging, we just need to read all chunk-* files in the directory and sort them by index. No extra mapping table is needed. Simple, reliable, and understandable at a glance.

Finally, the responsibility of our route handler POST /upload/chunk is very clear — confirm that the chunk has been successfully saved, and then return a confirmation response to the frontend:

router.post("/upload/chunk", uploadChunkMiddleware, (req, res) => {
  const { index, totalChunks, fileHash } = req.body;

  // Validate required fields
  if (index === undefined || !totalChunks || !fileHash) {
    return res.status(400).json({
      success: false,
      message: "Missing required chunk information (index/totalChunks/fileHash)",
    });
  }

  if (!req.file) {
    return res.status(400).json({
      success: false,
      message: "No chunk file received",
    });
  }

  res.json({
    success: true,
    message: "Chunk received successfully",
    data: { index, totalChunks, fileHash, size: req.file.size },
  });
});

Looking at this, you'll find that the core logic of this interface is actually completed in the middleware. The route handler just does a layer of validation and response wrapping. This pattern of "middleware does the dirty work, route does the light work" makes the code clearer and more maintainable.

Merging Chunks

When the frontend has finished uploading all chunks, it calls the merge interface POST /upload/chunk/merge, telling the backend "all chunks are here, you can piece them together now."

The route handler for the merge interface is as follows. First, we perform necessary parameter validation:

// Merge chunks into a complete file
router.post("/upload/chunk/merge", async (req, res) => {
  const { fileHash, filename, totalChunks } = req.body;
  if (!fileHash || !filename || totalChunks === undefined) {
    return res.status(400).json({
      success: false,
      message: "Missing required merge parameters (fileHash/filename/totalChunks)",
    });
  }
})

Next, we verify the number of chunks. This step is critical — if the number of chunks doesn't match, it means there was packet loss during transmission, and the merged file will inevitably be corrupted.

const chunkDir = getChunksDir(fileHash);
if (!fs.existsSync(chunkDir)) {
  return res.status(400).json({
    success: false,
    message: "Chunk directory does not exist",
  });
}

// Read all chunks and sort by index
const chunkFiles = fs
  .readdirSync(chunkDir)
  .filter((name) => name.startsWith("chunk-"))
  .sort((a, b) => {
    const indexA = Number(a.replace("chunk-", ""));
    const indexB = Number(b.replace("chunk-", ""));
    return indexA - indexB;
  });

if (chunkFiles.length !== Number(totalChunks)) {
  return res.status(400).json({
    success: false,
    message: `Chunk count mismatch, expected ${totalChunks}, got ${chunkFiles.length}`,
  });
}

The backend must sort chunks in index order when merging.

Once all the chunks are obediently lying in the server's temporary directory, we enter the most critical "puzzle" phase — piecing these scattered fragments back into a complete file. The following code is the core operation for this step. I'll walk you through the design thinking behind it line by line.

const uploadsDir = getUploadsDir();
const finalFilename = `${fileHash}-${filename}`;
const outputPath = path.join(uploadsDir, finalFilename);

// Write the merged file in order
const writeStream = fs.createWriteStream(outputPath);
for (const chunkFile of chunkFiles) {
  const chunkPath = path.join(chunkDir, chunkFile);
  const chunkBuffer = fs.readFileSync(chunkPath);
  writeStream.write(chunkBuffer);
}
writeStream.end();

await new Promise((resolve, reject) => {
  writeStream.on("finish", resolve);
  writeStream.on("error", reject);
});

The first three lines do something simple — decide where the merged file will be stored and what it will be called. Next, writeStream is the core of the merge, and also the most interesting part.

You can think of createWriteStream as a "pipe" connected to the disk. Each time a chunk is read, a segment of data is fed into this pipe via writeStream.write(). The other end of the pipe continuously writes this data to the disk file.

The benefit of this approach is: only the data of the currently processed chunk is kept in memory at any time, and it's released once processed. Whether the file is 1GB or 10GB, the memory usage during merging remains almost constant.

Instant Upload: How to "Upload 1GB in 1 Second"?

In reality, the essence of instant upload is not "fast transmission," but "no transmission needed." Its workflow is like this: when you select a file to upload, the system first calculates a unique "fingerprint" (i.e., fileHash) for that file, and then uses this fingerprint to ask the server: "Do you already have this file?"

• If the server says "Yes" — great, just tell the frontend "upload successful."
• If the server says "No" — then proceed with the normal upload process honestly.

Application Scenario: Instant upload is very practical in enterprise network drives, cloud storage, and content management systems. The simplest example — 10 people on your team all upload the same product promotional video. Without instant upload, this video would be stored 10 times on the server, wasting 10 times the storage space and bandwidth. With instant upload, the uploads for the 2nd to 10th person are all completed in an instant.

To implement instant upload, the backend must first have the ability to "find a file based on fileHash."

Add a file check interface:

router.get("/files/check", (req, res) => {
  const { fileHash } = req.query;
  if (!fileHash) {
    return res.status(400).json({
      success: false,
      message: "Missing fileHash parameter",
    });
  }

  const uploadsDir = getUploadsDir();
  if (!fs.existsSync(uploadsDir)) {
    fs.mkdirSync(uploadsDir, { recursive: true });
  }

  const files = fs
    .readdirSync(uploadsDir)
    .filter((filename) => !fs.statSync(path.join(uploadsDir, filename)).isDirectory());
  const matchedFilename = files.find((filename) =>
    filename.startsWith(`${fileHash}-`),
  );

  if (matchedFilename) {
    const fileInfo = getFileInfo(matchedFilename);
    return res.json({
      success: true,
      data: {
        exists: true,
        file: fileInfo,
      },
    });
  }

  res.json({
    success: true,
    data: {
      exists: false,
    },
  });
});

With the check interface, the frontend can determine if a file already exists "before uploading." But what if two users upload the same file simultaneously? A scenario could occur: User A's check request says "does not exist," and User B's check request also says "does not exist." Then A and B both start uploading, ultimately causing the file to be written twice.

We add a check in the chunk merge interface as a final safety net:

router.post("/upload/chunk/merge", async (req, res) => {
  // ... other processes ...
  // Instant upload fallback: If the final file already exists, reuse the existing file and clean up the chunk directory
  const uploadsDir = getUploadsDir();
  const finalFilename = `${fileHash}-${filename}`;
  const outputPath = path.join(uploadsDir, finalFilename);
  if (fs.existsSync(outputPath)) {
    fs.rmSync(chunkDir, { recursive: true, force: true });
    const stats = fs.statSync(outputPath);
    return res.json({
      success: true,
      message: "File already exists, instant upload successful",
      data: {
        filename: finalFilename,
        originalname: filename,
        size: stats.size,
        mimetype: getMimeType(finalFilename),
        path: outputPath,
        url: `/uploads/${finalFilename}`,
      },
    });
  }
})

The frontend modifications are mainly concentrated in the startUpload function. The core idea is: before judging the file size, uniformly calculate the hash and perform the instant upload check.

const startUpload = async () => {
  const fileSize = file.size;
  // File chunking
  const chunks = splitFile(file, CHUNK_SIZE);
  // Calculate file hash
  const fileHash = await calcFileHashSampling(chunks, fileSize);

  const checkResponse = await checkFileExists(fileHash);
  if (checkResponse.data.success && checkResponse.data.data?.exists) {
    uploadMessage.value = "File already exists, instant upload successful!";
    loadFiles();
    return;
  }
  // Continue with subsequent flow...
}

The entire process can be summarized in one sentence: The first time a file is transmitted, it's an upload; the second time, it's an instant upload.

Resumable Upload: No Need to Start Over if the Network Drops

Instant upload solves the case where "the file completely exists." Resumable upload solves the case where "the file was partially uploaded." Imagine you're downloading a large file, and the progress reaches 80% when the network suddenly drops — without resumable download, you'd have to restart from 0% after reconnecting. But with resumable download, you only need to fill in the missing 20%.

The same logic applies to uploading. We've already implemented chunked uploads, where each chunk is uploaded independently, providing a natural foundation for resumable uploads. The core idea of resumable uploads can be summed up in one sentence:

Only transmit what hasn't been uploaded, not what has already been transmitted.

The backend modifications are mainly concentrated in the /api/files/check interface. The original logic was simple: check if a complete file exists. The upgraded logic needs to ask one more question: "If the complete file doesn't exist, are there any partial chunks in the chunk directory?"

router.get("/files/check", (req, res) => {
  // Check the chunk directory, return uploaded and missing chunk indices
  const chunkDir = getChunksDir(fileHash);
  const uploadedChunks = [];

  fs.readdirSync(chunkDir)
    .filter((name) => name.startsWith("chunk-"))
    .forEach((name) => {
      const index = Number(name.replace("chunk-", ""));
      if (!isNaN(index)) {
        uploadedChunks.push(index);
      }
    });
  uploadedChunks.sort((a, b) => a - b);
  
  // All indices
  const allIndices = Array.from({ length: total }, (_, i) => i);
  // Set of uploaded indices
  const uploadedSet = new Set(uploadedChunks);
  // Missing indices
  const missingChunks = allIndices.filter((i) => !uploadedSet.has(i));
  
  res.json({
    success: true,
    data: {
      exists: false,
      totalChunks: total,
      uploadedChunks,
      missingChunks,
    },
  });
})

The frontend modifications for resumable upload are much more complex than the backend — the backend just added an interface logic, while the frontend needs to rewrite the entire control layer of the upload flow. To allow the pause and stop functions to truly "interrupt" ongoing network requests, we need to pass the AbortController's signal to each chunk request:

const abortControllers = ref<Map<number, AbortController>>(new Map());
const uploadLargeFile = async (file: File, pendingIndices: number[]) => {
  const controller = new AbortController();
  abortControllers.value.set(index, controller);

  const formData = new FormData();
  formData.append('chunk', chunk);

  return request.post<UploadChunkResponse>('/api/upload/chunk', formData, {
    headers: { 'Content-Type': 'multipart/form-data' },
    signal, // 👈 Key: Pass in the AbortSignal
  });
}

During upload, each chunk creates an independent AbortController, stored in the abortControllers Map. When pause is triggered, all controllers are cancelled:

const pauseUpload = () => {
  // Abort all ongoing requests
  for (const [index, controller] of abortControllers.value) {
    controller.abort();
  }
  abortControllers.value.clear();
};

When resuming the upload, we first determine if an instant upload is possible based on the data returned by the check interface. Then, based on the missing chunk indices, we re-initiate the upload requests.

const startUpload = async () => {
  const checkResponse = await checkFileExists();
  if (checkResponse.data.success && checkResponse.data.data?.exists) {
    showMessage("File already exists, instant upload successful!", "success");
    return;
  }
  missingChunks = checkResponse.data.data?.missingChunks || [];
  await uploadLargeFile(file, missingChunks);
}

Summary

After all this discussion, let's finally look back and sort out the key milestones on this path of large file uploading.

You might think the topic of "large file upload" sounds a bit cliché, with tons of articles and ready-made wheels available online with a quick search. But my biggest takeaway after writing this set of code is:

Truly figuring out this feature from start to finish, versus just grabbing a wheel to use, the difference isn't just a few lines of code — it's a sense of control over the entire process.

Think back: we started from the most basic concepts — clarifying the difference between Blob and File, understanding the role of ArrayBuffer in hash calculation. Then we used File.slice() to cut large files into small pieces, used SparkMD5 to calculate fingerprints, and used p-limit to control concurrency. Next, we sent the chunks to the backend, letting Express with Multer receive them piece by piece and merge them in order.

On this foundation, we added two more layers of "magic": instant upload and resumable upload.

Looking back, the implementation logic of this solution isn't actually complicated, but every step hits the mark: using hash values to give files an "ID card," using chunking to break down large tasks into small units, and using state records to make the upload process recoverable. These are the "three carriages" of large file uploads — missing one makes it incomplete, and only combined do they truly solve this "stubborn problem."

I hope this article helps you thoroughly understand large file uploads. The next time you encounter a scenario requiring large file uploads, you'll have confidence — knowing how it runs under the hood, and knowing where to start troubleshooting if something goes wrong.

Source Code Address

If you think this was well-written, please follow my Juejin Homepage. For more articles, please visit Xie Xiaofei's Blog