The Standard Chunked Upload Recipe Fails on Real Networks — Here's the Fix

In frontend interviews, when asked about large file uploads, almost everyone can recite the standard textbook answer: use Blob.slice to physically split the file into chunks, then upload them concurrently with Promise.all, and finally call a backend merge API.

Sounds flawless, right?

But in real business scenarios, when your code runs in a weak network environment on a high-speed train, or under a Starbucks public Wi-Fi with extremely high packet loss, this standard answer often triggers a series of online disasters. The upload progress bar freezes at the very end, the browser tab suddenly crashes to a white screen, and even the backend gateway may flag you as a malicious attack and blacklist you.

Why does seemingly perfect logic shatter into pieces when it encounters a complex physical network? Let's talk about this topic 👇

---

What are the problems with conventional chunked uploads?

Most online tutorials teach you purely toy-grade code. Their biggest problem is: completely ignoring the browser's underlying concurrency limits and the fragility of the physical network.

When you split a 2GB large file into 100 chunks, and then casually use Promise.all to hurl all 100 requests at the backend simultaneously, the disaster begins:

First, modern browsers have strict limits on the number of TCP concurrent connections to the same domain (usually 6). The 100 requests you sent will queue up in the browser's underlying network stack, crowding each other out for extremely limited bandwidth, causing a large number of requests to directly report timeout errors during the queuing phase.

Secondly, in a weak network environment, if just two or three chunks fail to upload due to packet loss, the entire Promise.all directly enters the reject state. The state of the 97 chunks that were painstakingly uploaded earlier is instantly lost, forcing the user to start over from scratch. This experience is simply devastating.

A chunked upload without concurrency control, fault-tolerant retries, or state persistence is the beginning of a disaster.

---

Simple implementation of a native concurrency queue with fallback retry

An upload module that can truly run in a production environment should absolutely not rely on fancy third-party heavy libraries, but should honestly use low-level JavaScript to precisely take over the entire upload lifecycle.

We don't need to introduce a massive state machine. Using a minimalist async task queue combined with recursive control, we can implement upload logic with error retry and precise concurrency control using pure native Web API:

// Large file chunk concurrent upload queue control
interface ChunkTask {
  chunk: Blob;
  index: number;
  retryCount: number;
}

export async function uploadChunksWithControl(
  tasks: ChunkTask[],
  maxConcurrency = 4, // Extremely restrained concurrency count, protecting network bandwidth
  maxRetries = 3      // Number of retries in weak network environments
): Promise<void> {
  let currentIndex = 0;
  let activeCount = 0;
  let hasError = false;

  return new Promise((resolve, reject) => {
    const enqueue = async () => {
      // If there is an unrecoverable retry exhaustion, or tasks have all been dispatched, exit directly
      if (hasError || currentIndex >= tasks.length) {
        if (activeCount === 0 && !hasError) resolve();
        return;
      }

      // Get the current task to process and move the pointer
      const task = tasks[currentIndex++];
      activeCount++;

      try {
        await uploadSingleChunk(task);
      } catch (err) {
        // If encountering network jitter, don't report an error, but silently retry
        if (task.retryCount < maxRetries) {
          console.warn(`Chunk ${task.index} upload failed, attempting retry ${task.retryCount + 1}...`);
          task.retryCount++;
          // Push the failed task back to the end of the queue
          tasks.push(task); 
        } else {
          hasError = true;
          reject(new Error(`Chunk ${task.index} completely failed after ${maxRetries} weak network retries`));
          return; // Cut off subsequent uploads to avoid wasting resources
        }
      } finally {
        activeCount--;
        enqueue(); // Recursively trigger the task for the next free slot
      }
    };

    // At initialization, fill the concurrency pool and start the specified number of tasks
    for (let i = 0; i < maxConcurrency && i < tasks.length; i++) {
      enqueue();
    }
  });
}

// Simulate an independent network request for a single chunk
async function uploadSingleChunk(task: ChunkTask): Promise<void> {
  const formData = new FormData();
  formData.append('file', task.chunk);
  formData.append('index', task.index.toString());
  
  const res = await fetch('/api/upload-chunk', {
    method: 'POST',
    body: formData,
  });

  if (!res.ok) throw new Error('Network Error');
}

There is no magic in this code. It uses activeCount to firmly cap concurrent requests within the threshold (e.g., 4). At the same time, if a chunk fails due to a weak network, it not only avoids a total collapse but is also pushed back into the queue for re-queuing. This fallback strategy is the prerequisite for ensuring upload success rates in mobile weak network environments.

---

Pitfalls that only surface on real devices

Even if you write the concurrency queue above 👆, once you run it on a real business line for large files, you will still encounter blind spots that only appear in the deep end. These are what separate architects from ordinary developers 🙌.

Silent page crash?

Many beginners, when slicing, like to convert all Blob objects into ArrayBuffer or Base64 and store them in a huge array. If a user uploads a 5GB high-definition video, this low-level operation will instantly blow up the browser's heap memory, causing the page to freeze or directly throw an OOM white screen.

So we must be clear on one point: File.slice() produces only a pointer reference to the file's physical memory, not a real data copy. Before a real network request occurs, absolutely do not call any FileReader method to read the content. Only at the moment of sending the request should you throw the reference into FormData.

Avoid freezing the main thread when calculating file hashes

To implement resumable uploads and instant-upload verification, we need to use the file's hash value (like MD5 or SHA-256) as a globally unique identifier. But for files several GB in size, if you use pure JavaScript to calculate the hash entirely on the main thread, the page will freeze for dozens of seconds, paralyzing any button interactions.

So for large files over tens of megabytes, we must throw them into a Web Worker for chunked asynchronous computation. A more advanced engineering practice is sampling hashing (only extracting the first 2MB, the last 2MB, plus several small segments from the middle for concatenation), which can compress the computation time from dozens of seconds to a few hundred milliseconds while maintaining an extremely low collision rate.

State persistence

Mobile browsers are extremely unreliable. If a user switches out to reply to a WeChat message, the system may kill your upload page in the background. When the user reopens the page, all chunk states in memory are lost, and they can only grit their teeth and start over from scratch.

If you introduce IndexedDB or localStorage as a local state machine, every time a chunk is successfully uploaded, a record is persistently written to the local database. After the user refreshes the page, by comparing hashes with local records, you can directly filter out the already successful chunk array and seamlessly take over the interrupted upload task.

---

Large file resumable upload architecture diagram (recommended to bookmark 😁)

In the daily business of large companies, many architectural optimizations that sound high-level actually have very simple underlying logic.

What we are fighting against is never some profound algorithmic mechanism, but the browser's pitiful concurrency limits, the fragile physical network, and the device's physical memory ceiling. Frontend engineers must not only know how to write code, but also maintain reverence for the underlying physical limits 🫡.

When you can deliver a massive file, under harsh conditions with network speeds of only 50KB/s and frequent disconnections, steadily and without missing a single piece to the server, you have truly touched the threshold of frontend engineering in this field 🫵.

Alright, that's all for today's sharing 🙌.