跪拜 Guibai
← Back to the summary

HatchZip's Truncation Trick Extracts a 2.2GB Archive in 300MB of Free Space

300MB Budget, Extract a 2.2GB Archive

A 2.2GB ZIP, when available budget is pushed to 300MB, normal extraction still requires freeing 2.2GB first; HatchZip just extracts it.

Three verifiable facts first:

HatchZip is an open-source decompressor written in Rust (MIT/Apache-2.0 dual license); I tested 0.1.8, and as of Nov 5 the repo has 2,497 stars. It doesn't invent a compression algorithm; it changes “how files are placed during extraction.”

一、 It's not streaming extraction, but “extract a batch, cut once”

Normal decompressors read the whole archive, write files one by one, and leave the archive intact. So the peak is always “archive + all output”; a 2.2GB archive needs at least 4.4GB of disk.

HatchZip does the opposite: it starts from the end of the archive, extracts a batch, then truncates the bytes of that batch from the archive. The truncated bytes immediately become available space for the next batch output.

This requires three prerequisites, without any of which it can't work.

First, the index must be saved in advance. ZIP's central directory is at the tail; truncating the tail directly destroys the index. So before starting, it copies the tail to <archive>.hatchzip-tail, records the CRC, and the reader combines the remaining archive plus the saved tail into a logical file. The suffix you see isn't junk; it's a backup of the index.

This tail is not just stored; it's read into memory and kept until extraction ends. The source limit is 1,000,000,000 bytes; exceeding that, it refuses before touching files, saying “index too large to hold in memory, archive not modified.” Normal ZIP central directories are far from that magnitude — my 22-entry archive is only a few hundred bytes — but archives with extremely many entries will hit this line. I didn't create such a large index; that's from source, not tested.

Second, the commit order can't be reversed. Truncation can only happen after output is actually flushed and the journal is written. The source commit order is: sync_batch flush → write journal → then set_len truncate. The comment states plainly: the log cannot claim a file that could be reclaimed after power loss. This order determines whether already committed batches count after an interruption.

Third, batch size must adapt to the disk. In engine.rs there are two constants: BATCH_MAX_UNITS = 512 and BATCH_MAX_BYTES = 2GiB. Each batch's budget is “available space minus --keep-free”, then min with 2GiB:

let budget = avail.min(BATCH_MAX_BYTES.max(first));

first is the size of the first file in the batch, so the budget floor is “at least one file.” The fuller the disk, the smaller the batch; the emptier, the larger, up to 2GiB. Whether space is saved depends on how small it's willing to cut.

This explains a counterintuitive point: on a wide disk it doesn't save. Because the batch hits the 2GiB cap, behavior is similar to normal unzip, even slightly higher due to the tail index copy and journal maintenance.

The check subcommand prints this budget directly without touching files; it's a dry run:

$ hatchzip check game.zip
game.zip  2.2 GB zip, 2.2 GB uncompressed in 22 entries
  available                    22.0 GB
  standard extraction needs    2.2 GB
  required                     104.9 MB (+2.0 GB reserved)
  available after              22.0 GB

Here required is 104.9MB, not 2.2GB. It doesn't need “space for all output” but “space for the first file of the batch.” The 2.0GB in parentheses is the default --keep-free, meaning it reserves 2GB untouched and uses the rest for batches. Together they show the true promise: not double space, but one file's margin plus a reservation you allocate.

二、 Three scenario tests

The test archive is synthetic: 21 × 100MiB /dev/urandom files, zip -0 stored uncompressed, totaling 2,202,013,120 bytes / 22 entries (21 files + 1 directory). It's not a real game package or real project files — this affects the boundary conclusions, explained later.

Environment: macOS 27.2, arm64, HatchZip 0.1.8, Info-ZIP zip 3.0 / UnZip 6.00. Each scenario run once, sampling script target 80ms interval, actual ~120–280ms (stat + du + df take time), peaks based on sampling points, all single measurements.

Scenario Tool Time Peak (archive+extracted) Max free drop
Wide disk, ~34GB free unzip 6.26s 4,404MB 1,966MB
Wide disk, ~29.5GB free hatchzip 7.79s 4,429MB 2,212MB
Budget 300MB hatchzip 6.97s 2,412MB 205MB
Budget ~70MB hatchzip refused, rc=3 — not measured

[Tested] The wide disk rows are key: it didn't save compared to unzip. Peak difference 25MB (0.6%), same magnitude. The df drops can't be directly compared: runs at different times, starting free 34GB vs 29.5GB, and df has a ~40MB step; so only “around 2GB”, no conclusion which is better.

[Tested] The third row is the interesting one. With --keep-free 26283MB pressing the available budget to 300MB, the same 2.2GB archive extracted. Peak 2,412MB is the max of “not-yet-truncated archive + already-extracted files”, about 210MB over the archive itself (2,202MB); df max drop 205MB, with a ~40MB step, so treat as a magnitude.

The archive length is a staircase, seen clearly in sampling (bytes):

2202013120 → 1992295869 → 1782580521 → 1572865173 → 1363149825
→ 1153434477 → 943719129 → 734003781 → 524288433 → 314573085 → 0

Most steps are about 200MiB (measured 209,715,348 bytes), i.e., two 100MiB files; the first batch has an extra ~2KB (tail index overhead), and the last step 314,573,085 bytes is 300MiB, three files. Batch sizes vary with remaining budget, not fixed two-per-batch; extract then cut, cut then extract.

This means the peak is not fixed: it equals “not-yet-truncated archive” plus “already-extracted files”, which trade off over time. 2.4GB is the highest sampled, not a theoretical value.

三、 When does it stop

The floor is one file's size, not zero. When the budget is pressed to ~70MB (--keep-free 20000MB), missing 34.5MB for a 104.9MB file, it refuses before creating the output directory:

Insufficient disk space: 34.5 MB more is required. Free up space or lower --keep-free.

Exit code 3. I ran it again with records: ck.zip SHA-256 unchanged (353ba6ac…), bytes unchanged (2,202,013,120), no output directory, no journal file. This is more important than “can extract”: failure is clean.

It can resume after budget exhaustion, but only if the budget suffices. I used --keep-free 24357MB, ran to 2.6s, this batch fell short; the tool self-reported not enough disk space to continue and exited: the archive reduced from 2,202,013,120 to 1,992,295,869, losing one 200MiB batch plus ~2KB tail index (measured diff 209,717,251 B), and the journal's committed_len matched the file length. Rerunning with a similar budget stops again — my second run with --keep-free 24000MB stopped at 0.42s at 1,677,722,847 (printing resuming: 2.0 GB of 2.2 GB remaining). After removing --keep-free and giving enough margin, it completed, printing:

resuming: 1.7 GB of 2.2 GB remaining

Then it finished, all 21 files verified, the journal auto-cleaned. These three steps prove “can resume after budget exhaustion”, given the next budget is actually sufficient.

SIGKILL I only half-verified. I killed at 3.59s, before the first batch committed: the archive length was unchanged (2,202,013,120), the journal's committed_len was still full length, no resumable progress; rerun printed resuming: 2.2 GB of 2.2 GB remaining, and it took 13.8s for a full redo. “Supports resume” doesn't mean “any interruption saves”; it depends on whether the first batch has been committed. Whether killing after commit can resume, and whether power loss leaves a bad state, I didn't test; this can't be taken as crash-safety evidence.

Path traversal is refused outright. A package with ../../evil.txt, hatchzip l reports would be written outside the destination folder; extraction refused, exit code 1. I checked before and after: the archive is still 342 bytes, SHA-256 unchanged (3456b288…), no output directory, and /tmp/hzb/evil.txt doesn't exist. It doesn't even list, let alone write outside.

四、 Costs and boundaries

It can only truncate from the tail. Truncation uses ftruncate / SetEndOfFile, cross-platform; but to delete bytes from the middle or head, it would need hole punch, which FAT32 and exFAT don't support. So this mechanism naturally applies only to formats where file data precedes the index and can be eaten from the back. The author documents this limitation.

It deletes the original archive by default. At success, the archive is gone. To keep it, add --keep-archive, but then it degrades to a normal decompressor, and the space requirement returns to “archive + all output”, losing the space saving. It's a binary choice, no middle ground.

Overwriting an existing directory requires an explicit flag. Existing files in the target path are untouched by default unless --replace is set; then old files are moved into .hatchzip-replaced inside the target, deleted after all extraction, and restore can move them back — except those whose paths are taken by newly extracted files. This is from the docs; I didn't test it, nor did I run restore.

Multiple archives in one run are checked sequentially. Each archive checks space before starting; the first that doesn't fit terminates the whole round. So batch extraction may fail mid-way, not at the start. Also from docs, not tested.

Format support is narrower than advertised. The README lists ZIP (incl. ZIP64, self-extracting exe), 7z, uncompressed tar, tar.gz/tgz, split, and RAR (solid only with --keep-archive). I only tested ZIP; the others I didn't run. Split and RAR resume behavior I haven't verified. Don't apply this article's conclusions to 7z.

Memory is not zero-cost. The tail index is resident, capped at 1GB; each batch's output buffer and CRC checks also go through memory. I didn't measure RSS, only disk, so this article doesn't make claims about memory usage. If an archive has a huge single file, the streaming buffer will amplify — you'll need to measure it yourself.

Real projects weren't tested. The test archive is 21 equal-sized 100MiB files, each a whole batch boundary, the ideal case. A real node_modules has tens of thousands of small files; batches will hit BATCH_MAX_UNITS = 512 instead of the byte limit, and the truncation count and metadata overhead will change. I didn't test that, and I won't guess.

A truly full disk wasn't achieved. I tried hdiutil to create a 260MB APFS image to simulate, but it reported “device not configured” and didn't succeed. The “300MB budget” above is simulated by --keep-free pressing the available quota, not a physically near-full disk. On a true full disk, filesystem fragmentation and allocation behavior may differ; I can't rule that out.

Timing should not be a conclusion. The two wide-disk runs were 6.26s and 7.79s; later paired runs gave 8.5s and 3.0s, wildly variable, clearly affected by page cache. In that paired test, hatchzip's output directory wasn't cleared (recorded 2.4GB, larger than the archive data), so it only shows “two runs of the same archive can differ by more than a factor of two”, not a comparison. Single measurements, no speed conclusion.

五、 When it's worth using

The judgment criterion is not “is the archive big” but “is the disk tight”.

If free space is double the archive, any tool works; HatchZip adds an extra index copy and journal overhead. When --keep-free is squeezed to only 100–200MB of budget while the archive is several GB, it's one of the few tools that can finish directly — provided the format is in the support list and you accept the original archive being eaten. True physical full disk wasn't tested; this is derived from the budget constraint.

One class of scenario is a natural fit: download-then-extract-in-place pipelines. When a download is half done and the disk is nearly full, normal unzip stalls at “need 2x space”; HatchZip can extract while eating the archive. The cost: on extraction failure, the original archive may be incomplete — it relies on the journal and restore subcommand for recovery, but that path wasn't tested.

If you occasionally extract a big archive, switching tools isn't worth it. If you often choose between “not enough disk” and “must extract now”, it changes the problem from “what to delete first” to “just run”.

One layer to reason about: the “space saving” is not unconditional; it's a “swap batch size for peak” trade-off. Default --keep-free 2GB means it initially uses only “available minus 2GB”; on a 30GB disk that budget hits the 2GiB cap, each batch maxes at 2GiB, and the peak climbs. To lower the peak, you actively raise --keep-free — at the cost of less margin for yourself; if the estimate is wrong, you hit the “34.5MB short” refusal. Space saving and “will definitely finish” are a pair of parameters you set, not auto-optimal.

My 300MB budget passing does not mean 300MB is a universal safe value. It depends on the largest file in the archive: the pre-check margin is one file's size; this archive has 100MiB files, so 300MB is ample. If a single 1.5GB file were present, the same 300MB budget would be refused at pre-check. Change the archive, and you must check again; don't use this article's numbers as a template.

Reproduction materials

All numbers are single runs on this machine. A single measurement can only say “what was observed this time”, not “it is stable at that value”.