How a 700 GB Model Fits Into 8 GB of VRAM
Consumer GPUs remain the default hardware for most developers, yet model sizes have outraced them by a factor of 20× or more. Quantization and distillation are the only practical bridges — without them, a 4070 owner simply cannot run a 7B model at all, let alone fine-tune one.
A 70-billion-parameter model at FP16 occupies roughly 130 GB of weights alone, and inference pushes VRAM past 150 GB — two orders of magnitude beyond a typical 8 GB consumer card. Quantization attacks the problem by representing each weight in fewer bits: INT4 drops a 7B model from 14 GB to about 3.5 GB, making it comfortably runnable on 8 GB. The loss is smaller than intuition suggests because weight distributions cluster near zero, over-parameterized networks absorb individual errors, and modern algorithms use per-channel scaling, calibration, and outlier protection rather than naive rounding.
Four quantization ecosystems serve different needs. GPTQ uses second-order information for high-precision post-training quantization but is slow and GPU-bound. AWQ identifies and protects the critical 1% of outlier weights, yielding better precision at the same bit width with faster processing. GGUF packages models into single files with multi-tier quantization levels (Q4_K_M being the sweet spot) and runs on CPU plus optional GPU offloading via llama.cpp or Ollama. bitsandbytes performs runtime dynamic quantization; its NF4 format plus LoRA (QLoRA) lets an 8 GB GPU fine-tune a 7B model.
Distillation takes the opposite approach: a small student model learns to imitate a large teacher's output probability distributions, absorbing inter-class similarity structure that hard labels never convey. Modern variants include data distillation — generating synthetic training data from the teacher — and white-box distillation from open-weight models like DeepSeek-R1, which has produced surprisingly capable 7B–14B math-reasoning models. The two techniques stack well: distill first, then quantize the result.
The '700 GB into 8 GB' framing is rhetorical, not literal for a single 70B model — INT4 on a 70B still yields ~33 GB. The real win is making 7B–13B models run on consumer hardware, or bringing a full-precision 405B-class model into the realm of a single high-end server GPU.
Quantization safety scales with model size: a 70B model at INT4 is nearly lossless for most tasks, while a 1B model at INT4 can become unusable. This inverts the intuition that smaller models should be easier to compress.
AWQ's 'protect 1% of weights' strategy is a concrete instance of a broader principle: in over-parameterized networks, most parameters are negotiable, and identifying the non-negotiable ones is the whole game.
GGUF's dominance in the local-running community is as much about distribution ergonomics (single file, no shards, tiered options) as about technical quantization quality — deployment UX matters as much as the algorithm.
Distillation's real power is not compressing a specific model but transferring a 'worldview': the student learns which concepts are similar, a structural knowledge that one-hot labels erase. This explains why distilled small models can outperform same-sized models trained from scratch on hard labels.
The checklist reveals a pragmatic truth: most developers should start with Ollama + a Q4 GGUF and only reach for AWQ/GPTQ/distillation when that baseline proves insufficient. The tooling hierarchy is as important as the theory.
The levers you can pull on the browser side are different. I have a pure front-end segmentation model, int8 weights 47MB. WebGPU runs at 768, fallback WASM runs at 512 — the weights are identical between the two, what's saved is entirely intermediate feature maps, not download size. On the web, you can only touch the activation value lever.