How a 700 GB Model Fits Into 8 GB of VRAM

You spent thousands on a 4070 with only 8 GB of VRAM; the model you want to run has a 700 GB weight file. This isn't "a little short" — it's two orders of magnitude apart. Today we'll thoroughly unpack quantization and distillation: how they turn 700 GB into 8 GB, and what trade-offs they make.

---

1. First, do the math: what makes a large model "large"

Before discussing compression, we need to be clear on one thing — what determines a model's size.

A neural network model is essentially a massive collection of floating-point parameters. "Parameter count" is simply how many trainable numbers exist in the model. For example, LLaMA-2 70B — the "70B" means 70 billion parameters. Those 70 billion numbers are all the knowledge the model has "learned."

How much space do 70 billion parameters occupy? The key is how many bits are used to store each parameter.

Deep learning training and inference default to FP32 (32-bit single-precision floating point) or FP16 (16-bit half-precision floating point). Let's calculate using the most common FP16:

70 billion parameters × 2 bytes = 140 billion bytes ≈ 130 GB

That means the weight file alone for a 70B model occupies about 130 GB of disk space.

VRAM requirements are even more brutal. During inference, beyond the weights themselves, you also need:

1. Weights: 130 GB — the biggest chunk;
2. KV Cache: intermediate results from the attention mechanism; the longer the context, the more it consumes;
3. Activations: intermediate outputs of every layer during forward propagation;
4. Framework overhead: VRAM consumed by PyTorch or similar frameworks.

So a 70B model can easily hit 150 GB+ during inference, without even accounting for batch size and long context. Meanwhile, a consumer GPU (like an RTX 4070) has only 8 GB of VRAM, and a single H100 has 80 GB.

"700 GB into 8 GB" — the 700 GB here likely refers to a super-large model (like a 405B LLaMA-3 at full precision), while 8 GB is an ordinary consumer GPU. The gap between them is what we need to bridge.

Core conclusion: a model is "large" because of parameter count × storage precision per parameter. To compress, there are exactly two directions — reduce the number of parameters (distillation), or reduce the precision of each parameter (quantization). Let's break each down.

---

2. Quantization: compressing "high precision" into "low precision"

2.1 What is quantization

The essence of quantization is one sentence: use fewer bits to approximate the original floating-point numbers.

An analogy. Suppose you have a high-definition photo (FP32), where each pixel uses 32-bit color and the detail is indistinguishable to the naked eye. Quantization is converting it to 16-bit, or even 4-bit color depth. The image gets a little "blurry," but visually the difference may be negligible, while the file size plummets.

Model quantization works the same way:

• FP16 → INT8: size halved, precision loss very small;
• FP16 → INT4: size becomes 1/4, precision loss exists but is usually "acceptable";
• More aggressive INT2 / INT3: even smaller, but model quality drops noticeably; rarely used.

For that 70B model above:

Precision	Size per parameter	Total weight size	Fits in 8 GB VRAM?
FP16	2 bytes	~130 GB	❌ Far from it
INT8	1 byte	~65 GB	❌ Far from it
INT4	0.5 bytes	~33 GB	❌ Still far from it

Wait — INT4 is still 33 GB and won't fit in 8 GB?

Here we need to clear up a common misconception: "700 GB into 8 GB" usually doesn't mean a single 70B model, but rather an originally 700 GB VRAM model (like a full-precision 405B giant) after quantization, or a 7B–13B class model after quantization, can run on 8 GB of VRAM.

Let's calculate with a more realistic example. A 7B model:

• FP16: 7B × 2 = 14 GB — won't run on 8 GB VRAM;
• INT8: 7 GB — barely, but still tight with KV Cache;
• INT4: 3.5 GB — add KV Cache and overhead, 8 GB VRAM is fully sufficient, with room left for context.

This is the real meaning of "fitting it in": through quantization, a model goes from "can't run" to "can run." Quantization isn't magic, but it's the most direct and effective compression technique.

2.2 Why "the loss isn't as bad as you'd think"

Many people, on first hearing about INT4 quantization, instinctively question: cutting a 32-bit number down to 4 bits drops 87.5% of the information — can the model still work?

The answer is: yes, and often far better than you'd imagine. Three reasons:

First, neural network weight distributions are naturally suited to quantization.

After training, model weights are not randomly distributed across the full floating-point range; they are highly concentrated near 0, roughly following a normal distribution. This means most weight values are very "small," and when represented at low precision, errors concentrate on a few "extreme values" — and those extreme values often don't affect overall inference results.

Second, neural networks have inherent "fault tolerance" (redundancy).

Modern large models have billions of parameters; tiny perturbations to individual parameters get "averaged out" by the vast number of other parameters. It's like randomly changing a few pixels' colors in a 1-megapixel photo — you simply can't tell. The more "over-parameterized" the model, the greater this redundancy, and the safer quantization becomes.

Third, quantization algorithms are not "brute-force truncation" but "smart mapping."

Early naive quantization (called "Round-to-Nearest") did indeed cause noticeable precision drops. But modern quantization methods incorporate many careful designs:

• Per-Channel / Per-Group quantization: instead of using a single scaling factor for the entire tensor, scaling is computed per column or even per group of a few dozen elements — error is much smaller;
• Calibration: run real data through the model before quantization, collect statistics on actual activation distributions, and determine the optimal scaling range and zero-point;
• Outlier handling: a small number of "outlier" weight values are preserved separately at high precision (this is the core idea behind AWQ).

It's these engineering optimizations that have brought INT4 quantization's performance loss from "unusable" down to "imperceptible on most tasks."

2.3 What quantization actually loses

Of course, quantization is not a free lunch. The losses mainly appear in a few areas:

1. Precision degradation on extreme tasks: math reasoning, code generation, long-chain logic — tasks sensitive to precision — INT4 may be 1–3 points worse than FP16;
2. Fragility of small models: the smaller the model, the less redundancy, and the more quantization hurts. A 1B small model quantized to INT4 may directly "go dumb"; but a 70B model quantized to INT4 is almost unnoticeable;
3. "Long-tail" loss from outliers: certain layers have a few exceptionally large weights where quantization error concentrates, potentially causing "occasional weirdness" on specific inputs.

One-sentence summary: the larger the model and the more "relaxed" the task (generation, chitchat, summarization), the safer quantization is; the smaller the model and the more "strict" the task (math, code, logic), the more dangerous quantization becomes.

---

3. What's the actual difference between quantization formats: GPTQ / AWQ / GGUF / bitsandbytes

The quantization landscape is varied, and newcomers easily get confused. They can actually be divided into two categories: "research/algorithm layer" and "engineering/deployment layer."

3.1 GPTQ: data-driven "post-training quantization"

GPTQ belongs to Post-Training Quantization (PTQ) — meaning the model is already trained, and you quantize it afterward without retraining.

Its core idea is layer-wise quantization based on second-order information (the Hessian matrix):

1. Feed a batch of calibration data to the model and record the "importance" of each layer's weights;
2. During quantization, "compensate" each layer's quantization error onto the remaining unquantized weights, rather than letting errors accumulate directly;
3. This significantly reduces the accumulated error from quantization.

GPTQ's strengths are high precision and INT4/INT3 support; its weaknesses are slow quantization, requires a GPU, and is sensitive to calibration data quality. It's mainly used in scenarios like "I have the original FP16 model and want to quantize an INT4 version myself" — for example, running the AutoGPTQ library.

3.2 AWQ: protecting the "critical 1%" of weights

AWQ (Activation-aware Weight Quantization) is an "evolved version" of GPTQ, proposing a counterintuitive but highly effective observation:

Only about 1% of weights in a model are "critical"; protect that 1% during quantization, and the remaining 99% can be compressed freely.

That 1% of critical weights are the "outliers" — weights with exceptionally large values that have an outsized impact on outputs. AWQ's approach:

• First, by analyzing activations, find the most important 1% of weight channels in each layer;
• Scale up these critical channels so they lose less when quantized to low precision;
• After quantization, apply a corresponding scaling compensation to the associated activations, canceling out the effect of the "scale-up."

It's like: give the top students "bonus points," then "curve everyone to the same line," and finally "deduct the bonus back" — the top students' grades are unaffected, while the weaker students' grades get compressed freely. This "importance-weighted" approach gives AWQ better precision than GPTQ at the same bit width, and quantization is faster.

Use case: you have the original model + a GPU and want to quickly get a high-quality INT4 model. The tool is AutoAWQ.

3.3 GGUF: a format built for "CPU + consumer GPU"

GGUF is the model format promoted by the llama.cpp project, and it's fundamentally different from the previous two:

• GPTQ/AWQ are primarily aimed at GPU inference, relying on CUDA;
• GGUF was designed from the start for CPU inference + Unified Memory, though it also supports GPU offloading.

GGUF's characteristics:

1. Single-file distribution: the entire model (including config, vocabulary, weights) is packaged into one .gguf file — download and use, no messing with a pile of shard files;
2. Multi-level quantization support: from Q8_0, Q6_K, Q5_K_M, Q4_K_M to Q2_K and more, offering different "precision/size" tiers; the K-quant series is a community-tuned mixed-precision scheme;
3. CPU-friendly: can run on pure CPU, or use the -ngl parameter to offload some layers to GPU, making full use of "8 GB VRAM + large system memory" machines.

For the average user, GGUF is nearly the optimal choice for "running large models locally" — download a Q4_K_M GGUF file from HuggingFace and run it with llama.cpp or Ollama.

3.4 bitsandbytes: "online quantization" for training and fine-tuning

bitsandbytes is the most common quantization library in the HuggingFace ecosystem, and its positioning differs from the previous three:

• GPTQ/AWQ are "quantize first, then use";
• bitsandbytes is "runtime dynamic quantization" — quantizing as it loads.

Its most classic feature is QLoRA: quantize the main model with NF4 (4-bit NormalFloat), then perform LoRA fine-tuning on top. This brings a 7B model that originally required 48 GB VRAM to fine-tune down to running on a single 8 GB GPU.

bitsandbytes' NF4 is an "information-theoretically optimal" 4-bit data format, specifically designed for normally-distributed weights, with lower quantization loss than naive INT4.

Use case: you don't have enough VRAM but want to fine-tune, or want to temporarily load a large model for experimentation. A single load_in_4bit=True line does it.

3.5 A table to clarify the four approaches

Approach	Positioning	Precision tiers	Target hardware	Typical tool/use
GPTQ	Post-training quantization	INT4/INT3/INT8	GPU	AutoGPTQ, quantize models yourself
AWQ	Protect critical weights	INT4/INT8	GPU	AutoAWQ, fast high-quality quantization
GGUF	Distribution/inference format	Multiple tiers	CPU+GPU	llama.cpp / Ollama, run locally
bitsandbytes	Runtime quantization	NF4/INT8	GPU	QLoRA fine-tuning, temporary loading

If you can't remember, remember this rule of thumb: to compress a model yourself, use AWQ; to download a ready-made one, use GGUF; to fine-tune, use bitsandbytes; for extreme precision, use GPTQ.

---

4. Distillation: a completely different path

If quantization is "compressing the same model smaller," then distillation is "making a small model smarter." The philosophies of the two paths are entirely different.

4.1 The essence of distillation: a small model is not a "shrunken large model"

This is the easiest point to misunderstand about distillation. Many people think "distillation" means pulling out some parameters from a large model to get a "mini version of the large model." Absolutely not.

The core idea of Knowledge Distillation is to have a small model "imitate" the output behavior of a large model, rather than directly inheriting its parameters.

An analogy:

• Quantization = reprinting a 700-page book in smaller type to fit 100 pages (content unchanged, just higher information density, possibly with some wear);
• Distillation = having a student read that 700-page book, then write their own 100-page "study notes" (content reorganized, capturing the "essence" rather than the original text).

Technically, distillation works like this:

1. There is an already-trained large model (Teacher);
2. There is a small model (Student) with far fewer parameters;
3. Feed the same inputs and have both teacher and student produce outputs;
4. When training the student, don't just make it "get the right answer" (hard labels); also make it imitate the teacher's "output probability distribution" (soft labels).

4.2 Why "imitating probability distributions" beats "memorizing answers"

This is the most elegant part of distillation, and its soul.

Suppose a classification task where the correct answer is "cat." Ordinary training only tells the student "this is a cat" (hard label, one-hot: cat=1, others=0). But the teacher model's output might look like:

cat = 0.85 (indeed a cat)
tiger = 0.12 (but looks like a tiger)
leopard = 0.02 (also a bit like a leopard)
dog = 0.01

This "soft label" contains information far beyond the correct answer — it tells the student: "cat" and "tiger" are close, "cat" and "leopard" are somewhat close, and "cat" and "dog" are completely unrelated. By imitating this probability distribution, the student learns a kind of "inter-class similarity structure" that hard labels can never provide.

That's why a distilled small model is often stronger than a "small model trained from scratch with hard labels" — it inherits the teacher model's "way of seeing the world," not just "the answers themselves."

4.3 Distillation in the modern large-model era: from "behavior imitation" to "data distillation"

In the large-model era, distillation has taken on new forms. The mainstream approaches include:

1. Soft-label distillation (classic method) As described above, train the student using the teacher's output probability distribution. In generative models, this means "having the student imitate the teacher's probability for each token."

2. Data distillation (currently the hottest) Have the large model generate massive amounts of high-quality training data, then use this "teacher-generated data" to train a small model. For example, use GPT-4 to generate millions of instruction-response pairs to train a 7B small model. This isn't "letting the student watch the teacher" but "having the student do problems written by the teacher." Many headlines about "small models catching up to large models" (various "small cannon" models) are essentially data distillation + high-quality data cleaning.

3. Feature distillation Have the student model imitate the teacher model's intermediate layer features (not just the final output), transferring more "internal knowledge." Training is more complex, but the transfer effect is sometimes better.

4. White-box distillation (e.g., DeepSeek-R1 → small models) Open-source large models (white-box, where weights and logits are visible) yield the best distillation results. For example, after DeepSeek-R1 was released, the community used it to distill a bunch of 7B and 14B small models that perform astonishingly well on math reasoning — this is the power of white-box distillation: you can access the teacher's complete, fine-grained output distribution.

4.4 Distillation vs. Quantization: how to choose

This is the most critical decision point in this article. The essential differences between the two paths:

Dimension	Quantization	Distillation
Compression target	Precision of the same model	Swap to a small model + transfer knowledge
Parameter structure	Unchanged (same number of parameters)	Changed (parameter count truly reduced)
Main cost	Precision loss	Requires training data/compute
Requires training?	Usually not (PTQ)	Yes
Suitable scenarios	Fast deployment, inference acceleration	Want a "small but strong" standalone model
Typical example	INT4 LLaMA-3	Various 7B "small cannons"

One-sentence summary:

• You want to quickly get your large model running → quantization (especially GGUF, download directly);
• You want a long-term, small-but-strong model and are willing to spend time training → distillation;
• Very often, stack both: first distill a small model, then quantize it — good results and good cost-effectiveness.

---

5. Practical decision-making: a checklist

All that theory — when it comes to actual practice, how do you choose? Here's a decision checklist you can copy directly:

Scenario 1: I only have an 8 GB VRAM GPU and want to run a large model locally

Recommendation: GGUF + INT4 quantization (Q4_K_M), using Ollama or llama.cpp.

• Directly download a ready-made GGUF model, choose the Q4_K_M tier (best precision/size balance);
• 8 GB VRAM can smoothly run 7B–8B Q4 models; 13B will be tight and needs -ngl to partially offload to system memory;
• Don't agonize over the tiny precision differences between GPTQ/AWQ; for daily use, GGUF's Q4 is fully sufficient.

Scenario 2: I have a 70B large model and want to run it on my server

Recommendation: AWQ (or GPTQ) INT4 quantization, deployed with vLLM or TensorRT-LLM.

• In a server GPU environment, prefer AWQ — fast quantization, high precision;
• 70B INT4 is about 33 GB; with vLLM's KV Cache management, a single 40 GB or 80 GB card can run it;
• For extreme throughput, use TensorRT-LLM; for development efficiency, use vLLM.

Scenario 3: I don't have enough VRAM but want to fine-tune a model

Recommendation: bitsandbytes QLoRA (NF4 quantization + LoRA).

• One load_in_4bit=True line + LoRA config brings 7B model fine-tuning down to runnable on 8 GB VRAM;
• After fine-tuning, you can "dequantize" and merge, or publish via GGUF.

Scenario 4: I want a "small but strong" long-term model

Recommendation: distillation (prefer white-box distillation), paired with high-quality data.

• Find an excellent open-source large model as teacher (e.g., white-box models like DeepSeek-R1);
• Use the teacher's outputs/data to train a 7B–14B small model;
• After training, apply INT4 quantization to balance size and performance.

Scenario 5: I don't know which to choose, just want to get something running and see

Recommendation: Ollama + a 7B/8B Q4 model.

• ollama run llama3 (or qwen, glm) — one command and done;
• It's already quantized by default, no fiddling required;
• Once it's running and you have a feel for it, then revisit AWQ/GPTQ/distillation as "advanced play."

---

6. Final words: compression isn't "cutting corners" — it's "engineering wisdom"

Back to the opening question: how do you fit a 700 GB model into 8 GB of VRAM?

We've now made the answer clear:

1. Quantization compresses each parameter's precision from 32 bits to 4 bits, trading "information density for space," using neural network redundancy to "hedge" the precision loss;
2. Distillation takes a different approach — instead of compressing the original model, it "teaches" a student model with far fewer parameters;
3. On the engineering side, formats and tools like GGUF, AWQ, GPTQ, bitsandbytes turn theory into "download-and-use" reality.

Thinking one layer deeper, what this reflects is a core contradiction in large-model deployment: models are getting stronger and larger, but the hardware that can actually run large models is always scarce and expensive. Quantization and distillation are the bridge between ordinary people and large models — they let an ordinary developer, with an ordinary consumer GPU, also enjoy the capabilities large models bring.

So stop thinking of "compression" as some kind of "downgrade." Knowing how to compress is truly knowing how to deploy. Understand quantization, and you'll know how to spell "cost-effectiveness" for running large models locally; understand distillation, and you'll know why those "small cannons" can catch up to large models.

I hope this article helps you break through that layer of fog. Next time you see words like Q4_K_M, AWQ, GGUF, you won't be "completely lost" — you'll have a mental ledger: what this thing is trading for what, and whether it's worth it.