INT4 cut GPU power but raised total energy 22.9% on GSM8K
Quantizing a reasoning model to four bits looks like an efficiency win. The model uses less memory and the GPU draws less power. A new preprint shows why those savings can disappear across a complete request.
On one GSM8K test, DeepSeek-R1-Distill-Llama-8B in FP16 drew 171.5 watts and used 4,955 joules per query. The same model under full INT4 quantization drew only 134.1 watts, but it generated 1,094 output tokens instead of 891. Total energy rose to 6,090 joules, an increase of 22.9%.
The Reasoning-Aware Compression preprint measures power, duration, output length, accuracy, and energy together. It also rewards a careful reading. Some of its most ambitious accuracy claims conflict with numbers in its own tables.
Lower watts did not mean lower energy
Power is a rate. Energy accumulates over time. A model that draws fewer watts can still consume more joules if it runs longer.
That distinction is easy to lose in model-compression discussions. Hugging Face's bitsandbytes documentation describes quantized linear layers as a way to reduce model memory. The paper used the library's NormalFloat 4 format across linear projections, with FP16 compute and double quantization disabled. It tested DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B. The Llama-8B model card identifies that model as one of six dense models distilled from DeepSeek-R1.
The researchers ran five benchmarks covering arithmetic, competition mathematics, formal logic, natural-language inference, and multi-hop question answering. Each model, dataset, and condition used greedy decoding. Most reported results averaged eight experiments of 30 samples. A calibration half was used to identify sensitive modules, while reported evaluation used a separate half.
Experiments ran on a Tesla V100-SXM2 with 32 GB of memory. A background thread sampled GPU power through NVML every 100 milliseconds. Energy per query was mean power multiplied by wall-clock inference time. NVIDIA's NVML documentation is the primary reference for that management and monitoring interface.
The GSM8K result makes the measurement problem concrete. Full INT4 reduced average power by 21.8% in the table, from 171.5 to 134.1 watts. Yet output grew by 22.8%, and total energy grew by 22.9%. The compression changed model behavior enough to erase the hardware-level saving.
The effect depended on model and task
The result was not universal. On the Qwen-7B version of the same GSM8K experiment, full INT4 used 3,662 joules compared with 3,681 joules for FP16, while accuracy changed from 92.92% to 92.50%. On MATH-500, Qwen-7B full INT4 reduced energy from 7,934 to 7,519 joules and matched the reported 66.25% FP16 accuracy.
Llama-8B behaved differently. Full INT4 reduced GSM8K accuracy from 80.42% to 74.17% while increasing energy. On MATH-500 it moved from 57.50% to 56.67% accuracy and from 8,092 to 8,469 joules. A deployment team cannot safely turn "use INT4" into one policy for every reasoning workload.
The authors also restored selected projection groups to FP16. Protecting attention projections produced the best grouped INT4 accuracy on both mathematics benchmarks for both models. But restoring MLP projections consumed about 21 to 22 GB, compared with roughly 15 to 16 GB for the FP16 baseline, because the mixed representation added overhead. A nominally compressed model can therefore become larger than the baseline if too much is restored carelessly.
The paper treats compression as a behavioral intervention, not merely a smaller weight file. A related problem appeared in BLOGish's test of Apple and NVIDIA token parity. Hardware comparisons are meaningful only when the generated work is held still or measured explicitly.
Selective restoration is promising, but the table needs repair
The researchers tested a more targeted method. They quantized each layer-projection pair individually on the calibration split, measured the accuracy drop, and ranked the pair's vulnerability. That sweep covered 196 pairs for Qwen-7B and 224 for Llama-8B. Selective variants then restored the top 10%, 20%, 30%, or 40% of pairs to FP16.
The preprint reports an eye-catching ProofWriter result for Qwen-7B: restoring the top 10%, or 19 of 196 pairs, reached 84.00% accuracy and 1,743 joules, versus 72.00% and 1,931 joules for FP16. The same table puts full INT4 at 82.00% and 1,796 joules.
Those values support the stated 12 percentage-point accuracy gain and 9.7% energy reduction. They do not prove a general method. The ranking failed to help several other model-task combinations, and MuSiQue accuracy was too low for the per-module sweep to detect useful sensitivity.
More importantly, the MATH-500 section of Table 5 contradicts its own prose and summary row. For Qwen-7B, the table lists FP16 at 66.25% and top-30% restoration at 59.17%, yet the summary calls this a 3.17-point gain. For Llama-8B, the table lists FP16 at 57.50% and top-20% at 49.17%, while the following paragraph compares 49.17% with a different 13.33% FP16 figure and calls it a 35.84-point gain.
Until the authors correct or explain those entries, the selective-compression claims should be treated as preliminary. The ProofWriter numbers are internally consistent. The MATH-500 improvement claims are not.
Measure the request, not the component
A practical compression test should record at least five outputs for every workload:
- Task accuracy or another product-specific quality metric.
- Input and output token counts.
- End-to-end latency, including any retries.
- Average power and total energy per completed request.
- Peak memory, not only the nominal weight precision.
Use the prompts and decoding settings that production actually uses. Separate calibration data from evaluation data. Repeat enough samples to show variance, especially when one correct answer can move the score by several points. Compare errors as well as averages. An INT4 model that saves energy on short arithmetic may spend more on long-form reasoning or recover through extra tokens.
Teams can use ish.chat to review the workload set and failure cases, then run the same request matrix across model configurations through api.ish.chat. Keep the model revision, quantization configuration, hardware, driver, power-sampling interval, prompt, decoding parameters, and stopping rules with every result.
An efficiency dashboard should therefore put watts beside tokens and seconds. A lower power trace is an incomplete result. The bill, and the environmental cost, follows joules per useful answer.
What the paper establishes, and what it does not
The study covers two distilled 7B to 8B reasoning models on one V100 GPU type. The authors note that results may differ on A100 or H100 hardware. Most conditions used small 30-sample batches, and a single correct MuSiQue answer could move accuracy by 3.33 points. The work is a preprint under review, not a settled benchmark.
Within that boundary, the Llama-8B GSM8K result is useful. Four-bit quantization lowered the observed power rate while increasing both reasoning length and total energy. Teams optimizing inference should test that failure mode before treating a smaller model file or a lower watt reading as an efficiency result.
Compression changes how long the generator keeps the GPU busy. Measure the complete answer, not only the component running it.
