ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #Qwen2.5-Coder #GGUF #Model Quantization #Code LLMs #Energy Efficiency

Benchmarking Quantized Qwen2.5-Coder: Smaller q4_K_M Models Deliver Better Energy per Solved Task

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A benchmark on a fanless MacBook Air M2 evaluates 24 Qwen2.5-Coder GGUF files across four model sizes using 542 EvalPlus tasks. By normalizing measured power by solved tasks, the study finds q4_K_M builds for the 1.5B and 3B models occupy the correctness–energy frontier at the 1 GB and 2 GB budgets. The only measured 7B file that fits the 3 GB budget, 7B-q2_K, produces no statistically significant gain over 3B-q4_K_M while requiring 2.03× the component energy per solved task. At 8 GB, 7B-q6_K yi


Core Finding

This study directly tests a common rule in local code-model selection: a larger model is often assumed to win even when it requires more aggressive quantization. It evaluates 24 Qwen2.5-Coder GGUF files at four model sizes on a fanless MacBook Air M2, using 542 EvalPlus tasks to measure correctness and energy per solved task.

| Ollama-listed file size | Recommended choice | Evidence | |---|---|---| | 1 GB | 1.5B q4_K_M | On both the correctness and energy frontiers | | 2 GB | 3B q4_K_M | On both frontiers under every energy-accounting method | | 3 GB | Keep the 2 GB 3B-q4_K_M option | The only measured 7B file that fits provides no significant correctness gain while consuming substantially more energy | | 8 GB | 7B-q6_K for quality-first workloads | Delivers a significant correctness gain over the 2 GB option, but at a higher energy cost |

The implication is that filling the available download budget is not itself an optimization objective. At a 3 GB budget, the smaller 3B-q4_K_M is preferable to downgrading to q2_K merely to fit a 7B model.

Mechanism: Pricing Energy per Solved Task

The study's central metric can be understood as:

E_solved = E_total / N_solved

Here, E_total is the energy consumed during an evaluation run, while N_solved is the number of successfully solved tasks. This is more representative of a coding assistant's cost than file size alone: generating invalid tokens or repeatedly producing answers that fail tests raises the energy required per successful task.

GGUF K-quants use grouped scales and mixed-precision representations to reduce weight size. For local inference, smaller weights generally mean less memory traffic and lower pressure on DRAM or unified-memory bandwidth, which is particularly important for CPU-bound inference. File size, however, does not shrink every runtime allocation. KV Cache, compute buffers, runtime metadata, and long code contexts can still dominate memory use and power consumption.

q2_K is an aggressive quantization regime. Its high compression can remove expressive capacity needed for syntax, identifier generation, and multi-token logic. q4_K_M uses a more conservative mixed 4-bit representation. In this experiment, beginning with the 1.5B scale, q4_K_M remained within three percentage points of the highest measured precision at the same model size according to 90% confidence intervals. q6_K retained more weight information, but required additional capacity and energy.

Why Two Energy Boundaries Matter

The paper separates component energy from whole-system power. Component accounting captures energy on CPU, GPU, and memory rails; whole-system measurement covers the power drawn by the complete machine during inference. These boundaries answer different questions. Component energy is useful for locating gains in kernels, memory bandwidth, and quantization, while system-level power better represents battery life and electricity cost.

This distinction is important on a fanless MacBook Air. Sustained inference does not immediately add the power cost of a large cooling fan, but the SoC, memory, SSD, and other peripherals still consume energy. As a result, a model gap measured on component rails can narrow at the wall. Even the lower 1.66×–1.81× whole-system ratio between 7B-q2_K and 3B-q4_K_M is enough to make the larger low-precision build unattractive for this workload.

Measured Results

  • At the 1 GB and 2 GB budgets, 1.5B-q4_K_M and 3B-q4_K_M occupy both the correctness and energy frontiers. Moderate quantization does not erase the cost advantage of the smaller models.
  • At 3 GB, 7B-q2_K is the only measured 7B file that fits, yet it offers no significant improvement in solved tasks over 3B-q4_K_M. Statistically, the possible deficit of 3B-q4_K_M is bounded at 2.4 percentage points, which does not compensate for the extra energy demand.
  • Under CPU, GPU, and memory component accounting, 7B-q2_K requires 2.03× the energy per solved task of 3B-q4_K_M. Whole-system accounting still yields a ratio of 1.66×–1.81×.
  • Five alternating HumanEval+ session pairs produce ratios of 1.925 and 1.956. Their proximity suggests that the main energy ranking was not caused by a single session or thermal-state drift, although the number of repeated pairs remains small.
  • At 8 GB, 7B-q6_K produces a significant correctness improvement over 3B-q4_K_M. Higher precision is therefore not inherently ineffective; it simply requires a larger capacity and energy budget.
  • The authors also release per-task results, power logs, and analysis scripts, allowing other researchers to inspect confidence intervals, energy boundaries, and the file-size mapping rather than relying only on aggregate conclusions.

Engineering Trade-offs and Benchmark Limits

First, the experiment covers one M2 machine. M2 uses a unified-memory architecture in which CPU, GPU, and memory share capacity and bandwidth, so its results cannot be directly extrapolated to discrete GPUs, x86 workstations, or power-hungry servers. Inference runtime, thread configuration, and quantization kernels can also change measured energy.

Second, 542 EvalPlus tasks are appropriate for function-level code generation but do not represent repository-scale development involving multi-file retrieval, dependency installation, tool use, context compression, and long-running repair loops. A coding assistant's practical value depends not only on whether generated code passes unit tests, but also on edit scope, compilation feedback, and subsequent rework.

Third, the correctness analysis provides 90% confidence intervals, but the supplied abstract does not fully disclose success-rate distributions, the number of energy-measurement repetitions, or the thermal-stabilization procedure. HumanEval+ replication includes only five alternating session pairs. Before extrapolating, reviewers should inspect the released logs for execution order, temperature changes, background load, and the treatment of failed tasks in the energy metric.

Fourth, Ollama's listed file size is a download and storage budget rather than peak process memory. GGUF metadata, KV Cache, compute graphs, and long contexts can make resident memory materially higher than the file size. Production evaluation should therefore record RSS or unified-memory occupancy alongside first-token latency and generation throughput.

Developer Recommendations

  1. With a 1 GB to 2 GB model budget on a laptop or edge device, test 1.5B or 3B q4_K_M first instead of dropping to q2_K merely to accommodate a larger model.
  2. Do not assume that more parameters are automatically better with a 3 GB budget. Compare q4_K_M and q2_K on real repository tasks before accepting the additional energy cost.
  3. When correctness has priority and roughly 8 GB is available, evaluate 7B-q6_K. Keep a separate latency budget because higher precision does not necessarily produce faster first-token responses.
  4. Reproductions should alternate candidate configurations and record wall power, component energy, peak memory, time to first token, tokens per second, task pass rate, and long-session thermal throttling. This avoids mistaking run-order bias for a model-level effect.
  5. Enterprise offline deployments should add repository-editing tasks and include human review and rework time in the total cost. Energy efficiency matters only when output quality does not degrade beyond an acceptable threshold.

Source: Hugging Face Daily Papers (2026-11-13)

— END —

Tags: #Qwen2.5-Coder #GGUF #Model Quantization #Code LLMs #Energy Efficiency

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…