ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Weight-Space Programming #LLM Evaluation #Model Editing #Gradient-Free Optimization #Agent Benchmark

WeightBench Evaluates LLMs as Weight-Space Programmers, Finds 7B-Parameter Editing Remains Hard

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:WeightBench formalizes weight-space programming, or WSP, as the ability of an LLM to inspect, repair, or directly generate another neural network's parameters without gradient descent. The benchmark contains 42 tasks spanning weight reading, fixing, writing, recognizing impossible edits, and data-free compression, with target networks ranging from 2K-parameter control policies to 7B-parameter language models. Claude Opus 5.5 reached a macro score of 0.535 across all 42 tasks, while GPT-6.1-sol


Core Finding

WeightBench turns the idea of letting an LLM manipulate another neural network's parameters into a systematic evaluation of weight-space programming, or WSP. Its 42 tasks cover five tracks: reading weights, fixing weights, writing weights from scratch, recognizing impossible edits, and compressing networks without data. The target networks range from 2K-parameter control policies to 7B-parameter language models.

The key result is not that language models can already edit large models reliably. Instead, the benchmark exposes a clear capability boundary. Frontier models can inspect some small networks and produce working control policies without gradient-based optimization, but patching 7B-class models, performing data-free compression, and distinguishing an impossible edit from a merely difficult one remain unresolved.

How WSP Differs from Ordinary Code Generation

For a repair task, the objective can be expressed as:

θ' = θ + Δ

The system must find an appropriate parameter delta Δ while preserving specified behavior and structural constraints. Conventional model editing typically solves this with gradient descent, a local optimizer, or a trainable hypernetwork. WSP instead asks the language model to act simultaneously as the optimizer, numerical code generator, and parameter serializer.

The model must therefore understand tensor shapes, parameter scales, inter-layer coupling, and numerical stability in addition to the natural-language task. Comparing old and new weights with Euclidean distance is also insufficient: weight-space symmetries such as neuron permutations or rescaling can substantially alter parameters while preserving the network function. The meaningful question is whether the edited network behaves correctly after execution, not whether its serialized weights match a reference tensor element by element.

Impossible-edit tasks additionally require the model to detect conflicting constraints rather than manufacture a patch. Data-free compression requires approximating functional behavior without a calibration distribution. Both demand something closer to an approximate optimizer and out-of-distribution generalization than reading a small network.

Benchmark Design and Evaluation Loop

WeightBench spans extremely small control policies and 7B language models to test whether capability degrades systematically with network scale, edit type, and the evaluated model's strength. A task registry fixes inputs, objectives, and success criteria, while an OS-level sandbox isolates code and weight artifacts submitted by the model.

The claim that models can write control policies without running them should not be interpreted as meaning that the benchmark skips functional validation. Executable candidates still have to run in an isolated environment to establish whether they work. The distinction is that the evaluated model's solution path does not rely on iterative execution feedback or gradient-based search. Production systems that permit such feedback should therefore be evaluated separately from one-shot WSP.

The code, task registry, sandbox harness, and live leaderboard are available through github.com/Aayush-Marasini/WeightBench, providing a common entry point for reproducing model and editing-system comparisons.

How to Read the Scores

The paper reports:

| Model | Tasks Covered | Macro Score | Note | |---|---:|---:|---| | Claude Opus 5.5 | 42 | 0.535 | Full task set | | Claude Sonnet 5.5 | 42 | 0.466 | Full task set | | Claude Haiku 5.5 | 42 | 0.402 | Full task set | | DeepSeek v4-flash | 42 | 0.215 | Full task set | | GPT-6.1-sol | 38 | 0.568 | 38-task subset |

The valid GPT-versus-Opus comparison is restricted to their 38 common tasks: 0.568 versus 0.546, an absolute difference of 0.022. GPT's 0.568 should not be directly compared with Opus's 0.535 across all 42 tasks because the task sets differ.

A macro score is not the same as the percentage of all 42 tasks passed. It likely aggregates normalized task- or track-level metrics, but the collected description does not disclose the exact normalization, per-track results, difficulty stratification, or confidence intervals. The results are sufficient to show that large-model editing remains a bottleneck, but not to determine which parameter operations each model handles reliably.

Engineering Trade-offs

1. Output Bandwidth Is a Hard Constraint

Storing 7B parameters in BF16 requires roughly 14GB, or about 28GB in FP32. Having a language model emit every weight is impractical in token count, latency, and cost. Real systems must restrict output to sparse patches, low-rank deltas, localized matrix edits, or compressed representations. Each choice changes task difficulty and can undermine comparability between approaches.

2. Numerical Serialization Can Introduce Error

Direct weight generation requires explicit rules for parameter ordering, data type, shape metadata, rounding, and NaN, Inf, or overflow handling. Text serialization adds another quantization layer. If the benchmark reports only a macro score without its numerical precision and serialization protocol, it becomes difficult to separate reasoning failures from formatting losses.

3. One-Shot Generation and Execution Feedback Need Separate Tracks

Pure WSP asks the model to produce usable parameters without gradients or iterative feedback, making it closer to a compiler or offline optimizer. A sandboxed agent can instead execute candidates, consume failure logs, and iteratively revise them. These configurations should not share one leaderboard, because retries, tool access, and feedback rounds can overwhelm the signal from the model's underlying weight-generation ability.

4. Parameter Distance Is Not Functional Equivalence

Weight spaces contain multiple functionally equivalent representations. If arbitrary permutations are legal but scoring requires exact parameter matching, a model may fail solely because it chose another representation. If scoring is entirely behavioral, it needs stricter hidden tasks and out-of-distribution tests to prevent speculative weights that merely fit public examples.

5. A Sandbox Is Not Automatically Secure

Submitted artifacts may contain malicious code, malformed tensors, or resource-exhaustion logic. An OS-level sandbox still needs CPU, memory, disk, process, and time limits, along with blocked network access by default. A reproducible leaderboard should preserve container digests, dependency lockfiles, and random seeds so environment drift does not invalidate comparisons.

Credibility and Independent Validation

The reported results come from the paper authors and have not yet been accompanied by full task-level scores, run-to-run variance, prompt and output budgets, retry policies, tool permissions, or complete cost data. Different models may also have used different inference configurations, so score differences cannot automatically be attributed to architecture or training methodology.

Task construction may also shape the conclusions. Impossible-edit tasks need strong hard negatives so that models cannot reject requests using superficial patterns. Data-free compression needs a clearly defined target distribution and hidden test set. If most tasks involve small, fully connected, or highly structured control policies, success there does not automatically transfer to 7B-class Transformers.

A live leaderboard can collect reproducibility data, but it becomes auditable only if model versions, API settings, prompt templates, output-token limits, execution permissions, and evaluation images are frozen.

Recommendations for Developers

  • For reproduction, pin both the repository commit and container digest. Compare models on the 38 shared tasks first, then report the full 42-task results separately. Include per-track scores, seeds, confidence intervals, token consumption, wall-clock latency, and failure categories.
  • Separate no-execution feedback, bounded sandbox feedback, and unrestricted tool use into distinct tracks so iterative search gains are not misattributed to one-shot weight generation.
  • Include conventional baselines. Model-editing systems should be compared with gradient-based editing, distillation, sparse or low-rank updates, quantization, and neural architecture search. Control-policy generation should be compared with ordinary reinforcement learning or evolutionary search.
  • In production, restrict WSP to small control networks for now and enforce strict parameter schemas, differential tests, candidate-weight validation, canary releases, and version rollback. Do not allow a model to overwrite 7B production weights directly.
  • Provide an explicit refusal interface and human-review path for impossible edits. The ability to produce a patch does not imply the ability to recognize when generation should stop.

WeightBench's main contribution is to turn WSP from isolated demonstrations into a reproducible research benchmark while showing that scalable language-model editing has not yet been solved. The usable signal today is concentrated in small-network reading and generation; reliable large-model repair still requires sparse representations, strict verification, and substantial human engineering controls.


Source: Hugging Face Daily Papers (2026-11-13)

— END —

Tags: #Weight-Space Programming #LLM Evaluation #Model Editing #Gradient-Free Optimization #Agent Benchmark

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…