WSP Edits Just 1,023 Parameters to Lift Math Accuracy by 29.45 Points

Summary:WSP-Scope presents a systematic study of Weight-Space Programming, in which reasoning agents inspect and directly modify neural-network parameters rather than relying on gradient-based adaptation. Across 16 editing trajectories, seven benchmarks, and Qwen3.5 models at 0.8B and 9B scale, the method reports gains of up to 29.45 points on MATH-500, 4.42 on GPQA Diamond, 36.72 on IFEval, and 17.11 on SciBench. One successful intervention changes only 1,023 parameters. A math-edited checkpoint also t

---

Core finding: weights can be edited directly, but reliable editing remains difficult

WSP-Scope presents a systematic study of Weight-Space Programming (WSP), where a reasoning agent inspects and directly modifies neural-network parameters instead of relying on gradient-based adaptation. The experiments cover Qwen3.5-0.8B and Qwen3.5-9B, seven benchmarks, and 16 editing trajectories.

The central result is not that agents can already rewrite large models reliably. Rather, successful interventions can be extremely localized. On full held-out evaluations, WSP reports gains of up to 29.45 points on MATH-500, 4.42 on GPQA Diamond, 36.72 on IFEval, and 17.11 on SciBench. In one case, a 29.45-point MATH-500 improvement comes from changing only 1,023 parameters.

This suggests that WSP offers a different adaptation path from conventional fine-tuning: an agent examines weight structure and evaluation feedback, locates a possible functional site, and writes a parameter-level patch. The edited payload is small, but the search and validation process is not necessarily inexpensive.

Mechanism: from parameter localization to verified writes

A WSP loop can be decomposed into four stages:

1. Inspect the weights: The agent accesses target tensors, layer structure, and numerical distributions to identify regions that may support the desired behavior.
2. Propose an intervention: It searches for parameter coordinates and replacement values expected to improve validation performance.
3. Apply the edit: New values are written into the model, producing a modified checkpoint or runtime weight version.
4. Validate reliability: The edit is accepted only when it exceeds a predefined threshold; otherwise, the original weights are retained.

The storage overhead of 1,023 scalar parameters is negligible under BF16 or FP16. Production costs arise elsewhere. Transformer inference also depends on KV caches, weight transfers, and fused kernels. A hot update that invalidates kernel caches or tensor layouts may require reloads or recompilation. Deployable weight patches also need concurrency isolation, version consistency, rollback mechanisms, and model-integrity checks.

WSP is not equivalent to pruning, sparse updates, or low-rank adaptation. LoRA confines updates to a low-rank matrix and receives a stable continuous optimization signal from gradients. WSP searches discrete coordinates and values. That search space can be much sparser, but candidate evaluation becomes noisier and validation costs more.

Transfer: math edits spill over, but not consistently

The second major finding concerns cross-task generalization. A math-edited checkpoint improves GPQA Diamond, IFEval, and SciBench by 3.03, 12.18, and 20.10 points, respectively, without further editing, while SciCode remains unchanged.

This indicates that localized weight changes can affect behavior beyond the target benchmark. Math-oriented editing may also strengthen instruction following, structured expression, or components of scientific reasoning. The transfer is nevertheless uneven: the GPQA gain is much smaller than the MATH-500 gain, and SciCode does not improve.

Such spillover should not automatically be interpreted as general capability transfer. It could reflect shared capabilities, benchmark-specific patterns, answer formatting, or response style. The reported results do not fully separate these mechanisms. A stronger evaluation would stratify tasks by family, output length, grading method, and prompt template, while checking safety behavior, refusal patterns, and factual calibration before and after each edit.

Engineering trade-offs: fewer updated parameters do not mean cheaper search

WSP's potential advantage lies in adaptation. A small parameter edit avoids the optimizer-state overhead of full fine-tuning and may eliminate the need to maintain a separate LoRA adapter for every task. It can also explore interventions that conventional parameterizations may not represent well.

The costs occur in three areas:

• Search: Every candidate intervention may require a model forward pass and evaluation. Iterative search can therefore cost far more than a single gradient update.
• Validation: Six of 16 trajectories retain the original weights because candidate improvements cannot be validated reliably, showing that acceptance thresholds directly determine the effective-edit rate.
• Maintenance: Direct weight writes disrupt conventional checkpoint boundaries, complicating auditing, reproduction, and rollback.

WSP is therefore better characterized as a controlled model-repair primitive than as a mature unsupervised self-improvement mechanism. Without clear baselines, regression suites, and non-negotiable safety constraints, automatically accepting local edits can crystallize evaluation noise into the base model.

Evaluation: promising gains, limited evidence

The study uses two model scales and 16 trajectories across math, scientific reasoning, instruction-following, and code tasks. Full held-out evaluations are more informative than training-set results, and the 1,023-parameter improvement is a striking engineering result.

The evidence nevertheless remains limited. Sixteen trajectories cannot cover every model family, parameter location, task difficulty, or search strategy. The maximum gains across four benchmarks do not come from one common intervention and should not be added together. The summary also leaves open the complete sample sizes, random seeds, candidate-search budget, baseline adaptation methods, significance tests, and contamination controls.

Direct parameter editing also has high variance. The reported trajectory-to-trajectory differences and conservative fallback behavior indicate that a fixed threshold, a single prompt, or one random trial is inadequate for judging WSP's stability. Reproductions should report multi-seed results, failure rates, search-call counts, checkpoint hashes, and acceptance criteria.

Deployment guidance

WSP is not yet suitable for unrestricted production agents with write access to model weights, but it can be tested in controlled environments:

• Use isolated copies: Write all interventions to branch checkpoints and preserve one-click rollback.
• Validate in layers: Run proxy tasks and regression suites before full held-out evaluation; never accept an edit based on one benchmark alone.
• Record provenance: Store modified coordinates, old and new values, agent trajectories, validation logs, and model versions.
• Enforce least privilege: Treat production weights as read-only and execute edits in an isolated environment before release.
• Cap search budgets: Set hard limits on forward passes, candidate count, and GPU time.
• Test for collateral damage: Re-evaluate safety behavior, factuality, tool-call formatting, long-context handling, and refusal quality.

The project also provides a research homepage and a GitHub repository for reproduction and extension. WSP's value is that it introduces a new primitive for model adaptation; whether it becomes practical depends on whether intervention discovery and reliable validation can be engineered beyond their current experimental state.

---

Source: Hugging Face Daily Papers (2026-11-06)

— END —

Tags: #WSP Follow #Weight-Space Programming Follow #Model Editing Follow #Qwen3.5 Follow #Reasoning Agents Follow