Even Identically Trained Language Models Break on Different Jailbreaks

Summary:A controlled study of 150 language models shows that jailbreak susceptibility is strongly tied to the specific pretraining run, rather than being determined only by the training-data recipe or safety post-training. Two post-trained copies of the same pretraining run have a per-prompt non-refusal correlation of 0.984, while changing the pretraining seed lowers it to 0.811. The best of 91 attack templates transfers to only 27% of model pairs after a new pretraining seed, compared with 67% after a

---

Jailbreak Susceptibility Depends on the Training Run

This study turns a commonly overlooked question into a measurable variable: do models trained on the same data and subjected to the same safety post-training actually share the same jailbreak weaknesses? The answer is no. Susceptibility to a particular jailbreak is strongly associated with the specific pretraining run, while changing the pretraining data recipe has a comparatively smaller effect.

The experiments cover 25 pretraining data recipes. For each recipe, the authors train three 1B-parameter base models with different pretraining seeds and apply the same safety post-training with two post-training seeds, yielding 150 models in total. They collect 20 responses from every model for a shared set of jailbreak prompts and compare per-prompt non-refusal rates across all model pairs.

The main results are:

• Two post-trained copies of the same pretraining run have an average correlation of 0.984;\n- Changing the pretraining seed lowers the correlation to 0.811;
• Changing the data recipe as well lowers it only slightly further, to 0.779;\n- The pattern holds across all 25 recipes and at both 150M and 300M parameters.

Safety post-training can therefore shape the refusal boundary of an existing checkpoint relatively consistently, but it does not erase vulnerabilities formed during pretraining. The data recipe determines what the model learns; pretraining randomness further determines which attacks happen to land.

The Best Jailbreak Is Usually Not a Universal Jailbreak

The study also evaluates attack-template transferability. Among 91 attack templates optimized on one checkpoint, the best template remains the best across only 27% of model pairs after a new pretraining seed. With a new post-training seed, that share rises to 67%.

A template ranking first on one checkpoint is therefore weak evidence that it will work reliably across models trained from the same recipe. Red teams that search prompts against a single checkpoint may overestimate general effectiveness and mistake an idiosyncratic weakness for a reproducible model vulnerability.

A useful evaluation should distinguish three layers:

1. Whether an attack induces a non-refusal;
2. Whether the attack remains the strongest among competing templates;
3. Whether the result generalizes across pretraining and post-training runs.

These layers can diverge substantially. Reporting only one attack success rate does not reveal whether the result is a general weakness, a property of the data recipe, or a checkpoint-specific training fingerprint.

Every Pretraining Run Leaves an Identifiable Fingerprint

The authors find that a model's per-prompt non-refusal rates across 3,211 prompts identify its pretraining run among all 75 runs every time. A random subset of 50 prompts still achieves 98% identification accuracy.

This means refusal behavior is not merely a uniform expression of learned harmfulness. Different pretraining runs leave stable behavioral fingerprints. For red teams, this raises the possibility that prompts optimized against one model can serve as run-level identification probes. For defenders, it also means that public jailbreak sets may become contaminated by training-trace information and potentially help attackers infer a checkpoint's training provenance or identify a near-duplicate model.

Successful run identification does not imply exact sample-level attribution. The remaining 2% error rate at 50 prompts shows that boundary cases, sampling noise, or insufficient prompt coverage still matter. Nor does the result establish that these fingerprints transfer across model scales, architectures, or publicly released checkpoints.

Important Experimental Limitations

The controlled design is a major strength, but the conclusions have clear boundaries:

• Restricted model scale: The main models are 150M, 300M, and 1B parameters. Whether pretraining variance grows, shrinks, or changes form at frontier scale remains open;
• A single post-training regime: Replacing the common safety procedure with different RLHF, DPO, or rule-based pipelines could change the 67% transfer rate;
• Refusal-classification noise: Non-refusal rates rely on output classifiers or human judgments, and classification errors affect both correlations and attack rankings;
• Prompt-set sensitivity: Conclusions may change with multilingual prompts, compositional attacks, long-context attacks, or multi-turn jailbreaks;
• Finite response sampling: Twenty responses per prompt estimate a probability rather than the exact refusal behavior. More samples improve precision but scale evaluation cost across 150 models;
• Substantial compute cost: The study requires three pretraining runs per recipe and two post-training runs per base model.

Accordingly, 0.811 and 0.984 should be treated as correlations under the tested regime, not universal constants for model safety.

Recommendations for Security Evaluation

1. Make seed replication part of the benchmark

A security benchmark should not publish results from one checkpoint alone. Evaluate multiple pretraining seeds and report post-training variance separately. Maintain the same data recipe and evaluation prompts across repetitions, then decompose variance into pretraining- and post-training-related components.

2. Report attack-ranking transfer

Beyond success rate, report rank correlation, Top-k hit rate, and paired transfer rates across models. This distinguishes broadly effective attacks from templates that only work against a small number of checkpoints.

3. Use a split evaluation suite

Fixed-checkpoint prompt search can adapt to training fingerprints. Use one subset of prompts for attack discovery and a held-out subset for final testing. Repeat sampling on every model to reduce overfitting to isolated non-refusal events.

4. Require cross-run evidence for deployment claims

For a safety patch or new defense, improving attack success on one checkpoint is insufficient. The defense should remain effective on new pretraining seeds, new post-training seeds, and attack templates that were not used during optimization.

5. Protect evaluation data from leakage

Because only 50 prompts can identify a pretraining run with 98% accuracy, public jailbreak sets can be used as model fingerprints. Benchmark maintainers should rotate prompts, conceal a portion of probes, and distinguish attack-search prompts from blind-test prompts.

Conclusion

The work shifts jailbreak benchmarking from a single model-and-prompt combination to a statistical distribution over training runs. Its most important result is not that a particular prompt works, but that a model's pretraining run deeply shapes its jailbreak boundary and post-training does not completely remove that path dependence.

Reliable robustness claims therefore require attack search across pretraining seeds, explicit reporting of ranking transfer, and separation of prompt-level effectiveness from model-specific training fingerprints.

---

Source: Hugging Face Daily Papers (2026-10-25)

— END —

Tags: #Language Model Safety Follow #Jailbreak Evaluation Follow #Pretraining Seeds Follow #Red Teaming Follow