BILBY Redefines Protein Motif Scaffolding Evaluation with Molecular Dynamics
Summary:Static refolding is frequently treated as sufficient evidence that a generated protein scaffold preserves a functional motif, although local structural recovery does not guarantee dynamical fidelity of the complete design. The study introduces two complementary benchmarks. BILBY-PRO systematically constructs 220 motif-scaffolding cases from 174 PROSITE-linked motif-pattern entries, while BILBY-MD evaluates designs through all-atom molecular dynamics against reference simulations. Among 55 static
Static Structural Accuracy Does Not Guarantee Dynamic Stability
Protein motif scaffolding aims to build a backbone that preserves the positions and local geometry of functionally important residues while providing a stable structural context. Conventional evaluations often ask whether a generated sequence can be refolded to a structure near the intended target. This provides useful evidence of local recovery, but it does not establish that the scaffold will resist motif drift, contact disruption, or abnormal dihedral-state populations under finite-temperature conditions.
BILBY addresses both benchmark scope and success criteria. BILBY-PRO systematically expands the motif test set, while BILBY-MD introduces a reference-based molecular-dynamics evaluation layer. The central distinction is that target coverage, unique-solution yield, and dynamical fidelity measure different capabilities and cannot be substituted for one another.
A Systematic Motif Benchmark
BILBY-PRO derives structurally conserved functional motifs from PROSITE-linked experimental structures, producing 220 cases across 174 PROSITE motif-pattern entries. Compared with benchmarks built from a small, manually curated list of motifs, this construction reduces dependence on a narrow set of commonly studied structures and provides broader functional coverage.
Evaluation of six recent scaffold-generation models reveals distinct capability profiles. RFDiffusion and Proteina solve more target motifs, while ESM3 generates more unique successful scaffolds among the targets it solves. This distinction matters because a model can repeatedly collapse to the same structure and still achieve a high static success score without expanding the usable conformational space.
The results should therefore be reported along two separate axes:
- Target coverage: whether a model can address a given functional motif;
- Unique-solution yield: whether successful outputs form a diverse set rather than converging on one template-like solution.
A single aggregate score can hide the difference between broad but repetitive generation and narrower but more diverse generation.
How BILBY-MD Tests Dynamical Fidelity
BILBY-MD is a reference-based, all-atom molecular-dynamics framework. Rather than relying only on coordinate RMSD, it compares generated designs with reference simulations through motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy landscapes.
Free-energy landscapes are especially informative. A design may match a reference in one frame while occupying low-probability states elsewhere in the conformational ensemble. Conversely, a structure with slightly worse coordinate RMSD may still preserve the population distribution of critical backbone dihedrals. Contact preservation identifies cases in which residues remain near one another but no longer maintain the interactions required for the motif's functional geometry.
The framework effectively changes the question from whether a design can refold to whether it can retain a functionally relevant conformational ensemble over a molecular-dynamics timescale. The cost is substantial: explicit-solvent all-atom simulations require force-field choices, protonation-state handling, adequate sampling, and repeated trajectories. Running such simulations for every generated candidate may be impractical at production scale.
Static Success Does Not Imply Dynamic Stability
Among 55 statically successful model-problem pairs from the original MotifBench, only 31 satisfy the study's MD-supported criterion, and only eight pass every check. Thus, roughly 43% of statically successful pairs fail the dynamics-supported threshold, while only about 15% pass the complete evaluation.
This does not negate the value of static refolding. It instead exposes its boundary: a structure predictor may recover local geometric constraints while failing to reproduce the reference conformational ensemble or state-transition behavior.
The failures also follow a meaningful structural pattern. Compact, self-contained motifs with local contact networks are more likely to receive MD support. Interface-like or context-dependent motifs fail more often because scaffold generation based only on local motif residues lacks the long-range constraints needed to stabilize the correct dynamics. A static predictor may still produce a plausible structure, while molecular dynamics reveals its instability.
Evaluation Validity and Open Questions
BILBY contributes an audit of both benchmark construction and success criteria, but several implementation details remain essential for reproducing and generalizing its conclusions:
- Reference-trajectory quality: Reference structures may themselves be imperfect, particularly for non-native states, flexible interfaces, or experimental conditions poorly represented by simulation.
- Simulation comparability: Protonation states, force fields, solvent models, temperature, and sampling length must be matched between design and reference systems.
- Statistical replication: A single MD trajectory can be dominated by initialization and metastability. Reliable decisions require repeated runs, uncertainty estimates, or block analysis.
- Threshold sensitivity: Only eight pairs pass all checks, so the strict-pass set is small and rankings may shift under alternative thresholds.
- Fair cross-model comparison: Training-data overlap, sampling budgets, and post-processing pipelines for RFDiffusion, Proteina, and ESM3 need consistent reporting.
Practical Guidance for Protein-Generation Teams
A production pipeline should not use refolding success as its sole selection criterion. A more defensible workflow is to use BILBY-PRO for broader candidate coverage, monitor unique-solution yield to retain diversity, and apply BILBY-MD-style dynamics checks to shortlisted designs.
Under tight compute budgets, teams can apply a tiered filter: static refolding for inexpensive rejection, contact-network and flexibility analysis as an intermediate layer, and repeated MD only for final candidates. For interface-dependent or context-sensitive motifs, the better intervention is to supply binding partners, ligands, or larger complex context rather than simply increasing the number of static samples.
BILBY ultimately changes the evaluation question. Protein generation should not only ask whether a structure looks like the target, but whether it can continuously preserve the target motif and its accessible conformations under realistic thermodynamic conditions.
— END —Source: Hugging Face Daily Papers (2026-11-05)
Tags: #Protein Design #Motif Scaffolding #Molecular Dynamics #Generative Model Evaluation #RFDiffusion
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments