HapBO Compiles Natural-Language Priors into Bounded Bayesian Optimization Beliefs
Summary:HapBO converts natural-language priors supplied by human experts, LLMs, or both into a reusable belief function π(x)∈[0,1] before optimization begins. The belief then modulates a nonnegative acquisition function through a bounded positive multiplier, preserving acquisition support while eliminating LLM calls from the optimization loop. For Expected Improvement, the paper proves that simple regret remains within a bounded factor of the standard rate even when the elicited belief is adversarial.
Core Takeaway
HapBO turns natural-language priors from outside the optimization loop into a reusable structured belief function π(x). An expert or LLM supplies the prior offline, a deterministic compiler produces π, and the online Bayesian optimizer consumes π like any other prior module—without LLM tokens, inference latency, or output jitter during optimization.
The key distinction is that the LLM does not select hyperparameters on every iteration. It participates in a single prior-calibration stage. Language-derived knowledge, the surrogate model, and the acquisition function remain decoupled, allowing the online loop to stay reproducible, low-latency, and auditable.
From Language to a Belief Function
In the offline stage, beliefs are elicited from a human expert, an LLM, or both, then converted into a structured specification. A deterministic compiler maps that specification to π:X→[0,1]. Because the representation is not restricted to a Gaussian family, it can represent multimodal regions, asymmetric preferences, local directions, or heterogeneous expert judgments.
Elicitation and compilation should be treated as separate operations. Even if the initial LLM call is stochastic, saving its structured output as a versioned input allows the same specification and compiler to produce the same π. Production systems should therefore retain the source specification, compiler version, model identifier, and final parameters rather than only a serialized weight vector.
In the online stage, the mechanism can be abstracted as follows:
- Let the base acquisition function be a(x)≥0.
- HapBO derives a bounded positive multiplier m(x) from π(x).
- The modified acquisition is ã(x)=m(x)a(x).
The supplied abstract does not disclose the multiplier's exact parameterization, but it confirms that the multiplier is bounded and strictly positive. Whenever the base acquisition is positive at a point, the modified acquisition remains positive there. HapBO therefore does not permanently remove an entire region from exploration merely because a prior labels it unlikely.
What the Theoretical Guarantee Protects
For maximization, Expected Improvement is commonly written as E[(f_best+ξ−f(x))_+], balancing exploitation around the incumbent with exploration of potential improvements. HapBO does not replace EI; it adjusts its sampling priorities through the belief function.
The paper proves that, under EI, simple regret remains within a bounded factor of the standard rate even if π is misleading or adversarial. The guarantee does not require the expert prior to be correct, which makes it relevant to LLM hallucinations, expert mistakes, and stale documentation.
The result should not be interpreted as a universal guarantee for every acquisition function, prior representation, and task. The stated regret-rate proof applies to EI. Other nonnegative acquisition functions may share the same multiplier interface, but extending the theoretical guarantee requires separate analysis.
Benchmark Results and Their Limits
The evaluation covers 40 HPOBench tasks, a synthetic suite with 6–15 dimensions, and two real-world applications: cortical neurostimulation and constrained solar design. The authors report the lowest aggregate HPOBench regret and strong real-world outcomes with only 12–20 queries. Because the LLM is used only during offline prior construction, the online token cost is zero and the wall-clock overhead is a small fraction of LLM-based alternatives.
The current abstract does not report absolute regret, per-task win rates, repeated-run counts, confidence intervals, or significance tests. A leading aggregate regret can conceal variance or regressions on a subset of tasks. HPOBench and synthetic functions are also proxies for physical optimization, while two applications are insufficient to establish broad domain-level generalization.
A reproduction should verify:
- Whether all methods use identical initial points, query budgets, and stopping rules;
- Whether baselines include standard EI, conventional prior-guided BO, and online LLM guidance;
- The LLM version, prompt, temperature, context, and token consumption;
- The expert elicitation protocol and the procedure for combining human and LLM beliefs;
- How the multiplier bounds are selected and whether they are tuned on test tasks;
- Whether wall-clock comparisons include prior elicitation and compilation;
- Whether code, configurations, and random seeds are available.
Engineering Trade-offs
Online uncertainty and cost approach zero. LLM-in-the-loop methods must handle network latency, rate limits, nondeterministic outputs, and context drift on every iteration. HapBO moves those risks offline; the online loop adds only a π(x) lookup and a multiplier evaluation.
Errors become persistent. A compiled π is reused throughout the run, so a faulty early specification may affect the entire query budget. Deterministic compilation guarantees repeatability, not correctness. Structured specifications should expose scope, direction, strength, uncertainty, conflicts, and missing information, while high-risk applications need a way to disable or decay the prior.
Bounded multipliers preserve support but constrain influence. A positive floor prevents an incorrect prior from sealing off search regions, but an overly large floor dilutes informative preferences. A high ceiling can let π overwhelm EI and force excessive reliance on natural-language descriptions. The bounds should be calibrated on historical tasks rather than treated as fixed defaults.
Expressiveness increases calibration difficulty. A non-Gaussian π can capture knowledge that traditional distributions cannot, but its probability semantics are harder to validate. The adversarial-belief theorem protects the optimization process; it does not automatically correct schema parsing errors, LLM bias, or errors introduced by the compiler.
Priors are not hard constraints. In neurostimulation and solar design, safety limits, hardware restrictions, and physical feasibility must be enforced by a constraint model. Encoding an infeasible region only as low belief improperly downgrades a hard requirement to a soft preference.
Deployment Guidance
- Generate a structured specification from documentation, prior experiments, and expert interviews, explicitly recording scope, direction, strength, and uncertainty for every claim.
- Version the specification, LLM identity, prompt, compiler, and final π together so every optimization run remains reproducible.
- Visualize π on historical tasks and run four ablations: no prior, expert-only, LLM-only, and combined prior. Do not compare only the final optimum.
- Keep hard constraints separate from the belief function and reject infeasible candidates through a feasibility model.
- Compare standard EI, conventional prior-guided BO, and online LLM guidance using identical initial points, budgets, and stopping rules. Record regret, success rate, variance, tokens, and end-to-end wall-clock time.
- For safety-critical applications such as neurostimulation, validate the method in simulation and offline approval workflows first. Natural-language priors should never directly control irreversible experiments.
HapBO's main contribution is not another prompt asking an LLM to choose hyperparameters. It establishes an auditable boundary: language models propose knowledge once, a structured compiler freezes that knowledge, and conventional Bayesian optimization handles uncertainty, exploration, and cost under a fixed query budget.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-21)
Tags: #Bayesian Optimization #Natural Language Priors #Hyperparameter Optimization #Large Language Models #Alpha-Pi BO
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments