Scaling hyperparameter search with multi-fidelity Bayesian approaches
Why multi-fidelity? Tuning hyperparameters can be slow when each evaluation requires full training on large datasets. Using lower-cost approximations that are informative about final performance can reduce compute by an order of magnitude while guiding search toward good regions. This article explains pragmatic ways to combine cheap fidelities with Bayesian-style surrogates so exploration becomes faster and more scalable.
The core idea is simple: treat fidelity as an input to the surrogate and allocate compute adaptively. Low-fidelity evaluations provide many cheap data points. High-fidelity runs confirm promising settings. A principled multi-fidelity pipeline trades off cost and information to find strong hyperparameters faster than naive grid or random search.
Key concepts in plain language
- Fidelity: any knob that controls evaluation cost and signal quality, for example training epochs, dataset fraction, or number of trees.
- Surrogate: a model that predicts validation loss as a function of hyperparameters and fidelity. Gaussian processes are common, but tree ensembles or neural nets may work.
- Acquisition: a rule to pick the next configuration to evaluate, balancing exploration and exploitation while accounting for cost.
- Adaptive allocation: start many cheap evaluations and progressively invest in the most promising ones.
Different combinations of these parts yield methods such as multi-fidelity Bayesian optimization, Hyperband with a surrogate, and combinations like BOHB. The practical choice often depends on available libraries, compute budget, and how fidelity can be controlled.
When this helps
- Large datasets where training on a subset or fewer epochs yields meaningful signal.
- Expensive models such as boosted trees with many rounds or deep nets where early epochs often indicate relative performance.
- Situations where parallel resources can evaluate many low-fidelity candidates concurrently.
Multi-fidelity approaches are not universally superior. If low-fidelity evaluations are uninformative, the method may waste cycles. It is helpful to validate that fidelity correlates with final performance on a small set of experiments first.
Concrete pipeline
Below is a practical pipeline that often works well in real projects:
- Define a fidelity dimension, for example train_fraction in (0.1, 0.25, 0.5, 1.0) or number_of_epochs.
- Run an initial space-filling design at low fidelity to explore broadly.
- Fit a surrogate that takes hyperparameters and fidelity as inputs. Model heteroskedastic noise if possible because low fidelity tends to be noisier.
- Use a cost-aware acquisition function to propose candidates that maximize expected improvement per unit cost or similar criterion.
- Allocate more resource to top candidates via successive halving or a scheduler that increases fidelity gradually.
- Refit the surrogate with new data and repeat until budget is exhausted.
This flow can be implemented from scratch for learning and control, or assembled using libraries that provide building blocks. The following Python sketch shows an approachable implementation pattern using scikit-learn and simple allocation logic. It is pedagogical rather than production-ready.
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from scipy.stats import uniform
# Simple objective that uses train fraction as fidelity
def train_evaluate(params, fidelity, X, y, seed=0):
# fidelity is fraction of training data used
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=seed)
n = max(10, int(fidelity * len(X_tr)))
X_sub, y_sub = X_tr[:n], y_tr[:n]
model = RandomForestRegressor(n_estimators=int(params['n_estimators']),
max_depth=int(params['max_depth']),
random_state=seed)
model.fit(X_sub, y_sub)
pred = model.predict(X_val)
rmse = mean_squared_error(y_val, pred, squared=False)
cost = n # proxy for compute cost
return rmse, cost
# Simple search loop combining random proposals at low fidelity and surrogate fitting
def multi_fidelity_search(X, y, budget=2000):
# parameter bounds
bounds = {'n_estimators': (10, 200), 'max_depth': (3, 20)}
observations = []
total_cost = 0
rf = RandomForestRegressor()
# initial random evaluations at low fidelity
for _ in range(20):
p = {'n_estimators': int(uniform.rvs(*bounds['n_estimators'])),
'max_depth': int(uniform.rvs(*bounds['max_depth']))}
rmse, cost = train_evaluate(p, fidelity=0.2, X=X, y=y)
observations.append((p['n_estimators'], p['max_depth'], 0.2, rmse))
total_cost += cost
if total_cost > budget:
break
# iterative surrogate-based proposals
while total_cost < budget:
arr = np.array(observations)
X_obs = arr[:, :3].astype(float) # n_estimators, max_depth, fidelity
y_obs = arr[:, 3].astype(float)
rf.fit(X_obs, y_obs)
# propose candidates by sampling and picking the best predicted RMSE per cost
candidates = []
for _ in range(200):
n_est = int(uniform.rvs(*bounds['n_estimators']))
md = int(uniform.rvs(*bounds['max_depth']))
for fid in (0.2, 0.5, 1.0):
pred = rf.predict(np.array([[n_est, md, fid]]))[0]
cost = int(fid * len(X)) # same cost proxy
score = pred / (1 + cost / 1000.0) # crude cost-aware score
candidates.append((score, {'n_estimators': n_est, 'max_depth': md}, fid))
candidates.sort(key=lambda x: x[0])
# pick top candidate and evaluate at its fidelity
_, params, fid = candidates[0]
rmse, cost = train_evaluate(params, fidelity=fid, X=X, y=y)
observations.append((params['n_estimators'], params['max_depth'], fid, rmse))
total_cost += cost
return observations
# Example run
X, y = load_diabetes(return_X_y=True)
obs = multi_fidelity_search(X, y, budget=5000)
print('Done', len(obs), 'observations')The sketch above uses a cheap surrogate and a heuristic cost-aware score. In practice, replacing the surrogate with a Gaussian process that models fidelity explicitly or with a multi-task GP may provide better acquisition decisions. Libraries such as BOHB, dragonfly, and Ray Tune implement more sophisticated schedulers and acquisition procedures that reduce implementation burden.
Design tips and pitfalls
- Choose fidelities that are cheap but still correlated with full training performance. Subsampling the dataset or limiting epochs often works.
- Model noise properly. Low-fidelity runs can be much noisier; heteroskedastic surrogates or modeling observation variance helps.
- Account for cost explicitly when selecting candidates. A candidate that slightly improves predicted loss but costs ten times more may not be worth it.
- Use successive halving or adaptive scheduling to boost promising configurations instead of running all candidates to full fidelity.
- Validate the best-found configuration at full fidelity and, if possible, repeat to check stability.
Also be cautious about leakage: if the fidelity reduces dataset diversity in a way that biases validation, results may not transfer. Simple sanity checks comparing low and high-fidelity rankings can reveal issues early.
Integrations and practical libraries
Some projects that offer multi-fidelity tools and schedulers include:
- BOHB: combines Bayesian optimization with Hyperband for efficient resource allocation.
- Ray Tune: flexible scheduling with several search algorithms and support for bandit-based early stopping.
- Dragonfly: supports multi-fidelity Gaussian processes for structured optimization problems.
Choosing between a self-contained implementation and a library depends on how much control is needed and how well the fidelity mechanism fits the application. For rapid experiments, using an established library can speed up iteration.
Final notes for practitioners
Multi-fidelity Bayesian approaches can reduce compute while keeping the search effective. They are most beneficial when low-cost evaluations provide meaningful signal about high-cost outcomes. Start with exploratory experiments to confirm fidelity correlation, use cost-aware acquisition, and include sanity checks when promoting configurations to full fidelity. With careful design, hyperparameter tuning becomes faster and scales to larger problems.