Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

B Bischl, M Binder, M Lang, T Pielok… - … : Data Mining and …, 2023 - Wiley Online Library
Most machine learning algorithms are configured by a set of hyperparameters whose values
must be carefully chosen and which often considerably impact performance. To avoid a time …

Taking the human out of the loop: A review of Bayesian optimization

B Shahriari, K Swersky, Z Wang… - Proceedings of the …, 2015 - ieeexplore.ieee.org
Big Data applications are typically associated with systems involving large numbers of
users, massive complex software systems, and large-scale heterogeneous computing and …

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

M Wortsman, G Ilharco, SY Gadre… - International …, 2022 - proceedings.mlr.press
The conventional recipe for maximizing model accuracy is to (1) train multiple models with
various hyperparameters and (2) pick the individual model which performs best on a held …

Advances and open problems in federated learning

P Kairouz, HB McMahan, B Avent… - … and trends® in …, 2021 - nowpublishers.com
Federated learning (FL) is a machine learning setting where many clients (eg, mobile
devices or whole organizations) collaboratively train a model under the orchestration of a …

Mobilenetv2: Inverted residuals and linear bottlenecks

M Sandler, A Howard, M Zhu… - Proceedings of the …, 2018 - openaccess.thecvf.com
In this paper we describe a new mobile architecture, mbox {MobileNetV2}, that improves the
state of the art performance of mobile models on multiple tasks and benchmarks as well as …

Learning transferable architectures for scalable image recognition

B Zoph, V Vasudevan, J Shlens… - Proceedings of the …, 2018 - openaccess.thecvf.com
Develo** neural network image classification models often requires significant
architecture engineering. In this paper, we study a method to learn the model architectures …

AutoML: A survey of the state-of-the-art

X He, K Zhao, X Chu - Knowledge-based systems, 2021 - Elsevier
Deep learning (DL) techniques have obtained remarkable achievements on various tasks,
such as image recognition, object detection, and language modeling. However, building a …

[PDF][PDF] Hyperparameter optimization

M Feurer, F Hutter - Automated machine learning: Methods …, 2019 - library.oapen.org
Recent interest in complex and computationally expensive machine learning models with
many hyperparameters, such as automated machine learning (AutoML) frameworks and …

Hyperband: A novel bandit-based approach to hyperparameter optimization

L Li, K Jamieson, G DeSalvo, A Rostamizadeh… - Journal of Machine …, 2018 - jmlr.org
Performance of machine learning algorithms depends critically on identifying a good set of
hyperparameters. While recent approaches use Bayesian optimization to adaptively select …

Deep networks with stochastic depth

G Huang, Y Sun, Z Liu, D Sedra… - Computer Vision–ECCV …, 2016 - Springer
Very deep convolutional networks with hundreds of layers have led to significant reductions
in error on competitive benchmarks. Although the unmatched expressiveness of the many …