Hyperparameter Tuning in Machine Learning: A Comprehensive Review
Justus A Ilemobayo
Samuel Ginn College of Engineering, Auburn University, USA.
Olamide Durodola *
Samuel Ginn College of Engineering, Auburn University, USA.
Oreoluwa Alade
Department of Physics, North Dakota State University, USA.
Opeyemi J Awotunde
Samuel Ginn College of Engineering, Auburn University, USA.
Adewumi T Olanrewaju
University of Florida, USA.
Olumide Falana
Samuel Ginn College of Engineering, Auburn University, USA.
Adedolapo Ogungbire
Department of Civil Engineering, University of Arkansas, USA.
Abraham Osinuga
Department of Chemical and Biomolecular Engineering, University of Nebraska-Lincoln, USA.
Dabira Ogunbiyi
Department of Biosystems and Agricultural Engineering, Oklahoma State, University, Stillwater, OK. USA
Ark Ifeanyi
Bredesen Center for Interdisciplinary Research and Graduate Education, University of Tennessee, USA.
Ikenna E Odezuligbo
Creighton University, Nebraska, USA.
Oluwagbotemi E Edu
Agricultural and Environmental Engineering Department, Obafemi Awolowo University, Nigeria.
*Author to whom correspondence should be addressed.
Abstract
Hyperparameter tuning is essential for optimizing the performance and generalization of machine learning (ML) models. This review explores the critical role of hyperparameter tuning in ML, detailing its importance, applications, and various optimization techniques. Key factors influencing ML performance, such as data quality, algorithm selection, and model complexity, are discussed, along with the impact of hyperparameters like learning rate and batch size on model training. Various tuning methods are examined, including grid search, random search, Bayesian optimization, and meta-learning. Special focus is given to the learning rate in deep learning, highlighting strategies for its optimization. Trade-offs in hyperparameter tuning, such as balancing computational cost and performance gain, are also addressed. Concluding with challenges and future directions, this review provides a comprehensive resource for improving the effectiveness and efficiency of ML models.
Keywords: Hyperparameter tuning, learning rate, batch size, grid search, random search, Bayesian optimization, meta-learning, neural networks