NCQS: Nonlinear Convex Quadrature Surrogate Hyperparameter Optimization

Abstract

Deep learning has revolutionized artificial intelligence and enabled breakthroughs across various domains. However, as deep learning models continue to grow in scale and complexity, optimizing their hyperparameters for efficient resource utilization becomes a critical challenge. Traditional optimization techniques often assume smooth and continuous loss functions, limiting their effectiveness in this context. In this work, we propose a novel data-driven approach to hyperparameter optimization using a convex quadrature surrogate. By leveraging a set of sampled hyperparameters and their corresponding performance, our method fits a multivariate quadratic surrogate model to identify the optimal hyperparameters. We demonstrate the practicality and effectiveness of our approach by improving the efficiency and performance of various hyperparameter strategies on both closed and open set benchmarks across diverse vision and tabular datasets. Additionally, we showcase its applicability in automatic target recognition tasks. This research contributes to the broader objective of resource-efficient deep learning for computer vision, fostering advancements in model efficiency, computational memory constraints, and latency considerations. Code available here.