Submodular Meta Data Compiling for Meta Optimization

Abstract

The search for good hyper-parameters is crucial for various deep learning methods. In addition to the hyper-parameter tuning on validation data, meta-learning provides a promising manner for optimizing the hyper-parameters, referred to as meta optimization. In all existing meta optimization methods, the meta data set is directly given or constructed from training data based on simple selection criteria. This study investigates the automatic compiling of a high-quality meta set from training data with more well-designed criteria and the submodular optimization strategy. First, a theoretical analysis is conducted for the generalization gap of meta optimization with a general meta data compiling method. Illuminated by the theoretical analysis, four criteria are presented to reduce the gap’s upper bound. Second, the four criteria are cooperated to construct an optimization problem for the automatic meta data selection from training data. The optimization problem is proven to be submodular, and the submodular optimization strategy is employed to optimize the selection process. An extensive experimental study is conducted, and results indicate that our compiled meta data can yield better or comparable performances than the data compiled with existing methods.