M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy

Abstract

Training state-of-the-art (SOTA) deep models often requires extensive data, resulting in substantial training and storage costs. To address these challenges, dataset condensation has been developed to learn a small synthetic set that preserves essential information from the original large-scale dataset. Nowadays, optimization-oriented methods have been the primary method in the field of dataset condensation for achieving SOTA results. However, the bi-level optimization process hinders the practical application of such methods to realistic and larger datasets. To enhance condensation efficiency, previous works proposed Distribution-Matching (DM) as an alternative, which significantly reduces the condensation cost. Nonetheless, current DM-based methods still yield less comparable results to SOTA optimization-oriented methods. In this paper, we argue that existing DM-based methods overlook the higher-order alignment of the distributions, which may lead to sub-optimal matching results. Inspired by this, we present a novel DM-based method named M3D for dataset condensation by Minimizing the Maximum Mean Discrepancy between feature representations of the synthetic and real images. By embedding their distributions in a reproducing kernel Hilbert space, we align all orders of moments of the distributions of real and synthetic images, resulting in a more generalized condensed set. Notably, our method even surpasses the SOTA optimization-oriented method IDC on the high-resolution ImageNet dataset. Extensive analysis is conducted to verify the effectiveness of the proposed method. Source codes are available at https://github.com/Hansong-Zhang/M3D.

Cite

Text

Zhang et al. "M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy." AAAI Conference on Artificial Intelligence, 2024. doi:10.1609/AAAI.V38I8.28784

Markdown

[Zhang et al. "M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy." AAAI Conference on Artificial Intelligence, 2024.](https://mlanthology.org/aaai/2024/zhang2024aaai-m/) doi:10.1609/AAAI.V38I8.28784

BibTeX

@inproceedings{zhang2024aaai-m,
  title     = {{M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy}},
  author    = {Zhang, Hansong and Li, Shikun and Wang, Pengju and Zeng, Dan and Ge, Shiming},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {2024},
  pages     = {9314-9322},
  doi       = {10.1609/AAAI.V38I8.28784},
  url       = {https://mlanthology.org/aaai/2024/zhang2024aaai-m/}
}