Timezone: »

 
Poster
Efficient Dataset Distillation using Random Feature Approximation
Noel Loo · Ramin Hasani · Alexander Amini · Daniela Rus

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #542
in Poster Session 6 »
Dataset distillation compresses large datasets into smaller synthetic coresets which retain performance with the aim of reducing the storage and computational burden of processing the entire dataset. Today's best performing algorithm, \textit{Kernel Inducing Points} (KIP), which makes use of the correspondence between infinite-width neural networks and kernel-ridge regression, is prohibitively slow due to the exact computation of the neural tangent kernel matrix, scaling $O(|S|^2)$, with $|S|$ being the coreset size. To improve this, we propose a novel algorithm that uses a random feature approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel which reduces the kernel matrix computation to $O(|S|)$. Our algorithm provides at least a 100-fold speedup over KIP and can run on a single GPU. Our new method, termed an RFA Distillation (RFAD), performs competitively with KIP and other dataset condensation algorithms in accuracy over a range of large-scale datasets, both in kernel regression and finite-width network training. We demonstrate the effectiveness of our approach on tasks involving model interpretability and privacy preservation.
Keywords: privacy · interpretability · Neural Tangent Kernel · Dataset Distillation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Noel Loo (MIT)
Ramin Hasani (MIT | Vanguard)
Alexander Amini (MIT)
Daniela Rus (Massachusetts Institute of Technology)

More from the Same Authors