Timezone: »

 
Poster
Data Pruning via Moving-one-Sample-out
Haoru Tan · Sitong Wu · Fei Du · Yukang Chen · Zhibin Wang · Fan Wang · Xiaojuan Qi

Thu Dec 14 03:00 PM -- 05:00 PM (PST) @ Great Hall & Hall B1+B2 #617
in Poster Session 6 »

In this paper, we propose a novel data-pruning approach called moving-one-sample-out (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and outperforms state-of-the-art methods by a large margin across various settings.

Author Information

Haoru Tan (HKU)
Sitong Wu (ucas)
Fei Du (Alibaba Group)
Yukang Chen (The Chinese University of Hong Kong)
Zhibin Wang (Alibaba Group)
Fan Wang (Alibaba Group)
Xiaojuan Qi (The University of Hong Kong)

More from the Same Authors