Timezone: »

 
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
Gowthami Somepalli · Avi Schwarzschild · Micah Goldblum · C. Bayan Bruss · Tom Goldstein

@
in Table Representation Learning »
Event URL: https://openreview.net/forum?id=FiyUTAy4sB8 »
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even performs competitively with gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over $30$ benchmark datasets in regression, binary classification, and multi-class classification tasks.
Keywords: pretraining · Contrastive · tabular

Author Information

Gowthami Somepalli (University of Maryland, College Park)
Avi Schwarzschild (University of Maryland)
Micah Goldblum (University of Maryland)
C. Bayan Bruss (Capital One)
Tom Goldstein (University of Maryland)

More from the Same Authors