Timezone: »

 
Poster
Altitude Training: Strong Bounds for Single-Layer Dropout
Stefan Wager · William S Fithian · Sida Wang · Percy Liang

Wed Dec 10 04:00 PM -- 08:59 PM (PST) @ Level 2, room 210D
in Poster Session »

Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks. This paper proposes a theoretical explanation for this phenomenon: we show that, under a generative Poisson topic model with long documents, dropout training improves the exponent in the generalization bound for empirical risk minimization. Dropout achieves this gain much like a marathon runner who practices at altitude: once a classifier learns to perform reasonably well on training examples that have been artificially corrupted by dropout, it will do very well on the uncorrupted test set. We also show that, under similar conditions, dropout preserves the Bayes decision boundary and should therefore induce minimal bias in high dimensions.

[ PDF] 

Author Information

Stefan Wager (Stanford University)
William S Fithian (Stanford University)
Sida Wang (Facebook AI Research)
Percy Liang (Stanford University)

More from the Same Authors