Timezone: »

Recurrent Convolutional Neural Networks Learn Succinct Learning Algorithms
Surbhi Goel · Sham Kakade · Adam Kalai · Cyril Zhang

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #716

Neural Networks (NNs) struggle to efficiently learn certain problems, such as parity problems, even when there are simple learning algorithms for those problems. Can NNs discover learning algorithms on their own? We exhibit a NN architecture that, in polynomial time, learns as well as any efficient learning algorithm describable by a constant-sized learning algorithm. For example, on parity problems, the NN learns as well as row reduction, an efficient algorithm that can be succinctly described. Our architecture combines both recurrent weight-sharing between layers and convolutional weight-sharing to reduce the number of \textit{parameters} down to a constant, even though the network itself may have trillions of nodes. While in practice the constants in our analysis are too large to be directly meaningful, our work suggests that the synergy of Recurrent and Convolutional NNs (RCNNs) may be more powerful than either alone.

Author Information

Surbhi Goel (Microsoft Research NYC)
Sham Kakade (Harvard University & Amazon)
Adam Kalai (Microsoft Research New England (-(-_(-_-)_-)-))
Cyril Zhang (Microsoft Research NYC)

More from the Same Authors