Representation Learning with Contrastive Predictive Coding

Seminar | September 14 | 11 a.m.-12 p.m. | 310 Sutardja Dai Hall

 Aaron van den Oord, Research Scientist, Deepmind


Representation Learning with Contrastive Predictive Coding

While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Aaron van den Oord works as a research scientist at DeepMind, London. His research focuses on generative models and representation learning. Aaron completed his PhD at the University of Ghent in Belgium where he worked on generative models, image compression and music recommendation. After Aaron joined DeepMind in 2015 he made important contributions to the field of generative modeling with autoregressive networks, including PixelRNN, PixelCNN and WaveNet. He also developed new techniques for speeding up generative models for text-to-speech synthesis, which are now used in various Google products such as the Google Assistant. In Aaron's most recent work he focused on representation learning with VQ-VAE and Contrastive Predictive Coding.