CBMM Virtual Seminar: Marco Baroni

Jun 23, 2020 - 2:00 pm
Venue:  Zoom Speaker/s:  Marco Baroni, Facebook AI Research (Paris) and Catalan Institute for Research and Advanced Studies (Barcelona)

Title:
Is compositionality over-rated? A view from emergent neural network language analysis

Abstract:

Compositionality is the property whereby linguistic expressions that denote new composite meanings are derived by a rule-based combination of expressions denoting their parts. Linguists agree that compositionality plays a central role in natural language, accounting for its ability to express an infinite number of ideas by finite means.

"Deep" neural networks, for all their impressive achievements, often fail to quickly generalize to unseen examples, even when the latter display a predictable composite structure with respect to examples the network is already familiar with. This has led to interest in the topic of compositionality in neural networks: can deep networks parse language compositionally? how can we make them more sensitive to compositional structure? what does "compositionality" even mean in the context of deep learning?

I would like to address some of these questions in the context of recent work on language emergence in deep networks, in which we train two or more networks endowed with a communication channel to solve a task jointly, and study the communication code they develop. I will try to be precise about what "compositionality" mean in this context, and I will report the results of proof-of-concept and larger-scale experiments suggesting that (non-circular) compositionality is not a necessary condition for good generalization (of the kind illustrated in the figure). Moreover, I will show that often there is no reason to expect deep networks to find compositional languages more "natural" than highly entangled ones. I will conclude by suggesting that, if fast generalization is what we care about, we might as well focus directly on enhancing this property, without worrying about the compositionality of emergent neural network languages.

 

Please click the link below to join the webinar: 

https://mit.zoom.us/j/93213662313?pwd=N0F2eXUxT1gvRklCeFdDVzBZd0N5Zz09

Password: brains

Organizer:  Jean Lawrence Organizer Email:  cbmm-contact@mit.edu

CBMM Virtual Research Meeting: Hossein Mobahi

Jun 9, 2020 - 2:00 pm
Venue:  Zoom Speaker/s:  Hossein Mobahi, Google Research

TITLE:

Improving Generalization Performance by Self-Training and Self-Distillation
 
ABSTRACT:

In supervised learning we often seek a model which minimizes (to epsilon optimality) a loss function over a training set, possibly subject to some (implicit or explicit) regularization. Suppose you train a model this way and read out the predictions it makes over the training inputs, which may slightly differ from the training targets due to the epsilon optimality. Now suppose you treat these predictions as new target values, and retrain another model from scratch using those predictions instead of the original target values. Surprisingly, the second model can often outperform the original model in terms of accuracy on the test set. Actually, we may repeat this loop a few times, and each time see an increase in the generalization performance. This might sound strange as such a supervised self-training process (aka self-distillation) does not receive any new information about the task and solely evolves by retraining itself. In this talk, I argue such self-training process induces additional regularization, which gets amplified in each round of retraining. In fact, I will rigorously characterize such regularization effects when learning the function in Hilbert space. The latter setting can relate to neural networks with infinite width. I will conclude by discussing some open problems in the area of self-training and self-distillation.

 

Link to Talk: https://mit.zoom.us/j/95694089706?pwd=TnplQVRTbWMxZmJKdS84NXRoY3k2QT09

Password: brains

Organizer:  Jean Lawrence Organizer Email:  cbmm-contact@mit.edu

CBMM Virtual Research Meeting: Noga Zaslavsky

May 19, 2020 - 2:00 pm
Photo of Noga Zaslavsky
Address:  https://mit.zoom.us/j/99968215057?pwd=dVJsRzFXcFVYNzZnSUY1d05lcDVRdz09 Password: compress Speaker/s:  Noga Zaslavsky

 

Title: Efficient compression and linguistic meaning in humans and machines

 

Abstract: In this talk, I will argue that efficient compression may provide a fundamental principle underlying the human capacity to communicate and reason about meaning, and may help to inform machines with similar linguistic abilities. I will first address this at the population level, showing that pressure for efficient compression may drive the evolution of word meanings across languages, and may give rise to human-like semantic representations in artificial neural networks trained for vision. I will then address this at the agent level, where local context-dependent interactions influence the meaning of utterances. I will show that efficient compression may give rise to human pragmatic reasoning in reference games, suggesting a novel and principled approach to informing machine learning systems with pragmatic skills.

 

https://mit.zoom.us/j/99968215057?pwd=dVJsRzFXcFVYNzZnSUY1d05lcDVRdz09

Password: compress

Organizer:  Jean Lawrence Organizer Email:  cbmm-contact@mit.edu

CBMM Virtual Research Meeting: Max Tegmark

May 5, 2020 - 4:00 pm
Photo of Max Tegmark
Venue:  Zoom Address:  Link for Talk: https://mit.zoom.us/j/94413961955?pwd=Ni9TeSt3a2xpajkraGlJanJkOERBQT09 Speaker/s:  Max Tegmark, MIT

 

Title: AI for physics & physics for AI

 

Abstract: After briefly reviewing how machine learning is becoming ever-more widely used in physics, I explore how ideas and methods from physics can help improve machine learning, focusing on automated discovery of mathematical formulas from data. I present a method for unsupervised learning of equations of motion for objects in raw and optionally distorted unlabeled video. I also describe progress on symbolic regression, i.e.,  finding a symbolic expression that matches data from an unknown function. Although this problem is likely to be NP-hard in general, functions of practical interest often exhibit symmetries, separability, compositionality and other simplifying properties. In this spirit, we have developed a recursive multidimensional symbolic regression algorithm that combines neural network fitting with a suite of physics-inspired techniques that discover and exploit these simplifying properties, enabling significant improvement of state-of-the-art performance.

Link for talk: https://mit.zoom.us/j/94413961955?pwd=Ni9TeSt3a2xpajkraGlJanJkOERBQT09

Password included in announcement email

Organizer:  Jean Lawrence Organizer Email:  cbmm-contact@mit.edu

Research Meeting: Youssef Mroueh

Apr 28, 2020 - 2:00 pm
Photo of Youssef Mroueh
Venue:  Zoom Address:  https://mit.zoom.us/j/94030585358?pwd=bWVwaXQ5RE5NNC9mbU5JT0UzT1lzZz09 Speaker/s:  Youssef Mroueh, MIT-IBM Watson AI lab

 

 

Title of the talk: Sobolev Independence Criterion: Non-Linear Feature Selection with False Discovery Control.    

 

Abstract: In this talk I will show how learning gradients help us designing new non-linear algorithms for feature selection, black box sampling and also, in understanding neural style transfer. In the first part of the talk, I will present Sobolev Independence Criterion (SIC), that relates to saliency based method in deep learning. SIC is an interpretable dependency measure that gives rise to feature importance scores. Sparsity inducing gradient penalties are crucial regularizers for the SIC objective and in promoting the desired non-linear sparsity. SIC can subsequently be used in feature selection and false discovery rate control.

 

Paper: http://papers.nips.cc/paper/9147-sobolev-independence-criterion.pdf Joint work with Tom Sercu, Mattia Rigotti, Inkit Padhi and  Cicero Dos Santos

 

Bio: Youssef Mroueh is a research staff member in IBM Research and a principal investigator in the MIT-IBM Watson AI lab. He received his PhD in computer science in February 2015 from MIT, CSAIL, where he was advised by   Professors Tomaso Poggio and Lorenzo Rosasco. In 2011, he obtained his engineering diploma from Ecole Polytechnique Paris France, and a master of science in Applied Maths from Ecole des Mines de Paris. He is interested in Deep Learning, Machine Learning, Statistical Learning Theory, Computer Vision.  He conducts Modeling and Algorithmic research in Multimodal Deep Learning.

 

Link to Talk: https://mit.zoom.us/j/94030585358?pwd=bWVwaXQ5RE5NNC9mbU5JT0UzT1lzZz09

Password for Talk in Email Annoucement 

Organizer:  Jean Lawrence Hector Penagos Organizer Email:  cbmm-contact@mit.edu

Pages

Subscribe to Front page feed