Abstract: Supervised learning is the problem of estimating a function from input and output samples. But how many samples are needed to achieve a prescribed accuracy?

This question can be answered only by restricting the class of problems—for example, considering functions that don’t vary much. But in even this case, we find that the number of needed samples depends exponentially on the dimensions of each input—the so-called curse of dimensionality.

Since neural nets seem to learn well with much less data, it is natural to postulate that the underlying problems (functions) have more structure beyond bounded variations.  The search for the right notion of “structure” has been quite elusive thus far, and I will discuss some recent results that emphasize the role of sparsity and compositions.

Bio: Lorenzo Rosasco is a professor at the University of Genova, a research affiliate at MIT, and a visiting scientist at the Italian Technological Institute (IIT). He is a founder and coordinator of the Machine Learning Genova center (MaLGa) and the Laboratory for Computational and Statistical Learning, focusing on the theory, algorithms, and applications of machine learning. He obtained his PhD in 2006 from the University of Genova and was a visiting student at the Center for Biological and Computational Learning at MIT, the Toyota Technological Institute at Chicago (TTI-Chicago), and the Johann Radon Institute for Computational and Applied Mathematics. From 2006 to 2013, he worked as a postdoc and research scientist at the Brain and Cognitive Sciences Department at MIT. He is a fellow at Ellis and serves as the co-director of the "Theory, Algorithms and Computations of Modern Learning Systems" program as well as the Ellis Genoa unit. Lorenzo has received several awards, including an ERC consolidator grant.

Abstract: How do neurons, in their collective action, beget cognition, as well as intelligence and reasoning? As Richard Axel recently put it, we do not have a logic for the transformation of neural activity into thought and action; discerning this logic as the most important future direction of neuroscience. I will present a mathematical neural model of brain computation called NEMO, whose key ingredients are spiking neurons, random synapses and weights, local inhibition, and Hebbian plasticity (no backpropagation). Concepts are represented by interconnected co-firing assemblies of neurons that emerge organically from the dynamical system of its equations. It turns out it is possible to carry out complex operations on these concept representations, such as copying, merging, completion from small subsets, and sequence memorization. NEMO is a neuromorphic computational system that, because of its simplifying assumptions, can be efficiently simulated on modern hardware. I will present how to use NEMO to implement an efficient parser of a small but non-trivial subset of English, and a more recent model of the language organ in the baby brain that learns the meaning of words, and basic syntax, from whole sentences with grounded input. In addition to constituting hypotheses as to the logic of the brain, we will discuss how principles from these brain-like models might be used to improve AI, which, despite astounding recent progress, still lags behind humans in several key dimensions such as creativity, hard constraints, energy consumption.

Abstract: 

This discussion will feature panelists from the Poggio and Fiete labs describing ongoing work on deep classifiers and hippocampal-inspired networks, respectively. Following brief presentations by each panelist, Ila Fiete will steer a discussion to explore strengths, limitations and neural plausibility of each approach.

Sugandha Sharma and Sarthak Chandra (Fiete lab).

Hippocampal circuits in the brain enable two distinct cognitive functions: construction of spatial maps for navigation and storage of sequential episodic memories. This dual role of the hippocampus remains an enduring enigma. While there have been advances in modeling the spatial representation properties of the hippocampus, we lack good models of its role in episodic memory. Here we present a neocortical-entorhinal-hippocampal network model that exhibits high-capacity general episodic memory and spatial memory without the memory cliff of existing neural memory models. Instead, the circuit exhibits a graceful tradeoff between number of stored items and detail, achieved by factorizing content storage from the dynamics of generating error-correcting stable states. The exponentially large space avoids catastrophic forgetting. Next, we show that pre-structured representations are an essential feature for constructing episodic memory: unlike existing episodic memory models, they enable high-capacity memorization of sequences by abstracting the chaining problem into one of learning transitions within a rigid low-dimensional grid cell scaffold. Finally, we show that previously learned spatial sequences in the form of location-landmark associations can themselves be re-usably leveraged as robust scaffolds and associated with neocortical inputs for a high-fidelity one-shot memory, providing the first circuit model of the ``memory palaces'' used in the striking feats of memory athletes.

Akshay Rangamani and Tomer Galanti (Poggio lab).

We conduct an empirical study of the feature learning process in deep classifiers. Recent research has identified a training phenomenon called Neural Collapse (NC), in which the top- layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer’s weights align with those features. Our study aims to investigate if these properties extend to intermediate layers. We empirically study the evolution of the covariance and mean of representations across different layers and show that as we move deeper into a trained neural network, the within-class covariance decreases relative to the between-class covariance. Additionally, we find that in the top layers, where the between-class covariance is dominant, the subspace spanned by the class means aligns with the subspace spanned by the most significant singular vector components of the weight matrix in the corresponding layer. Finally, we discuss the relationship between NC and Associative Memories.

The fundamental problem of machine learning is often formulated as the problem of function approximation. For example, we have data of the form {(xj,yj)}, where yj is the class label for xj, and we want to approximate the class label as a function of the input x. The standard way for this approximation is to minimize a loss functional, usually with some regularization. Surprisingly, even though the problem is posed as a problem of function approxi- mation, approximation theory has played only a marginal role in this theory. We describe our efforts to explore why this might be the case, and also to develop approximation theory/harmonic analysis tools more meaningfully and directly applicable to machine learning. We also argue that classification problems are better treated as generalized signal separation problems rather than function approximation problems.