From Associative Memories to Deep Networks

TitleFrom Associative Memories to Deep Networks
Publication TypeCBMM Memos
Year of Publication2021
AuthorsPoggio, T
Date Published01/2021

About fifty years ago, holography was proposed as a model of associative memory. Associative memories with similar properties were soon after implemented as simple networks of threshold neurons by Willshaw and Longuet-Higgins. In these pages I will show that today’s deep nets are an incremental improvement of the original associative networks. Thinking about deep learning in terms of associative networks provides a more realistic and sober perspective on the promises of deep learning and on its role in eventually understanding human intelligence. As a bonus, this discussion also uncovers connections with several interesting topics in applied math: random features, random projections, neural ensembles, randomized kernels, memory and generalization, vector quantization and hierarchical vector quantization, random vectors and orthogonal basis, NTK and radial kernels.


Download:  PDF icon CBMM-Memo-114.pdf
CBMM Memo No:  114

Associated Module: 

CBMM Relationship: 

  • CBMM Funded