Forward learning with top-down feedback: empirical and analytical characterization

TitleForward learning with top-down feedback: empirical and analytical characterization
Publication TypeJournal Article
Year of Publication2023
AuthorsSrinivasan, RFrancesco, Mignacco, F, Sorbaro, M, Refinetti, M, Kreiman, G, Dellaferrera, G
Date Published02/2023

“Forward-only” algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation. Here, we first discuss the similarities between two “forward-only” algorithms, the Forward- Forward and PEPITA frameworks, and demonstrate that PEPITA is equivalent to a Forward- Forward framework with top-down feedback connections. Then, we focus on PEPITA to address compelling challenges related to the “forward- only” rules, which include providing an analytical understanding of their dynamics and reducing the gap between their performance and that of backpropagation. We propose a theoretical analysis of the dynamics of PEPITA. In particular, we show that PEPITA is well-approximated by an “adaptive-feedback-alignment” algorithm and we analytically track its performance during learning in a prototype high-dimensional setting. Finally, we develop a strategy to apply the weight mirroring algorithm on “forward-only” algorithms with top-down feedback and we show how it impacts PEPITA’s accuracy and convergence rate.


Associated Module: 

CBMM Relationship: 

  • CBMM Funded