%0 Journal Article %J arXiv %D 2023 %T Forward learning with top-down feedback: empirical and analytical characterization %A Ravi Francesco Srinivasan %A Francesca Mignacco %A Martino Sorbaro %A Maria Refinetti %A Gabriel Kreiman %A Giorgia Dellaferrera %X

“Forward-only” algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation. Here, we first discuss the similarities between two “forward-only” algorithms, the Forward- Forward and PEPITA frameworks, and demonstrate that PEPITA is equivalent to a Forward- Forward framework with top-down feedback connections. Then, we focus on PEPITA to address compelling challenges related to the “forward- only” rules, which include providing an analytical understanding of their dynamics and reducing the gap between their performance and that of backpropagation. We propose a theoretical analysis of the dynamics of PEPITA. In particular, we show that PEPITA is well-approximated by an “adaptive-feedback-alignment” algorithm and we analytically track its performance during learning in a prototype high-dimensional setting. Finally, we develop a strategy to apply the weight mirroring algorithm on “forward-only” algorithms with top-down feedback and we show how it impacts PEPITA’s accuracy and convergence rate.

%B arXiv %8 02/2023 %G eng %U https://arxiv.org/abs/2302.05440