Classical generalization bounds are surprisingly tight for Deep Networks

TitleClassical generalization bounds are surprisingly tight for Deep Networks
Publication TypeCBMM Memos
Year of Publication2018
AuthorsLiao, Q, Miranda, B, Hidary, J, Poggio, T
Date Published07/2018

Deep networks are usually trained and tested in a regime in which the training classification error is not a good predictor of the test error. Thus their working regime seems far from generalization, defined as convergence of the empirical to the expected error. Here we show that, when normalized appropriately, deep networks trained on exponential type losses show an approximately linear dependence of test loss on training loss. The observation, motivated by a previous theoretical analysis of over-parameterization and over-fitting, not only demonstrates the validity of classical generalization bounds for deep learning but suggests that they are tight, directly contradicting the claims of a recent, much cited paper titled “Understanding deep learning requires rethinking generalization”



CBMM Memo No: 


Research Area: 

CBMM Relationship: 

  • CBMM Funded