Classical generalization bounds are surprisingly tight for Deep Networks

TitleClassical generalization bounds are surprisingly tight for Deep Networks
Publication TypeCBMM Memos
Year of Publication2018
AuthorsLiao, Q, Miranda, B, Hidary, J, Poggio, T
Date Published07/2018

Deep networks are usually trained and tested in a regime in which the training classification error is not a good predictor of the test error. Thus the consensus has been that generalization, defined as convergence of the empirical to the expected error, does not hold for deep networks. Here we show that, when normalized appropriately after training, deep networks trained on exponential type losses show a good linear dependence of test loss on training loss. The observation, motivated by a previous theoretical analysis of overparametrization and overfitting, not only demonstrates the validity of classical generalization bounds for deep learning but suggests that they are tight. In addition, we also show that the bound of the classification error by the normalized cross entropy loss is empirically rather tight on the data sets we studied.




CBMM Memo No:  091

Research Area: 

CBMM Relationship: 

  • CBMM Funded