Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

TitleFisher-Rao Metric, Geometry, and Complexity of Neural Networks
Publication TypeReport
Year of Publication2017
AuthorsLiang, T, Poggio, T, Rakhlin, A, Stokes, J
Date Published11/2017
Other NumbersarXiv:1711.01530v1
Keywordscapacity control, deep learning, Fisher-Rao metric, generalization error, information geometry, Invariance, natural gradient, ReLU activation, statistical learning theory

We study the relationship between geometry and capacity measures for deep  neural  networks  from  an  invariance  viewpoint.  We  introduce  a  new notion  of  capacity — the  Fisher-Rao  norm — that  possesses  desirable  in- variance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures.  We  discuss  upper  bounds  on  the  generalization  error  induced by  the  proposed  measure.  Extensive  numerical  experiments  on  CIFAR-10 support  our  theoretical  findings.  Our  theoretical  analysis  rests  on  a  key structural lemma about partial derivatives of multi-layer rectifier networks.

Download:  PDF icon 1711.01530.pdf

Research Area: 

CBMM Relationship: 

  • CBMM Related