A Homogeneous Transformer Architecture

TitleA Homogeneous Transformer Architecture
Publication TypeCBMM Memos
Year of Publication2023
AuthorsGan, Y, Poggio, TA
Date Published09/2023

While the Transformer architecture has made a substantial impact in the field of machine learning, it is unclear what purpose each component serves in the overall architecture. Heterogeneous nonlinear circuits such as multi-layer RELU networks are interleaved with layers of soft-max units. We introduce here a homogeneous architecture based on Hyper Radial Basis Function (HyperBF) units. Evalua- tions on CIFAR10, CIFAR100, and Tiny ImageNet demonstrate a performance comparable to standard vision transformers.



Download:  PDF icon CBMM-Memo-143.pdf
CBMM Memo No:  143

Associated Module: 

CBMM Relationship: 

  • CBMM Funded