Acknowledgement

This text represents a humble attempt to summarise and synthesise decades of

existing knowledge in deep learning architectures, through the geometric lens

of invariance and symmetry. We hope that our perspective will make it easier

both for newcomers and practitioners to navigate the field, and for researchers

to synthesise novel architectures, as instances of our blueprint. In a way, we

hope to have presented “all you need to build the architectures that are all you

need”—a play on words inspired by Vaswani et al. (2017).

The initial part of the text was written during the coronavirus pandemic

(2020–2021). As it often happens, we had thousands of doubts whether the

birthplace of the Erlangen Program!—owing to a kind invitation from Andreas

Maier. The feedback we received for these talks was enormously invaluable to

keeping our spirits high, as well as polishing the work further. Last, but cer-

tainly not least, we thank the organising committee of ICLR 2021, where our

work featured in a keynote talk, delivered by Michael.

We should note that reconciling such a vast quantity of research is seldom

enabled by the expertise of only four people. Accordingly, we would like to

give due credit to all of the researchers who have carefully studied aspects of

our text as it evolved, and provided us with careful comments and references:

Yoshua Bengio, Charles Blundell, Andreea Deac, Fabian Fuchs, Francesco Di

Williamson, Peter Wirnsberger and David Wong. Their expert feedback was

invaluable to solidifying our unification efforts and making them more useful

to various niches. Though, of course, any irregularities within this text are our

responsibility alone.