Acknowledgement

This text represents a humble attempt to summarise and synthesise decades of

existing knowledge in deep learning architectures, through the geometric lens

of invariance and symmetry. We hope that our perspective will make it easier

both for newcomers and practitioners to navigate the ﬁeld, and for researchers

to synthesise novel architectures, as instances of our blueprint. In a way, we

hope to have presented “all you need to build the architectures that are all you

need”—a play on words inspired by Vaswani et al. (2017).

The initial part of the text was written during the coronavirus pandemic

(2020–2021). As it often happens, we had thousands of doubts whether the

whole picture makes sense, and used opportunities provided by our colleagues

to help us break our “stage fright” and present early versions of our work,

which saw the light of day in Petar’s talk at Cambridge (courtesy of Pietro

Liò) and Michael’s talks at Oxford (courtesy of Xiaowen Dong) and Imperial

College (hosted by Michael Huth and Daniel Rueckert). Petar was also able to

present our work at Friedrich-Alexander-Universität Erlangen-Nürnberg—the

birthplace of the Erlangen Program!—owing to a kind invitation from Andreas

Maier. The feedback we received for these talks was enormously invaluable to

keeping our spirits high, as well as polishing the work further. Last, but cer-

tainly not least, we thank the organising committee of ICLR 2021, where our

work featured in a keynote talk, delivered by Michael.

We should note that reconciling such a vast quantity of research is sel-

dom enabled by the expertise of only four people. Accordingly, we would

like to give due credit to all of the researchers who have carefully studied

aspects of our text as it evolved, and provided us with careful comments

and references: Yoshua Bengio, Charles Blundell, Andreea Deac, Fabian

Fuchs, Francesco Di Giovanni, Aleksa Gordi

c, Marco Gori, Raia Hadsell, Will

Hamilton, Maksym Korablyov, Marc Lackenby, Abbas Mehrabian, Christian

Merkwirth, Maks Ovsjanikov, Razvan Pascanu, Sébastien Racaniere, Bruno

Ribeiro, Anna Scaife, Jürgen Schmidhuber, Marwin Segler, Csaba Szepesvári,

Corentin Tallec, Ngân V

u, Geordie Williamson, Peter Wirnsberger and David

Wong. Their expert feedback was invaluable to solidifying our uniﬁcation

efforts and making them more useful to various niches. Though, of course,

any irregularities within this text are our responsibility alone.