Acknowledgement
This text represents a humble attempt to summarise and synthesise decades of
existing knowledge in deep learning architectures, through the geometric lens
of invariance and symmetry. We hope that our perspective will make it easier
both for newcomers and practitioners to navigate the field, and for researchers
to synthesise novel architectures, as instances of our blueprint. In a way, we
hope to have presented “all you need to build the architectures that are all you
need”—a play on words inspired by Vaswani et al. (2017).
The initial part of the text was written during the coronavirus pandemic
(2020–2021). As it often happens, we had thousands of doubts whether the
whole picture makes sense, and used opportunities provided by our colleagues
to help us break our “stage fright” and present early versions of our work,
which saw the light of day in Petar’s talk at Cambridge (courtesy of Pietro
Liò) and Michael’s talks at Oxford (courtesy of Xiaowen Dong) and Imperial
College (hosted by Michael Huth and Daniel Rueckert). Petar was also able to
present our work at Friedrich-Alexander-Universität Erlangen-Nürnberg—the
birthplace of the Erlangen Program!—owing to a kind invitation from Andreas
Maier. The feedback we received for these talks was enormously invaluable to
keeping our spirits high, as well as polishing the work further. Last, but cer-
tainly not least, we thank the organising committee of ICLR 2021, where our
work featured in a keynote talk, delivered by Michael.
We should note that reconciling such a vast quantity of research is seldom
enabled by the expertise of only four people. Accordingly, we would like to
give due credit to all of the researchers who have carefully studied aspects of
our text as it evolved, and provided us with careful comments and references:
Yoshua Bengio, Charles Blundell, Andreea Deac, Fabian Fuchs, Francesco Di
Giovanni, Aleksa Gordi
´
c, Marco Gori, Raia Hadsell, Will Hamilton, Maksym
Korablyov, Marc Lackenby, Abbas Mehrabian, Christian Merkwirth, Razvan
Pascanu, Sébastien Racaniere, Bruno Ribeiro, Anna Scaife, Jürgen Schmid-
huber, Marwin Segler, Csaba Szepesvári, Corentin Tallec, Ngân V
˜
u, Geordie
Williamson, Peter Wirnsberger and David Wong. Their expert feedback was
invaluable to solidifying our unification efforts and making them more useful
to various niches. Though, of course, any irregularities within this text are our
responsibility alone.
