Further Beyond

•

Many behaviours we would like to align neural networks to are not

invertible, nor fully composable. Modelling them escapes the axioms of

group theory required by the geometric deep learning framework.

•

Lifting the invertibility constraint leads us to monoids; an excellent

starting point for alignment to computer science primitives.

•

Lifting compositionality leads us to groupoids, expressing GDL-

consistent models that behave differently across distinct input orbits.

•

Lifting both at once unleashes the full generality of category theory,

embodied within the categorical deep learning framework.

•

It is not always obvious where relevant structures for applying GDL

(groups, or beyond) would come from. Several relevant techniques exist

to approximate or derive such structure, and we cover many of them here.

Groups, representations and equivariance offer a reliable, well-studied basis

for formalising the constraints of neural network architectures. This has been

observed throughout our book’s Chapters. The aim of the introductory mate-

rial of Part I was to provide a solid foundation in the key concepts necessary to

leverage geometric deep learning principles, whereas Part II presented diverse

case studies to show how powerful this framework truly can be for improv-

ing our understanding of neural network architectures—while avoiding getting

bogged down in details for any particular domain of GDL.

While these parts (hopefully!) accomplish the pedagogical aims we had, we

also acknowledge that we’ve covered virtually every popular architecture in

use for deep learning at the time of writing: from graph neural networks and

large language models, through convolutional and recurrent neural networks,

all the way to exotic-domain models such as spherical CNNs, tensor ﬁeld net-

works, or gauge-equivariant mesh CNNs. Given this breadth, it wouldn’t be

all too unexpected for a newly-minted GDL practitioner to presume there is no

point in studying anything beyond GDL! Yet, generalisations of GDL certainly

exist – groups are, relatively speaking, quite simple mathematical structures

258 Chapter 9

– and thinking outside of the Erlangen Program’s conﬁnes can indeed open

horizons in neural network architectures that our blueprint does not cover.

While keeping in mind this is a highly nascent direction, with many roads

left uncharted, this ﬁnal Chapter aims to provide a useful sketch of a roadmap

to navigate it.

9.1 Why Go Beyond Groups?

At its core, equivariance helps architectures gain generalisation capability;

that is, it guarantees a certain type of generalisation (w.r.t. the actions of a

given symmetry group G) must be respected over all possible model inputs.

The key reason to move beyond groups, then, is when they provide insufﬁcient

ﬂexibility to represent a desirable form of generalisation.

To see where such shortcomings might come from, we draw attention to two

speciﬁc group axioms:

g ∈G ∧h ∈G =⇒ gh ∈G (closure / total compositionality)

g ∈G =⇒ g

–1

∈G, s.t. gg

–1

= g

–1

g = e (invertibility)

These axioms felt natural when we aimed to capture the intuitive constraint

that “symmetries should leave objects unchanged”. However, many types of

generalisation we might care about concern transformations that do fundamen-

tally alter the objects they act on, sometimes irreversibly so! Such capabilities

cannot be characterised using the formalisms of GDL we built up so far.

Example 1: Generalising across horizons As a simple example, we are

often interested in designing deep learning architectures capable of size gener-

alisation

: they should perform consistently well across inputs of a wide range

of input sizes—even ones never seen during training. This is a notorious pain

point for neural networks, given that most of their success stories are obtained

when generalising in-distribution, i.e. when the patterns found in their training

data properly anticipate the ones found at inference time.

A great inspiration for relevant capabilities of this kind comes from learning

robust computer science primitives, such as algorithms and data structures,

from data (Veli

ckovi

c and Blundell 2021). For the learnt models to behave

robustly here, the target algorithm’s invariants must be preserved within these

models, irrespective of, e.g., the size or generative distribution of the input.

This implies that any training set we have will likely not cover all possible

scenarios encountered in practical inference.

How can we express a meaningful constraint involving the model’s

behaviour across different input lengths? If we want to follow the spirit of

GDL, we need to ﬁnd a collection of transformations (akin to a symmetry

Further Beyond 259

ﬁlter (̸=

)

ﬁlter (̸=

)

Figure 9.1

Filter equivariance (Lewis et al. 2025) is a way to express functions which are length-

invariant through removing items by value. Generally, a ﬁlter-equivariant function f

may be composed with ﬁlter in any order, yielding the same results at the end. In this

example, we ﬁlter by removing all elements with a “green” colour attribute.

group) which characterise a “change in length”, then enforcing the model

to provide predictably transformed results whenever such a length-change

operation is applied.

Two immediate options arise for a set of length-altering transformations:

adding elements to the input, or removing them from the input. The latter turns

out much easier to characterise: for any ﬁnite input of n elements, we have





possible, well-speciﬁed, k-element removals; conversely, there are inﬁnite

ways to add new elements into a list—for certain input data types, even adding

a single element presents unbounded possibilities. Additionally, while there

are many strategies for specifying deletion, a common content-based strategy

is the ﬁlter operation in functional programming, which removes all elements

from the input which match a particular predicate.

Given the above collection of ﬁltering operations, it is then possible to

express a particular kind of size generalisation by discussing “ﬁlter equivari-

ant

” functions: functions whose answer is identical regardless of whether we

ﬁlter their input or their output, across all possible predicates (see Figure 9.1).

It should be easy to see why the collection of all ﬁlters does not satisfy

the axioms of a group: deleting items from a set is a non-invertible operation!

Clearly, we must go beyond GDL to analyse ﬁlter equivariant functions. On

this, we fully defer to Lewis et al. (2025) where many beautiful properties

of such functions, and an algorithm for reconstructing them from data, are

260 Chapter 9

provided. What is worthy of noting is that, if we know our function is ﬁlter-

equivariant, only a small number of carefully chosen data points are needed to

reconstruct such functions exactly.

Example 2: Graph-bespoke GNNs Throughout all of our discussions of

graph neural network design – be it relating to generic combinatorial graphs

from Chapter 5, or to geometric graphs from Chapter 8 – we had always

assumed that their parametric components (e.g., the message function, ψ, and

the update function, ϕ, from Equation 5.56) were always fully shared across

all (pairs of) nodes, within all of the graphs we have.

While this is an elegant way of representing a broad range of permutation-

equivariant functions on graphs efﬁciently, strictly speaking, there is no need

for every (non-isomorphic) graph to be processed the same way – each graph

emits its own permutation equivariance constraint, after all, and these do not

interact with each other.

This points to a clear opportunity to build more expressive GNNs: allow-

ing for their computations to arbitrarily vary for different input graphs, as long

as those graphs do not have conﬂicting equivariance constraints. Most gener-

ally, this allows us to formulate graph-bespoke GNNs, F

: X

→Y

, which

transform feature spaces X

(e.g. node feature matrices) into outputs Y

in a

manner speciﬁc to the input graph. Parameter sharing across two GNNs, F

and F

′

, is enforced only if there exists an isomorphism (node permutation)

between them.

Note that much of the machinery of permutation equivariance is still present

here, i.e., all of the transformations in our collection of interest are still just

permutations, and hence, unlike the ﬁltering example above, they are invert-

ible. However, now that our neural networks’ parameters depend directly on

the speciﬁc connectivity of the input graph, we only care about permutations

that are valid isomorphisms between speciﬁc pairs of graphs—that is, permu-

tations that map the edges of a source graph G directly onto the edges of a

target isomorphic graph G

′

. Because a permutation that validly maps G

’s

edges to G

’s edges will generally not map a different graph G

onto G

we can no longer treat our symmetries as untyped elements of S

; instead,

we must explicitly track each transformation alongside the speciﬁc graphs it

connects, as a typed permutation σ

G→G

′

: G →G

′

. Such permutations yield

Further Beyond 261

1 2

constrains F

→G

swap 1 ↔ 3

1 2

with F

∼

1 2

constrains F

→G

swap 1 ↔ 3

1 2

with F

1 2

“σ

→G

◦ σ

→G

” : G

−→ G

cannot exist, as G

∼

1 2

Figure 9.2

A simple example demonstrating why, when tracking a collection of typed permuta-

tions of graphs, σ

G→G

′

: G →G

′

, total compositionality no longer holds. We observe

two isomorphic pairs of three-node graphs: (G

, G

) with a single edge, and (G

, G

)

with two edges. Clearly, G

∼

; i.e., these two graphs are not isomorphic due to a dif-

ferent edge set cardinality. In this setting, the composition of any σ

→G

with σ

→G

is impossible, as no typed permutations (isomorphisms) can exist between G

and G

which would be the necessary endpoints of such a function composition. This is a key

aspect of working with typed permutations—if we, instead, treated σ

→G

and σ

→G

as elements of the (untyped) permutation group, S

, they will compose perfectly ﬁne (in

this example, producing the identity permutation).

corresponding commutative diagrams:

′

(σ

G→G

′

)

(σ

G→G

′

)

′

(9.157)

where ρ

and ρ

provide the real representations of σ

G→G

′

, just as in the usual

equivariance diagrams.

However, tracking a collection of typed permutations no longer allows us

to freely combine them; instead, we can only compose σ

: G

→G

with σ

→G

into σ

◦σ

: G

→G

if G

= G

(i.e., the target graph of the ﬁrst

transformation matches the source graph of the second!). See also Figure 9.2.

The fact that not all pairs of transformations in our collection compose into

transformations in the collection violates the compositionality group axiom

above! Hence, we are going to need a more generic object than a group to

reason about such transformations.

262 Chapter 9

9.2 What Lies Beyond?

Precisely speaking, to transcend this boundary, we might want to allow certain

transformations g to not have deﬁned inverses g

–1

, or to have pairs g, h for

which the composition gh is not deﬁned, or both of these properties at once.

Depending on which of these constraints we choose to drop, we get one of

three possible generalisations of groups (the monoid, the groupoid, and the

category, respectively)—each with its own set of situations where they might

be particularly useful. They relate to each other by generality as follows:

monoid category

group groupoid

partial compositionality

no invertibility

partial compositionality

no invertibility

(9.158)

We have already hinted at examples of two of these structures: the collection

of ﬁlter operations forms a monoid, whereas the collection of typed permu-

tations forms a groupoid. It is, of course, possible to drop both conditions,

leading to the general notion

of a category. Category theory represents one of

the pinnacles of abstract mathematics, which allowed mathematicians to con-

nect many seemingly disparate results across ﬁelds, due to them having similar

compositional structures—and it is not unexpected that similar efforts are now

inﬂuencing deep learning

as well (Gavranovi

c 2024).

One important issue that arises when migrating from groups into these less

constrained objects is that it becomes harder to resolve the induced constraints

(as the space of possible solutions grows). Hence, to illustrate the breadth

of possible approaches, we present speciﬁc examples where the derivations

yielded a new kind of architecture. In all cases, there will still be residual ambi-

guity where it is unclear that we chose the best possible class of solutions; we

will clearly indicate these moments in what follows.

9.2.1 Monoids: asynchrony invariance

The ﬁrst object we focus on is the monoid—a “group without inverses”.

Monoids are a great abstraction to study computational primitives, as there

exist exact correspondences between monoid actions and state machines.

As such, it is only ﬁtting that we now elaborate on the asynchrony-invariant

GNN of Dudzik et al. (2024) we hinted at in Chapter 5. As a quick reminder

on motivation: while GNNs are typically implemented fully synchronously

(all nodes transmit and receive messages in parallel, simultaneously), many

computations one might want to do over a graph do not require all nodes to

Further Beyond 263

transmit messages at the same time; a certain degree of asynchrony in the exe-

cution might be preferred. With asynchrony invariance, we instead elect to

keep the model fully synchronous for scalability reasons, but design its com-

ponents such that the ﬁnal embeddings and predictions are the same as if the

messages are being received, and potentially sent, out of sync with each other.

The message, state and argument monoids The forward pass of a GNN

from Chapter 5 (Equation 5.56) exchanges information at varying stages—

to allow us to reason about the order of computation execution, we need to

formalise each of these stages as a separate collection (even though, in practice,

they are all usually latent embeddings in R

). In particular, we model different

stages of data processing as the following sets:

•

The messages, M, formalising information sent across an edge.

•

The node state, S, iteratively updated as messages arrive.

•

The node arguments, A, which govern the inputs each node provides to the

message function, ψ.

We further assume a simple but important caveat: both messages and argu-

ments can be iteratively assembled, allowing them a monoid structure. Follow-

ing the usual monoid notation as a triplet of (set of values, binary operation,

identity element), we can write these two monoids as (M, ·, 1) and (A, +, 0),

respectively. Note that the identity elements, or the operation, do not have to be

distinct at all across these two monoids. (The eager reader might wonder why

arguments, A, are important to explicitly track in this way, as in most practical

GNNs, what goes into the message function is the node state, i.e., S = A. Treat-

ing arguments separately will prove critical when reasoning about invariance

to out-of-order message creation and sending.)

Monoid actions and cocycles We can formalise the effect that an incoming

message, m ∈M, has on the state, s ∈S, as follows:

(m, s) 7→(m •s, δ

(s)) (9.159)

where •: M×S →S transforms the state using the message, and δ : M×

S →A emits an argument for future message function calls that include this

node. Both of these operations are transformations induced by messages.

Similarly to how we deﬁned group actions axiomatically in Chapter 3, we

can deﬁne monoid actions, as these axioms did not depend on invertibility. As

it relates to how results are affected by composition, if we constrain • and δ

via the monoid action and composition laws, we already can induce a form of

264 Chapter 9

asynchrony invariance! Concretely, these transformations now must satisfy:

1 •s = s δ

(s) = 0 (9.160)

·m

) •s = m

•(m

•s) δ

·m

(s) = δ

(s) + δ

•s) (9.161)

The ﬁrst row of constraints concerns the interaction of identity elements, and

is as one would expect – an identity message must not change its target’s state,

and it must trigger an identity argument. The composition law for • also fol-

lows the expected form quite closely: given two messages, we either update

the state twice, compositionally, or instead, we compose the messages ﬁrst and

then act on the state with the composed resulting message. This represents an

asynchrony invariance constraint, as once it is satisﬁed, the model may either

transform states with incoming messages immediately as they arrive, or it may

wait to aggregate them ﬁrst before transforming—the outcome is guaranteed

to be the same in both cases.

A more interesting type of compositionality occurs when decomposing

·m

: it implies we ﬁrst generate the argument and transform the state using

(yielding a new state, m

•s, and an emitted argument δ

(s)). Then,

we generate another argument by acting on this transformed state with m

•s). We can then use the monoid operation of A to combine these

arguments into one.

This type of condition is popular in mathematics, as satisfying it allows for

combining multiple structures (in this case, the monoids M and A and state

space S) into one coherent system. It is often referred to as a (1-)cocycle,

derivation or a crossed homomorphism, depending on context—we will use

the cocycle term. Much like with the previous constraint, knowing that δ

a cocycle induces asynchrony invariance, allowing us to emit arguments from

incoming messages at any level of synchrony desired.

Monoid homomorphisms for messages Once arguments, a ∈A, are pre-

pared in particular node(s), they can be used to compute new messages. As a

simplifying assumption, we ﬁrst consider message functions, ψ : A→M, of

only one argument—that is, messages that only depend on the sender node—as

is the case with, e.g., GCNs.

As message functions transition between two monoids, one might form a

cocycle condition on them as well, to support asynchrony invariance on how

incoming argument changes are processed. Speciﬁcally:

ψ(0) = 1 (9.162)

ψ(a + b) = ψ(a) ·ψ(b) (9.163)

Further Beyond 265

Note that, as ψ in GNNs is typically assumed to be a stateless function, we do

not need to explicitly track s in the above equations, and the cocycle amounts

to a monoid homomorphism: respecting the structure across A to M.

Further, to cover the case where, e.g., the receiver node’s argument also

participates in the message computation, we can extend the notion of a stateless

cocycle to arbitrary numbers of input arguments: ψ : A

×A

×···×A

→

M. In this case, asynchrony invariance is preserved so long as ψ is a monoid

multimorphism; keeping any k – 1 of its inputs ﬁxed, it must act as a monoid

homomorphism in the remaining one.

Asynchrony-invariant GNNs The monoid action and cocycle conditions

speciﬁed so far are fully abstractiﬁed. To work towards actually creating an

asynchrony-invariant GNN, we need to ground the monoids in concrete real

vector spaces, and carefully trace the conditions discovered.

As a guiding example, consider a graph with four nodes, V = {a, b, c, d} and

four edges, E = {(a, b), (b, b), (c, b), (b, d)}. In this simpliﬁed setting, consider

what happens around node b over several computational time-steps:

t = 0 t = 1 t = 2 t = 3 . . .

′

ψϕ

Note there are three (colour-coded) synchronisation points here:

•

Message aggregation: All incoming message vectors from N

must be

provided before invoking the aggregator,

;

•

State update: The messages must be aggregated (into a single vector, m

x∈N

) before calculating the next state using the update function, ϕ;

•

Message computation: Node b’s state must be updated before messages

sent to its outgoing neighbour, d, can be computed using the message function,

ψ, and sent across to it.

Asynchrony invariance is then gradually ascertained by placing constraints on

, ϕ and ψ such that it becomes possible to call these functions “ahead-of-

time” without modifying the end result. To be able to place these constraints,

266 Chapter 9

we need to translate them from the abstract monoid space studied before into

the space of actual GNN computations.

Message aggregation asynchrony Our message monoid is a real vector

space tied to the properties of the operator ⊕; speciﬁcally, we can denote it

by M= (R

, ⊕, 0

⊕

). Note that the identity element 0

⊕

is not always the same

as a zero-vector; this depends on choice of ⊕. For example, 0

⊕

= 0 when ⊕= +,

but 0

⊕

= 1 when ⊕= ⊙, the elementwise multiplication.

Note that, for all GNNs we studied so far, ⊕ was permutation invariant, and

M was a commutative monoid (Ong and Veli

ckovi

c 2022). That is, for any

two messages, m, n ∈M,

m ⊕n = n ⊕m (9.164)

With this condition already satisﬁed, message aggregation asynchrony invari-

ance is achieved: messages can be aggregated as they arrive, rather than waiting

on all of them being available. Accordingly, the following computational

diagram produces identical embeddings to the synchronised GNN:

t = 0 t = 1 t = 2 t = 3 t = 4 . . .

b{a,b}

′

While this is a useful start, it arguably doesn’t buy a lot of expressivity, as

the success of the node update function, ϕ, is still blocked on this process fully

completing. To truly unblock this constraint, we need to enable ϕ to be callable

iteratively, immediately as new messages arrive.

State update asynchrony We assume that our node states, x

, are simply

l-dimensional real vectors, i.e., S = R

in the abstract framework.

To allow ϕ to asynchronously inter-operate with ⊕, we need it to be a

monoid action on S, as described by Equations 9.160–9.161. This translates

Further Beyond 267

into the following intuitive constraints:

ϕ(x, 0

⊕

) = x (9.165)

ϕ(x, m ⊕n) = ϕ(ϕ(x, n), m) (9.166)

which, once satisﬁed, allows us to repeatedly invoke ϕ with every new mes-

sage, rather than waiting to aggregate them. This rule is not easy to solve for in

general; one arbitrary-but-intuitive choice is to set ϕ = ⊕, and then rely on the

commutativity and associativity of ⊕, which is guaranteed by its commutative

monoid structure.

Our equivalent asynchronous computational diagram becomes further

desynchronised as a result of this constraint being satisﬁed:

t = 0 t = 1 t = 2 t = 3 t = 4 . . .

′

b{a}

′

b{a,b}

′

ϕ ψ

Message computation asynchrony The remaining bottleneck to full asyn-

chrony invariance is obvious from the diagram above. Under the constraints

so far, the message function, ψ, can only be called once all messages from the

current layer have been fully processed. To ﬁnish our journey, we need to ﬁnd

a way to allow ψ to be called at arbitrary points during the computation. This

poses a particular headache for two reasons, which we address separately.

The ﬁrst issue is that, while our previous constraint guarantees the end prod-

uct of iteratively calling ϕ is correct, there is no guarantee that the intermediate

values (e.g. x

′

b{a,b}

in the diagram above) are relevant as inputs for ψ. To cross

that bridge, we need to reintroduce the argument monoid, A.

Recall that the main point of emitted arguments is that they will be actual

inputs for message computation. Further, assume that the argument monoid is

(A, ⊗, 1

⊗

) for some binary vector operation ⊗ with identity element 1

⊗

. To

guarantee S and A can asynchronously operate when transformed by M, we

ﬁrst translate our cocycle conditions from Equations 9.160–9.161 for a general

268 Chapter 9

argument emission function δ

(x):

⊕

(x) = 1

⊗

(9.167)

n⊕m

(x) = δ

(x) ⊗δ

(ϕ(x, m)) (9.168)

These constraints guarantee that, as new messages arrive, our iteratively-

assembled argument continues to match what comes out of equivalent syn-

chronous state updates. However, in the vast majority of GNN models, the

argument passed to ψ is exactly the updated node state, i.e., A= S and

(x) = ϕ(x, m). Notice a subtle algebraic tension here: our monoid action

law requires ϕ(x, 0

⊕

) = x, whereas setting δ

(x) = ϕ(x, m) in Equation 9.167

would force ϕ(x, 0

⊕

) = 1

⊗

for all x! Intuitively, an identity message leaves the

state unchanged, but a naive node that always emits its state would still re-emit

x rather than emitting “nothing” (1

⊗

Dudzik et al. (2024) resolve this tension by showing that if we make the

common assumption

that M= S = A and ⊕= ⊗, we can drop the strict iden-

tity requirement (moving from monoids to semigroups) provided we make ϕ

an idempotent function; that is, repeatedly updating with the same message

leaves the state fully unchanged:

ϕ(ϕ(x, m), m) = ϕ(x, m) (9.169)

Once we remove the bottleneck on ψ, nothing stops a node from repeatedly

sending the same (potentially partially ready) state multiple times, even when

no new information has arrived. Idempotence resolves this issue: it guarantees

that re-emitting existing states or receiving duplicate messages will not affect

the end result, rendering the strict identity law in Equation 9.167 unnecessary.

One typical way to ensure idempotence, particularly in algorithmic reasoning

research (Veli

ckovi

c et al. 2020; Tang et al. 2020), is to set ϕ = ⊕= max.

The second issue to resolve is that, even if we ensure that our

iteratively-emitted arguments for ψ are all valid, we need to also make

sure that iteratively-produced messages will match synchronously-produced

ones. Fortunately, this is the same as satisfying the monoid homomor-

phism/multimorphism condition for ψ from Equations 9.162–9.163. Trans-

lated to our current setting, the relevant conditions are as follows:

ψ(1

⊗

) = 0

⊕

(9.170)

ψ(a ⊗b) = ψ(a) ⊕ψ(b) (9.171)

If we plug everything we have built up so far into our solution (ϕ = ⊕= ⊗=

max), this amounts to a single condition on ψ (as Equation 9.170 is dropped

Further Beyond 269

alongside the identity elements

ψ(max(x

, x

)) = max(ψ(x

), ψ(x

)) (9.172)

and to satisfy this, one option is to make ψ a tropical linear layer: essentially,

a single matrix multiplication ψ(x) = trop(W, x), where “+” is replaced with

max, and “×” is replaced with +:

trop(W, x)

= max

1≤j≤k

+ x

(9.173)

While such operations can be neatly related to a higher-dimensional variant

of the path-ﬁnding primitives

in the Bellman-Ford algorithm (Bellman 1958),

they have recently been both discovered (Gaubert and Vlassopoulos 2024) and

deployed (Hashemi et al. 2025) within attentional models.

All taken together, and translated into the usual message-passing style of

Equation 5.56, our asynchrony-invariant GNN has the following form:

= max



, max

v∈N

trop(W, x

)



(9.174)

Once deployed, we can ﬁnally decouple message computation from the full

state assembly, and achieve the most relaxed computational structure that is

still equivalent to the synchronous GNN above:

t = 0 t = 1 t = 2 t = 3 t = 4 . . .

(1)

′

b{a}

(2)

′

b{a,b}

(3)

′

(4)

Note that here we depicted the computation of partial messages m

(t)

at the

same time-step as the computation of partial states x

′

b{... }

merely to make

the diagram cleaner to read. With all the relevant conditions satisﬁed, these

messages and state updates can be computed in any order, and even multiple

messages can be sent in succession from the same state.

Before concluding this analysis, it’s worth reiterating that the GNN pre-

sented in Equation 9.174 is only one possible solution pathway, and resulted

270 Chapter 9

from a sequence of arbitrary decisions! One particularly exciting direction that

was completely ignored here is a more bespoke treatment of the argument

generator, δ

(x), and we believe exploring other mechanisms for it is a key

direction for further work.

Aside: Deep scale spaces As mentioned before, monoids typically arise

when reasoning about computer science constructs, where they serve as a

very neat model of state machines. Further, cocycle conditions on them (a

more generic form of equivariance!) are naturally well-geared towards mod-

elling discrete computational phenomena. However, the ideas discussed in this

subsection can and do ﬁnd application in more perceptual, multimodal input

processing, when dealing with slightly more lossy operations is helpful.

A great example of this are deep scale spaces: image models incorpo-

rating “equivariance” to scaling and coarsening actions (“zooming in/out”)

from Worrall and Welling (2019). In a naïve continuous world, rescaling an

image by a factor s > 0 is a fully invertible operation, and in fact forms a

group. On a discrete pixel grid, however, coarsening an image requires low-

pass ﬁltering (blurring) and subsampling to avoid aliasing. Once ﬁne-grained,

high-frequency details are blurred away, they are irrevocably lost—there is no

inverse operation that can recover them.

Consequently, image coarsening is naturally modelled as a monoid of non-

negative log-scales, ([0, ∞), +, 0). Let S

denote the coarsening operator at

log-scale t ≥0 (conventionally implemented via Gaussian blurring of vari-

ance ∝e

– 1, followed by downsampling). Acting on an input image X, these

operators satisfy our monoid action axioms from Equations 9.160–9.161:

•X = X (9.175)

•X = S

•(S

•X) (9.176)

That is, coarsening at scale 0 leaves the image untouched, and coarsening by

followed by t

is identical to coarsening by t

+ t

in a single step, due to

operating in log-space. Note that we can only further add to the scale, as going

backwards after a coarsening operation is no longer possible.

Rather than requiring invertibility, Worrall and Welling (2019) design con-

volutional layers, F, that are “monoid equivariant”. Our message function, ψ,

being a homomorphism via Equation 9.163, allowed us to push monoid oper-

ations through the message function. The convolutional layers, F, must pass

that same test by commuting with S

F(S

•X) =

•F(X) (9.177)

Further Beyond 271

•

H(t, u)

F(H)(0, u)

F(H)(1, u)

F(H)(2, u)



•



•

F(H)(t, u)

W(0, v)

W(1, v)

−2

W(0, 2

−1

−2

W(1, 2

−1

−4

W(0, 2

−2

Figure 9.3

An illustration of a single dilated convolutional layer employed by deep scale spaces

(Worrall and Welling 2019). The input signal is represented as a “tower” of images

at progressively coarsening (log-)scales, and is convolved with a local 3 ×3 ﬁlter, W

(shown as coloured cells. Note that, as the ﬁlter is applied in higher levels of the tower,

exponentially more zeroes are inserted in-between its weights, following the pattern of

dilated convolutions (Kalchbrenner et al. 2016)). Latent embeddings at every output

scale are computed by appropriately combining ﬁltered information at their own scale

with scales that are above it. The ﬁgure shows only a few such connections, following

parts of Equation 9.178, to preserve clarity (e.g., the ﬁlter W(2, v) is omitted).

where

is a corresponding monoid action on the latent feature space. As

any coarsening operation would irrevocably alter the representations within,

deep scale space models store multi-scale latent states. Speciﬁcally, they lift

the 2D image into a 3D scale-space feature map H(t, u) (adding an axis for

(log-)scale, t ≥0, alongside 2D spatial coordinates, u ∈Z

) and process it via

scale-transformed convolutions:

F(H)(t, u) =

τ≥0

v∈Z

–2t

W(τ , 2

–t

v) H(t + τ , u + v) (9.178)

This essentially amounts to a form of ﬁlter matching that is very similar to

the group CNNs (T. Cohen and Welling 2016) we encountered in Chapter

7. The added caveats are that we restrict the scale summation to τ ≥0 (the

monoid structure only allows us to further coarsen, never go backwards) and

explicitly account for the scaling operation via 2

–t

. The base of 2 is chosen

because, ultimately, a discrete ﬁltering operation is desired across pixels—

and this choice allows for a practical implementation of Equation 9.178 with

272 Chapter 9

dilated convolutions (Kalchbrenner et al. 2016) with dilation factor 2

; see

Figure 9.3.

While, in principle, Equation 9.178 is well-deﬁned for all τ ∈[0, +∞),

practical implementations will only store these embeddings up to a certain

maximal scale, as eventually the coarsenings and dilation factors will become

uninformative for useful representation learning.

9.2.2 Groupoids: natural graph networks

Next up, we will discuss the groupoid—a “collection of group-like clus-

ters”. Groupoids can make an excellent abstraction for “typed equivariance”:

enforcing a symmetry constraint across a domain while allowing us to treat cer-

tain collections of input domains differently to each other. Speciﬁcally, input

domains of the same “type” will be processed by the same equivariant model.

As we previously discussed in this Chapter, a particularly neat opportunity

for doing so arises with graph neural networks and permutation equivariance,

as in principle, there is no requirement for two non-isomorphic graphs to be

processed with exactly the same parametric functions. It was de Haan, Cohen,

and Welling (2020) who ﬁrst discovered this opportunity

, and realised it

within the natural graph network (NGN) framework, which we discuss next.

The adjacency matrix groupoid To allow for the kind of typed equivariance

we need, we’re going to consider each graph as its binary adjacency matrix,

A ∈{0, 1}

n×n

, and we’ll induce equivariance constraints only via isomor-

phisms, f : {0, 1}

n×n

↔{0, 1}

n×n

, between them. For convenience of notation,

we will also use f

to denote that our isomorphism is induced by a node

permutation, σ ∈S

. Note that some of those isomorphisms may also be auto-

morphisms, if they do not actually modify the input matrix at all (f (A) = A);

see Figure 9.4. Given this structure, we can conclude that the adjacency matrix

groupoid must (locally) look something like this:

C E

A B D

–1

that is, all adjacency matrices are partitioned into “equivalence classes”, with

isomorphic matrices forming each partition, and no mappings are allowed

between the partitions. Since each transformation induces one equivariance

Further Beyond 273

0 1 1 1

1 0 0 1

1 0 0 0

1 1 0 0

⎡

⎢

⎣

⎤

⎥

⎦

0 1 0 1

1 0 1 1

0 1 0 0

1 1 0 0

⎡

⎢

⎣

⎤

⎥

⎦

⊤

(2, 1, 3, 4)

0 1 1 1

1 0 0 1

1 0 0 0

1 1 0 0

⎡

⎢

⎣

⎤

⎥

⎦

⊤

= A

(1, 4, 3, 2)

Figure 9.4

Considering a graph with four nodes and undirected edges {(1, 2), (1, 3), (1, 4), (2, 4)},

we visualise two of its isomorphisms in the adjacency matrix groupoid: above, one (f

)

that swaps nodes 1 and 2; below, one (f

) that swaps nodes 2 and 4. Note that the latter

is also an automorphism, as the adjacency matrix is unchanged by this swap.

constraint, this ensures that there are no blockers to decoupling parameters

across non-isomorphic inputs.

Additionally, it is interesting to consider which message functions must be

shared within one particular graph structure, under such constraints. To study

this, we will denote by ψ

u→v

the message function between the graph’s nodes

u ∈V and v ∈V. Clearly, whenever an isomorphism, f

: A ↔A

′

, is applied

to a graph with speciﬁed message functions, this “transports” the correspond-

ing message functions: ψ

(A)

u→v

= ψ

′

)

σ(u)→σ(v)

in the new graph. This, in and of

itself, does not constrain the choices of ψ in any way, unless f

is also an

automorphism. Because automorphisms do not alter the graph’s adjacency

representation, there is no way to tell whether the original or transported mes-

sage should be in each edge—hence, they must be identical! Formally, every

automorphism f

: A ↔A induces constraints of the form:

u→v

= ψ

τ(u)→τ(v)

(9.179)

forcing weight sharing between edges that fall within the same orbit of the

graph’s automorphisms

. Edges (u, v) ∈E and (w, z) ∈E are on the same orbit

274 Chapter 9

σ = (2, 1, 3, 4)

τ = (1, 4, 3, 2)

Figure 9.5

The graph whose adjacency matrix was studied in Figure 9.4 has exactly two auto-

morphisms, f

; those are the identity, and swapping 2 ↔4. Under the assumption that

u→v

= ψ

v→u

, these induce exactly one non-trivial constraint on the message functions

utilised by any GNN on this graph: ψ

1→4

= ψ

1→2

. This is due to the fact that edges (1, 4)

and (1, 2) are on the same orbit induced by f

. In comparison, any other isomorphism

merely transports these message functions to their new locations, without constraining.

if there exists an automorphism τ such that τ (u) = w and τ(v) = z. See also

Figure 9.5.

Local neighbourhood groupoids, and local NGNs The concept of (global)

natural graph networks (NGNs), built over the adjacency matrix groupoid,

is remarkably powerful. Indeed, it allows us to sharply move away from the

GNNs where every single message function must be the same, while still

being built under the foundational principles of equivariance. However this

proves to be a double-edged sword: when given so much choice, it becomes

computationally intractable to meaningfully take advantage of it in full (with

the exception of speciﬁc problems for which we know the graph structures of

interest well in advance). Furthermore, without signiﬁcant weight sharing, we

strongly risk any ability to generalise outside of the training data distribution.

To make practical NGNs possible, we hence need to move away from global

and into local constraints, as de Haan, Cohen, and Welling (2020) did.

The key insight for local NGNs is simple: rather than conditioning our

model’s message functions on the entire graph’s automorphisms, we focus on

small (e.g., k-hop) local neighbourhoods around that edge. Speciﬁcally, let G

denote the k-hop subgraph around u ∈V, and G

u→v

= G

∪G

be the subgraph

Further Beyond 275

1 2

1 3

1 2

1 3

(u)

(v)



0 1 0 0 0

1 0 1 1 1

0 1 0 1 0

0 1 1 0 1

0 1 0 1 0





0 1 0 0 0

1 0 1 1 1

0 1 0 1 0

0 1 1 0 1

0 1 0 1 0





0 0 1 0 0

0 0 1 1 0

1 1 0 1 1

0 1 1 0 1

0 0 1 1 0



u→v

w→z

ρ(f

(u)

)

X ↦→ P

A = P

⊤

′

(v)

)

ρ(f

(u)

)

X ↦→ P

A ↦→ P

⊤

′

(v)

)

Automorphism Isomorphism

Figure 9.6

A visualisation of the equivariance constraints in a local natural graph network (NGN).

Middle: A local subgraph G

u→v

around edge (2, 4) in a given graph. Right: An isomor-

phism f

: G

u→v

↔G

w→z

between two 1-hop edge neighbourhoods (with tagged sender

and receiver nodes) restricts to isomorphisms f

(u)

and f

(v)

on the respective node neigh-

bourhoods, requiring the message functions ψ

u→v

and ψ

w→z

to commute with their

representations. Left: When the transformation is an automorphism of a single edge

neighbourhood, f

: G

u→v

↔G

u→v

, this commutative diagram constrains the internal

structure of ψ

u→v

itself.

around the directed edge (u, v) ∈E. Please note: to avoid ambiguity, all the sub-

graphs explicitly label which node(s) correspond to the central node or edge.

Hence, G

will explicitly label which node is u, and G

u→v

will explicitly label

both u and v.

In full generality, working with local subgraphs upgrades what our mes-

sage functions, ψ

u→v

, actually operate on. Rather than merely mapping a

single vector x

at node u to a message vector for node v, a local NGN

allows each node to store the entire feature set of its corresponding subgraph,

. Under this lens, ψ

u→v

: R

|×k

×R

|×|G

→R

|×l

is a subgraph-to-

subgraph transformation that takes features and adjacency structure across the

sender node’s neighbourhood, G

, and maps them to updated features across

276 Chapter 9

the receiver node’s neighbourhood, G

; that is, ψ

u→v

, A

) = M

. Mul-

tiple such “message tensors” can then be combined or aggregated later on, just

as before.

We can now construct a local neighbourhood groupoid whose elements are

these local subgraphs. The transformations we track are then isomorphisms

between these subgraphs, with an important constraint on preserving their

tagged nodes. Hence, if, e.g., f

: G

u→v

→G

w→z

, then σ(u) = w and σ(v) = z.

Enforcing typed equivariance over this local groupoid yields a local NGN.

As illustrated in Figure 9.6, whenever there is an isomorphism between two

edge neighbourhoods, f

: G

u→v

↔G

w→z

, this yields node neighbourhood iso-

morphisms f

(u)

: G

↔G

and f

(v)

: G

↔G

, through restricting the subgraphs.

For the message functions, ψ

u→v

, to be equivariant to the edge subgraph

isomorphism, they should commute with these node subgraph isomorphisms:

w→z

◦ρ(f

(u)

) = ρ

′

(v)

) ◦ψ

u→v

(9.180)

where ρ and ρ

′

denote the usual permutation representations acting on the

sender and receiver subgraph features and adjacency matrix, respectively. Sim-

ilarly, for any automorphism of the edge neighbourhood, f

: G

u→v

→G

u→v

we can derive a corresponding constraint on ψ

u→v

◦ρ(f

(u)

) = ρ

′

(v)

) ◦ψ

u→v

(9.181)

This approach elegantly resolves the bottlenecks with global NGNs: small k-

hop subgraphs co-occur frequently

across different graphs—allowing mean-

ingful generalisation—and they can be made small enough to have tractable

computational complexity for ﬁnding the relevant isomorphisms.

Practical local NGNs: GCN

At ﬁrst glance, managing this collection of

constraints on all of the possible ψ

u→v

functions might feel overwhelming.

A closer, intuitive look at what the commutative diagrams in Figure 9.6 are

looking for reveals that the situation is far from being dire. What we actually

need is a learnable function computing ψ

u→v

, A

) = M

in a way that is

equivariant to any permutations in the graph speciﬁed by (X

, A

). But we

already know how to build functions like that—we can simply leverage graph

neural networks, executed locally on G

This insight led to the original practical instantiation of local NGNs from de

Haan, Cohen, and Welling (2020): the graph convolutional network squared

(GCN

). In GCN

, computing the message along an edge (u, v) amounts to

running an “inner GNN” (in this case, using the GCN from Kipf and Welling

(2016), but any other GNN would work here) over the subgraph G

, with u and

v explicitly marked with an additional input feature. The output embeddings

computed by the local GNN can then be processed further to obtain a message

Further Beyond 277

1 2

1 0

0 0 0 1



w ∈ {2, 3}

inner

1 2

2 4

1 0

0 0

0 1



w ∈ {2, 3, 5}

inner

1 2

1 0

0 0 0 1



w ∈ {2, 5}

inner

/∈ G



u ∈ {2, 3, 5}

Figure 9.7

Visualising the dataﬂow of the GCN

model (de Haan, Cohen, and Welling 2020) allow-

ing for one layer only in the inner GNN. Assuming we want to update the embeddings

of node v (node 4 in this case), we ﬁrst construct all possible local subgraphs of senders

to v. Executing a GNN over those subgraphs, we can compute the message m

as the

ﬁnal representation of node v in node u’s subgraph. These messages are then aggregated

and processed onwards within another GNN. Note that, in all of the local subgraphs G

we append two extra bits of information, indicating which nodes are u and v.

vector for the receiver node, v; the simplest option is simply to pull out the

embedding of the receiver from M

, or to average them all.

Once we have extracted the message vector, m

, we can then use it as any

usual message, within an “outer GNN”, to update node v’s embedding from x

to h

= ϕ







u∈N

GNN

inner



, A



| {z }

u→v

with extraction of m







(9.182)

For illustrative purposes, we expand this for a single-layer inner GNN using

the same aggregation function, and computing messages by just taking out the

target node from M

= ϕ





u∈N

inner





(u→v)

w∈N

∩G

inner



(u→v)











(9.183)

where

(u→v)

= [x

∥I

w=u

∥I

w=v

] is node w’s feature vector augmented with the

two aforementioned edge indicator features (see Figure 9.7).

278 Chapter 9

By nesting a local GNN inside the message function of a global GNN,

GCN

directly satisﬁes the requirements of the local groupoid equivariance

constraints, while sharing parameters across all isomorphic subgraphs

Much like the case of monoid constraints and asynchrony invariance, here

too the space of solutions is signiﬁcantly beyond what the GCN

equation

could explain. The decision to use an inner GNN is itself an arbitrary way

to solve for the constraints presented, and even that decision opens up a lot

of possible optimisations of inner/outer architecture interplay. We are hopeful

that this exposition serves as a useful roadmap for the interested reader, as

exploring the space of NGNs is deﬁnitely an exciting direction of research.

9.2.3 Categorical deep learning

Looking at our journey through this Chapter so far, a curious meta-pattern

emerges. Regardless of which abstraction we used to organise our collection of

transformations – be it groups, monoids or groupoids – the overarching recipe

remained largely the same:

1. Specify our domain

and our signal (including in latent spaces);

2. Specify a collection of transformations of interest we want our models to be

“robust” against (group, monoid, groupoid... );

3. Specify how the transformations act on the data domain / signals;

4. Enforce that any neural network layers, f , commute with these actions:

X X

Y Y

action in X

action in Y

(9.184)

Indeed, all of our “equivariance” constraints throughout the book eventually

had exactly this kind of shape (potentially rotated by 90

◦

): f ◦A

= A

◦f .

This invites a natural

concluding question for our current analysis: rather

than inventing a bespoke mathematical framework for every transformation

collection of interest, can we capture all symmetry principles – and beyond

– within a single, universal algebraic construct? As established by Gavra-

novi

c et al. (2024), the answer is a resounding yes! By stepping up one ﬁnal

level of abstraction into categorical deep learning (CDL), we can reveal that

every constraint we’ve studied so far is a special case of a single concept from

category theory: the monad algebra homomorphism.

We felt strongly that this construction warranted inclusion in this book, and

thought long and hard about how best to present it without simply restating

Further Beyond 279

the formalisms of Gavranovi

c et al. (2024). Because a fully rigorous bottom-

up treatment of CDL requires deﬁning a substantial amount of categorical

machinery—lying outside the core focus of this book—we have opted for a

more pedagogical path. Hence, what follows introduces only the minimal cat-

egory theory background needed to build an intuitive grasp of the relevant

concepts and how they map to GDL, in the hope of inspiring interested readers

to explore the foundational literature further.

Group actions as a monad We will build our intuition directly from the

familiar concept of monoid/group actions. Suppose we have a monoid (or

group) of transformations, M, and a feature space, X. Before we even spec-

ify how a transformation m ∈M modiﬁes a vector X ∈X, we can specify

intent to apply this transformation, through pairs (m, X) ∈M×X. This rep-

resents a pending action—an input, X, waiting to be acted upon by m. Within

category theory, this “contained computation” construct around a space, X, is

universally expressed using a monad structure, which we will denote by M(X).

We will not expand on the full formal deﬁnition of a monad in category

theory

, and instead opt for a functional programming-aligned exposition, to

keep matters more streamlined. Speciﬁcally, if we follow its signature in lan-

guages such as Haskell, a monad data structure over a space X should support

two kinds of functions, allowing us to manipulate these containers:

•

return : X →M(X)

•

>>= : M(X) →(X →M(Y)) →M(Y)

We can think of return as wrapping an identity computation around X, to cre-

ate a basic monad we can further edit, whereas >>= (also known as “bind”)

allows us to directly operate on the contents of a monad (via its second argu-

ment’s function) as well as potentially composing any wrapped computational

structure through f as well.

Let’s outline how we can represent group actions as monads, as was hinted

previously. We already mentioned that the underlying data structure, M = G ×

–, should be:

•

Monad elements are pairs of a group element and an input signal: (g, X) ∈

M(X), representing “intent to transform X with g”.

•

return X = (e, X); intending to act with the identity creates a “base monad”.

•

To calculate (g, X) >>= f , ﬁrst compute f (X) = (h, Y). Then we can com-

pose:

(

(g, X) >>= f

)

= (gh, Y). Note how both contents and intended group

actions are chained and composed through f .

– If we would like to only modify the input with some f : X →Y without

modifying the computational wrapper at all, we can simply combine the

280 Chapter 9

two operators: (g, X) >>= (return ◦f ) will yield (g, f (X)). We will denote this

operation by M(f ) : M(X) →M(Y); in Haskell it is referred to as fmap.

Representations as monad algebra Of course, merely pairing (g, X) ∈

M(X) only allows us to track intended group actions before they’re executed

– it lacks the machinery to perform these actions on our features. We need to

augment our monad with a structure map, a : M(X) →X, which executes its

contained computation. A pair, (X, a), consisting of a space and its structure

map, is then called a monad algebra (or M-algebra), if a respects the monad’s

internal structure:

X M(X) M(M(X)) M(X)

X M(X) X

return

M(a)

– >>= id

The ﬁrst axiom says that executing the computation of the monad returned by

return must yield the same object we started from (which is sensible, as we

assume return to create a monad with “identity” computation). The second

ensures that, when using a to execute computation in a doubly-nested monad,

M(M(X)), it does not matter whether we make two applications of a, or if we

ﬁrst collapse to a singly-wrapped monad using >>= id, and then execute it.

When M is the previously discussed group action monad M(X) = G ×X,

what is an appropriate structure map a : G ×X →X? It is simply a function

that takes a transformation g ∈G and an input X ∈X, and produces the trans-

formed output in X. When X is a vector space, this is precisely multiplication

by the group’s representation, ρ

. That is:

a(g, X) = ρ

(g)X (9.185)

Consequently, the (G ×–)-algebra laws reduce exactly to the two foundational

axioms of a group representation that we have employed throughout this book:

(e)X = X and ρ

(g)ρ

(h)X = ρ

(gh)X.

Armed with monads and their algebras, we can now start to describe neural

networks which might respect the structure in the algebra.

Equivariant layers are monad algebra homomorphisms Now we have a

categorical construction for describing a collection of transformations we care

about (monads) and for explaining how to execute such transformations over a

particular space (algebras).

What are neural networks within this framework? Well, they map between

two spaces: f : X →Y. If f is equivariant to some collection of transformations,

Further Beyond 281

we would expect it to respect the structure of those transformations, across

both of these spaces. Monads, being wrappers over spaces, neatly allow us

to deﬁne both M(X) and M(Y). Algebras, however, are deﬁned with respect

to a space. This means we need to specify separate M-algebras for the input,

(X, a), and the output (Y, b). (Note this is very similar to how we sometimes

needed to represent separate representations ρ

and ρ

for input and output

spaces when handling equivariance.)

Of course, f should not be just any mapping between these two spaces—to

qualify as equivariant, it should respect all of the structure contained within

the two algebras. Such a map is called a monad algebra homomorphism, and

it needs to make the following diagram commute:

M(X) M(Y)

X Y

M(f )

This is simply stating that, if we have a computation-wrapped input and we’d

like to apply f to it, we can either ﬁrst execute the computation on the input

while it’s in X space (using the structure map a : M(X) →X) and then call f on

the result, or we can ﬁrst call f on the input, and then execute the computation

on the output in the Y space (using the structure map b : M(Y) →Y).

What happens when M = G ×–, the group action monad? Our commutative

diagram becomes:

G ×X G ×Y

X Y

G×f

(–)(–)

which yields the constraint f ◦ρ

(g) = ρ

(g) ◦f : the key equivariance condi-

tion we’ve been using throughout the book! Hence, geometric deep learning is

precisely the study of group action monad algebra homomorphisms.

Beyond symmetry Why did we go through the effort of repackaging the

familiar equivariance condition into the language of monads, algebras and

homomorphisms? The answer is simple: the monad algebra homomorphism

diagram we discovered works for any monad algebra—not only the group

282 Chapter 9

action monad. For example, we can easily swap in monoid actions, or con-

sider monads over families of spaces indexed by groupoid objects, to recover

the other advanced examples studied within this Chapter.

We can also consider examples that go fully beyond what even monoids

or groupoids can express. By swapping out the monad M for other founda-

tional monads in computer science, the exact same homomorphism square

b ◦M(f ) = f ◦a directly transitions into algorithmic and stateful computation.

For instance, choosing M to be the list monad M(X) = X

∗

(which bundles

elements into arbitrary-length sequences) turns M-algebra homomorphisms

into sequence-processing folding operations that underpin recurrent neural

networks (RNNs) and state-space models (SSMs).

Viewed from this perspective, categorical deep learning does not replace the

geometric principles we have studied throughout this book; rather, it reveals

GDL to be the symmetry-pilled branch of a much broader algebraic theory of

neural architectures (Gavranovi

c et al. 2024). Whenever we constrain a neural

network to respect the structure of its domain—whether that structure is a 3D

rotation group, a collection of message-processing monoids, a local subgraph

groupoid, or a data structure in theoretical computer science—we are, at the

heart of it, building a monad algebra homomorphism.

9.3 Where Does the Group Come From?

Over the ﬁve years since we ﬁrst released our geometric deep learning proto-

book (Michael M. Bronstein et al. 2021) to the time of writing this ﬁnal

Chapter, we’ve delivered countless lectures, talks and tutorials on the subject.

On virtually every single one of those presentations, one question from the

audience presents itself time and time again: “What if we do not know G?”

This is an important, foundational question about the GDL setup. We side-

stepped it entirely for most of the book by simply assuming the domain, Ω,

the signal space, X, and their symmetry group, G, are known and ﬁxed. Yet,

in many important real-world domains, we may not have easy access to G, or

ﬁnding G might be one of the key scientiﬁc outcomes of the machine learn-

ing system in the ﬁrst place. (The same conclusion, of course, holds for more

general structures than groups that we discussed earlier in this Chapter.)

Rather than viewing this issue as a boundary past which no GDL research

can meaningfully take place, there’s been a vibrant – yet painfully small – fron-

tier of research that tackles this issue head-on from a wide variety of angles. We

opt to use the ﬁnal bit of real estate in our book to survey and credit their efforts,

in the hopes that they serve as a chart for GDL techniques to meaningfully

transition into these “messier” regimes of interest to deep learning.

Further Beyond 283

Direct symmetry discovery from data When the symmetry group acting

on our inputs is unknown, one direct approach to still leveraging it is to explic-

itly parametrise and learn either (a) the generating parameters of the group,

such as basis matrices for Lie algebras (Yang et al. 2023, LieGAN), or (b) data

augmentation distributions that leave the training distribution or task labels

invariant (Benton et al. 2020, Augerino). This allows for learning relevant

group parameters directly from training data, purely through gradient descent.

When we additionally have the beneﬁt of learning over dynamical sys-

tems, we can exploit a physics backdoor coming from one of the heroines

of Chapter 1: Noether’s theorem (Noether 1918). This theorem establishes

a fundamental correspondence between the continuous symmetries of a sys-

tem and its conserved quantities over time (such as energy or momentum).

Inspired by this duality, Alet et al. (2021)’s Noether networks bypass the need

to explicitly parametrise group elements, by instead meta-learning a conserva-

tion law. Given a sequential prediction model f

7→

t+1

, they meta-learn a

parametric embedding g

: X →R

alongside a “conservation loss”:

Noether

1:T

; g

) =

t=1

∥g

(

) – g

(

t–1

)∥

(9.186)

Here,

= x

is the initial ground-truth state, and g

is meta-learned using

second-order optimisation: its parameters, ϕ, are chosen such that adapting

the predictor’s weights, θ, to minimise L

Noether

at test time directly improves

the quality of the predicted rollout

1:T

. Remarkably, even on raw videos of

objects sliding down ramps or damped real-world pendulums where energy

is only imperfectly conserved, Noether networks recover meaningful physical

invariants directly from data.

Making sense of partially observed geometry The previous subsection

focussed on scenarios where symmetries are completely unknown at the onset.

What if, instead, we are fully aware of what G is, but we are fundamentally

unable to observe in full how it acts on our input signal? This is a remarkably

common scenario that arises in most image and video inputs. For example,

we observe a three-dimensional world, on which previously discussed spatial

groups such as SE(3) operate seamlessly. However, making sense of how those

actions map to any 2D projections (e.g. in pictures or video frames) is substan-

tially less trivial due to partial observability of the entire scene at any one point

in time. Earlier works tackled this by explicitly performing SO(3) or SE(3)

actions on latent bottlenecks behind an explicit renderer; see, e.g., Falorsi et

al. (2018) and Dupont et al. (2020). A growing body of recent work attempts

284 Chapter 9

to unearth this unseen geometry, or demonstrate under which conditions it

emerges, within general-purpose 2D neural network backbones.

Leveraging self-supervised learning and interesting inductive biases, Simion

et al. (2026) demonstrate that our understanding of 3D viewpoint symme-

tries can be meaningfully distilled into two-dimensional vision backbones. The

key innovation of their method, 3DPoV, is the emergence of internal symme-

try representations through differentiable sorting and self-supervision. First,

3DPoV extracts keypoints in the input by using an off-the-shelf point track-

ing system (Karaev et al. 2025); then, it explicitly tracks the similarity of the

embeddings of those keypoints against a selection of reference frames. Finally,

self-supervision is performed to enforce that the patch embeddings are equally

ranked by level of correspondence, in both a teacher and a student network.

Ranks are themselves extracted using differentiable sorting, to allow gradients

to propagate through the ranking. Preserving this relative ordering of interme-

diate patch similarities causes an approximately viewpoint-invariant geometric

structure to emerge within a neural network’s representation space: a structure

that is hard to uncover by forcing raw feature vectors to match across severe

changes in perspective.

Taking these ideas a step further, the SeeSE3 framework (C. Chen et

al. 2026) probes whether large, passively trained vision encoders (such as

DINOv2 (Oquab et al. 2024)) spontaneously organise their internal repre-

sentations to mirror the geometry of physical space. While raw differences

between these representations are not directly isomorphic to SE(3), applying a

lightweight Siamese “Poincaré adapter”

reveals that relative camera motions

in the Lie algebra se(3) can be linearly decoded from the displacements of

latent features within a neural network. In effect, even when the full extent of

a group’s action is hidden behind a camera lens, sufﬁciently rich multi-view

prediction forces the network’s latent manifold to rediscover SE(3) on its own.

Approximate equivariance Even when we do know the symmetry group, G,

of a domain, Ω, and its effects are fully observable in our signals, X, enforc-

ing its equivariance via strict architectural constraints is not always optimal.

Real-world data frequently exhibits symmetry breaking (in physics, e.g., due

to external ambient force ﬁelds, boundary conditions, or measurement noise),

and the efforts needed to set up an explicitly equivariant model, both in terms of

deriving it and its computational complexity, may not always be warranted—

see Petrache and Trivedi (2023)—especially often when taking into account

the use of (relatively) unrestricted, efﬁcient architectures, such as ones relying

on Transformer backbones.

Further Beyond 285

In Chapter 8 we had already discussed arguably the most striking example

of this: the transition of the AlphaFold model series from leveraging invari-

ant point attention, explicit reference frames and torsion angles (Jumper et

al. 2021), all the way to an approximately equivariant diffusion Transformer

operating directly on raw atomic coordinates (Abramson et al. 2024).

As a transition towards discussing these kinds of models, we start with a

brief overview of elegant ways to retain exact equivariance while decoupling it

from architectural priors (meaning that the model itself can be generic, as the

harness around it guarantees equivariance of the entire system). Such methods

work by either employing a carefully constructed average over a subset of

group-transformed inputs (Puny et al. 2022) or learning a way to represent each

input canonically, so group actions are not needed anymore (Kaba et al. 2023).

When the symmetry itself is only assumed to be approximate, however,

an increasingly popular alternative is to relax it. One might approach this

relaxation architecturally: e.g., by explicitly blending fully-equivariant and

unconstrained layers (Finzi, Benton, and Wilson 2021), or by introducing

insufﬁciently constrained ﬁlter banks in an otherwise equivariant backbone

(Wang, Walters, and Yu 2022). Another option which started gaining traction

recently is to approximate equivariance through the training objective only,

allowing for maximal speedup beneﬁts from generic accelerated foundation

model architectures. Following this route, Elhag et al. (2025) leave the archi-

tectural backbone entirely unconstrained, and instead frame equivariance as

an auxiliary loss term, penalising deviations from the equivariance constraint,

∥f (ρ

(g)X) – ρ

(g)f (X)∥

9.4 Closing Remarks

Seeing symmetries relaxed into loss terms, delegated to lightweight canonical-

isation harnesses, or discovered implicitly inside unconstrained Transformers

might tempt the reader to conclude that we never really needed any of this geo-

metric machinery in the ﬁrst place. We argue that the exact opposite is true.

One cannot meaningfully relax, canonicalise, probe for, or regularise towards a

structure without ﬁrst having a rigorous language to deﬁne what that structure

is. Even when a modern architecture chooses to trade strict equivariance for

raw hardware scaling – though even this principle may have caveats (Brehmer

et al. 2025) – geometric deep learning provides the framework to reason about

which inductive biases are being considered, how the data or learning objec-

tive should compensate for any removed ones, and how to measure whether

the model is meaningfully compensating for them within its representations.

And with that realisation, our journey through this book comes full cir-

cle. What began in Chapter 1 as a deep learning adaptation of Felix Klein’s

286 Chapter 9

1872 Erlangen Programme—unifying virtually all of the popular deep learn-

ing architectures in use today through the lens of group equivariance—has

expanded in this ﬁnal Chapter to encompass generalisations to monoids,

groupoids, categories, and even regimes where the symmetry itself may not

be immediately available from raw observations.

Whether a symmetry is hard-wired into a message-passing layer, wrapped

in a monad algebra, hidden in the weights of the neural network itself (as

mentioned in Chapter 3), or loosely injected into a model through data augmen-

tation, a conservation loss, or even fully self-emergent in foundation models,

the foundational lesson of geometric deep learning remains: to understand how

to learn in a complex world, we must ﬁrst understand the structure of what

stays the same.