Deep Learning Symmetries and Sets

Symmetries: Invariance vs Equivariance

• Invariance: the output remains exactly the same even if the input is transformed
  • Example: if you have an image of a cat and shift the cat to the right, the label should still simply be “cat”.
• Equivariance: the output changes in the exact same way that the input was transformed:
  • Example: if you are detecting the position of the cat, shifting the cat to the right in the image should shift the predicted coordinates to the right by the same amount.

Geometric Deep Learning Blueprint

1. Linear Equivariant Layer: applies a learnable transformation that respects the data’s symmetry
2. Nonlinearity
3. Local pooling: coarsens the tensor to aggregate local information
4. Global pooling (invariant): reduces the entire tensor into a final invariant prediction vector

Learning on sets and permutation invariance

• Because sets are unordered, a NN processing a set must possess permutation invariance.

Deep Sets architecture

• Linear equivariant layer
• Global Pooling
• Final MLP

def mlp_inference(input, weights, biases):
	x = input
	for W, b in zip(weights, biases):
		x = np.dot(x, W) + b
		x = np.maximum(0, x) # ReLU activation
	return x
def deep_sets_inference(input_set, mlp_weights, mlp_biases, pred_weight, pred_bias, pooling_fn):
	embeddings = mlp_inference(input_set, mlp_weights, mlp_biases) # equivariant layer
	pooled = pooling_fn(embeddings) # invariant pooling
	return mlp_inference(pooled, pred_weight, pred_bias) # final prediction

Pooling is the core operation for sets: common pooling functions include sum, mean, max, min, variance, and count. Principal aggregation functions are when you combine multiple pooling functions together, which can capture more complex interactions between elements in the set. For example, you could concatenate the mean and max pooling outputs to create a richer representation of the set, allowing the model to capture both average trends and extreme values within the data.