Gating Mechanisms

Gating mechanisms control information flow by selective filtering. A gate typically passes data through a Sigmoid function and multiplies by the input: $output=\sigma (Wx+b)\odot x$. If the sigmoid outputs 0, the gate is closed, if it’s 1, it is fully open.

Gates as Learned Interpolation

The most useful view of a gate is as a learned convex combination. Given two candidate states $a$ and $b$ and a gate $g=\sigma (\cdot )\in (0,1)$:

$$\text{out}=g\odot a+(1-g)\odot b$$

The network learns, per element and per input, how much to keep from each source. Because $g$ is elementwise, different feature channels can be gated independently. This single idea underlies recurrent memory, highway networks, and modern transformer feed forward blocks.

import torch
import torch.nn as nn
 
class Gate(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.to_gate = nn.Linear(dim, dim)
 
    def forward(self, a, b):
        g = torch.sigmoid(self.to_gate(a))
        return g * a + (1 - g) * b        # learned interpolation

Gates in Recurrent Networks

Gating was introduced to fix vanishing gradients in recurrent nets by creating a near identity path through time.

• The LSTM uses three sigmoid gates (forget, input, output) around a protected cell state.
• The GRU uses two (update, reset), and its update rule $h_t=(1-z_t)\odot h_{t-1}+z_t\odot {\tilde{h}}_t$ is precisely the interpolation form above.

When a gate stays near the “keep” value, the state is copied forward almost unchanged, so gradients do not decay across many steps.

Gated Linear Units in Transformers

Modern transformer MLPs replace the plain feed forward with a Gated Linear Unit (GLU), where one linear projection gates another:

$$\text{GLU}(x)=(x{W}_1)\odot \sigma (x{W}_2)$$

SwiGLU swaps the sigmoid for a Swish (SiLU) activation and is used in LLaMA and PaLM:

$$\text{SwiGLU}(x)=(x{W}_1)\odot \text{SiLU}(x{W}_3),\text{SiLU}(z)=z\sigma (z)$$

import torch.nn.functional as F
 
class SwiGLU(nn.Module):
    def __init__(self, dim, hidden):
        super().__init__()
        self.w1 = nn.Linear(dim, hidden, bias=False)   # value branch
        self.w3 = nn.Linear(dim, hidden, bias=False)   # gate branch
        self.w2 = nn.Linear(hidden, dim, bias=False)   # project back
 
    def forward(self, x):
        return self.w2(self.w1(x) * F.silu(self.w3(x)))

Parameter budget

A GLU block splits the hidden layer into two projections, so to keep parameters roughly constant the hidden dimension is scaled by $\frac{2}{3}$ (for example $\frac{8}{3}d$ instead of $4d$) relative to a standard MLP.

FiLM: Feature-wise Linear Modulation

FiLM: Feature-wise Linear Modulation. A conditioning method that applies a learned affine transformation to the feature maps of a neural network based on some external input.

$FiLM(x,z)=\gamma (z)\odot x+\beta (z)$

It allows one input (z) to conditional lscale and shift another input (x), which can be useful for tasks like visual question answering, where the question (z) can modulate the image features (x) to focus on relevant information.

Sigmoid saturation

Sigmoid gates saturate: when the pre-activation is large in magnitude the gradient ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$ vanishes and the gate stops learning. Initializing gate biases so gates start mostly open (recurrent nets often set the forget gate bias positive) keeps early training healthy.