Activation Derivatives

Let $\sigma (x)=1/(1+e^{-x})$.

Activation	$f(x)$	$f^{\prime }(x)$	Note
Sigmoid	$\sigma (x)$	$\sigma (x)(1-\sigma (x))$	max slope $0.25$, saturates both ends
Tanh	$\tanh x$	$1-{\tanh }^{2}x$	max slope $1$, zero-centered
ReLU	$\max (0,x)$	$1[x>0]$	undefined at $0$, take $0$
Leaky ReLU	$\max (\alpha x,x)$	$1$ if $x>0$ else $\alpha$	no dead units
ELU	$x$ or $\alpha (e^x-1)$	$1$ if $x>0$ else $f(x)+\alpha$	reuse the forward value
GELU	$x\Phi (x)$	$\Phi (x)+x\varphi (x)$	$\varphi$ is the normal pdf
SiLU / Swish	$x\sigma (x)$	$\sigma (x)(1+x(1-\sigma (x)))$	non-monotone near $0$
Softplus	$\log (1+e^x)$	$\sigma (x)$	smooth ReLU

Two useful identities

Both let you compute the derivative from the forward output, so you store one tensor instead of two:

${\sigma }^{\prime }=\sigma (1-\sigma ),{\tanh }^{\prime }=1-{\tanh }^2$

From scratch

import torch
 
def check(f, df, x):
    """Compare an analytic derivative against autograd."""
    x = x.clone().requires_grad_(True)
    f(x).sum().backward()
    return (x.grad - df(x.detach())).abs().max()
 
sigmoid   = lambda x: torch.sigmoid(x)
d_sigmoid = lambda x: torch.sigmoid(x) * (1 - torch.sigmoid(x))
 
silu      = lambda x: x * torch.sigmoid(x)
d_silu    = lambda x: (s := torch.sigmoid(x)) * (1 + x * (1 - s))
 
gelu      = lambda x: x * torch.distributions.Normal(0., 1.).cdf(x)
d_gelu    = lambda x: (torch.distributions.Normal(0., 1.).cdf(x)
                       + x * torch.exp(-x**2 / 2) / (2 * torch.pi)**0.5)
 
x = torch.randn(1000, dtype=torch.double)
assert all(check(f, d, x) < 1e-9 for f, d in
           [(sigmoid, d_sigmoid), (silu, d_silu), (gelu, d_gelu)])