Dropout

Dropout is a regularization technique used to prevent overfitting in neural networks.

• During training, dropout randomly sets a fraction of the input units to zero at each update, which helps to break up co-adaptations between neurons and encourages the network to learn more robust features.
• The dropout rate is a hyperparameter that determines the fraction of input units to drop, typically set between 0.2 and 0.5.
• During inference, dropout is turned off, and the weights are scaled by the dropout rate to account for the missing units during training.
• This technique has been shown to improve the generalization performance of neural networks, especially in cases where the model is prone to overfitting due to a large number of parameters or limited training data.

$\text{Dropout}(x)=\frac{x\cdot \text{Bernoulli}(p)}{p}$

Where:

• $x$ is the input to the dropout layer.
• $\text{Bernoulli}(p)$ is a random variable that takes the value 1 with probability $p$ (the keep probability) and 0 with probability $1-p$ (the dropout rate).
• The division by $p$ is used to scale the activations during training, ensuring that the expected value of the activations remains the same during inference when dropout is turned off.

import numpy as np
def dropout(x, p=0.5):
    mask = np.random.binomial(1, p, size=x.shape)  # Create a dropout mask
    return (x * mask) / p  # Apply the mask and scale the activations