Multilayer Perceptrons

Each layer $l$ contains a set of nodes $d^{(l)}$. The input to a node $i$ in layer $l$ is $s_i^{(l)}$ and its corresponding output is $x_i^{(l)}$. The network’s learnable parameters are its weights (and biases), collectively denoted as $W=w_{ij}^{(l)}$ , which represent the connection strength between node $j$ in layer $l-1$ and node $i$ in layer $l$.

The objective of training is to the find optimal set of weights $W^{\ast }$ that minimize the loss function over a given training set. $W^{\ast }=\arg {\min }_W\frac{1}{N}\Sigma err(x_i,y_i)$

We use Gradient Descent to update the weights. Backpropagation is used to compute the exact gradients required for the SGD update step.

Other details that are important to note:

1. Input Normalization. Before you train your model, you want your input data components to have zero mean and unit variance.
   1. $x_i(j)\leftarrow \frac{(x_i(j)-\mu (j))}{\sigma (j)}$
2. Data augmentation
3. Weight initialization: proper weight initialization is very important to avoid issues such as vanishing/exploding gradients.
   1. Initializing weights with small random numbers can work for shallow networks, but often leads to vanishing gradients in deeper architectures.
   2. Xavier Initialization nn.init
      1. Weights are drawn from a normal distribution with variance scaled by the number of inputs to the node.
      2. For a weight $w_{ij}^{(l)}$, the initialization is $w_{ij}^{((l))}\sim N(0,1/n)$ where $n$ is the # of edges incident on the node.
   3. Initialization with ReLU:
      1. $w_{ij}^{(l)}\sim N(0,2/n)$
4. Regularization with Dropout
   1. During the forward pass of training, randomly set the output of any given node to zero with a specified probability $p$.
   2. Reduces overfitting