Adam

Adam (Adaptive Moment Estimation) is a popular optimization algorithm for training neural networks. It combines the advantages of two other popular optimization algorithms: AdaGrad and RMSProp.

Adam computes adaptive learning rates for each parameter by keeping track of the first and second moments of the gradients. The algorithm updates the parameters using the following equations: $m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$ $v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$ ${\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t}$ ${\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}$ ${\theta }_t={\theta }_{t-1}-\alpha \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$ Where:

• $m_t$ is the first moment (mean) of the gradients.
• $v_t$ is the second moment (uncentered variance) of the gradients.
• ${\beta }_1$ and ${\beta }_2$ are hyperparameters that control the decay rates of the first and second moments, respectively.
• $\alpha$ is the learning rate.
• $ϵ$ is a small constant to prevent division by zero.
• ${\theta }_t$ are the parameters being optimized.
• $g_t$ is the gradient of the loss function with respect to the parameters at time step $t$.
• ${\hat{m}}_t$ and ${\hat{v}}_t$ are bias-corrected estimates of the first and second moments, respectively.

import numpy as np
class Adam:
    def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
        self.learning_rate = learning_rate
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.m = None
        self.v = None
        self.t = 0
 
    def update(self, params, grads):
        if self.m is None:
            self.m = np.zeros_like(params)
            self.v = np.zeros_like(params)
 
        self.t += 1
        self.m = self.beta1 * self.m + (1 - self.beta1) * grads
        self.v = self.beta2 * self.v + (1 - self.beta2) * (grads ** 2)
 
        m_hat = self.m / (1 - self.beta1 ** self.t)
        v_hat = self.v / (1 - self.beta2 ** self.t)
 
        params -= self.learning_rate * m_hat / (np.sqrt(v_hat) + self.epsilon)