Autoencoders

Autoencoders are a type of neural network architecture used for unsupervised learning, primarily for dimensionality reduction and feature learning. They consist of two main components: an encoder and a decoder. The encoder compresses the input data into a lower-dimensional representation (called the latent space), while the decoder reconstructs the original data from this compressed representation.

The model is trained purely to minimize the reconstruction error (typically MSE) between the input and the output.

Variations and Applications:

• Sparse Autoencoders: Introduce a sparsity constraint on the latent space, encouraging the model to learn more meaningful and interpretable features.
• Denoising Autoencoders: Train the model to reconstruct the original input from a corrupted version, which helps the model learn robust features that are less sensitive to noise.
• Anomaly Detection: Autoencoders can be used to identify anomalies by training on normal data and then flagging inputs that have high reconstruction error as anomalies.

Standard AEs are good for compression, but terrible for generation. This is because the latent space can be irregular and not smooth, meaning that sampling from it may not yield meaningful outputs. Variational Autoencoders (VAEs) address this issue by imposing a probabilistic structure on the latent space, allowing for better generalization and sample generation.

import torch
import torch.nn as nn
 
class Autoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim),
            nn.Sigmoid()  # Assuming input is normalized between 0 and 1
        )
 
    def forward(self, x):
        latent = self.encoder(x)
        reconstructed = self.decoder(latent)
        return reconstructed