NN Engineering

Data

• Data splitting should be partitioned into Train, Validation, and Test sets
• The training set is used to optimize weights, the validation set is used to make modeling decisions (hyperparameters), and the test set is kept strictly untouched until final evaluation to prevent overly optimistic estimates.
• Cross-Validation: for smaller datasets, you can evaluate the model across multiple different folds of the data.
• Stratified splitting: when dealing with imbalanced labels, stratification ensures that each data splits maintains the exact same class proportions as the original dataset.
• Adversarial Splitting: To test how robust a model truly is (its ability to extrapolate), you should split data by leaving out entire clusters, specific labels, or time periods (e.g., training on past data, testing on future data). This evaluates the model under challenging, out-of-distribution scenarios
• Data augmentation: you can artificially increase your dataset size and bake inductive biases into the model by applying relevant transformations: for text this could be using synonyms; for images, this could be rotations, flips, cropping, color shifts.

Evaluation metrics and calibration

• Regression: evaluated using MAE/MSE, R^2, Pearson’s r.
• Ranking: evaluated using Spearman’s rank correlation, Kendall Tau (pair-based ordering)
• Classification: evaluated using hard metrics (accuracy, f1-score), AUROC (ranking-based metric where 1.0 is a perfect classifier and 0.5 is random guessing)

Optimization & Training Strategies

• Adam
• Learning Rate Decay
• Gradient Clipping
• Early Stopping

Handling Imbalanced Data

• Weighted Loss: You can multiply the loss calculation by weights inversely proportional to class frequency, forcing the model to care more about the rare minority classes.
  • $\text{Weighted Loss}=-{\sum }_{i=1}^N{w}_i{y}_i\log (p_i)$
• Focal Loss:
  • $\text{FL}(p_t)=-{\alpha }_t(1-p_t{)}^{\gamma }\log (p_t)$
  • This loss function down-weights easy examples and focuses training on hard negatives, which is especially useful in cases of extreme class imbalance. The hyperparameter $\gamma$ controls the strength of this effect, while ${\alpha }_t$ can be used to balance the importance of positive vs negative examples. If $\gamma =0$, focal loss reduces to standard cross-entropy loss. As $\gamma$ increases, the loss focuses more on hard examples and less on easy ones.
• Resampling: You can either oversample the minority class (e.g., using SMOTE) or undersample the majority class to create a more balanced training set. However, be cautious of overfitting when oversampling, and losing valuable information when undersampling.

Regularization and Architecture Tricks

• BatchNorm
• LayerNorm
• Skip Connections
• Dropout
• Regularization