NeRF

A Neural Radiance Field (NeRF) represents a 3D scene as a continuous function stored in the weights of a small MLP, and renders novel views by shooting rays through that function and integrating color along them. Given a handful of posed photos, it lets you synthesize any new camera view.

Neural radiance fields for view synthesis

• Input pictures ⇒ synthesize any view
  • gets you depth maps Neural Net input:
• 3D coordinate vector $\text{x}=(x,y,z)$
• $d=(\theta ,\varphi )$ viewing direction

Output:

• color $(r,g,b)$
• volume density $\sigma$ (describes how opaque the point is)

The MLP radiance field

The scene is a function $F_{\Theta }:(\mathbf{x},\mathbf{d})\mapsto (\mathbf{c},\sigma )$. Density $\sigma$ depends only on position (a point is equally solid from any angle), while color $\mathbf{c}$ depends on both position and viewing direction, which is what lets NeRF capture view-dependent effects like specular highlights. In practice the MLP first predicts $\sigma$ and a feature vector from $\mathbf{x}$, then concatenates the direction $\mathbf{d}$ to predict color.

Positional encoding of coordinates

An MLP fed raw $(x,y,z)$ produces blurry results because networks are biased toward low-frequency functions. NeRF lifts each coordinate into a high-frequency Fourier basis first:

$\gamma (p)=(\sin (2^0\pi p),\cos (2^0\pi p),\dots ,\sin (2^{L-1}\pi p),\cos (2^{L-1}\pi p))$

This encoding is what allows the MLP to represent sharp geometry and fine texture. It is a close cousin of the Positional Encoding used in transformers.

Ray sampling

For each pixel, cast a ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ from the camera origin $\mathbf{o}$ through that pixel. Sample points $t_i$ along the ray between near and far bounds. NeRF uses hierarchical sampling: a coarse network with stratified samples finds where mass is, then a fine network places more samples near high-density regions so compute is not wasted on empty space.

The rendering integral

Expected color along a ray is the volume-rendering integral, accumulating emitted color weighted by how much light reaches that point and how opaque it is:

$C(\mathbf{r})={\int }_{t_n}^{t_f}T(t)\sigma (t)\mathbf{c}(t)dt,T(t)=\exp \left(-{\int }_{t_n}^t\sigma (s)ds\right)$

Here $T(t)$ is transmittance, the probability the ray travels to $t$ unobstructed. The discrete form used in practice, with ${\delta }_i=t_{i+1}-t_i$:

$C(\mathbf{r})\approx {\sum }_i{T}_i(1-e^{-{\sigma }_i{\delta }_i}){\mathbf{c}}_i,T_i=\exp (-{\sum }_{j<i}{\sigma }_j{\delta }_j)$

Because this whole pipeline is differentiable, the model is trained by simple photometric loss: render the training rays and minimize squared error against the real pixel colors.

$\mathcal{L}={\sum }_{\mathbf{r}}\Vert \hat{C}(\mathbf{r})-C_{\text{gt}}(\mathbf{r}){\Vert }_2^2$

Relation to Gaussian Splatting

NeRF is accurate but slow: rendering requires many MLP evaluations per ray. Gaussian Splatting represents the scene as an explicit set of 3D Gaussians rasterized directly to the screen, achieving real-time rendering while keeping differentiability, and has largely displaced NeRF where speed matters.

Common pitfalls

• Requires accurate camera poses (usually from structure-from-motion); pose error smears the reconstruction.
• A vanilla NeRF overfits one static scene and does not generalize to new scenes.
• Training and rendering are compute-heavy; without positional encoding results are blurry.