Neural 3D Rendering: NeRF, Gaussian Splatting, and the Tools That Power Them

What Is Neural 3D Rendering?

Traditional 3D pipelines rely on explicit geometry — meshes, textures, and rasterization engines. Neural 3D rendering takes a different approach: a neural network learns to represent a scene implicitly, either as a continuous function or as a set of learnable primitives, and then renders novel views from that representation. The result is photorealistic novel-view synthesis from a sparse set of photographs, with no manual 3D modeling required.

Two techniques dominate the field in 2026: Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). They solve the same problem — reconstruct a 3D scene from 2D photos and render it from new angles — but use fundamentally different scene representations and rendering pipelines. Understanding both requires a common starting point: volume rendering.

---

Foundation: Volume Rendering

Volume rendering models a scene as a participating medium where light can be absorbed, emitted, or scattered at every point along a ray. For neural rendering, the relevant equation is the volume rendering integral:

For a ray r(t) = o + td from camera origin o in direction d, the expected color is:

C(r) = ∫[t_near to t_far] T(t) · σ(r(t)) · c(r(t), d) dt

where:
  T(t) = exp(-∫[t_near to t] σ(r(s)) ds)   — transmittance (how much light survives)
  σ(r(t))                                   — volume density at point r(t)
  c(r(t), d)                                — emitted color at point r(t) viewed from direction d

The transmittance T(t) decreases as rays pass through denser regions. High density points contribute more to the final color; the integral accumulates color weighted by both density and remaining transmittance. In practice this integral is approximated by stratified sampling: the ray is divided into N intervals, a sample point is chosen in each, and the integral is replaced by a weighted sum.

---

NeRF: Neural Radiance Fields

Scene Representation

NeRF, introduced by Mildenhall et al. in 2020, represents a scene as a Multi-Layer Perceptron (MLP) that maps a 5D input — a 3D spatial coordinate (x, y, z) and a 2D viewing direction (θ, φ) — to a 4D output: RGB color and volume density σ.

F_θ : (x, y, z, θ, φ) → (r, g, b, σ)

Density σ is predicted from position only (view-independent), while color is predicted from both position and direction (view-dependent). This split allows the network to represent view-dependent effects like specular highlights and reflections.

Positional Encoding

Neural networks struggle to learn high-frequency functions from low-dimensional inputs. NeRF encodes inputs with sinusoidal positional encoding before passing them to the MLP:

γ(p) = (sin(2⁰πp), cos(2⁰πp), sin(2¹πp), cos(2¹πp), ..., sin(2^(L-1)πp), cos(2^(L-1)πp))

With L=10 for position and L=4 for direction, the input dimension expands from 5 to 60. This lets the network capture fine geometric and texture detail that a raw MLP would miss.

Hierarchical Sampling

Uniform sampling along a ray wastes compute on empty space. NeRF uses a coarse-to-fine strategy: a coarse network samples 64 points uniformly and produces a weight distribution along the ray; a fine network then importance-samples 128 additional points where the coarse network predicted high density. Final color comes from the combined 192 sample points fed through the fine MLP.

Training

Training requires a set of images with known camera poses (typically extracted via Structure from Motion). For each training step, rays are cast through known image pixels, colors are rendered via the volume integral, and the network is optimized with a simple photometric loss:

L = Σ_r ‖C_rendered(r) − C_true(r)‖²

Original NeRF trains for 1–2 days on a single GPU. The MLP learns to allocate density and color across space purely from RGB supervision — no depth maps, masks, or 3D annotations needed.

NeRF Variants Worth Knowing

• Instant-NGP (2022): Replaces the large MLP with a multi-resolution hash grid — a compact spatial data structure that stores learned features. Reduces training time from days to seconds. Still one of the fastest NeRF implementations.

• Mip-NeRF 360 (2022): Addresses aliasing by casting conical frustums instead of infinitesimally thin rays and encoding integrated positional encoding over each frustum. Handles unbounded outdoor scenes gracefully.

• Zip-NeRF (2023): Combines Instant-NGP's hash grid with Mip-NeRF 360's anti-aliased formulation — currently one of the highest-quality unbounded NeRF models.

• NeRFStudio: A modular training framework that wraps many NeRF variants behind a common API, making it easy to swap scene representations and rendering strategies.

NeRF Limitations

• Rendering requires querying the MLP for hundreds of samples per ray — slow at inference time.

• Scenes are baked into network weights; editing geometry is non-trivial.

• Requires accurate camera poses; works best with controlled capture setups.

---

3D Gaussian Splatting

Scene Representation

3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) replaces the implicit MLP with an explicit set of 3D Gaussians. Each Gaussian primitive stores:

• Position (x, y, z) — center of the Gaussian

• Covariance matrix Σ — encodes shape (anisotropic ellipsoid), decomposed as Σ = RSSTRT where R is rotation and S is a diagonal scale matrix

• Opacity α — transparency

• Spherical harmonic coefficients — encode view-dependent color (typically degree 3, giving 48 coefficients per RGB channel)

A scene is represented by N such Gaussians (often 1–6 million for complex scenes). Unlike NeRF weights, these are directly interpretable geometric primitives.

Rasterization-Based Rendering (Splatting)

3DGS does not use ray marching. Instead it projects 3D Gaussians onto the 2D image plane — a process called splatting — and composites them front-to-back using alpha blending:

1. Project each 3D Gaussian to a 2D Gaussian on the image plane
2. Sort Gaussians by depth (back-to-front)
3. For each pixel, accumulate color using alpha compositing:

   C = Σ_i c_i · α_i · Π_{j<i} (1 - α_j)

The 2D projection of a 3D Gaussian is itself a 2D Gaussian — making the splatting step analytically computable. Each tile of the image accumulates contributions only from the Gaussians whose projected footprint overlaps that tile, enabling highly parallelized GPU rasterization.

Training Pipeline

Training starts from a sparse point cloud (from SfM, same as NeRF) and then optimizes all Gaussian parameters end-to-end:

1. Initialize: Place one Gaussian at each SfM point, with small spherical covariance and random SH coefficients.

2. Render: Rasterize the current Gaussians to produce a training image.

3. Loss: Photometric loss (L1 + SSIM) between rendered and ground-truth images.

4. Backpropagate: Gradients flow through the differentiable rasterizer back to all Gaussian parameters.

5. Adaptive density control: Every N iterations, split large Gaussians covering large view-space footprints, clone small Gaussians in under-reconstructed regions, and prune transparent or tiny Gaussians.

Training converges in 30–60 minutes on a modern GPU. Rendering is real-time (30–120+ fps on consumer GPUs) because it is GPU rasterization, not ray marching.

3DGS Variants

• Mip Splatting (2023): Fixes aliasing artifacts in 3DGS by bounding the 3D Gaussian footprint to the pixel resolution — objects no longer appear as needles when seen at oblique angles or far away.

• 4D Gaussian Splatting: Extends primitives to 4D to represent dynamic scenes (moving objects, people). Each Gaussian has a temporal dimension controlling when it is active.

• GaussianAvatars: Ties Gaussians to the surface of a deformable head model (FLAME) — enables photorealistic head avatar animation.

• LangSplat: Distills CLIP/SAG language features into the Gaussian primitives, enabling open-vocabulary 3D segmentation and part selection.

3DGS vs NeRF at a Glance

PropertyNeRF (MLP)3D Gaussian SplattingScene representationImplicit (MLP weights)Explicit (Gaussian primitives)Rendering methodRay marching (volumetric)Rasterization (splatting)Training timeHours to days (seconds with Instant-NGP)30–60 minutesRendering speedSlow (seconds/frame) to real-time with bakingReal-time (30–120+ fps)MemoryCompact (MLP weights)Large (per-Gaussian attributes, 100–500 MB)EditabilityHard — weights are opaqueEasier — primitives can be moved, removed, insertedView-dependent effectsFull (learned per-ray)Good (spherical harmonics)Dynamic scenesVariants exist (D-NeRF)4D Gaussian Splatting

---

Commercial Services

Luma AI (lumalabs.ai)

Luma's mobile app and web platform let users capture a scene by walking around it with a smartphone. The platform runs NeRF or 3DGS reconstruction in the cloud and returns an interactive 3D scene that can be embedded or exported. Luma also provides a Dream Machine video model that uses its 3D understanding. Used widely by product photographers, real estate, and game developers for asset creation. Free tier available; Pro at $29.99/month.

Polycam

Polycam offers photogrammetry, LiDAR-assisted scanning, and NeRF-based capture on iOS devices. It targets architecture, design, and e-commerce use cases. Exports to GLB, OBJ, USD, and NeRF formats. Direct integrations with Revit, Blender, and Unreal. Business plan at $79/month with unlimited scans and team access.

Postshot

A desktop application (Mac/Windows) for local 3D Gaussian Splatting. Users import their own photos or video, Postshot runs reconstruction locally on the GPU, and the result is a real-time splat viewer. Targets prosumer photographers and indie game developers who need fast turn-around without uploading data. One-time purchase ~$99.

NVIDIA Instant NeRF / Omniverse

NVIDIA released Instant-NGP as an open reference, but its production path is through Omniverse. The RTX AI toolkit includes tools for converting NeRF captures into USD assets compatible with Omniverse's physics simulation and rendering pipeline. Targeted at enterprise digital twin and industrial metaverse workflows.

Adobe Substance 3D / Project Stardust

Adobe has integrated generative 3D capabilities into Substance 3D Sampler, allowing material extraction from photos. Project Stardust (Photoshop) uses scene understanding to enable object-aware inpainting, and Adobe has previewed NeRF-based depth estimation inside Firefly 3D. Tight integration with the Creative Cloud subscription.

RealityCapture (by Epic Games)

Not NeRF-based — uses classical photogrammetry — but the industry-standard comparison point for any new neural rendering pipeline. Epic acquired it and offers it free for revenue under $1M/year. Still faster than most NeRF pipelines for large-scale scenes and well-established in film VFX and game asset pipelines.

---

Open-Source Tools

Nerfstudio

The most actively maintained NeRF framework as of 2026. Modular architecture: swap scene representations (NeRF, Instant-NGP, Zip-NeRF, Mip-NeRF 360, 3DGS) without changing the training loop. Ships with a browser-based viewer for live training visualization. Supports COLMAP for camera pose estimation, multiple export formats, and a plugin API for custom models.

pip install nerfstudio
ns-process-data images --data ./photos/ --output-dir ./data/
ns-train nerfacto --data ./data/

3D Gaussian Splatting (original repo)

The reference implementation from Inria and the University of Tübingen. Requires COLMAP for initial point cloud and camera poses. Rasterization is implemented as a custom CUDA kernel — requires an NVIDIA GPU. The repo includes the full training loop, evaluation tools, and a viewer.

git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
python train.py -s <path to COLMAP dataset>
python render.py -m <trained model path>

Gaussian Opacity Fields (GOF)

An extension of 3DGS that enables accurate surface reconstruction by learning opacity fields on Gaussian surfaces. Fills the main gap in vanilla 3DGS: while it renders well, it does not produce clean meshes. GOF exports watertight meshes compatible with standard 3D pipelines.

gsplat (by Nerfstudio team)

A clean, well-documented Python library wrapping the 3DGS rasterization CUDA kernels. Designed to be embedded in other frameworks rather than used as a standalone trainer. Provides differentiable splatting as a building block for research and production pipelines. pip-installable.

pip install gsplat

COLMAP

Not a rendering tool, but the standard Structure-from-Motion pipeline that almost every NeRF and 3DGS workflow depends on. Given a set of unordered photos, COLMAP extracts feature matches, estimates camera intrinsics and extrinsics, and produces a sparse point cloud. Most neural rendering tools call COLMAP automatically during pre-processing.

WebGL / Three.js Splat Viewers

Several libraries render trained 3DGS scenes in the browser at interactive framerates using WebGL or WebGPU:

• antimatter15/splat: Minimal vanilla JS/WebGL viewer, good for embedding.

• mkkellogg/GaussianSplats3D: Three.js-based, full scene graph integration.

• Spline: Design tool with native 3DGS import and web export — targeted at web designers rather than 3D engineers.

---

Where the Field Is Heading

The boundary between NeRF-style volumetric representations and Gaussian-based explicit representations is blurring. Several 2024–2025 papers combine hash-grid features (Instant-NGP lineage) with Gaussian splatting rasterizers. Separately, diffusion-model-driven scene generation (DreamFusion, Wonder3D) uses NeRF as a 3D consistency backbone for text-to-3D synthesis.

The practical split in 2026: NeRF-family methods remain dominant for research, quality-critical unbounded scenes, and workflows where memory is constrained. Gaussian Splatting is winning in applications that need real-time rendering — virtual production, game asset capture, live e-commerce, and spatial computing apps on Apple Vision Pro and Samsung Galaxy XR.

Both approaches are converging on the same challenge: making neural 3D representations editable — allowing artists to move objects, change lighting, and composite scenes — without retraining from scratch. Solving that will be the inflection point from research curiosity to mainstream production pipeline.

---

References: Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" (ECCV 2020) · Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (SIGGRAPH 2023) · Müller et al., "Instant Neural Graphics Primitives" (SIGGRAPH 2022) · Barron et al., "Mip-NeRF 360" (CVPR 2022) · Nerfstudio documentation (2024) · gsplat library docs (2025).