Project 4

Neural Radiance Field!

Part 0: Calibrating Your Camera and Capturing a 3D Scan

Part 0.1: Calibrating Your Camera

• Captured 53 calibration images at a fixed zoom.
• Detected ArUco corners and computed intrinsics/distortion via cv2.calibrateCamera.
• Skipped frames without detections to avoid corrupting calibration.
• Used ProCam to lock exposure, white balance, shutter speed, and ISO; kept autofocus enabled for reliable tag detection.
• My printer could not print true-to-scale, so I remeasured the printed sheet and used these dimensions in calibration: ArUco tag length 53.5 mm, horizontal separation 27 mm, vertical separation 14 mm.

Part 0.2: Capturing a 3D Object Scan

• Captured 59 object images next to a printed ArUco tag.
• Used the same ProCam setup to lock exposure, white balance, shutter, and ISO; left autofocus on for robust tag detection and minimized motion blur.
• For the single reference tag used during capture, the measured side length was 95.5 mm.

Part 0.3 + 0.4: Estimating Camera Pose and Dataset

• Used cv2.solvePnP with detected corners and intrinsics to recover pose.
• Converted to camera-to-world matrices for visualization and later parts.
• Visualized camera frustums in 3D in Viser.
• Undistorted images with cv2.undistort (later, I skipped the undistort step, as this yielded better results).
• Using recovered intrinsics and poses (c2w), I packaged everything into a final .npz dataset for training.

Part 1: Fit a Neural Field to a 2D Image

In this part, I train a neural field (an MLP with sinusoidal positional encoding) to map 2D pixel coordinates to RGB, fitting an image by minimizing mean squared error. The output quality is measured with Peak Signal-to-Noise Ratio (PSNR), which increases as MSE decreases:

• Implemented an MLP with sinusoidal positional encoding to map 2D coords → RGB in PyTorch.
• Used a random pixel sampling dataloader; trained with MSE (Adam 1e-2).
• Reported PSNR and tuned width and max PE frequency.

Part 2: Fit a Neural Radiance Field from Multi-view Images

Part 2.1: Create Rays from Cameras

Implemented the full camera/ray geometry pipeline and data loading used throughout NeRF:

• Loaded a .npz dataset via numpy.load, extracted images_train, c2ws_train, and focal, and normalized images to [0,1] float32.
• Converted camera-space points to world space using homogeneous coordinates and batched matrix multiply, then divided by w with a small clamp for stability.
• Inverted the pinhole model with K^{-1}[u,v,1]^T (depth scale s applied afterward), precomputing K^{-1} and building homogeneous [u,v,1] per pixel.
• Ray origin is the camera center (c2w[:3,3]). Pixels are mapped to world-space points at unit depth, then ray_d = normalize(x_w − ray_o), with everything fully batched.

Part 2.2: Sampling

Sampled both rays and 3D points along each ray:

• An infinite generator returns batches of (ray_o, ray_d, pixel_color). Randomly sampled image indices and pixel coordinates, gathered RGB targets, selected per-pixel K and c2w, and called pixel→ray to get batched origins and directions.
• Used stratified sampling between [near, far] with num_samples bins. With perturbation, added uniform noise within each interval and returned the 3D points, unit directions, and step_size = (far − near)/num_samples.

Part 2.3: Putting the Dataloading All Together

The combined pipeline produces per-batch ray origins/directions and colors, and I added a small visualization script (Viser) to render cameras, rays, and sampled 3D points for sanity checks.

Part 2.4: Neural Radiance Field

Implemented a NeRF-style MLP conditioned on positional encodings of 3D points and view directions:

• Used sin/cos bands with frequencies 2^k π, concatenated with the input. Separate levels for positions and directions.
• A stack of fully-connected layers (configurable depth/width) with a skip connection that concatenates the encoded position mid‑network.
• Mapped hidden features to σ via softplus (with positive bias) for stable, non‑negative densities.
• Produced a feature, concatenated encoded view direction, and output RGB with sigmoid to constrain [0,1].

For Part 2 I kept the architecture essentially the same as the reference NeRF: a stacked MLP with a skip connection, separate density and color heads, and positional encodings for both 3D coordinates and view directions. My changes focused on hyperparameters only (e.g., network width, coordinate positional-encoding levels L, and view-direction encoding levels L_dir).

Part 2.5: Volume Rendering

Implemented the discrete volumetric rendering equations in PyTorch for color, depth, and opacity:

• Given per-sample densities σ_i and colors c_i with uniform Δ, computed α_i = 1 − exp(−σ_i Δ) and transmittance T_i = exp(−∑_j<i σ_j Δ). The weights are w_i = T_i α_i and the rendered color is ∑ w_i c_i.
• Reused the same weights to compute accumulated opacity acc = ∑ w_i and expected depth ∑ w_i t_i / max(acc, ε).
• Used exclusive cumulative sums for T_i, included shape checks, and ensured full differentiability.

Part 2.6: Training with your own data

Training ties all components together: per‑step I sample rays and 3D points, run the NeRF, volume‑render, compute MSE against ground‑truth pixels, and optimize with Adam. I log PSNR (−10 log10(MSE)), periodically render full images (with optional depth/opacity), and save checkpoints.

I trained a NeRF on my Part 0 dataset, rendered a circling camera GIF, and tracked loss with intermediate renders.

During training, the training MSE continued to decrease while the validation error largely stagnated. This behavior is consistent with the earlier observation that my calibration is not perfectly accurate—validation rays are evaluated under slightly mismatched intrinsics/poses. Even though the validation curves did not keep improving, the rendered reconstructions still looked progressively better to a human observer, suggesting that the model was learning a useful representation of the scene despite these imperfections.

I rescaled images to a width of 800 px and scaled the camera intrinsics accordingly. I chose a higher resolution because my calibration was not perfect: in Viser, back‑projected rays from the camera origin did not intersect exactly at the ArUco tag borders (as they would with ideal intrinsics/poses). My hypothesis was that a higher resolution would reduce pixel‑level quantization error and provide more samples per object area, helping NeRF tolerate small calibration errors. In practice, training at the higher resolution produced better results, which likely supports this rationale (with the trade‑off of increased compute).

Bells & Whistles

Depth Map Rendering with Accumulated Opacity Threshold

Along each camera ray, NeRF samples points with predicted densities σ (and colors). Volumetric rendering assigns each sample an opacity α_i = 1 − exp(−σ_i Δ_i) over interval Δ_i and a transmittance T_i = exp(−∑_j<i σ_j Δ_j). The per‑sample weight is w_i = T_i · α_i. The depth map is the ray‑wise expectation of distance: Depth = ∑ w_i · t_i (optionally normalized for display).

The accumulated opacity (confidence) that a ray intersects a surface is acc = ∑ w_i, with values near 0 indicating empty space and near 1 indicating an opaque hit. A threshold acc_thresh masks rays with insufficient confidence (acc < acc_thresh) and normalizes depths over the remaining pixels, suppressing floaters and background leakage in low‑confidence regions.

I tuned acc_thresh empirically. Lower values kept semi‑transparent floaters, while higher values started knocking out thin structures. Settling around acc_thresh ≈ 0.3 consistently produced cleaner, more stable depth maps for my scenes.

Learnings

• Dataset creation was the hardest part. Reliable intrinsics/poses and disciplined capture matter more than small model tweaks.
• Lego trained easily; my own data was more challenging. High‑quality results required precise calibration and consistent capture geometry.
• Surprisingly, skipping undistortion improved my results—likely because the iPhone lens has minimal distortion and/or applies in‑camera correction.