Physics-Informed Dynamic 3D Reconstruction: From Gaussian Splatting and Rigid-Body Simulation to Inverse Material Estimation

Abstract

3D Gaussian Splatting (3DGS) is a leading representation for photorealistic reconstruction and novel view synthesis, with explicit, real-time scene models. Three limitations constrain the standard formulation: it depends on sparse Structure-from-Motion point clouds for initialization, is restricted to static scenes, and is disconnected from the physics simulators that would consume its output. Reviewing dynamic 3DGS methods shows reliance on hand-designed temporal regularizers rather than physical laws, so their motion is plausible but not physically produced. Across four studies of increasing complexity, this thesis brings physical reasoning to the pipeline from 3DGS reconstruction to physics simulation.

We first evaluate five point cloud upsampling strategies and a depth-guided point lifting method for static 3DGS initialization. On Mip-NeRF360 and Replica, geometry-aware upsampling consistently improves reconstruction quality, with average best-per-scene PSNR gains of 0.26 dB and 0.65 dB respectively, and yields scene-characteristic selection guidelines.

Next we ask whether dynamic reconstructions can be made physics-compatible after the fact. A dual-representation system pairing fixed-topology meshes for collision detection with Gaussians for rendering attains a 4.65× simulation speedup, but converting varying-topology reconstructions to fixed topology incurs 65 to 80% geometric degradation. This negative result means physics compatibility cannot be retrofitted by post-processing and must be expressed in the reconstruction objective.

Acting on this finding, PersistGS couples differentiable rigid-body simulation with 4D Gaussian Splatting to maintain object permanence through occlusion. Estimating friction and initial velocity from visible frames and positioning Gaussians along physics-predicted SE(3) trajectories, it achieves +2.46 dB PSNR over the best causal kinematic baseline and comes within 0.19 dB of the ground-truth upper bound.

At the continuum, non-differentiable end, the contribution is inverse material estimation. We recover the material parameters of the AnisoMPM food-fracture simulator, whose Lagrangian particle field parallels the 3DGS primitive. A goal-conditioned reinforcement learning policy in a normalizing-flow latent space amortizes estimation across arbitrary targets, achieving 0.642 simulator-validated recovery in a single pass, while a warm-start hybrid reaches the highest overall recovery, 0.828.

Together, the studies move reconstruction from photometric fidelity alone toward joint fidelity and physical consistency, and finally beyond 3DGS to continuum-fracture material parameters.