Physically Embodied Gaussian Splatting: Embedding Physical Priors into a Visual 3D World Model For Robotics

Jad Abou-Chakra1, Krishan Rana1, Feras Dayoub2, Niko Sünderhauf1,
1Queensland University of Technology, 2University of Adelaide

Visual Forces correcting the physical state of the scene.


Robots require a "mental model" of the world to anticipate and plan for future interactions. This model should capture and reason over physical phenomena such as collisions, forces, and object properties. Current representations compress the 3D structure of the world into latent vectors limiting their ability to incorporate known physical priors. This requires them to learn these known physical priors soley from data.

We build on our earlier work ParticleNeRF which combines NeRFs with a particle-based physics system that maintains the 3D structure of the world. This allows us to directly incorporate explicit physical, structural, and kinematic priors. Our dual Gaussian-Particle model provides a bidirectional bridge to update our world model with observations from the real world.



Modelling Step To initialize the system, the robot scans the environment with an RGBD camera and initializes particles based on the point cloud data. Semantic information is provided by a segmention network and used to induce objectness between particles. Particles representing the robot itself are initialised based on the kinematic state and a URDF model of the robot. Finally, particles are used to initialise Gaussians in their proximity.


Update Step Our dual Gaussian-Particle representation captures visual (Gaussians) and physical (particles) aspects of the world and enables forward prediction of robot interactions with the world. A photometric loss between rendered Gaussians and observed images is computed (Gaussian Splatting, left) and converted into visual forces. These and other physical phenomena such as gravity, collisions, and mechanical forces are resolved by the always-active physics system (right) and applied to the particles, which in turn influence the position of their associated Gaussians.

Related Links

ParticleNeRF introduces the idea of using particles as both the primitive for rendering NeRFs and the primitive on which a physics system acts.

PAC-NeRF also uses a combination of a physics method and a NeRF rendering method for the purposes of estimating physical parameters and unknown geometry.