Abstract
Robots require a "mental model" of the world to anticipate and plan for future interactions. This model should capture and reason over physical phenomena such as collisions, forces, and object properties. Current representations compress the 3D structure of the world into latent vectors limiting their ability to incorporate known physical priors. This requires them to learn these known physical priors soley from data.
We build on our earlier work ParticleNeRF which combines NeRFs with a particle-based physics system that maintains the 3D structure of the world. This allows us to directly incorporate explicit physical, structural, and kinematic priors. Our dual Gaussian-Particle model provides a bidirectional bridge to update our world model with observations from the real world.
Method
Modelling Step To initialize the system, the robot scans the environment with an RGBD camera and initializes particles based on the point cloud data. Semantic information is provided by a segmention network and used to induce objectness between particles. Particles representing the robot itself are initialised based on the kinematic state and a URDF model of the robot. Finally, particles are used to initialise Gaussians in their proximity.
Update Step Our dual Gaussian-Particle representation captures visual (Gaussians) and physical (particles) aspects of the world and enables forward prediction of robot interactions with the world. A photometric loss between rendered Gaussians and observed images is computed (Gaussian Splatting, left) and converted into visual forces. These and other physical phenomena such as gravity, collisions, and mechanical forces are resolved by the always-active physics system (right) and applied to the particles, which in turn influence the position of their associated Gaussians.