For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation - modelling geometry, physics, and visual observations - that informs perception, planning, and control algorithms. We propose a novel dual "Gaussian-Particle" representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates "visual forces" that correct the particle positions while respecting known physical constraints. By integrating predictive physical modeling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Our system runs in realtime at 30Hz using only 3 cameras.

The particles of a rope in a simulated scene.

The particles of a rope in a real scene.

The TBlock is pushed over by the robot and the gravity prior is able to keep the block within the corrective ability of the visual forces.

A mug is slightly pushed by the robot before being picked up. Both the push and the pickup are captured by the representation.

The Gaussian-Particle representation synchronizing with a rope as it is deformed.

Multiple objects being tracked by our system.

The particles and Gaussians can be built for complex shapes.