Gaussian splats capture the visual detail of real places, but a captured environment is typically static. Object interaction, a newly available scene variation feature available in beta, identifies objects within an environment and makes them independently selectable and movable in simulation. The goal is to preserve visual character while adding object-level structure, including collision information and simplified, estimated physical properties, so teams can work with more than a fixed view of a place.
Train robots for a changing world with object-level variation
Object interaction turns a captured environment into a set of elements that can be handled individually. At a high level, the workflow has four steps:
-
Identify candidate objects. Environment understanding finds whole, movable objects that are suitable for separation.
-
Extract each object. The selected object is separated from the surrounding area while retaining its visual appearance and collision information.
-
Review and clean up. The extraction is checked and refined where needed so the object can be used in the environment.
-
Rearrange the environment. Objects are represented individually, then can be selected, moved, or removed to create a different layout. Simplified physical properties provide a baseline for how objects interact.
Robots must handle environments that change after capture: a chair moves, equipment shifts, or clutter is rearranged. Training in a single static environment can leave a policy dependent on details that may not hold in the real world.
Object interaction lets robotics teams create varied arrangements within a recognizable, real-world environment. Teams can use these scenarios to test navigation, obstacle avoidance, and object interaction across changing layouts, then assess performance through their existing simulation and validation workflows. This variation can also help reduce a policy’s reliance on any one arrangement.
Combined with relighting, object interaction enables teams to vary both layout and lighting while keeping environments grounded in the real world, all without rebuilding from scratch.
We’re inviting a small number of design partners with high-value object interaction use cases and representative captured environments to evaluate how object-level variation can help test robot policies and shape what comes next.