SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

Our SERF map maintains a 4D scene representation of both the environment and the robot body in a shared latent feature space. We build the map using neural points, which are 3D points with learnable features trained to reconstruct dense DINO embeddings.

To track changes, we construct 3D keypoint correspondences between consecutive observations, estimate an object-level rigid transform, and update the corresponding points. We extend the neural points to the robot body by sampling surface points from a URDF and positioning them via forward kinematics at each step.

We condition the VLA policy on SERF map tokens in addition to RGB observations, proprioceptive state, and the task embedding. We tokenize the SERF map across multiple reference frames, including global, end-effector, and robot-base frames, to capture both local and global context. These tokens provide the SERF VLA policy with allocentric scene memory and egocentric robot-environment context for long-horizon action decisions.