Apple researchers introduce RayRoPE for multi-view AI vision tasks

Multi-view transformers—AI models that process multiple posed input images together—require a way to encode the spatial position of image patches that respects both the multi-view geometry and the mathematical properties needed for stable attention. Prior positional encoding schemes, whether absolute (fixed coordinates) or relative (distances between patches), fall short on three fronts: they do not uniquely identify each patch across views, they do not preserve SE(3) invariance (the property that the encoding remains stable under 3D rotations and translations), and they do not adapt to the actual geometry of the 3D scene being reconstructed.

RayRoPE solves this by anchoring patch positions to rays—the imaginary lines from camera centers through pixels—and using a predicted point along each ray rather than just the ray direction. To maintain SE(3) invariance, the method computes similarity in a query-frame projective coordinate system, allowing multi-frequency attention patterns that respect the geometry. The method also includes a principled way to handle uncertainty: when the predicted 3D point is imprecise, RayRoPE analytically computes the expected position encoding rather than failing or defaulting to a fixed approximation.

Validation on novel-view synthesis (generating new views of a scene from existing images) and stereo depth estimation (inferring 3D depth from two views) shows that RayRoPE consistently outperforms existing encoding schemes. Notably, it can also incorporate RGB-D input—color plus depth—which alternative methods cannot positionally encode, yielding even larger improvements. This suggests RayRoPE may become a useful component for 3D reconstruction and multi-view AI systems.