AI and XR are converging, with spatial information becoming the key to connecting the virtual and real worlds.

By VR Gyroscope Wickey

With the development of multimodal AI, models' understanding of the real world is gradually extending from 2D visual recognition to 3D spatial reasoning and physical interaction.

On the morning of October 14, Google officially released Gemini 4 Argon, demonstrating its capabilities in spatial understanding and physical simulation. The Real2Sim technology can combine information such as robot camera footage and human first-person operation videos to reconstruct a 3D simulated environment that includes the geometric structure, spatial relationships, and physical properties of objects. It simulates object motion and interactions using MuJoCo.

Although primarily aimed at the robotics field, this technology also reveals possibilities for deeper virtual-real fusion in the XR industry. Currently, devices like PICO 4 Ultra, Quest 3, and Apple Vision Pro can already place virtual objects in real environments and simulate basic collisions and interactions. With improvements in models like Gemini 4 Argon's understanding of the physical world, future XR devices are expected to better understand the properties and motion patterns of real objects, allowing virtual content to blend more naturally into the real environment and further enhancing the MR experience.

01 For the Virtual World to "Land," XR Must First Understand Real Space

For the XR industry, naturally blending virtual objects with the real world has always been a key technical direction.

In Meta Quest 3's MR game First Encounters, virtual creatures can break through real walls and move and bounce around furniture. With Apple Vision Pro's RealityKit, developers can make virtual blocks land on real tables and produce simulated collision effects through hand gestures.

Apple RealityKit physics simulation example. Source: Apple

These experiences already enable virtual content to participate in interactions within real space, supported by the spatial awareness technologies accumulated in the XR industry over time.

Early VR devices primarily relied on Inertial Measurement Units (IMUs) for 3DoF tracking, capable of sensing only rotational movements such as pitch, yaw, and roll of the head. With the popularization of 6DoF tracking technology, displacement sensing in three directions—forward/backward, left/right, and up/down—was added. Using external optical tracking, Visual-Inertial Odometry (VIO), or SLAM technologies, devices can estimate their position and orientation in real time, allowing virtual content to maintain a relatively stable spatial position as users move.

To further enable virtual-real interactions, XR devices also need to understand the geometric structure of the surrounding environment. Through cameras, depth sensors, and scene reconstruction algorithms, devices can identify structures like walls, floors, and tables, generating spatial meshes that provide capabilities such as collision, occlusion, and spatial anchoring for virtual content.

Taking First Encounters as an example, Meta uses Scene API and Scene Mesh to obtain geometric information about the room, enabling virtual creatures to interact with the user's real environment. Apple's RealityKit can also utilize spatial meshes generated through scene reconstruction to enable physical collisions between virtual objects and real surfaces.

However, these experiences still heavily rely on spatial geometric information, physics engines, and pre-configured interaction logic by developers. For example, when a virtual ping-pong ball lands on a real table, the XR device can determine the timing of the collision based on the table's geometric position and use preset physical parameters to calculate the bounce trajectory. However, when faced with different surfaces like wooden tables, glass, or carpets, the system may not automatically determine their elasticity, friction, and other physical properties to generate interaction effects that comply with real-world laws.

This limitation also makes virtual-real fusion largely dependent on developers' pre-design of the scene. Virtual objects can interact with the real space according to set rules, but to automatically understand object properties and generate reasonable physical feedback in vastly different real environments, there is still a long way to go. With multimodal AI extending into 3D spatial reasoning and physical understanding, new solutions to this problem are beginning to emerge.

02 Google Enables AI to Reconstruct the Physical World, Bringing New Variables to Virtual-Real Fusion

In recent years, the integration of AI and XR has evolved from traditional voice-based Q&A to intelligent interactions that incorporate the real environment.

As early as 2024, Google Research proposed the XR-Objects research prototype, combining object recognition with multimodal large models to allow users to directly obtain information or generate virtual content around real objects. In January of this year, Meta further introduced the MR language learning application Spatial Lingo, which identifies and locates real objects through Quest and uses the Llama model to generate corresponding language learning content.

Meta Spatial Lingo technical architecture. Source: Meta Developers

The above explorations demonstrate AI's ability to combine object recognition, spatial positioning, and semantic understanding. However, to further understand the real world, AI also needs to grasp the spatial relationships between objects and even infer their motion patterns and physical interaction processes.

Google's newly released Gemini 4 Argon further showcases AI's spatial understanding and physical reasoning capabilities. The model achieved a score of 87% on the Embodied Reasoning (ERQA) benchmark and, through Real2Sim technology, reconstructs visual information such as first-person videos into 3D simulated environments that include physical properties like collisions and friction, used to simulate robotic grasping, movement, and other operations.

Real2Sim reconstructs first-person videos into 3D physical simulation environments. Source: Google for Developers

Another demonstration is even more noteworthy for the XR industry. Google used operation videos shot from a human first-person perspective to identify objects and hand movements, mapped human operation trajectories onto a virtual humanoid robot with a dexterous hand structure, and completed physical simulations using MuJoCo.

This means that an operation process in the real world can be transformed, with the help of AI, into a digital environment with 3D structure and physical interaction capabilities.

Gemini 4 Argon guides a dexterous hand to complete 33 consecutive operations through an iterative video world model, ultimately spelling out "ARGON." Source: Google for Developers

Although this technology is currently primarily aimed at robotics and physical AI research and has not been directly applied to XR devices, it provides ideas for spatial awareness in XR devices. It may be possible to use visual information collected by ordinary cameras and complete partial spatial reconstruction and object relationship recognition through AI, reducing reliance on complex sensors. Its combination in some MR application scenarios, especially in collisions and contacts between 3D objects and actual surface materials, may yield different results.

On the other hand, while AI is currently difficult to replace real-time spatial tracking with high-precision multi-cameras, its improved spatial understanding capabilities can also provide a new technical path for lightweight XR devices. Of course, the data collected by spatial awareness and 3D reconstruction technologies is also an important data source for training AI models. The VLM-3R research, accepted by CVPR 2026, combines 3D reconstruction with visual-language models to construct over 200,000 spatial understanding training Q&As, covering information such as object positions, distances, directions, and spatial relationships, providing a new training method to enhance the model's 3D spatial reasoning capabilities.

Source: VLM-3R

From this perspective, spatial awareness and AI also form a mutually reinforcing relationship. SLAM, depth perception, and environmental mapping provide geometric information and spatial constraints for AI, while the 3D data generated by spatial reconstruction can also be used for model training, further enhancing AI's spatial understanding and reasoning capabilities.

03 The Trade-Off Between Computing Power and Weight: How XR Devices Allocate Spatial Computing Capabilities

As spatial awareness and AI understanding capabilities continue to enrich, higher requirements are also placed on the hardware configurations of XR devices. However, compared to VR and MR headsets, AR and AI glasses face stricter limitations in terms of size, weight, and power consumption. Therefore, it is necessary to make reasonable trade-offs between spatial awareness capabilities and computing resources based on actual application scenarios.

Generally, basic information display may only require 3DoF pose tracking, while spatial navigation, virtual content anchoring, and AI environmental interactions require a combination of 6DoF positioning, object recognition, and spatial information to achieve accurate associations between digital content and the real environment.

In response to the positioning accuracy and power consumption requirements of different scenarios, manufacturers have begun to explore more flexible perception solutions. In May of this year, Google introduced a low-power positioning mode in ARCore for Jetpack XR, achieving basic geographic positioning using only IMUs and GPS, allowing some applications to operate without relying on complex spatial awareness systems, thereby reducing computing overhead.

Google introduced a low-power positioning mode for ARCore for Jetpack XR, relying solely on IMUs and GPS for geographic positioning information. Source: Google

Moreover, spatial tracking algorithms themselves are also evolving toward low power consumption. The LEVIO research announced this year optimizes visual-inertial odometry for resource-constrained devices like smart glasses, achieving 20 FPS 6DoF tracking with power consumption below 100mW. Through algorithmic and hardware co-optimization, complex spatial tracking tasks can run on lower-power platforms.

LEVIO visual-inertial odometry architecture, achieving low-power 6DoF spatial tracking through the fusion of camera and IMU data. Source: LEVIO research team

The above explorations indicate that lightweight spatial computing for XR glasses can be achieved by selecting positioning modes on demand or relying on algorithmic optimizations to reduce the power consumption of continuous tracking. However, spatial tracking, environmental awareness, and AI processing still require significant computing resources. How to accommodate these tasks within the limited frame volume also prompts manufacturers to explore different computing architectures.

Currently, split-type architectures have become the primary choice for manufacturers to reduce frame weight. XREAL AURA, whose price was officially announced on October 7, adopts this approach. The device weighs approximately 95 grams, with the glasses end equipped with the X1S spatial co-processor for low-latency spatial display and sensor processing. The external computing unit is equipped with the Snapdragon Reality Elite platform, responsible for running Android XR and performing complex computing tasks. Through this division of labor, AURA can retain capabilities such as 6DoF spatial tracking, gesture interaction, and AI visual perception within a relatively lightweight frame.

Source: XREAL

Meta's recently released VR Glasses also transfer some computing components and batteries to an external unit, keeping the glasses' weight at around 100 grams. However, the two devices differ in display and environmental awareness solutions. XREAL AURA uses optical see-through, allowing users to directly observe the real environment; Meta VR Glasses use video see-through, requiring cameras to capture and present external scenes in real time.

Image source: Meta

In contrast, Snap SPECS opts to integrate computing, sensors, display, and battery within the glasses themselves. SPECS weighs approximately 132 to 136 grams, with an official claimed mixed-use battery life of up to 4 hours. It employs dual processors to handle tasks such as computer vision and AR applications separately, enabling independent spatial interaction. This all-in-one design eliminates the need for an external computing unit but requires balancing hardware integration, weight, and battery life within the frame.

Regardless of the architecture, lightweight spatial awareness should focus more on determining which tasks must run in real-time, which computations can be offloaded to external devices, and whether AI can leverage fewer sensors to obtain sufficient spatial information.

04 Conclusion

AI is expanding the application value of XR spatial awareness, enabling devices to not only determine the position of virtual content but also understand its interactive relationship with real-world objects, allowing digital content to blend more naturally into the real environment.

However, there is still a gap between three-dimensional simulation capabilities and practical applications. Faced with dynamically changing real-world environments, XR devices must ensure the accuracy and real-time performance of spatial tracking and AI inference while also addressing issues such as computing power, power consumption, and hardware integration.

As related technologies gradually mature, XR devices are expected to combine user location, environmental information, and specific tasks to provide more natural and practical spatial interaction experiences, further integrating spatial computing into daily life.