Although Apple barely mentioned the Vision Pro's operating system during its main stage presentation at WWDC 2026, developer documentation and technical sessions reveal that visionOS 27 is one of the most substantial upgrades to date. New features include pre-configured handheld object tracking, an infrared LED-based tracking accessory framework, gaze-free screen recording, and numerous underlying technical improvements.

It is reported that visionOS 27 will be available for M2 and M5 models of Apple Vision Pro this fall. The first beta is now available, but some features will require the stable version.

I. AI and Intelligent Interaction

Siri AI and Visual Intelligence

While it won't be available in the initial release, Apple has stated that a reimagined Siri AI, powered by large language models, will arrive on visionOS later this year. Driven by next-generation Apple Foundation Models—developed in collaboration with Google (reportedly under a $1 billion annual agreement)—Siri AI will handle some tasks on-device while offloading complex tasks to Private Cloud Compute.

In visionOS, Siri AI appears as a persistent sphere that can be anchored in physical space. Users can simply look at it and speak without a wake word. It also supports visual intelligence, enabling searches, questions, and actions related to both physical and virtual environments. Its capabilities are positioned between Meta AI (Quest 3) and Gemini (Samsung Galaxy XR).

Additionally, visionOS 27 introduces a dedicated Siri app for reviewing conversation history, serving as a pre-installed alternative to third-party AI chatbots.

Comment: Presenting the AI assistant as an "anchorable spatial object" feels more natural than traditional floating panels or voice bars, highlighting the advantages of eye-tracking technology.

Notably, the next-generation Foundation Models also bring: more realistic image generation in Image Playground, automatic Safari tab organization, one-click password strength upgrades, and smart replies in Messages.

II. Spatial Environment and Visual Experience

Spatial Panoramic Environments

visionOS 2 added stereoscopic effects to 2D images, while visionOS 26 transformed 2D images into depth-enhanced "spatial scenes" using Gaussian splatting. visionOS 27 extends this capability to panoramic photos taken with iPhones. Users can set these panoramas as the main environment in shared spaces, with windows and virtual content appearing against a realistic backdrop.

New Environment: Þórsmörk (Thorsmork)

visionOS 27 introduces a 10th virtual environment—the Þórsmörk valley in Iceland (meaning "Valley of Thor"), considered one of the most beautiful landscapes on Earth. The nighttime version features aurora effects.

Comment: The visionOS environment library now covers diverse scenes, including the moon, volcanoes, deserts, forests, and auroras.

Curved Windows (Apple Native Apps Only)

Previously, only Mac virtual displays in ultra-wide mode featured curvature. visionOS 27 adds curved window support to Safari, Freeform, and Apple TV's multi-view modes. Safari's tab browsing experience slightly wraps around the user. However, this uses a private API, preventing third-party developers from using it in shared spaces.

Safari on visionOS 27 supports curved windows

Comment: This limitation may reflect Apple's desire for system consistency, but it raises anticipation for future third-party access.

III. Tracking and Accessory Ecosystem

High-Frame-Rate Object Tracking

visionOS 2 already supported tracking preset 3D objects, primarily for static items at ~5Hz. visionOS 27 introduces a high-frame-rate mode (~30Hz) for moving and handheld objects. For handheld tracking, Apple recommends "extended training," which significantly boosts performance but increases training time (e.g., ~18 hours for a fish-shaped toy).

Comment: While 30Hz remains lower than hand-tracking frame rates, it suffices for many interactive scenarios.

Open Infrared LED Tracking Accessories

visionOS 26 hardcoded support for the PS VR2 Sense controller and Muse digital pen. visionOS 27 introduces a new API allowing any company to create infrared LED tracking accessories, akin to SteamVR's Lighthouse ecosystem. Tracking fuses with IMU data for 90Hz optical tracking quality.

Tracking infrared LED accessories in visionOS 27

These accessories integrate in two ways:

Companion App: Hardware makers publish a visionOS app to register devices for other apps to call (ideal for consumer products).

Built-in App: Developers embed device data directly into their apps (ideal for enterprises).

Additionally, DFRobot and MikroE will release reference kits and development boards to lower entry barriers.

visionOS 27 supports tracking accessories from DFRobot and MikroE

Comment: This marks Apple's significant step toward an open accessory ecosystem, potentially enabling specialized controllers, styluses, and even motion-sensing devices.

IV. Recording and Developer Tools

Gaze-Free Screen Recording

Since Vision Pro always uses foveated rendering, screen recordings blur non-gaze areas. visionOS 27 adds a "high-quality recording" option for up to 3 minutes of gaze-free 4K footage, entirely on the headset without a Mac. This is one of the most anticipated features for developers and content creators.

Mac 3D Object Preview API

macOS 27 introduces an API for generating and previewing 3D objects directly in visionOS 27. Apple's built-in "Preview" app already supports it, and developers can add this functionality to their Mac apps with just a few lines of code. It supports bidirectional editing (changes in visionOS sync back to macOS) and SharePlay collaboration.

Additionally, Cinema 4D and SketchUp are integrating this feature.

Comment: This significantly lowers the barrier for adapting Mac-based 3D creation tools for Vision Pro.

Device Hub (Developers)

Xcode 27 introduces the "Device Hub" app for developers to remotely launch Vision Pro experiences without guiding users. This fills the gap left by iPhone/iPad companion apps, which couldn't launch apps.

Wi-Fi Initialization Speed Improved by 3x

Apple acknowledged slow Wi-Fi connection times after Vision Pro startup. visionOS 27 triples the connection speed.

V. Interface and Interaction Improvements

New Control Center

Divided into three panels with a spacious interface:

Left: Date/time, battery, Wi-Fi, notifications

Center: Quick toggles for Wi-Fi/Bluetooth/AirPlay/Airplane Mode, volume slider, Mac virtual display controls

Right: Environment controls (day/night, passthrough intensity)

Accessory Widgets (Extra-Small)

visionOS 26 introduced anchorable mini-windows. visionOS 27 adds smaller "accessory widgets" for battery levels, timers, stock prices, and smart home switches. Apple also provides a first-party Mac virtual display widget for one-click connections even when the Mac is locked.

Comment: If smart home apps support it, you could place a virtual switch beside each physical device in your home and control it with a pinch gesture while looking at it.

Gaze to Expand Notifications

Simply look at a notification, and it will automatically expand to display more information and actions, without the need to pinch.

Auto Video Captions (Accessibility)

Uses on-device speech recognition models to generate real-time captions for any video.

Vehicle Motion Cues

Previously seen on iOS—simulating vehicle acceleration, deceleration, and turns through animated dots along the screen edges to reduce motion sickness—now arrives on visionOS. However, the specific form of expression (manifestation) on the headset remains unclear.

Eye-Controlled Electric Wheelchair

visionOS 27 supports native eye-tracking control for specific electric wheelchairs. The first batch supports Tolt and LUCI drive systems (USA), with Bluetooth and wired connectivity. Apple states it will continue to expand compatible wheelchair drive systems.

Comment: This is an outstanding application of spatial computing in the accessibility field—using sensors originally designed for gaze interaction to directly translate into mobility.

VI. Major Upgrades to RealityKit's Core Technology

RealityKit is the underlying engine for visionOS shared spaces and many full-space applications. visionOS 27 brings the following key upgrades:

1. Physical Space Lighting

Apps can now not only cast shadows in physical environments but also project light (point lights and spotlights). Supports projected textures and animated patterns, enabling effects like stained glass, underwater caustics, and starfield projections. Leverages LiDAR-scanned 3D meshes and ambient light sensors at the bottom layer (foundation) level.

2. Native Gaussian Splatting Rendering

Previously, developers had to write their own rendering code for Gaussian Splatting. Now, RealityKit natively supports it, significantly lowering the barrier to achieving high-quality 3D reconstructions. Apple has already used this technology in "Spatial Scenes" and Personas.

3. Real-Time Cloth Simulation

Integrates deformable fabric simulation: cloth can drape, fold, wrinkle, collide with objects under gravity, and be pulled or interacted with. Adjustable stiffness, friction, and material parameters.

4. Immersive Audio Upgrades (Reverb Grid API)

Apps can describe the geometry and acoustic properties of virtual environments (assigning parameters like absorption, scattering, and reflection to surfaces). RealityKit simulates sound propagation and reverberation in the space, aligning audio more closely with visual scenes.

Comment: Interestingly, Meta's XR Audio SDK was "frozen" in development earlier this year, while Valve's Steam Audio SDK continues to receive updates. Apple's introduction of similar functionality at this time demonstrates its emphasis on professional XR audio.

VII. Other Noteworthy Updates

Safari Web Environment: Developers can specify a 3D environment for web pages (similar to a favicon), which users can optionally enable, unrelated to WebXR.

Microsoft/Meta Similar Features Unreleased: The article mentions in passing that Microsoft and Meta announced similar functionality for Windows 11 and Horizon OS in 2024, but neither has been released yet.

M5 Vision Pro Experience: The author describes the M5 model as "extremely fast," with the only issue—slow Wi-Fi startup—now fixed.

Conclusion

visionOS 27 is not a minor update but a comprehensive evolution spanning AI, tracking ecosystems, developer tools, and the underlying rendering engine. While Apple only spent a few minutes discussing it during the keynote, the details revealed in documentation and developer sessions suggest this may be the most substantial system update for Vision Pro since its launch.

Whether it's eye-controlled wheelchairs, an open accessory ecosystem, or Gaussian Splatting and real-time cloth simulation, all represent significant strides toward making Vision Pro a "universal platform for spatial computing."

By akua

All uncredited images sourced from the internet.