Visionos Immersive Media Developer

visionOS Immersive Media Developer

Description and Goals

This skill provides a practical playbook for building immersive and spatial media experiences on Apple Vision Pro. Start with AVKit when the system player experience fits your app. Use RealityKit’s VideoPlayerComponent when you need custom placement, custom environments, or deeper scene integration for video playback that can move between window/portal playback and immersive playback.

It also outlines how Apple’s ImmersiveMediaSupport framework fits into authoring and packaging Apple Immersive Video content.

Goals

• Help you choose the right playback surface (2D window, portal, progressive immersive, full immersive).
• Implement RealityKit video playback using VideoPlayerComponent correctly.
• Handle immersive viewing mode transitions safely (including scene transitions when needed).
• Respond to video playback events and comfort mitigation events on visionOS 26+.
• Provide a foundation for Apple Immersive Video authoring workflows where relevant.

What This Skill Should Do

When asked to add immersive/spatial video to a visionOS app, this skill should:

1. Clarify the media experience
   • Is this a “video on a surface” experience, a portal, or full immersive playback?
   • Is the content mono, stereo, spatial video, or immersive media?
   • Do you need passthrough tinting, captions/subtitles, or multi-user playback sync?
2. Pick a playback architecture
   • Prefer AVKit when the system playback interface matches the product requirements.
   • Use RealityKit VideoPlayerComponent when the experience needs custom placement, custom environments, or scene-aware playback behavior.
   • Escalate to CompositorServices only for custom engines or bespoke full-space rendering.
3. Implement playback + UI
   • Create a video entity, attach VideoPlayerComponent, and manage UI affordances.
   • Handle immersive-viewing mode transitions (portal ↔ immersive space).
4. Handle events and comfort
   • Subscribe to VideoPlayerEvents for viewing mode changes, transitions, and comfort mitigation.
5. Validate comfort and accessibility
   • Provide clear exit paths, reduce motion where appropriate, and support captions.

Load the appropriate reference file from the tables below for detailed usage, code examples, and best practices.

Information About the Skill

Core Concepts

Choose AVKit first, then RealityKit when needed

AVKit provides the preferred system playback experience on visionOS. Use RealityKit’s VideoPlayerComponent when you need the video to participate directly in a custom 3D scene, custom environment, or bespoke interaction model.

Immersive viewing modes

For immersive media types, visionOS supports multiple immersive viewing modes:

• Portal: immersive media rendered as a portal window matching the containing entity transform.
• Progressive: partial-to-full coverage controlled via the Digital Crown (not used for Spatial Video).
• Full: immersive media fills the field of view.

You should treat switching between these modes as a scene orchestration problem as well as a component configuration problem.

Comfort mitigation (visionOS 26+)

visionOS can detect video comfort violations and trigger mitigation. Your app should subscribe to the event and respond:

• pause playback
• reduce immersion
• continue (play) depending on what the system reports and what UX you want

Authoring / packaging Apple Immersive Video

The ImmersiveMediaSupport framework provides tools and a workflow for authoring and packaging Apple Immersive Video content (typically as part of a content pipeline, not day-to-day app UI).

Reference Tables

Implementation Patterns

Basic VideoPlayerComponent setup

import AVFoundation
import RealityKit

let videoEntity = Entity()
let player = AVPlayer(url: url)

var video = VideoPlayerComponent(avPlayer: player)
video.isPassthroughTintingEnabled = true

videoEntity.components.set(video)
player.play()

Subscribe to immersive viewing mode transitions

import RealityKit

var subscription: EventSubscription?

subscription = content.subscribe(to: VideoPlayerEvents.ImmersiveViewingModeWillTransition.self) { event in
    // Disable non-essential UI while transitioning.
}

Subscribe to comfort mitigation (visionOS 26+)

subscription = content.subscribe(to: VideoPlayerEvents.VideoComfortMitigationDidOccur.self) { event in
    switch event.comfortMitigation {
    case .pause:
        // Update UI to indicate playback paused.
        break
    case .reduceImmersion:
        // Offer a UI path back to portal / lower immersion.
        break
    case .play:
        break
    }
}

Pitfalls and Checks

• Don’t treat immersive mode switching as “just a property change”; you may need to transition between window scenes and immersive spaces deliberately.
• Make sure you have an obvious “exit” affordance from immersive playback.
• Always test Spatial Video vs immersive media behavior (they differ; progressive isn’t used for Spatial Video).
• Subscribe to transitions/events early; many UX issues appear only during mode changes.
• For visionOS 26+, handle comfort mitigation events and align your UI with the system’s mitigation choice.