RealityKit Development Guide

Purpose: Build 3D content, AR experiences, and spatial computing apps using RealityKit's Entity-Component-System architecture iOS Version: iOS 13+ (base), iOS 18+ (RealityView on iOS), visionOS 1.0+ Xcode: Xcode 15+

When to Use This Skill

Use this skill when:

• Building any 3D experience (AR, games, visualization, spatial computing)
• Creating SwiftUI apps with 3D content (RealityView, Model3D)
• Implementing AR with anchors (world, image, face, body tracking)
• Working with Entity-Component-System (ECS) architecture
• Setting up physics, collisions, or spatial interactions
• Building multiplayer or shared AR experiences
• Migrating from SceneKit to RealityKit
• Targeting visionOS

Do NOT use this skill for:

• SceneKit maintenance (use axiom-scenekit)
• 2D games (use axiom-spritekit)
• Metal shader programming (use axiom-metal-migration-ref)
• Pure GPU compute (use Metal directly)

1. Mental Model: ECS vs Scene Graph

Scene Graph (SceneKit)

In SceneKit, nodes own their properties. A node IS a renderable, collidable, animated thing.

Entity-Component-System (RealityKit)

In RealityKit, entities are empty containers. Components add data. Systems process that data.

Entity (identity + hierarchy)
  ├── TransformComponent (position, rotation, scale)
  ├── ModelComponent (mesh + materials)
  ├── CollisionComponent (collision shapes)
  ├── PhysicsBodyComponent (mass, mode)
  └── [YourCustomComponent] (game-specific data)

System (processes entities with specific components each frame)

Why ECS matters:

• Composition over inheritance: Combine any components on any entity
• Data-oriented: Systems process arrays of components efficiently
• Decoupled logic: Systems don't know about each other
• Testable: Components are pure data, Systems are pure logic

The ECS Mental Shift

2. Entity Hierarchy

Creating Entities

// Empty entity
let entity = Entity()
entity.name = "player"

// Entity with components
let entity = Entity()
entity.components[ModelComponent.self] = ModelComponent(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .blue, isMetallic: false)]
)

// ModelEntity convenience (has ModelComponent built in)
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: true)]
)

Hierarchy Management

// Parent-child
parent.addChild(child)
child.removeFromParent()

// Find entities
let found = root.findEntity(named: "player")

// Enumerate
for child in entity.children {
    // Process children
}

// Clone
let clone = entity.clone(recursive: true)

Transform

// Local transform (relative to parent)
entity.position = SIMD3<Float>(0, 1, 0)
entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0))
entity.scale = SIMD3<Float>(repeating: 2.0)

// World-space queries
let worldPos = entity.position(relativeTo: nil)
let worldTransform = entity.transform(relativeTo: nil)

// Set world-space transform
entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)

// Look at a point
entity.look(at: targetPosition, from: entity.position, relativeTo: nil)

3. Components

Built-in Components

Custom Components

struct HealthComponent: Component {
    var current: Int
    var maximum: Int

    var percentage: Float {
        Float(current) / Float(maximum)
    }
}

// Register before use (typically in app init)
HealthComponent.registerComponent()

// Attach to entity
entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)

// Read
if let health = entity.components[HealthComponent.self] {
    print(health.current)
}

// Modify
entity.components[HealthComponent.self]?.current -= 10

Component Lifecycle

Components are value types (structs). When you read a component, modify it, and write it back, you're replacing the entire component:

// Read-modify-write pattern
var health = entity.components[HealthComponent.self]!
health.current -= damage
entity.components[HealthComponent.self] = health

Anti-pattern: Holding a reference to a component and expecting mutations to propagate. Components are copied on read.

4. Systems

System Protocol

struct DamageSystem: System {
    // Define which components this system needs
    static let query = EntityQuery(where: .has(HealthComponent.self))

    init(scene: RealityKit.Scene) {
        // One-time setup
    }

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query,
                                        updatingSystemWhen: .rendering) {
            var health = entity.components[HealthComponent.self]!
            if health