RealityKit + ARKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world tracking. Covers RealityView, entity management, raycasting, scene understanding, and gesture-based interactions. Targets Swift 6.2 / iOS 26+.

Contents

• Setup
• RealityView Basics
• Loading and Creating Entities
• Anchoring and Placement
• Raycasting
• Gestures and Interaction
• Scene Understanding
• Common Mistakes
• Review Checklist
• References

Setup

Project Configuration

1. Add NSCameraUsageDescription to Info.plist
2. For iOS, RealityKit uses the device camera by default via RealityViewCameraContent (iOS 18+, macOS 15+)
3. No additional capabilities required for basic AR on iOS

Device Requirements

AR features require devices with an A9 chip or later. Always verify support before presenting AR UI.

import ARKit

guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}

Key Types

RealityView Basics

RealityView is the SwiftUI entry point for RealityKit. On iOS, it provides RealityViewCameraContent which renders through the device camera for AR.

import SwiftUI
import RealityKit

struct ARExperienceView: View {
    var body: some View {
        RealityView { content in
            // content is RealityViewCameraContent on iOS
            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 50cm in front of camera
            content.add(sphere)
        }
    }
}

Make and Update Pattern

Use the update closure to respond to SwiftUI state changes:

struct PlacementView: View {
    @State private var modelColor: UIColor = .red

    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }

        Button("Change Color") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}

Loading and Creating Entities

Loading from USDZ Files

Load 3D models asynchronously to avoid blocking the main thread:

RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}

Programmatic Mesh Generation

// Box
let box = ModelEntity(
    mesh: .generateBox(size: [0.1, 0.2, 0.1], cornerRadius: 0.005),
    materials: [SimpleMaterial(color: .gray, isMetallic: true)]
)

// Sphere
let sphere = ModelEntity(
    mesh: .generateSphere(radius: 0.05),
    materials: [SimpleMaterial(color: .blue, roughness: 0.2, isMetallic: true)]
)

// Plane
let plane = ModelEntity(
    mesh: .generatePlane(width: 0.3, depth: 0.3),
    materials: [SimpleMaterial(color: .green, isMetallic: false)]
)

Adding Components

Entities use an ECS (Entity Component System) architecture. Add components to give entities behavior:

let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)

// Make it respond to physics
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))

// Add collision shape for interaction
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))