# How to Set Up Real-World Occlusion Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/setup_real_world_occlusion/?platform=swift On iOS, Niantic Spatial depth can be used to occlude virtual content in several ways, depending on how much control you need over rendering and synchronization. This guide explains one specific technique: **occlusion using a full-screen occluder mesh** rendered with Metal. This approach is functionally equivalent to Unity's **NSDK Occlusion Extension -> Preferred Occlusion Technique: Occlusion Mesh**. ## Overview In this sample, real-world occlusion is achieved by: 1. Converting the depth image into a GPU texture 2. Reconstructing a **world-space mesh** from that depth 3. Rendering the mesh **into the depth buffer only** 4. Drawing virtual content afterward so it is automatically occluded ## Full-Screen Occluder Mesh The **occluder mesh** is a screen-aligned grid with **one vertex per depth pixel**. The mesh itself is invisible, but it writes depth values that block virtual objects. - Each vertex stores a `(u, v)` coordinate into the depth texture - Triangles connect adjacent pixels into a continuous surface - The mesh is generated **once**, after the first depth frame arrives - Only the depth texture and transforms change per frame Conceptually: **Depth Image -> Occluder Mesh -> Depth Buffer** Any virtual geometry rendered **after** the occluder mesh that participates in normal depth testing, will be automatically hidden wherever its geometry is behind the mesh. ## Data Required to Build the Occluder Mesh To reconstruct and render the occluder mesh, the renderer needs **five pieces of data** every frame: the **depth image** (GPU texture), **depth intrinsics** and **extrinsics**, **ARKit view** and **projection** matrices. These inputs together define how depth pixels are converted into world-space geometry and then projected into the current camera view. The NSDK occlusion sample collects and updates these resources inside the render loop so they are always used together when drawing the occluder mesh. ```swift private func fetchDepthData() -> Bool { guard let depthResult = /* Acquire the latest DepthResult from NSDK */, let frame = /* Acquire the latest ARFrame from ARKit*/ else { return false } // Update the depth texture when a new depth frame arrives if depthResult.frameId != lastDepthFrameId { lastDepthFrameId = depthResult.frameId TextureUtils.createOrUpdateTexture( from: depthResult.image, texture: &depthTexture, device: device ) intrinsicsMatrix = depthResult.intrinsics extrinsicsMatrix = depthResult.pose } // Update the current AR camera matrices viewMatrix = frame.camera.viewMatrix(for: orientation) projectionMatrix = frame.camera.projectionMatrix( for: orientation, viewportSize: mtlView.drawableSize, zNear: 0.01, zFar: 10.0 ) return true } ``` ### What Each Resource Is Used For - **Depth Texture**: The depth image is uploaded to a Metal texture and sampled in the vertex shader. Each pixel represents the distance from the depth camera to real-world surfaces. This texture provides the raw depth values used to construct the occluder geometry. - **Depth Intrinsics**: The depth intrinsics describe how pixel coordinates in the depth image map to camera-space rays. They are used to unproject each depth pixel into a 3D position in the depth camera's coordinate system. - **Depth Camera Pose** (Extrinsics): The depth camera pose represents the position and orientation of the camera at the moment the depth image was captured. This matrix is used to transform points from depth camera space into world space, placing the reconstructed geometry correctly in the scene. - **ARKit View Matrix**: The ARKit view matrix describes the inverse transform of the current camera pose. It is used to transform world-space geometry into the current camera's view space for rendering. - **ARKit Projection Matrix**: The projection matrix defines the camera's lens parameters and viewport mapping. It projects view-space geometry into clip space so the occluder mesh aligns correctly with the live camera image. ## Depth-Only Rendering The occluder mesh is rendered **only into the depth buffer**. This is esnured by specifying and empty write mask to the color attachment of the render pipeline used to draw the mesh. ```swift let occDesc = MTLRenderPipelineDescriptor() ... occDesc.depthAttachmentPixelFormat = mtlView.depthStencilPixelFormat occDesc.colorAttachments[0].pixelFormat = mtlView.colorPixelFormat occDesc.colorAttachments[0].writeMask = [] // depth-only (bypass color) ``` As a result: - The mesh is invisible - Depth values are written - Any geometry drawn afterward is occluded automatically ## Depth Unprojection (Vertex Shader) In the occlusion vertex shader, each vertex: 1. Samples the depth texture 2. Unprojects the pixel using depth intrinsics 3. Transforms the point into world space using the depth camera pose (extrinsics) 4. Projects it using the current ARKit view-projection matrix ```swift float depth = depthTexture.sample(depthSampler, uv).r; // The values cx, cy, fx, and fy come from the intrinsics of the depth image. // Additionally, px and py represent the coordinates of the current pixel. float x_cam = (px - cx) * depth / fx; float y_cam = (py - cy) * depth / fy; float z_cam = -depth; float4 camPos = float4(x_cam, y_cam, z_cam, 1.0); float4 worldPos = extrinsics * camPos; out.position = viewProj * worldPos; ``` This reprojects the depth geometry from the depth camera's capture pose into the current AR camera view, stabilizing occlusion even when depth updates arrive at a lower frame rate. For full implementation details, see the Swift Occlusion Sample in the Niantic Spatial iOS samples project. For more information, see the [Occlusion Feature page](https://www.nianticspatial.com/docs/nsdk/features/occlusion/) ## More Information For more information, see the [Occlusion Feature page](https://www.nianticspatial.com/docs/nsdk/features/occlusion/) and [Adding Occlusion to Your Project](https://www.nianticspatial.com/docs/nsdk/how-to/ar/adding_occlusion/).