# Displaying Depth Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/depth/display_depth/?platform=swift This guide demonstrates how to visualize the environment depth image provided by **NSDK** using **Metal**. We'll use the NSDK's NsdkDepthSession to obtain metric depth data, convert it into a Metal texture, and render it on-screen through a shader that encodes depth as color. For the full source code, please download the [NsdkSamples](https://github.com/nianticspatial/nsdk-samples-csharp) project. --- ## 1. Acquiring Depth and Creating a Metal Texture Subscribe to `$result` on the depth session to receive depth frames reactively each time a new frame is available: ```swift depthSession.$result .compactMap { state -> DepthResult? in if case .success(let result) = state { return result } else { return nil } } .receive(on: DispatchQueue.main) .sink { result in let cpuImage = result.image // Create or update your Metal texture from cpuImage } .store(in: &cancellables) ``` The CPU image contains *float32* values, so the Metal texture is created with the *r32Float* pixel format: ```swift let descriptor = MTLTextureDescriptor.texture2DDescriptor( pixelFormat: .r32Float, width: width, height: height, mipmapped: false ) descriptor.usage = [.shaderRead] descriptor.storageMode = .shared depthTexture = device.makeTexture(descriptor: descriptor) ``` The replace(`region:mipmapLevel:withBytes:bytesPerRow:`) call uploads the depth buffer directly to the GPU without any conversion. ```swift let pixelCount = Int(cpuImage.width * cpuImage.height) let bytesPerRow = cpuImage.width * MemoryLayout.size let floatPtr = cpuImage.data.bindMemory(to: Float.self, capacity: pixelCount) let region = MTLRegionMake2D(0, 0, cpuImage.width, cpuImage.height) depthTexture.replace(region: region, mipmapLevel: 0, withBytes: floatPtr, bytesPerRow: bytesPerRow) ``` --- ## 2. Displaying the Metal Texture To display the Metal texture, you can either: - Follow Apple's [Using Metal to Draw a View's Contents](https://developer.apple.com/documentation/metal/using-metal-to-draw-a-view%27s-contents) tutorial to build a custom MTKView rendering pipeline, or; - Use the TextureView class included in the sample -- a convenience wrapper around MTKView that handles fullscreen rendering, blending, and shader setup for you. ### Using TextureView TextureView simplifies rendering by automatically configuring a fullscreen quad, setting up the Metal pipeline, and drawing the assigned texture each frame. You can inject the vertex and fragment shaders you want it to use when creating the view ```swift imageView = TextureView( frame: view.bounds, vertexShader: "depthVertexShader", fragmentShader: "depthFragmentShader" ) ``` Once your view is created, assign the Metal texture each frame to update the display: ```swift if let texture = depthTexture { imageView.setTexture(copyFrom: texture) } ``` --- ## 3. Understanding the Depth Display Shader The **depth shaders included in the sample project** convert depth values into color using a hue-based gradient and ensure that the image is correctly oriented and scaled on-screen. ### Converting Depth to Color The fragment shader visualizes depth values by mapping them to colors in the **HSV color space** -- a common way to represent hue (color), saturation, and brightness. Nearby surfaces appear in warm colors (red-yellow), while distant ones shift toward cool colors (green-blue). The helper function `HSVtoRGB` performs this conversion: ```swift static inline float3 HSVtoRGB(float3 hsv) { float4 K = float4(1.0, 2.0/3.0, 1.0/3.0, 3.0); float3 P = abs(fract(hsv.xxx + K.xyz) * 6.0 - K.www); return hsv.z * mix(K.xxx, saturate(P - K.xxx), hsv.y); } ``` This function provides a compact, GPU-efficient way to compute smooth transitions between hues -- derived from the standard GLSL HSV-to-RGB formula. In the fragment shader, the hue is computed by normalizing the metric depth value to a 0-1 range: ```swift float lerpFactor = (depth - _MinDistance) / (_MaxDistance - _MinDistance); float hue = mix(0.70, -0.15, saturate(lerpFactor)); if (hue < 0.0) hue += 1.0; ``` This linearly maps nearby pixels (smaller depth) toward red and distant pixels toward blue, producing an intuitive visualization of scene geometry. --- ### Mapping Image Coordinates to the Viewport In the vertex shader, each vertex carries both a screen position and a texture coordinate (texCoord). Before sampling, these coordinates are transformed by a **3x3 display matrix** (uvTransform) supplied through the shader's uniform buffer: ```swift float3 uvh = float3(in.texCoord, 1.0); out.texCoord = uniforms.uvTransform * uvh; ``` This transform aligns the depth texture to the current viewport resolution and orientation. It compensates for rotation, aspect ratio differences, or camera alignment, ensuring the displayed depth image matches the underlying AR camera view.