# How to Set Up Real-World Occlusion Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/setup_real_world_occlusion/?platform=kotlin On Android, Niantic Spatial depth can be used to occlude virtual content in several ways, depending on how much control you need over rendering and synchronization. This guide explains one specific technique: **occlusion using a full-screen depth material** rendered with Filament. ## Prerequisites - A Kotlin or Compose app configured with the Niantic Spatial SDK (NSDK) for Android. - Access to `NSDKSession` so you can acquire a `DepthSession`. - A rendering layer (Filament, OpenGL, etc.) where you can draw depth information for occlusion. ## Overview In this sample, real-world occlusion is achieved by: 1. Creating the depth material and applying it to the **ARCameraStream** 2. Registering for frame updates from the **NSDKSessionManager's ARSessionManager** 3. Collecting depth updates from the `DepthSession` and uploading each new depth buffer to the depth texture 4. Drawing virtual content afterward so it is automatically occluded ## Setting up the Depth Buffer Occlusion in Kotlin starts with creating a depth buffer from the NSDK `DepthSession`. To do this first obtain the instance of the `DepthSession` from the `NSDKSessionManager` and configure it for use: ```kotlin private val engine: Engine private val session: depthSession = nsdkSessionManager.session.depthSession.acquire() private val cameraStream: ARCameraStream? private val materialLoader: MaterialLoader private var viewportSize : Size private var _depthTexture: Texture? = null private var _nsdkMaterial: Material? = null depthSession.configure( DepthConfig( framerate = 30, featureMode = AwarenessFeatureMode.UNSPECIFIED ) ) ``` Once configured, start the session and collect its `depthUpdates` flow. The flow polls the session on its own schedule while it has a collector and emits each new depth buffer once, so store the latest buffer for the frame listener to use: ```kotlin fun startDepth() { if (isRunning) return depthSession.start() isRunning = true pollJob = coroutineScope.launch { depthSession.depthUpdates.collect { result -> when (result) { is NSDKResult.Success -> currentDepthBuffer = result.value is NSDKResult.Error -> Log.w(_TAG, "No depth frame received: ${result.code}") } } } } fun stopDepth() { if (!isRunning) return pollJob?.cancel() pollJob = null depthSession.stop() isRunning = false currentDepthBuffer = null } ``` Errors are emitted once per distinct code, for example `NOT_READY` while depth estimation initializes. Cancelling the collecting job is what ends the polling, so cancel it before stopping the session. Do not poll `latestDepth()` in a loop: every call copies the depth image into a new `FloatArray`. ## Creating the CameraStream and Listening for Depth Buffer Updates To use the depth data received from the `DepthSession`, set up an `ARCameraStream` that will contain the material the depth texture will write to. Create a depth material using the NSDK depth material (found within the [NSDK Sample Project](https://www.nianticspatial.com/docs/nsdk/sample_projects/): "materials/camera_stream_live_depth.filamat"), and begin listening for frame updates: ```kotlin init { nsdkSessionManager.arManager.addFrameUpdateListener(this) cameraStream?.let { stream -> _originalMaterialInstances = stream.materialInstances val buffer = context.assets.open(_DEPTH_MATERIAL).use { ByteBuffer.wrap(it.readBytes()) } _nsdkMaterial = materialLoader.createMaterial(buffer).apply { defaultInstance.apply { setParameter(kUVTransformParameter, Transform()) setExternalTexture(kCameraTextureParameter, stream.cameraTexture) } } stream.setMaterialInstances(_nsdkMaterial!!.defaultInstance) materialInstance = _nsdkMaterial!!.defaultInstance } } ``` ## Updating the Depth Texture Each frame, the depth texture received from the `Depth Session` will need to be transformed to align with the current camera orientation. The frame update listener checks if the `depthBuffer` has changed since the last timestamp, and if so process the depth texture so it can be updated within the `CameraStream`: ```kotlin fun onFrameUpdate(frame: Frame) { val mi = materialInstance ?: return val depthBuffer = currentDepthBuffer ?: return val currentViewportSize = viewportSize ?: return if (depthBuffer.timestampMs != _lastDepthTimestamp) { _updateDepthTexture(depthBuffer, mi) _lastDepthTimestamp = depthBuffer.timestampMs } _updateUvTransform(depthBuffer, frame, currentViewportSize, mi) } ``` To update the depth texture, you first need to get the latest pixel data from the buffer, and update the material: ```kotlin /** * Allocates or reuses a Filament R16F texture and uploads the latest depth * image from [depthBuffer]. Recreates the texture if dimensions have changed. */ private fun _updateDepthTexture(depthBuffer: DepthBuffer, mi: MaterialInstance) { try { val width = depthBuffer.imageWidth val height = depthBuffer.imageHeight if (_depthTexture == null || _depthTexture?.getWidth(0) != width || _depthTexture?.getHeight(0) != height) { _depthTexture?.let { engine.safeDestroyTexture(it) } _depthTexture = Texture.Builder() .width(width).height(height) .sampler(Texture.Sampler.SAMPLER_2D) .format(Texture.InternalFormat.R16F) .levels(1) .build(engine) mi.setParameter("depthTexture", _depthTexture!!, _depthTextureSampler) } val byteBuffer = ByteBuffer.allocateDirect(depthBuffer.image.size * Float.SIZE_BYTES) .order(ByteOrder.nativeOrder()) byteBuffer.asFloatBuffer().put(depthBuffer.image) byteBuffer.rewind() _depthTexture?.setImage(engine, 0, Texture.PixelBufferDescriptor(byteBuffer, Texture.Format.R, Texture.Type.FLOAT)) } catch (e: Exception) { Log.e(_TAG, "Failed to update depth texture: ${e.message}", e) } } ``` Additionally, the data needs to be transformed to match the current display: ```kotlin /** * Computes the combined display + reprojection UV transform and writes it * to the [MaterialInstance] as a mat3 parameter. * * Two corrections are applied each frame: * - Display orientation: the NSDK depth image is in sensor orientation and must * be rotated/flipped to match the current display orientation and viewport aspect ratio. * - Temporal reprojection: NSDK depth runs asynchronously and lags behind the * AR frame. The depth was captured from a slightly different camera pose, so * each screen pixel must be remapped to where it fell in the depth image at * capture time. Without this, fast motion causes depth to drift against the scene. */ private fun _updateUvTransform(depthBuffer: DepthBuffer, arFrame: Frame, viewportSize: Size, mi: MaterialInstance) { try { val imageSize = depthBuffer.intrinsics.getImageDimensions() val display = ImageMath.displayTransform( orientation = nsdkSessionManager.arManager.lastImageOrientation, viewportSize = viewportSize, imageSize = Size(imageSize[0], imageSize[1]) ) * ImageMath.affineInvertVertical() val reprojection = _calculateReprojection(depthBuffer, arFrame) val uvTransform = Matrix() (display * reprojection).invert(uvTransform) uvTransform.getValues(_uvTransformMatrix) mi.setParameter("depthUvTransform", MaterialInstance.FloatElement.MAT3, _uvTransformMatrix, 0, 1) } catch (e: Exception) { Log.e(_TAG, "Failed to update UV transform: ${e.message}", e) } } /** * Computes the perspective reprojection matrix from the depth capture pose * to the current camera pose, accounting for the depth intrinsics. * * Because NSDK depth is captured asynchronously, [depthBuffer.pose] and the * current [arFrame] camera pose will differ whenever the device has moved since * the last depth capture. This matrix re-aligns depth samples to the current * viewpoint so occlusion edges remain correctly placed under motion. */ private fun _calculateReprojection(depthBuffer: DepthBuffer, arFrame: Frame): Matrix { val reference = depthBuffer.pose.inverse() val target = arFrame.camera.pose.inverse() reference.toMatrix(_referenceView, 0) target.toMatrix(_targetView, 0) val imageSize = depthBuffer.intrinsics.getImageDimensions() val focalLengthY = depthBuffer.intrinsics.getFocalLength()[1] return ImageMath.reprojection( aspect = imageSize[0].toFloat() / imageSize[1].toFloat(), fovRadians = 2f * atan(imageSize[1].toFloat() / (2f * focalLengthY)), zNear = 0.2f, zFar = 100f, referenceView = _referenceView, targetView = _targetView ) } ``` The end result of this process will be a depth buffer aligned with the current camera orientation. Any virtual 3d geometry drawn afterwards will be automatically occluded based on the depth buffer information. For full implementation details, see the Kotlin Occlusion Sample in the [NSDK Sample Project](https://www.nianticspatial.com/docs/nsdk/sample_projects/). For more information, see the [Occlusion Feature page](https://www.nianticspatial.com/docs/nsdk/features/occlusion/) ## More Information For more information, see the [Occlusion Feature page](https://www.nianticspatial.com/docs/nsdk/features/occlusion/) and [Adding Occlusion to Your Project](https://www.nianticspatial.com/docs/nsdk/how-to/ar/adding_occlusion/).