# How to Query Scene Segmentation and Highlight Semantic Channels Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/query_semantics_real_objects/?platform=kotlin ## Prerequisites - A Kotlin or Compose app configured with the Niantic Spatial SDK (NSDK) for Android. - Access to `NSDKSession` so you can acquire a `SemanticsSession`. - (Optional) A rendering layer (Filament, OpenGL, etc.) where you can draw confidence data returned by semantics. In the sections below we will build a manager that owns the semantics session and walk through the math needed to project the confidence map onto the current camera feed. Feel free to adapt each piece to match your architecture. (image: Kotlin semantics overlay with adjustable opacity) ## Build a Semantics Manager The manager encapsulates the `SemanticsSession` lifecycle, polls for channel metadata, emits confidence buffers, and surfaces warnings. The implementation below follows the same structure as the Swift sample: a `start()` function configures the session, channel polling runs until metadata is available, and a second coroutine polls `latestConfidence()` for the active channel. ### Session Lifecycle Spin up coroutines, expose immutable `StateFlow`s, and keep track of jobs/warnings without tying yourself to a specific UI or lifecycle base class. ```kotlin class SemanticsManager( nsdkSessionManager: NSDKSessionManager, parentScope: CoroutineScope? = null, ) { private val semanticsSession = nsdkSessionManager.session.semantics.acquire() private val scope = parentScope?.let { CoroutineScope(it.coroutineContext + SupervisorJob()) } ?: CoroutineScope(Dispatchers.Main.immediate + SupervisorJob()) private val _channels = MutableStateFlow>(emptyList()) val channels: StateFlow> = _channels.asStateFlow() private val _selectedChannelIndex = MutableStateFlow(-1) val selectedChannelIndex: StateFlow = _selectedChannelIndex.asStateFlow() private val _latestResult = MutableStateFlow(null) val latestResult: StateFlow = _latestResult.asStateFlow() ``` ### Configure and Start the Session Initialize the semantics feature with your desired frame rate, start it on a worker thread, then kick off channel discovery. ```kotlin suspend fun start() { if (_sessionState.value is SemanticsSessionState.Streaming) return _sessionState.value = SemanticsSessionState.LoadingChannels withContext(Dispatchers.Default) { val config = SemanticsConfig().apply { frameRate = 20 mode = AwarenessFeatureMode.UNSPECIFIED } semanticsSession.configure(config) semanticsSession.start() } // Poll channels on start pollChannels() } ``` ### Poll for Channel Metadata Retry `channelNames()` until the model reports labels, choose a default channel (for example `ground`), and transition the manager into a streaming state. ```kotlin private fun pollChannels() { channelJob = scope.launch(Dispatchers.Default) { while (isActive && isSessionRunning()) { val result = semanticsSession.channelNames() when (result) { is NSDKResult.Success -> { val names = result.value.toList() if (names.isNotEmpty()) { _channels.value = names _selectedChannelIndex.value = names.indexOfFirst { it.contains("ground", ignoreCase = true) } .takeIf { it >= 0 } ?: 0 updateStreamingState() startResultPolling() return@launch } } is NSDKResult.Error -> emitWarning(SemanticsWarningEvent.ChannelQueryError(result.code)) } delay(CHANNEL_POLL_DELAY_MS) } } } ``` ### Poll for Confidence Buffers Fetch `latestConfidence(channelIndex)` every frame (or at your chosen cadence), publish only newer timestamps, and surface errors through the warning flow. ```kotlin private fun startResultPolling() { resultJob = scope.launch(Dispatchers.Default) { var lastTimestamp = 0L while (isActive && isSessionRunning()) { val index = _selectedChannelIndex.value if (index >= 0) { when (val result = semanticsSession.latestConfidence(index)) { is NSDKResult.Success -> { val buffer = result.value if (buffer.timestampMs > lastTimestamp) { lastTimestamp = buffer.timestampMs _latestResult.value = buffer } } is NSDKResult.Error -> emitWarning( SemanticsWarningEvent.LatestConfidenceResultError(index, result.code) ) } } delay(RESULT_POLL_DELAY_MS) } } } ``` This manager intentionally polls the SDK because semantics does not push updates on its own. Use warning events to inform the UI when metadata or buffers are unavailable. #### Click to reveal the full `SemanticsManager` implementation ```kotlin import android.util.Log import androidx.lifecycle.LifecycleOwner import com.nianticspatial.nsdk.NSDKResult import com.nianticspatial.nsdk.AwarenessFeatureMode import com.nianticspatial.nsdk.AwarenessStatus import com.nianticspatial.nsdk.awareness.semantics.SemanticsConfig import com.nianticspatial.nsdk.awareness.semantics.SemanticsResult import com.nianticspatial.nsdk.awareness.semantics.SemanticsSession import com.nianticspatial.nsdk.externalsamples.NSDKSessionManager import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.Job import kotlinx.coroutines.SupervisorJob import kotlinx.coroutines.cancel import kotlinx.coroutines.delay import kotlinx.coroutines.flow.MutableSharedFlow import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.SharedFlow import kotlinx.coroutines.flow.StateFlow import kotlinx.coroutines.flow.asSharedFlow import kotlinx.coroutines.flow.asStateFlow import kotlinx.coroutines.isActive import kotlinx.coroutines.launch import kotlinx.coroutines.withContext sealed interface SemanticsWarningEvent { data class ChannelQueryFailed(val cause: Throwable) : SemanticsWarningEvent data class ChannelQueryError(val status: AwarenessStatus) : SemanticsWarningEvent data object ChannelsUnavailable : SemanticsWarningEvent data class LatestConfidenceQueryFailed(val channelIndex: Int, val cause: Throwable) : SemanticsWarningEvent data class LatestConfidenceResultError(val channelIndex: Int, val status: AwarenessStatus) : SemanticsWarningEvent data class StopFailed(val cause: Throwable) : SemanticsWarningEvent data object Cleared : SemanticsWarningEvent } sealed interface SemanticsSessionState { data object Idle : SemanticsSessionState data object LoadingChannels : SemanticsSessionState data class Streaming(val channelName: String?) : SemanticsSessionState data object Stopping : SemanticsSessionState data class Failed(val cause: Throwable?) : SemanticsSessionState } class SemanticsManager( nsdkSessionManager: NSDKSessionManager, parentScope: CoroutineScope? = null, ) { companion object { private const val TAG = "SemanticsManager" private const val CHANNEL_POLL_DELAY_MS = 100L private const val RESULT_POLL_DELAY_MS = 16L private const val MAX_CHANNEL_POLLS = 1000 private const val DEFAULT_ALPHA = 0.5f } private val semanticsSession: SemanticsSession = nsdkSessionManager.session.semantics.acquire() private val scope: CoroutineScope = parentScope?.let { parent -> val parentJob = parent.coroutineContext[Job] CoroutineScope(parent.coroutineContext + SupervisorJob(parentJob)) } ?: CoroutineScope(Dispatchers.Main.immediate + SupervisorJob()) private var channelJob: Job? = null private var resultJob: Job? = null private val _channels = MutableStateFlow>(emptyList()) val channels: StateFlow> = _channels.asStateFlow() private val _selectedChannelIndex = MutableStateFlow(-1) val selectedChannelIndex: StateFlow = _selectedChannelIndex.asStateFlow() private val _sessionState = MutableStateFlow(SemanticsSessionState.Idle) val sessionState: StateFlow = _sessionState.asStateFlow() private val _warnings = MutableSharedFlow(replay = 0) val warnings: SharedFlow = _warnings.asSharedFlow() private val _latestResult = MutableStateFlow(null) val latestResult: StateFlow = _latestResult.asStateFlow() private val _overlayOpacity = MutableStateFlow(DEFAULT_ALPHA) val overlayOpacity: StateFlow = _overlayOpacity.asStateFlow() private var isPaused = false suspend fun start() { val current = _sessionState.value if ( current is SemanticsSessionState.LoadingChannels || current is SemanticsSessionState.Streaming || current is SemanticsSessionState.Stopping ) { return } _sessionState.value = SemanticsSessionState.LoadingChannels runCatching { configureSession() } .onFailure { error -> Log.e(TAG, "Failed to configure semantics", error) _sessionState.value = SemanticsSessionState.Failed(error) return } runCatching { withContext(Dispatchers.Default) { semanticsSession.start() } }.onFailure { error -> Log.e(TAG, "Failed to start semantics", error) _sessionState.value = SemanticsSessionState.Failed(error) return } pollChannels() } fun stop() { when (_sessionState.value) { is SemanticsSessionState.Idle, is SemanticsSessionState.Failed, SemanticsSessionState.Stopping -> return else -> Unit } _sessionState.value = SemanticsSessionState.Stopping channelJob?.cancel() channelJob = null resultJob?.cancel() resultJob = null runCatching { semanticsSession.stop() }.onFailure { error -> Log.e(TAG, "Failed to stop semantics", error) emitWarning(SemanticsWarningEvent.StopFailed(error)) } _channels.value = emptyList() _selectedChannelIndex.value = -1 _latestResult.value = null _sessionState.value = SemanticsSessionState.Idle } fun selectChannel(index: Int) { if (index == _selectedChannelIndex.value) return if (index < 0 || index >= _channels.value.size) return _selectedChannelIndex.value = index _latestResult.value = null updateStreamingState() } fun setOverlayOpacity(value: Float) { val clamped = value.coerceIn(0f, 1f) _overlayOpacity.value = clamped } override fun onPause(owner: LifecycleOwner) { super.onPause(owner) val running = isSessionRunning() isPaused = running if (running) { stop() } } override fun onResume(owner: LifecycleOwner) { super.onResume(owner) if (isPaused) { scope.launch { start() } isPaused = false } } override fun onDestroy(owner: LifecycleOwner) { super.onDestroy(owner) stop() scope.cancel() runCatching { semanticsSession.close() } .onFailure { error -> Log.e(TAG, "Failed to close semantics session", error) } } private suspend fun configureSession() { withContext(Dispatchers.Default) { val config = SemanticsConfig().apply { frameRate = 20 mode = AwarenessFeatureMode.UNSPECIFIED } semanticsSession.configure(config) } } private fun pollChannels() { channelJob?.cancel() channelJob = scope.launch(Dispatchers.Default) { var polls = 0 while (isActive && isSessionRunning() && polls < MAX_CHANNEL_POLLS) { val result = runCatching { semanticsSession.channelNames() } .onFailure { error -> Log.e(TAG, "Channel query failed", error) emitWarning(SemanticsWarningEvent.ChannelQueryFailed(error)) } .getOrNull() val channelNames = when (result) { is NSDKResult.Success -> result.value.toList() is NSDKResult.Error -> { Log.e(TAG, "Channel query error: ${result.code}") emitWarning(SemanticsWarningEvent.ChannelQueryError(result.code)) emptyList() } else -> emptyList() } if (channelNames.isNotEmpty()) { withContext(Dispatchers.Main.immediate) { _channels.value = channelNames val defaultIndex = channelNames.indexOfFirst { it.contains("ground", ignoreCase = true) } _selectedChannelIndex.value = if (defaultIndex >= 0) defaultIndex else 0 updateStreamingState() } clearWarning() startResultPolling() return@launch } polls++ delay(CHANNEL_POLL_DELAY_MS) } if (_channels.value.isEmpty()) { emitWarning(SemanticsWarningEvent.ChannelsUnavailable) } } } private fun startResultPolling() { resultJob?.cancel() resultJob = scope.launch(Dispatchers.Default) { var lastTimestamp = 0L var previousIndex = -1 while (isActive && isSessionRunning()) { val index = _selectedChannelIndex.value if (index != previousIndex) { previousIndex = index lastTimestamp = 0L } if (index >= 0) { val result = try { semanticsSession.latestConfidence(index) } catch (error: Exception) { Log.e(TAG, "latestConfidence threw", error) emitWarning( SemanticsWarningEvent.LatestConfidenceQueryFailed(index, error) ) delay(RESULT_POLL_DELAY_MS) continue } when (result) { is NSDKResult.Success -> { val confidence = result.value if (confidence.timestampMs > lastTimestamp) { lastTimestamp = confidence.timestampMs withContext(Dispatchers.Main.immediate) { _latestResult.value = confidence } clearWarning() } } is NSDKResult.Error -> { Log.e(TAG, "Semantics error: ${result.code}") emitWarning( SemanticsWarningEvent.LatestConfidenceResultError( index, result.code ) ) } } } delay(RESULT_POLL_DELAY_MS) } } } private fun updateStreamingState() { if (!isSessionRunning()) return val channelName = _channels.value.getOrNull(_selectedChannelIndex.value) _sessionState.value = SemanticsSessionState.Streaming(channelName) } private fun emitWarning(event: SemanticsWarningEvent) { if (!_warnings.tryEmit(event)) { scope.launch { _warnings.emit(event) } } } private fun clearWarning() { emitWarning(SemanticsWarningEvent.Cleared) } private fun isSessionRunning(): Boolean = when (_sessionState.value) { SemanticsSessionState.LoadingChannels, is SemanticsSessionState.Streaming -> true else -> false } } ``` ## Processing the Confidence Map Once the manager begins streaming, you will receive `SemanticsResult` objects that contain both the confidence buffer and the pose/intrinsics that were used when the inference ran. To display those pixels on the current screen you need two transforms: 1. **Reprojection** - compensates for camera motion between the inference timestamp and the current frame. 2. **Display Transform** - adapts the model output resolution to the device viewport (aspect fill, rotation, Y inversion). ```kotlin import android.graphics.Matrix import android.util.Size import androidx.core.graphics.times import com.google.ar.core.Frame import com.nianticspatial.nsdk.Orientation import com.nianticspatial.nsdk.utils.ImageMath import kotlin.math.atan fun updateOverlay( currentFrame: Frame, viewportSize: Size, deviceOrientation: Orientation ) { // 1. Pull the latest semantics result from your manager. val result = semanticsManager.latestResult.value ?: return // 2. Prepare parameters for reprojection. val referenceView = FloatArray(16) val targetView = FloatArray(16) // Calculate the View matrices (inverse of Pose). val reference = result.pose.inverse() val target = currentFrame.camera.pose.inverse() reference.toMatrix(referenceView, 0) target.toMatrix(targetView, 0) // Get dimensions and focal length from intrinsics. val imageSize = result.intrinsics.getImageDimensions() val focalLengthY = result.intrinsics.getFocalLength()[1] // fy // 3. Compute the reprojection matrix: maps past inference pixels to current camera pixels. val reprojectionMatrix = ImageMath.reprojection( aspect = imageSize[0].toFloat() / imageSize[1].toFloat(), fovRadians = 2f * atan(imageSize[1].toFloat() / (2f * focalLengthY)), zNear = 0.2f, zFar = 100.0f, referenceView = referenceView, targetView = targetView ) // 4. Compute the display transform: maps normalized image coordinates to the screen. // We also apply a vertical flip to match the shader's UV coordinate system. val displayMatrix = ImageMath.displayTransform( orientation = deviceOrientation, viewportSize = viewportSize, imageSize = Size(imageSize[0], imageSize[1]) ) * ImageMath.affineInvertVertical() // 5. Combine and invert so the shader can sample the texture correctly. // M_forward = M_display * M_flip * M_reprojection // M_shader = M_forward^-1 val uvTransform = Matrix() (displayMatrix * reprojectionMatrix).invert(uvTransform) // 6. Hand the data to your renderer of choice. renderer.updateConfidenceTexture(result.image) // Note: Depending on your rendering engine, you may need to convert the Android Matrix // to a float array (e.g. using uvTransform.getValues()) or a native matrix type. renderer.setShaderUniform("uvTransform", uvTransform) } ``` ## Why the Confidence map needs these transforms The semantics model runs asynchronously on a separate timeline from the render loop. This means the confidence map you receive corresponds to a camera frame from the past (typically a few milliseconds ago). To overlay this data accurately on top of the current camera feed, we need to account for two main discrepancies: 1. **Camera Movement (Reprojection)**\ As the user moves the device, the camera's position and rotation change. If we simply drew the past confidence map over the current frame, the semantic overlay would appear to "lag" behind the real-world objects. **The Solution:** We compute a **homography matrix** that warps pixels from the *inference pose* (where the camera was when the model ran) to the *current render pose* (where the camera is now). This "reprojects" the semantic data to match the current view. 2. **Screen vs. Texture Coordinates (Display Transform)**\ The confidence map is a texture with its own resolution and aspect ratio (often lower resolution than the screen). The device screen, however, has a different aspect ratio and orientation (Portrait vs. Landscape). **The Solution:** We calculate an affine transform that maps the normalized coordinates of the confidence texture [0, 1] to the screen's viewport. This handles scaling, cropping (aspect fill), and rotation. **Coordinate Systems:** Rendering engines typically use UV coordinates where (0,0) is bottom-left, whereas image processing often assumes top-left. A vertical flip is usually applied here to align them. **Combined Update Logic**\ By multiplying the display transform and the reprojection matrix, and then inverting the result, we create a single `uvTransform` matrix. In your fragment shader, you can use this matrix to look up the correct semantic value for any given screen pixel: `textureCoordinate = uvTransform * screenCoordinate` ## Try it Out 1. Start the semantics manager when your AR session begins. 2. Wait for `channels` to populate, then select the semantic class you want to monitor. 3. Feed the latest `SemanticsResult` into the processing function above and verify that the overlay tracks the scene even as you rotate or move the device.