# How to Query Scene Segmentation and Highlight Semantic Channels Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/query_semantics_real_objects/?platform=swift This how-to covers: - Querying scene segmentation to detect what is on screen - Select a semantic channel to highlight in AR ## Prerequisites You will need an Xcode project with NSDK installed and `BaseARViewController` set up. For more information, see [Set up the NSDK in Swift](https://www.nianticspatial.com/docs/nsdk/setup/#set-up-the-nsdk-in-swift). ## Setting Up Scene Segmentation ### Create a Feature Manager To set up scene segmentation, begin by creating a new script called SemanticsManager. This class will start the scene segmentation feature and read semantics data from the feature for rendering in AR. ```swift import Foundation import NSDK import ARKit import Metal final class SemanticsManager { let session: NsdkSemanticsSession // GPU resources private let device: MTLDevice private var confidenceTexture: MTLTexture? // Helpers private var cachedTextureWidth: Int = 0 private var cachedTextureHeight: Int = 0 private var lastFrameIndex: UInt64 = 0 init(nsdk: NsdkSession) { device = MTLCreateSystemDefaultDevice()! session = nsdk.createSemanticsSession() } func start() { let config = NsdkSemanticsSession.Configuration() do { try session.configure(with: config) } catch { print("Error: failed to configure session - \(error)") return } session.start() } func stop() { session.stop() } ``` ### Get the Supported Semantic Channels Add a method to get the list of supported semantic channels. ```swift // Retrieves the list of available semantic segmentation channel names from the model. // Returns: An array of classification labels (`String`) if available, otherwise `nil`. func channelNames() -> [String]? { let resultState = session.channelNames() switch resultState { case .inProgress(nil): return nil case .failure(let error): print("NSDK Semantics Error: \(error)") return nil case .success(let names): return names default: print("Unexpected state in semantics channel name retrieval: \(resultState)") return nil } } ``` ### Read the Confidence of One Channel Add a method to get the confidence map for a specific semantic channel. If the data is available, it will be passed to a helper function to be turned into a Metal texture. The current device pose is probably different than the one used to create the most recent semantics data, so this function also returns a transform to reproject the buffer for the latest device pose. This function is designed to be called for every render frame; if no new semantics data is available, only the reprojection matrix will be updated, and `isNew` will return false. ```swift // Retrieves the latest confidence map for the specified channel as a Metal texture and computes a // 3x3 homography matrix to reproject pixels into the coordinate system of the given camera pose. // Validates that the semantics feature and latest buffer are available, otherwise returns nil. // Reuses the existing texture unless a new semantics frame was received, in which case it updates it. // Returns: A tuple containing the confidence `MTLTexture` (if available), // and the reprojection `matrix_float3x3` to the target camera pose. func getTextureAndReprojection(for channelIndex: Int, using cameraPose: matrix_float4x4) -> (texture: MTLTexture?, reprojection: matrix_float3x3?, isNew: Bool) { let featureStatus = session.featureStatus() if (!featureStatus.isOk()) { print("Semantics Feature Status Error: \(featureStatus)") return (nil, nil, false) } // Throws error if channelIndex is out of bounds. Here, we assume it's not. let resultState = try! session.latestConfidence(channelIndex: channelIndex) switch resultState { case .inProgress(nil): return (nil, nil, false) case .failure(let error): print("NSDK Semantics Error: \(error)") return (nil, nil, false) case .success(let buffer): // Calculate the reprojection transform let reprojection = SemanticsManager.calculateReprojection(for: buffer, to: cameraPose) // Determine if the texture needs to be updated let wasSemanticsUpdated = buffer.frameId != lastFrameIndex lastFrameIndex = buffer.frameId // Acquire the latest image let texture = wasSemanticsUpdated ? createOrUpdateTexture(from: buffer.image) : confidenceTexture return (texture, reprojection, wasSemanticsUpdated) default: print("Unexpected state in semantics texture and reprojection retrieval: \(resultState)") return (nil, nil, false) } } ``` ### Add Helper Functions Now, add a helper function to turn the raw semantics confidence data buffer into a Metal texture that we can render. ```swift // Creates or updates a Metal texture from a semantic segmentation confidence map `RawImage` with `Float32` pixel data. // Returns the updated texture, or `nil` if creation/upload fails. private func createOrUpdateTexture(from rawImage: RawImage?) -> MTLTexture? { guard let image = rawImage else { print("Cannot update the confidence texture from the provided image (nil).") return nil } guard image.type == .semanticsConfidence else { print("Unsupported image type.") return nil } let width = Int(image.width) let height = Int(image.height) // Only recreate texture if size changed if confidenceTexture == nil || cachedTextureWidth != width || cachedTextureHeight != height { let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor( pixelFormat: .r32Float, width: width, height: height, mipmapped: false ) textureDescriptor.usage = [.shaderRead] textureDescriptor.storageMode = .shared guard let texture = device.makeTexture(descriptor: textureDescriptor) else { print("Failed to create Metal texture") return nil } confidenceTexture = texture cachedTextureWidth = width cachedTextureHeight = height } guard let texture = confidenceTexture else { print("No confidence texture available") return nil } // Upload raw float data directly to GPU - no CPU conversion needed! let pixelCount = Int(image.width * image.height) let floatPtr = image.data.bindMemory(to: Float.self, capacity: pixelCount) let region = MTLRegionMake2D(0, 0, width, height) let bytesPerRow = width * MemoryLayout.size // 4 bytes per pixel for R32Float texture.replace(region: region, mipmapLevel: 0, withBytes: floatPtr, bytesPerRow: bytesPerRow) return texture } ``` To finish off the class, add a helper function to calculate a reprojection matrix from the device pose of the original semantics frame to the current pose. ```swift /// Computes a 3x3 homography that reprojects the provided semantics image into the coordinate frame /// of the given camera pose. /// - Returns: A 3x3 homography matrix, or `nil` if the computation fails. private static func calculateReprojection(for buffer: SemanticsResult?, to cameraPose: matrix_float4x4) -> matrix_float3x3? { guard let buffer = buffer, let image = buffer.image else { return nil } // Projection params let aspect = Float(image.width) / Float(image.height) let focalLength = buffer.intrinsics.columns.1.y let fovRadians = 2.0 * atan(Float(image.height) / (2.0 * focalLength)) let zNear: Float = 0.2 let zFar: Float = 100.0 // Reference and target view let reference = buffer.pose.inverse let target = cameraPose.inverse return ImageMath.reprojection( aspect: aspect, fovRadians: fovRadians, zNear: zNear, zFar: zFar, referenceView: reference, targetView: target, backProjectionDistance: 0.9 ) } } ``` #### Click to reveal the full `SemanticsManager` script ```swift import Foundation import NSDK import ARKit import Metal final class SemanticsManager { let session: NsdkSemanticsSession // GPU resources private let device: MTLDevice private var confidenceTexture: MTLTexture? // Helpers private var cachedTextureWidth: Int = 0 private var cachedTextureHeight: Int = 0 private var lastFrameIndex: UInt64 = 0 init(nsdk: NsdkSession) { device = MTLCreateSystemDefaultDevice()! session = nsdk.createSemanticsSession() } func start() { let config = NsdkSemanticsSession.Configuration() do { try session.configure(with: config) } catch { print("Error: failed to configure session - \(error)") return } session.start() } func stop() { session.stop() } // Retrieves the list of available semantic segmentation channel names from the model. // Returns: An array of classification labels (`String`) if available, otherwise `nil`. func channelNames() -> [String]? { let resultState = session.channelNames() switch resultState { case .inProgress(nil): return nil case .failure(let error): print("NSDK Semantics Error: \(error)") return nil case .success(let names): return names default: print("Unexpected state in semantics channel name retrieval: \(resultState)") return nil } } // Retrieves the latest confidence map for the specified channel as a Metal texture and computes a // 3x3 homography matrix to reproject pixels into the coordinate system of the given camera pose. // Validates that the semantics feature and latest buffer are available, otherwise returns nil. // Reuses the existing texture unless a new semantics frame was received, in which case it updates it. // Returns: A tuple containing the confidence `MTLTexture` (if available), // and the reprojection `matrix_float3x3` to the target camera pose. func getTextureAndReprojection(for channelIndex: Int, using cameraPose: matrix_float4x4) -> (texture: MTLTexture?, reprojection: matrix_float3x3?, isNew: Bool) { let featureStatus = session.featureStatus() if (!featureStatus.isOk()) { print("Semantics Feature Status Error: \(featureStatus)") return (nil, nil, false) } // Throws error if channelIndex is out of bounds. Here, we assume it's not. let resultState = try! session.latestConfidence(channelIndex: channelIndex) switch resultState { case .inProgress(nil): return (nil, nil, false) case .failure(let error): print("NSDK Semantics Error: \(error)") return (nil, nil, false) case .success(let buffer): // Calculate the reprojection transform let reprojection = SemanticsManager.calculateReprojection(for: buffer, to: cameraPose) // Determine if the texture needs to be updated let wasSemanticsUpdated = buffer.frameId != lastFrameIndex lastFrameIndex = buffer.frameId // Acquire the latest image let texture = wasSemanticsUpdated ? createOrUpdateTexture(from: buffer.image) : confidenceTexture return (texture, reprojection, wasSemanticsUpdated) default: print("Unexpected state in semantics texture and reprojection retrieval: \(resultState)") return (nil, nil, false) } } // Creates or updates a Metal texture from a semantic segmentation confidence map `RawImage` with `Float32` pixel data. // Returns the updated texture, or `nil` if creation/upload fails. private func createOrUpdateTexture(from rawImage: RawImage?) -> MTLTexture? { guard let image = rawImage else { print("Cannot update the confidence texture from the provided image (nil).") return nil } guard image.type == .semanticsConfidence else { print("Unsupported image type.") return nil } let width = Int(image.width) let height = Int(image.height) // Only recreate texture if size changed if confidenceTexture == nil || cachedTextureWidth != width || cachedTextureHeight != height { let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor( pixelFormat: .r32Float, width: width, height: height, mipmapped: false ) textureDescriptor.usage = [.shaderRead] textureDescriptor.storageMode = .shared guard let texture = device.makeTexture(descriptor: textureDescriptor) else { print("Failed to create Metal texture") return nil } confidenceTexture = texture cachedTextureWidth = width cachedTextureHeight = height } guard let texture = confidenceTexture else { print("No confidence texture available") return nil } // Upload raw float data directly to GPU - no CPU conversion needed! let pixelCount = Int(image.width * image.height) let floatPtr = image.data.bindMemory(to: Float.self, capacity: pixelCount) let region = MTLRegionMake2D(0, 0, width, height) let bytesPerRow = width * MemoryLayout.size // 4 bytes per pixel for R32Float texture.replace(region: region, mipmapLevel: 0, withBytes: floatPtr, bytesPerRow: bytesPerRow) return texture } /// Computes a 3x3 homography that reprojects the provided semantics image into the coordinate frame /// of the given camera pose. /// - Returns: A 3x3 homography matrix, or `nil` if the computation fails. private static func calculateReprojection(for buffer: SemanticsResult?, to cameraPose: matrix_float4x4) -> matrix_float3x3? { guard let buffer = buffer, let image = buffer.image else { return nil } // Projection params let aspect = Float(image.width) / Float(image.height) let focalLength = buffer.intrinsics.columns.1.y let fovRadians = 2.0 * atan(Float(image.height) / (2.0 * focalLength)) let zNear: Float = 0.2 let zFar: Float = 100.0 // Reference and target view let reference = buffer.pose.inverse let target = cameraPose.inverse return ImageMath.reprojection( aspect: aspect, fovRadians: fovRadians, zNear: zNear, zFar: zFar, referenceView: reference, targetView: target, backProjectionDistance: 0.9 ) } } ``` ### Add a View Controller In a new file called **SemanticsViewController.swift**, add a view controller that uses the manager to overlay the reprojected semantic channel detection. This class depends on the `BaseARViewController` class in [Set up the NSDK in Swift](https://www.nianticspatial.com/docs/nsdk/setup/#set-up-the-nsdk-in-swift). #### Click to reveal the full `SemanticsViewController` script ```swift import UIKit import ARKit import RealityKit import NSDK import Metal class SemanticsViewController: BaseARViewController, UIPickerViewDataSource, UIPickerViewDelegate { // Image supplier private var manager: SemanticsManager? // Toggle image button private var showImageButton: UIButton? // Image display private var imageView: TextureView! // UI components private var transparencySlider: UISlider = UISlider() private var sliderLabel: UILabel = UILabel() private var channelPicker: UIPickerView = UIPickerView() private var channelButton: UIButton = UIButton() // State variables and helpers private var isChannelPickerVisible: Bool = false private var channels: [String] = [] private var selectedChannelIndex: Int = 1 // Default to channel 1 (ground) private var areChannelsLoaded: Bool = false override func viewDidLoad() { super.viewDidLoad() // Start semantics inference manager = SemanticsManager(nsdk: nsdkManager!.session) manager?.start() } override func setupUI() { super.setupUI() self.title = "Semantics" helpLabel.text = "Semantics Sample Help\n\n This sample uses our semantic feature and a shader to represent them coloring pink where found and blue where not.\n\n TO USE: \n select a semantic channel from the drop down menu, and set the transparency to see the color highlight." self.showImageButton = addButton(buttonTitle: "Hide", onClickAction: #selector(handleShowImageTap)) // Set up the image view imageView = TextureView( frame: view.bounds, vertexShader: "semanticVertexShader", fragmentShader: "semanticFragmentShader" ) // Set transparency imageView.opacity = 0.5 imageView.isOpaque = false imageView.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0) imageView.translatesAutoresizingMaskIntoConstraints = false arView.addSubview(imageView) NSLayoutConstraint.activate([ imageView.leadingAnchor.constraint(equalTo: view.leadingAnchor), imageView.topAnchor.constraint(equalTo: view.topAnchor), imageView.trailingAnchor.constraint(equalTo: view.trailingAnchor), imageView.bottomAnchor.constraint(equalTo: view.bottomAnchor) ]) // Observe text changes to show/hide the label sampleInfoLabel.addObserver(self, forKeyPath: "text", options: .new, context: nil) // Set up transparency slider transparencySlider.minimumValue = 0.0 transparencySlider.maximumValue = 1.0 transparencySlider.value = 0.5 // Start at 50% transparency transparencySlider.addTarget(self, action: #selector(transparencySliderChanged), for: .valueChanged) transparencySlider.translatesAutoresizingMaskIntoConstraints = false view.addSubview(transparencySlider) // Set up slider label sliderLabel.text = "Transparency: \(Int(imageView.opacity * Float(100)))%" sliderLabel.textColor = .white sliderLabel.font = .systemFont(ofSize: 12) sliderLabel.textAlignment = .center sliderLabel.translatesAutoresizingMaskIntoConstraints = false view.addSubview(sliderLabel) // Set up channel picker channelPicker.dataSource = self channelPicker.delegate = self channelPicker.backgroundColor = UIColor.black.withAlphaComponent(0.7) channelPicker.layer.cornerRadius = 8 channelPicker.translatesAutoresizingMaskIntoConstraints = false channelPicker.isHidden = true // Initially hidden view.addSubview(channelPicker) // Set up channel button channelButton.setTitle("Loading channels...", for: .normal) channelButton.setTitleColor(.white, for: .normal) channelButton.backgroundColor = .systemGray channelButton.layer.cornerRadius = 8 channelButton.titleLabel?.font = .boldSystemFont(ofSize: 16) channelButton.addTarget(self, action: #selector(toggleChannelPicker), for: .touchUpInside) channelButton.isEnabled = false // Initially disabled channelButton.translatesAutoresizingMaskIntoConstraints = false view.addSubview(channelButton) NSLayoutConstraint.activate([ // Slider constraints transparencySlider.leadingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.leadingAnchor, constant: 20), transparencySlider.trailingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.trailingAnchor, constant: -20), transparencySlider.bottomAnchor.constraint(equalTo: showImageButton!.topAnchor, constant: -20), // Slider label constraints sliderLabel.centerXAnchor.constraint(equalTo: transparencySlider.centerXAnchor), sliderLabel.bottomAnchor.constraint(equalTo: transparencySlider.topAnchor, constant: -5), // Channel button constraints channelButton.leadingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.leadingAnchor, constant: 20), channelButton.trailingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.trailingAnchor, constant: -20), channelButton.bottomAnchor.constraint(equalTo: transparencySlider.topAnchor, constant: -20), channelButton.heightAnchor.constraint(equalToConstant: 44), // Channel picker constraints (positioned above the button) channelPicker.leadingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.leadingAnchor, constant: 20), channelPicker.trailingAnchor.constraint(equalTo: view.safeAreaLayoutGuide.trailingAnchor, constant: -20), channelPicker.bottomAnchor.constraint(equalTo: channelButton.topAnchor, constant: -5), channelPicker.heightAnchor.constraint(equalToConstant: 120), ]) } override func viewWillDisappear(_ animated: Bool) { super.viewWillDisappear(animated) manager?.stop() // Remove observer to prevent memory leaks sampleInfoLabel.removeObserver(self, forKeyPath: "text") } override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) { if keyPath == "text", let label = object as? UILabel, label == sampleInfoLabel { DispatchQueue.main.async { // Hide the label if text is empty or nil, show if there's text self.sampleInfoLabel.isHidden = (self.sampleInfoLabel.text?.isEmpty ?? true) } } } override func session(_ session: ARSession, didUpdate frame: ARFrame) { super.session(session, didUpdate: frame) guard let manager = manager else { return } // Check whether the channel names (classifications) are loaded if !areChannelsLoaded { // Try to get channel names from the semantics session guard let channelNames = manager.channelNames() else { updateInfoLabel(text: "Loading semantic segmentation labels...") return } // Update UI with the list of channel names DispatchQueue.main.async { self.updateChannelsWithNames(channelNames) } updateInfoLabel(text: "") areChannelsLoaded = true } // Skip acquiring the texture if the view is disabled guard !imageView.isHidden else { return } // Acquire the new texture let (texture, reprojection, isNewImage) = manager.getTextureAndReprojection(for: selectedChannelIndex, using: frame.camera.transform) // Update the view using the texture if isNewImage, let texture = texture { imageView.setTexture(copyFrom: texture) } // Optional: Reproject the image to the current devcie pose imageView.setReprojection(reprojection ?? matrix_identity_float3x3) } private func updateChannelsWithNames(_ channelNames: [String]) { // Format channel names: capitalize and remove underscores channels = channelNames.map { channelName in return channelName .replacingOccurrences(of: "_", with: " ") .capitalized } // Update UI channelButton.isEnabled = true channelButton.backgroundColor = .systemBlue // Set default selection (try to find "ground" or use first channel) if let groundIndex = channels.firstIndex(where: { $0.lowercased().contains("ground") }) { selectedChannelIndex = groundIndex } else { selectedChannelIndex = 0 } // Update button title let selectedChannel = channels[selectedChannelIndex] channelButton.setTitle("Semantic Channel: \(selectedChannel) ▼", for: .normal) // Reload picker channelPicker.reloadAllComponents() channelPicker.selectRow(selectedChannelIndex, inComponent: 0, animated: false) print("Loaded \(channels.count) semantic channels: \(channels)") } // MARK: - UIPickerViewDataSource func numberOfComponents(in pickerView: UIPickerView) -> Int { return 1 } func pickerView(_ pickerView: UIPickerView, numberOfRowsInComponent component: Int) -> Int { return channels.count } // MARK: - UIPickerViewDelegate func pickerView(_ pickerView: UIPickerView, titleForRow row: Int, forComponent component: Int) -> String? { guard row < channels.count else { return "Unknown" } return channels[row] } func pickerView(_ pickerView: UIPickerView, attributedTitleForRow row: Int, forComponent component: Int) -> NSAttributedString? { guard row < channels.count else { return NSAttributedString(string: "Unknown", attributes: [.foregroundColor: UIColor.white]) } return NSAttributedString(string: channels[row], attributes: [.foregroundColor: UIColor.white]) } func pickerView(_ pickerView: UIPickerView, didSelectRow row: Int, inComponent component: Int) { guard row < channels.count else { return } selectedChannelIndex = row let selectedChannel = channels[row] channelButton.setTitle("Semantic Channel: \(selectedChannel) ▼", for: .normal) print("Selected semantic channel: \(selectedChannel) (index: \(row))") } @objc private func transparencySliderChanged(_ sender: UISlider) { imageView.opacity = sender.value let percentage = Int(sender.value * 100) sliderLabel.text = "Transparency: \(percentage)%" } @objc private func handleShowImageTap() { imageView.reset() imageView.isHidden = !imageView.isHidden showImageButton?.setTitle(imageView.isHidden ? "Show" : "Hide", for: .normal) } @objc private func toggleChannelPicker() { guard areChannelsLoaded else { return } isChannelPickerVisible = !isChannelPickerVisible UIView.animate(withDuration: 0.3) { self.channelPicker.isHidden = !self.isChannelPickerVisible self.channelButton.alpha = self.isChannelPickerVisible ? 0.7 : 1.0 } // Update button title to show expand/collapse state guard selectedChannelIndex < channels.count else { return } let currentChannel = channels[selectedChannelIndex] let arrow = isChannelPickerVisible ? "▲" : "▼" channelButton.setTitle("Semantic Channel: \(currentChannel) \(arrow)", for: .normal) } } ``` ### Try it Out Launch the view controller in your app and move the camera around. After the feature starts, the list should populate with NSDK semantic channels. When you select a semantic channel from the list, it should be highlighted in AR according to the confidence of its presence in the camera view.