# Adding Meshing to Your Project Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/meshing/adding_meshing/?platform=swift Niantic Spatial SDK (NSDK) exposes its meshing functionality on native iOS through the **`NsdkMeshingSession`** API. When the application calls `NSDKSession.update()`, an active `NsdkMeshingSession` ingests depth and device pose data and builds a mesh from it. > **Note:** > > Niantic Spatial Meshing works on lidar and non-lidar iOS devices. > For support on lidar devices, ensure that `NSDKSession` is configured with **`useLidar=true`** and that `sceneDepth` is added to `ARWorldTrackingConfiguration.frameSemantics`. `DefaultSessionDataSource` reads `ARFrame.sceneDepth` automatically when available. > See the [Swift sample project](https://www.nianticspatial.com/docs/nsdk/sample_projects/) for a complete example. > > Lidar depth will only support a maximum integration distance of around five meters. > If you wish to generate mesh blocks farther away from the user, disable `useLidar` to use NSDK depth from RGB frames instead. ## Mesh Chunks To lighten the application's compute workload as the mesh grows, NSDK breaks the mesh up into chunks. The three-dimensional world is divided into a regular grid of "blocks" of a fixed size. When meshing is running, new mesh chunks will be created as the user explores their surroundings. Mesh chunks are continually updated as new data is added to the 3D representation. The application should regularly read `NsdkMeshingSession.updatedMeshInfos` to get the list of updated mesh chunks: - **`ids`:** An array of all mesh chunk IDs - **`updated`:** An array indicating whether each chunk was updated. For each corresponding ID in `ids`, this array indicates: - **Non-zero value**: The chunk has been updated since the last call to `meshDataById`. Call `meshDataById` to read the latest mesh data. Calling `meshDataById` will reset the update status for that chunk to 0. - **Zero value**: The chunk has not been updated since the last call to `meshDataById`, so there is no need to read its mesh data. To get the latest mesh data, call `NsdkMeshingSession.meshDataById` for each updated chunk. `meshDataById` returns a `MeshData` containing the raw data for one mesh chunk: - **`verticesPtr`:** A pointer to a float array in native memory of all vertices in the mesh chunk. - The position of each vertex is given by three floats in the vertices array. For example, the position of the first vertex in the chunk is stored in the first three elements of the vertices array in (x, y, z) order. - **`indicesPtr`:** A pointer to a float array in native memory of all indices, or faces, in the mesh chunk. - Each face of the mesh is defined by three indices in the indices array. For example, the first face in the chunk is defined by the first three elements of the indices array. The index values correspond to the indices of the vertices array. - **`normalsPtr`:** A pointer to a UInt32 array in native memory of all vertex normals in the mesh chunk. - Each vertex in the vertices array has a normal which corresponds to three floats in the normals array. For example, the normal for the first vertex in the vertices array is stored in the first three elements of the normals array in (x, y, z) order. - `uvsPtr`: *Only available for VPS mesh downloader--not used for the live meshing feature.* There is also a helper function, `MeshData.toSCNGeometry`, that creates a SceneKit `SCNGeometry` from the mesh chunk. --- ## Creating and Configuring a Meshing Session **`NsdkMeshingSession`** manages the meshing feature lifecycle and data retrieval. It is created from an existing `NsdkSession` instance and works independently of other AR features. ```swift // Assume an NSDK session is set up let nsdk = NSDKSession() // Enable meshing functionality for this session let meshingSession = nsdk.acquireMeshingSession() // Configure and start meshing var config = NSDKMeshingSession.Configuration() // Optionally set any config options config.fuseKeyframesOnly = true config.enableDistanceBasedVolumetricCleanup = true do { try meshingSession.configure(with: config) meshingSession.start() } catch { print("Failed to configure meshing session: \(error)") } ``` ## Retrieving Mesh Data Subscribe to `meshUpdates` to receive incremental mesh changes each frame. Each emission contains a non-empty array of chunk insertions, updates, and removals since the previous frame. ```swift meshingSession.meshUpdates .receive(on: DispatchQueue.main) .sink { updates in for chunkUpdate in updates { switch chunkUpdate { case .insert(let id, let data), .update(let id, let data): // Process the inserted or updated mesh chunk // e.g., create or update a scene entity from `data` case .remove(let id): // Remove the mesh chunk from the scene } } } .store(in: &cancellables) ``` Each `MeshChunkUpdate` is one of: - **.insert(id:data:)** -- A new chunk has been added to the mesh. - **.update(id:data:)** -- An existing chunk has been updated with new geometry. - **.remove(id:)** -- A chunk has been removed from the mesh. The `MeshData` provided with each insert or update contains the raw geometry (`verticesPtr`, `indicesPtr`, `normalsPtr`) and helpers for creating scene geometry: `toSCNGeometry` for SceneKit and `toMeshResource` for RealityKit. ## Configuring Meshing NSDK provides `NsdkMeshingSession.Configuration` to allow you to tweak meshing rules for distance, quality and clean-up: - **frameRate:** The number of times per second to run the mesh update routine. This should be no higher than the ARSession update rate. - **fuseKeyframesOnly:** Enabling this improves mesh accuracy at the cost of a lower update frequency. - **AR Fusion Parameters:** - **maximumIntegrationDistance:** The far distance threshold from the device sensor for integrating depth samples into the 3D scene, in meters. New mesh blocks will not be generated further than this distance from the camera. - **voxelSize:** The size of individual voxel elements in the scene, in meters. Higher values will save memory but also reduce the precision of the surfaces. - **enableDistanceBasedVolumetricCleanup:** Enable this to save memory and smooth latency by cleaning up already-processed elements in the feature's volumetric representation (once they move outside the region where new meshes are generated). This will not remove previously-generated mesh. - **AR Meshing Parameters:** - **meshBlockSize:** The size of the mesh blocks used for generating the mesh filter and mesh collider. - **meshCullingDistance:** The distance from the user where mesh blocks will be removed from the scene. Set to 0 to disable distance-based culling. - **enableMeshDecimation:** Enable this to save memory by removing excess triangles from the mesh. - **Mesh Filtering:** - **filterMeshWithSemantics:** This enables semantic mesh filtering. See [Mesh Filtering](https://www.nianticspatial.com/docs/nsdk/how-to/ar/meshing/semantic_mesh_filtering/) for more information. - **enableAllowlist:** If populated, only these semantic classes will be allowed in the mesh. - **packedAllowlist:** If enableAllowlist is true, only these semantic classes will be allowed in the mesh. (bitmask) - **enableBlocklist:** If populated, these semantic classes will be excluded from the mesh. - **packedBlocklist:** If enableBlocklist is true, these semantic classes will be excluded from the mesh. (bitmask) - **Experimental Meshing Options:** - **numVoxelLevels:** Setting this to >= 1 enables the experimental level of detail meshing feature which saves memory and reduces latency. For more information, see the [Meshing Level of Detail](https://www.nianticspatial.com/docs/nsdk/experimental/level_of_detail_meshing/) page. `NsdkMeshingSession.configure` should be called before starting the feature. ## Long-Distance Meshing > **Caution:** > > These settings are only recommended for high-end devices. > For information on supported devices, see the [Apple ARKit device list](https://www.apple.com/augmented-reality). You can configure your application to mesh over much longer distances by increasing the **Voxel Size**, turning on **Distance-Based Volumetric Cleanup** and increasing the **Maximum Integration Distance** and **Mesh Culling Distance** to between 20 and 40 meters. For example, try out the following settings: - **frameRate**: 20 - **maximumIntegrationDistance**: 40 - **voxelSize**: 0.05 - **enableDistanceBasedVolumetricCleanup**: True - **meshBlockSize**: 1.4 - **meshCullingDistance**: 40 (should be >= Maximum integration distance) - **enableMeshDecimation**: True This will increase CPU, GPU and memory usage, so we recommend testing thoroughly on a variety of devices. ## More Information The [NSDK sample project](https://www.nianticspatial.com/docs/nsdk/sample_projects/) includes an example of running meshing. Also, see [How to Exclude Semantic Channels with Mesh Filtering](https://www.nianticspatial.com/docs/nsdk/how-to/ar/meshing/semantic_mesh_filtering/) for a guide of how to set up a project with meshing.