# Adding Meshing to Your Project Source: https://www.nianticspatial.com/docs/nsdk/how-to/ar/meshing/adding_meshing/?platform=kotlin Niantic Spatial SDK (NSDK) exposes its meshing functionality on native Android through the **`MeshingSession`** API. When the application calls `NSDKSession.update()`, an active `MeshingSession` ingests depth and device pose data and builds a mesh from it. > **Note:** > > Niantic Spatial Meshing currently does not support lidar on Android. > Ensure that `NsdkSession` is configured with **`useLidar=false`** before attempting meshing. ## Mesh Chunks To lighten the application's compute workload as the mesh grows, NSDK breaks the mesh up into chunks. The three-dimensional world is divided into a regular grid of "blocks" of a fixed size. When meshing is running, new mesh chunks will be created as the user explores their surroundings. Mesh chunks are continually updated as new data is added to the 3D representation. The application should regularly read the latest `MeshingUpdateInfo` from `MeshingSession.getUpdatedInfos` to get the list of updated mesh chunks: - **`ids`:** An array of all mesh chunk IDs - **`updated`:** An array indicating whether each chunk was updated. For each corresponding ID in `ids`, this array indicates: - **Non-zero value**: The chunk has been updated since the last call to `MeshingSession.getData`. Call `getData` to read the latest mesh data. Calling `getData` will reset the update status for that chunk to 0. - **Zero value**: The chunk has not been updated since the last call to `MeshingSession.getData`, so there is no need to read its mesh data. To get the latest mesh data, call `MeshingSession.getData` for each updated chunk. `getData` returns a `MeshData` containing the raw data for one mesh chunk: - **`vertices`:** A FloatArray of all vertices in the mesh chunk. - The position of each vertex is given by three floats in the vertices array. For example, the position of the first vertex in the chunk is stored in the first three elements of the vertices array in (x, y, z) order. - **`uvs`:** A FloatArray of all UVs in the mesh chunk. Note that this array can be null if there is no UV data. - **`indicesPtr`:** An IntArray of all indices, or faces, in the mesh chunk. - Each face of the mesh is defined by three indices in the indices array. For example, the first face in the chunk is defined by the first three elements of the indices array. The index values correspond to the indices of the vertices array. - **`normalsPtr`:** A FloatArray of all vertex normals in the mesh chunk. Note that this array can be null if there is no Normal data. - Each vertex in the vertices array has a normal which corresponds to three floats in the normals array. For example, the normal for the first vertex in the vertices array is stored in the first three elements of the normals array in (x, y, z) order. --- ## Creating and Configuring a Meshing Session **`MeshingSession`** manages the meshing feature lifecycle and data retrieval. It is created from an existing `NsdkSession` instance and works independently of other AR features. ```kotlin // Set up the NSDKSession. Note that for Android, useLidar must be false val nsdkSession = NSDKSession(apiKey = "YOUR_API_KEY", useLidar = false) // Enable meshing functionality for this session val meshingSession = nsdkSession.meshingSession.acquire() // Configure and start meshing meshingSession.configure( MeshingConfig( fuseKeyframesOnly = true, ) ) meshingSession.start() ``` ## Retrieving Mesh Data Use `MeshingUpdatedInfo` to check which mesh chunks have been updated. ```kotlin val updatedMeshInfo = meshingSession.getUpdatedInfos() if (updatedMeshInfo != null) { val updateInfo: MeshingUpdateInfo = updatedMeshInfo // Iterate through the IDs provided in the updateInfo updateInfo.ids.forEachIndexed { index, meshId -> val isUpdated = (updateInfo.updated.getOrNull(index) == 1.toByte())// Get corresponding updated flag, convert to bool // Get the MeshData for this specific meshId if its updated if (isUpdated) { val meshDataResult = meshingSession.getData(meshId) // Use the new mesh data // ... } } } ``` It is recommended to keep track of the **chunk IDs** to delete portions of the mesh that have been removed by the NSDK feature. A **`lastMeshUpdateTime`** counter is also available to avoid unnecessary copies of the `MeshingUpdatedInfo`. The following function is an example of starting the meshing session, and launching a coroutine that will listen for updates to the mesh info, gather the changes in an array, and pass it to a callback for your app to use. ```kotlin val meshIdToMeshData = mutableStateMapOf() var meshingStarted by mutableStateOf(false) var lastUpdateTime by mutableLongStateOf(0) fun startMeshing(updateMeshCallback: (MutableMap) -> Unit) { meshingStarted = true meshingSession.start() // Periodically poll the meshingSession for updated info, and pass that along to the View. // This will also post messages about the meshing update status that Views can listen for. coroutineScope.launch { while (meshingStarted) { delay(100L) // Update every 100ms val latestUpdateTime = meshingSession.getLastUpdateTime() if (latestUpdateTime != null) { if (latestUpdateTime > lastUpdateTime) { lastUpdateTime = latestUpdateTime Log.d("Meshing", "New Meshing Update Time: $latestUpdateTime") val updatedMeshInfo = meshingSession.getUpdatedInfos() if (updatedMeshInfo != null) { Log.d("Meshing", "Updated Mesh Infos: ${updatedMeshInfo}") val updateInfo: MeshingUpdateInfo = updatedMeshInfo // Type is MeshingUpdateInfo Log.d("Meshing", "Received MeshingUpdateInfo: $updateInfo") // Iterate through the IDs provided in the updateInfo updateInfo.ids.forEachIndexed { index, meshId -> val isUpdated = (updateInfo.updated.getOrNull(index) == 1.toByte())// Get the value as a bool // Get the MeshData for this specific meshId if its updated if (isUpdated) { try { val meshDataResult = meshingSession.getData(meshId) val meshData: MeshData = meshDataResult!! meshIdToMeshData[meshId] = meshData // Add/update in your state map } catch (e: NsdkStatusException) { Log.d("Meshing", "Exception getting MeshData for ID $meshId: $e") } } } updateMeshCallback.invoke(meshIdToMeshData) } else { Log.d("Meshing", "No mesh info available.") } } else { Log.d("Meshing", "Waiting for new meshing update") } } else { Log.d("Meshing", "Mesh update time not yet available.") } } } } ``` ## Configuring Meshing NSDK provides `MeshingConfig` to allow you to tweak meshing rules for distance, quality and clean-up: - **frameRate:** The number of times per second to run the mesh update routine. This should be no higher than the ARSession update rate. - **fuseKeyframesOnly:** Enabling this improves mesh accuracy at the cost of a lower update frequency. - **AR Fusion Parameters:** - **maximumIntegrationDistance:** The far distance threshold from the device sensor for integrating depth samples into the 3D scene, in meters. New mesh blocks will not be generated further than this distance from the camera. - **voxelSize:** The size of individual voxel elements in the scene, in meters. Higher values will save memory but also reduce the precision of the surfaces. - **enableDistanceBasedVolumetricCleanup:** Enable this to save memory and smooth latency by cleaning up already-processed elements in the feature's volumetric representation (once they move outside the region where new meshes are generated). This will not remove previously-generated mesh. - **AR Meshing Parameters:** - **meshBlockSize:** The size of the mesh blocks used for generating the mesh filter and mesh collider. - **meshCullingDistance:** The distance from the user where mesh blocks will be removed from the scene. Set to 0 to disable distance-based culling. - **disableMeshDecimation:** Enable this will disable the default optimization setting of removing excess triangles from the mesh. Disabling decimation consumes more memory. - **Mesh Filtering:** - **filterMeshWithSemantics:** This enables semantic mesh filtering. See [Mesh Filtering](https://www.nianticspatial.com/docs/nsdk/how-to/ar/meshing/semantic_mesh_filtering/) for more information. - **enableAllowlist:** If populated, only these semantic classes will be allowed in the mesh. - **packedAllowlist:** If enableAllowlist is true, only these semantic classes will be allowed in the mesh. (bitmask) - **enableBlocklist:** If populated, these semantic classes will be excluded from the mesh. - **packedBlocklist:** If enableBlocklist is true, these semantic classes will be excluded from the mesh. (bitmask) - **Experimental Meshing Options:** - **numVoxelLevels:** Setting this to >= 1 enables the experimental level of detail meshing feature which saves memory and reduces latency. For more information, see the [Meshing Level of Detail](https://www.nianticspatial.com/docs/nsdk/experimental/level_of_detail_meshing/) page. `MeshingSession.configure` should be called before starting the feature. ## Long-Distance Meshing > **Caution:** > > These settings are only recommended for high-end devices. You can configure your application to mesh over much longer distances by increasing the **Voxel Size**, turning on **Distance-Based Volumetric Cleanup** and increasing the **Maximum Integration Distance** and **Mesh Culling Distance** to between 20 and 40 meters. For example, try out the following settings: - **frameRate**: 20 - **maximumIntegrationDistance**: 40 - **voxelSize**: 0.05 - **enableDistanceBasedVolumetricCleanup**: True - **meshBlockSize**: 1.4 - **meshCullingDistance**: 40 (should be >= Maximum integration distance) - **enableMeshDecimation**: True This will increase CPU, GPU and memory usage, so we recommend testing thoroughly on a variety of devices. ## More Information The [NSDK sample project](https://www.nianticspatial.com/docs/nsdk/sample_projects/) includes an example of running meshing.