# Place virtual content with VPS2 Source: https://www.nianticspatial.com/docs/nsdk/how-to/vps2/placing_virtual_content/?platform=kotlin This guide explains how to place virtual content in an AR scene relative to a [localized asset](https://www.nianticspatial.com/docs/nsdk/core_concepts/#anchor-and-place-content) or at known geo coordinates. Virtual content is any digital object rendered in AR, such as a 3D model, label, or visual effect. To appear in the correct location and remain stable as the user moves, it needs a reliable spatial reference. Depending on the placement approach, the app either attaches the content to a localized asset's origin or repeatedly calculates its AR pose from geo coordinates. When VPS2 localizes to a Site, it matches the camera view against the Site's **VPS map asset**. The asset's origin defines a tracked position and orientation in the real world. Placing virtual content relative to that origin allows the AR system to keep the content aligned with the environment over time. VPS2 handles the conversions between AR space, [map-relative space](https://www.nianticspatial.com/docs/nsdk/core_concepts/#map-relative-space), and [global space](https://www.nianticspatial.com/docs/nsdk/core_concepts/#global-space), so content placed against an asset can be expressed in whichever frame your app needs. Because the asset is a mapped real-world location, content placed relative to it appears consistently across sessions and devices. Follow this workflow: 1. [Choose a placement approach](#choose-how-to-place-virtual-content) based on where the content's position comes from and how your app needs to restore it. 2. Follow the readiness guidance for that approach. For [asset-relative placement](#place-content-relative-to-an-anchor), localize to the Site and evaluate the localized asset's tracking data and confidence. For [geo-coordinate placement](#place-content-using-geo-coordinates), evaluate the device's VPS2 tracking state and reported geolocation accuracy. 3. Use the instructions for the selected approach to place the content and save the data needed to restore it. 4. [Test the placement](#test-your-placement) and verify that the content remains stable as the AR session updates. Choose the approach once based on your application's placement and persistence requirements. If its readiness conditions are not met, wait for localization to improve or show a localizing state rather than switching approaches. --- ## Prerequisites Before you start, make sure you already have: - a project with NSDK installed and a basic AR scene set up - a Site in Scaniverse for workflows that localize to a Site or use Site-derived content For setup details, see [Set up the Niantic SDK](https://www.nianticspatial.com/docs/nsdk/setup/), [Set up a basic AR scene](https://www.nianticspatial.com/docs/nsdk/setup/#set-up-a-basic-ar-scene), and [First Localization with NSDK](https://www.nianticspatial.com/docs/nsdk/first_localization/). --- ## Choose how to place virtual content There are multiple ways to place virtual content in an AR scene. Each approach defines how **content is positioned**, how it **behaves over time**, and **what data is stored** so it can be reused. Choose an approach based on where the content's position comes from and how it must be restored. The current device or asset tracking state does not determine which approach to use. #### Key considerations - **Using multiple approaches** - Different features in your app can use different placement approaches. You do not need to choose only one for the entire application. - **Persistence** - Each approach defines what data, if any, is stored for reuse, and whether that reuse is limited to the current session or can work across sessions and devices: - **VPS-based placement** stores data relative to a real-world location, which can allow content to appear in the same place across sessions and devices. - **Local placement** can keep content stable during the current session, including user-driven placement such as hit tests, but it does not necessarily restore to the same real-world position later unless your app persists additional data. Use the following table to compare the available approaches based on when to use them, what data they save, and how each one restores content later. | Approach | Best for | Saved data | Restoration | | --- | --- | --- | --- | | [Place content relative to a localized asset](#place-content-relative-to-an-anchor) | You localize to a Site (by Site ID, or by coordinates for nearby Sites), then want content to stay attached to that asset's origin across sessions. | The Site ID, the asset ID, and each content item's asset-local transform. | Re-localizes to the same Site and reapplies the saved asset-local transform. | | [Place content using geo coordinates](#place-content-using-geo-coordinates) | Content is authored at a known latitude, longitude, and altitude. | Geolocation data such as latitude, longitude, altitude, and optional heading. | Reconstructs placement from geospatial coordinates and reapplies the resulting pose. | ## Know which readiness signal your approach uses The two approaches read **different** readiness signals, and using the wrong one is a common source of misplaced content: | Approach | Check this before placing | Why | | --- | --- | --- | | Place content relative to a localized asset | The **asset's** tracking data | The content is attached to the asset's origin, so the asset's pose is what has to be reliable. | | Place content using geo coordinates | The **device's** `Vps2TrackingState` | The content's position comes from the device's localization, not from a tracked asset. | These two states change independently -- the VPS2 tracking state can be `precise` while an asset is not yet returning tracking data. See [Tracking states](https://www.nianticspatial.com/docs/nsdk/features/vps2/#tracking-states). --- ### Place content relative to a localized asset Use this approach when the user decides where content should appear during a localized session. Save the content's local transform relative to the asset's origin, then apply that same local transform again the next time the asset is tracked. When you check an asset's tracking data, follow these rules: | Rule | Why it matters | What to do | |---|---|---| | **Evaluate each asset separately.** | If you localize to more than one Site, each asset is tracked on its own. Do not combine them into a single session-wide flag, or a distant asset that is not tracked will misreport a nearby asset that is. | Store and evaluate readiness by asset ID. | | **Tracking data means a pose is available; its confidence says how good it is.** | When VPS2 returns asset tracking data, the origin pose is available to place content against. The tracking confidence (`trackingConfidence`, `0.0`-`1.0`) estimates how likely the pose is to be correct. It rises as localizations agree, and falls as they disagree or grow stale while the device moves. | Place content once tracking data is available. If your content must align closely with the real world, also require a minimum tracking confidence, tuned by testing at your Site. | | **Check again after AR tracking is disturbed.** | If the camera is covered, the device is moved abruptly, or the session is interrupted, the asset can stop returning tracking data while content stays on screen at an out-of-date position. | Hide the placed content while tracking data is unavailable, and show it again once it returns. | Follow this workflow to place, save, and later restore content relative to a localized asset: 1. Localize to a Site with one of the `localize` overloads. Localization is asynchronous; VPS2 reports the localized asset on `localizationUpdates` once the device matches a Site's VPS map. - To localize to a specific Site, [create a Site](https://www.nianticspatial.com/docs/nsdk/first_localization/#create-a-private-site) in [Scaniverse](https://scaniverse.nianticspatial.com/signin) and get its Site ID (from Scaniverse or the [Sites API](https://www.nianticspatial.com/docs/nsdk/how-to/sites/getting_started/#3-query-sites)), then call [`localize(siteId)`](https://www.nianticspatial.com/docs/api/kotlin/com.nianticspatial.nsdk.vps2.Vps2Session.localize). - To localize to nearby Sites, call [`localize(latitude, longitude, radiusMeters)`](https://www.nianticspatial.com/docs/api/kotlin/com.nianticspatial.nsdk.vps2.Vps2Session.localize) with the device's location. You do not need a Site ID. 2. Create a helper that owns a `PoseNode` for the localized asset and applies the latest asset transform to it. The following code example shows the helper type that manages the localized asset's `PoseNode`: ```kotlin class AssetRelativeContentPlacement( private val engine: Engine, private val arChildNodes: MutableList, ) { private var poseNode: PoseNode? = null } ``` 3. Collect `localizationUpdates` and, on each emission, poll `getAssetTrackingData(assetId)` for the localized asset. When it returns data, fill a column-major `FloatArray(16)` with `tracking.assetToLocalTrackingTransform.toMatrix(matrix, 0)` and apply it to the `PoseNode`. `getAssetTrackingData` returns data only while the asset is tracked, so the `PoseNode` updates only while a pose is available. Check its `trackingConfidence` if your content must align closely with the real world. The following code does that: ```kotlin vps2Session.localizationUpdates .onEach { localization -> val assetId = localization.localizedAsset?.assetId val tracking = assetId?.let { vps2Session.getAssetTrackingData(it) } if (tracking == null) { placement.setContentVisible(false) // hide while the asset is not tracked return@onEach } val matrix = FloatArray(16) tracking.assetToLocalTrackingTransform.toMatrix(matrix, 0) placement.applyAssetMatrix(matrix) placement.setContentVisible(true) } .launchIn(scope) ``` 4. Add your content as a child node under that `PoseNode`. Adding the content under the `PoseNode` keeps it aligned as VPS2 refines the asset's pose. ```kotlin val poseNode = placement.getOrCreatePoseNode() val contentNode = CubeNode( engine = engine, size = Float3(0.2f, 0.2f, 0.2f), center = Float3(0f, 0f, 0f), materialInstance = refinedGreenMaterial, ) poseNode.addChildNode(contentNode) ``` 5. Persist the Site ID, asset ID, and the child node's local pose so you can restore it later. ```kotlin savedSiteId = siteId savedAssetId = assetId savedLocalPosition = contentNode.position savedLocalRotation = contentNode.rotation ``` Expand the following example for a runnable Kotlin controller that covers this workflow -- the helper file plus the wiring to place and restore content in the sample app: #### Runnable example: Place and restore content relative to a localized asset 1. Create a new file named `AssetRelativeContentPlacement.kt`. If you are using the sample app, create it in `kotlin/NsdkSamples/src/main/java/com/nianticspatial/nsdk/externalsamples/vps2/` next to `VPS2View.kt`, `VPS2Manager.kt`, and `VPS2Route.kt`. The following defines the controller that follows the localized asset and the helper that keeps its `PoseNode` updated: ```kotlin package com.nianticspatial.nsdk.externalsamples.vps2 import com.nianticspatial.nsdk.NSDKSession import com.nianticspatial.nsdk.vps2.Vps2Session import io.github.sceneview.node.Node import dev.romainguy.kotlin.math.Float4 import dev.romainguy.kotlin.math.Mat4 import io.github.sceneview.ar.node.PoseNode import com.google.android.filament.Engine import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Job import kotlinx.coroutines.flow.launchIn import kotlinx.coroutines.flow.onEach class AssetRelativeContentPlacementController( nsdkSession: NSDKSession, engine: Engine, arChildNodes: MutableList, private val scope: CoroutineScope, ) { private val vps2Session: Vps2Session = nsdkSession.vps2.acquire() val placement = AssetRelativeContentPlacement(engine, arChildNodes) // The ID of the currently tracked asset, saved alongside the placement. var currentAssetId: String? = null private set // getAssetTrackingData returns data only while the asset is tracked, so the // PoseNode updates only while a pose is available. fun startTrackingLocalizedAsset(): Job { return vps2Session.localizationUpdates .onEach { localization -> val assetId = localization.localizedAsset?.assetId val tracking = assetId?.let { vps2Session.getAssetTrackingData(it) } if (assetId == null || tracking == null) { placement.setContentVisible(false) // hide while the asset is not tracked return@onEach } currentAssetId = assetId val matrix = FloatArray(16) tracking.assetToLocalTrackingTransform.toMatrix(matrix, 0) placement.applyAssetMatrix(matrix) placement.setContentVisible(true) } .launchIn(scope) } fun isAssetTracked(assetId: String): Boolean = vps2Session.getAssetTrackingData(assetId) != null } class AssetRelativeContentPlacement( private val engine: Engine, private val arChildNodes: MutableList, ) { private var poseNode: PoseNode? = null fun getOrCreatePoseNode(): PoseNode { return poseNode ?: PoseNode(engine = engine).also { poseNode = it arChildNodes.add(it) } } fun setContentVisible(visible: Boolean) { poseNode?.isVisible = visible } fun applyAssetMatrix(matrix: FloatArray) { getOrCreatePoseNode().worldTransform = Mat4( Float4(matrix[0], matrix[1], matrix[2], matrix[3]), Float4(matrix[4], matrix[5], matrix[6], matrix[7]), Float4(matrix[8], matrix[9], matrix[10], matrix[11]), Float4(matrix[12], matrix[13], matrix[14], matrix[15]), ) } } ``` 2. In your VPS2 screen, after localizing to the Site, create the controller and start tracking the localized asset. Hold the saved placement state in remembered state so both the place and restore actions can read it: ```kotlin import io.github.sceneview.math.Position import io.github.sceneview.math.Rotation import androidx.compose.runtime.rememberCoroutineScope val coroutineScope = rememberCoroutineScope() var savedSiteId by remember { mutableStateOf(null) } var savedAssetId by remember { mutableStateOf(null) } var savedLocalPosition by remember { mutableStateOf(null) } var savedLocalRotation by remember { mutableStateOf(null) } val controller = remember { AssetRelativeContentPlacementController( nsdkSession = nsdkManager.session, engine = engine, arChildNodes = arChildNodes, scope = coroutineScope, ).also { it.startTrackingLocalizedAsset() } } ``` 3. Add a `Place Content` button that attaches content under the asset's `PoseNode` and saves the Site ID plus the content's local pose: ```kotlin Button(onClick = { val poseNode = controller.placement.getOrCreatePoseNode() val contentNode = CubeNode( engine = engine, size = Float3(0.2f, 0.2f, 0.2f), center = Float3(0f, 0f, 0f), materialInstance = refinedGreenMaterial, ) poseNode.addChildNode(contentNode) savedSiteId = siteId savedAssetId = controller.currentAssetId savedLocalPosition = contentNode.position savedLocalRotation = contentNode.rotation }) { Text("Place Content") } ``` 4. To restore in a later session, localize to the same Site again, wait for the asset's `PoseNode` to update, then add a `Restore Content` button that reattaches a node with the saved local pose: ```kotlin Button(onClick = { val assetId = savedAssetId ?: return@Button val localPosition = savedLocalPosition ?: return@Button val localRotation = savedLocalRotation ?: return@Button if (!controller.isAssetTracked(assetId)) return@Button // wait until the saved asset is tracked again val poseNode = controller.placement.getOrCreatePoseNode() val restoredNode = CubeNode( engine = engine, size = Float3(0.2f, 0.2f, 0.2f), center = Float3(0f, 0f, 0f), materialInstance = refinedGreenMaterial, ).apply { position = localPosition rotation = localRotation } poseNode.addChildNode(restoredNode) }) { Text("Restore Content") } ``` 5. Build to Android and validate on a real device. The on-screen validation order is: - open `VPS2` - localize to the Site and wait until the asset's cube appears - tap `Place Content` - move the device, then tap `Restore Content` If you want to compare this explanation against a working implementation, review `VPS2Manager.kt` -- see `pollAssetTrackingData` and `updateAssetMesh` -- and `VPS2View.kt` in [nsdk-samples-kotlin](https://github.com/nianticspatial/nsdk-samples-kotlin). Expected result: after you save local pose data and localize to the same Site again, the content returns to the same real-world location. ## Place content using geo coordinates When you place content from geo coordinates, follow these rules: | Rule | Why it matters | What to do | |---|---|---| | **At a Site, wait for the precise state before placing.** | `getPose` returns a pose even in the `coarse` tracking state, but it is only accurate enough to anchor virtual content once the state is `precise`. The state can fall back to `coarse`, for example when AR tracking resets. | Gate content placement on the `precise` state, keep checking it on each update, and treat `coarse` poses as approximate. | | **Expect `coarse` away from mapped Sites.** | Only localizing to a VPS map reaches `precise`. Local sensor fusion and universal localization stop at `coarse`, so content gated on `precise` does not appear away from mapped Sites. The runnable examples below gate on `precise`. | For content outside mapped Sites, place it in the `coarse` state and size or fade it using the reported horizontal and heading accuracy. | Follow this workflow to place content from geo coordinates: 1. Define the target latitude, longitude, and altitude for the real-world location where content should appear. These values can come from authored data, backend content, or a known location. 2. Pass those coordinates to a helper that owns your VPS2 session. The helper builds a [`GeolocationData`](https://www.nianticspatial.com/docs/api/kotlin/com.nianticspatial.nsdk.GeolocationData) from them. 3. On each localization update, convert that `GeolocationData` to an AR pose with [`getPose`](https://www.nianticspatial.com/docs/api/kotlin/com.nianticspatial.nsdk.vps2.Vps2Session.getPose) and apply its pose to your content. `getPose` uses the latest localization and returns `null` until VPS2 has localized. The returned `Vps2Pose` carries its `trackingState`; when your content needs accurate placement, wait for the `PRECISE` state, then apply `pose.pose`. 4. Repeat step 3 on every localization update to keep content aligned as localization improves. The following runnable example shows one complete helper that converts a fixed geolocation into an AR pose: #### Runnable example: Place content from geo coordinates Create a new file named `GeoPositionedObjectHelper.kt` and add the following code: ```kotlin import com.google.ar.core.Pose import com.nianticspatial.nsdk.GeolocationData import com.nianticspatial.nsdk.vps2.Vps2Session import com.nianticspatial.nsdk.vps2.Vps2TrackingState class GeoPositionedObjectHelper( private val vps2Session: Vps2Session, latitude: Double, longitude: Double, altitude: Double, ) { private val targetLocation = GeolocationData( latitude = latitude, longitude = longitude, altitude = altitude, heading = 0.0, orientationEdn = floatArrayOf(0f, 0f, 0f, 1f), ) fun updateContentPose(onPoseUpdated: (Pose) -> Unit) { // getPose uses the latest localization and returns null until VPS2 has localized. // The returned Vps2Pose carries the tracking state it was computed from. val pose = vps2Session.getPose(targetLocation) ?: return // The pose is only accurate enough to place content once its tracking state is precise. if (pose.trackingState != Vps2TrackingState.PRECISE) return onPoseUpdated(pose.pose) } } ``` Create the helper with your target coordinates where you own the `Vps2Session`, then call `updateContentPose` from your localization-updates callback and apply the returned `Pose` to a visible node: ```kotlin private val geoHelper = GeoPositionedObjectHelper( vps2Session = vps2Session, latitude = targetLatitude, longitude = targetLongitude, altitude = targetAltitude, ) // Inside your vps2Session.localizationUpdates handler: geoHelper.updateContentPose { pose -> val matrix = FloatArray(16) pose.toMatrix(matrix, 0) // apply `matrix` (column-major) to your rendered node's world transform } ``` The Kotlin sample app shows the full rendering wiring -- a visible `PoseNode` and `CubeNode` in `VPS2View.kt`, and the callback plumbing in `VPS2Manager.kt`. For reading the device's geolocation, see [Use geoposition with VPS2](https://www.nianticspatial.com/docs/nsdk/how-to/vps2/getting_vps2_geoposition/). ## Test Your Placement In addition to testing your app on-location, you can also use Playback in order to iterate quickly without traveling to a VPS-enabled Site. Playback runs NSDK against a prerecorded AR session dataset instead of live camera input. It can exercise the full asset-relative and geo-coordinate workflows, as long as the dataset was recorded at the Site. Still, Playback is only a simulation of a live session, so it is best for iteration and debugging rather than final validation. On-location device testing is the most accurate way to confirm that content appears in the correct real-world place for the actual Site. See [How to set up Playback](https://www.nianticspatial.com/docs/nsdk/how-to/playback/setting_up_playback/).