# Set up the Niantic SDK Source: https://www.nianticspatial.com/docs/nsdk/setup/?platform=kotlin # Set up the NSDK in Kotlin The NSDK Android library is distributed as an AAR served from a GitHub Maven repository and is added through Gradle. Before you begin, ensure your Android project uses `minSdk` **24** or higher, Kotlin version **2.0.0** or higher, and JDK **17** to run Gradle (required by Android Gradle Plugin 8.x; the app itself can still target Java 11). Kotlin DSL (`build.gradle.kts`) and Groovy (`build.gradle`) are both supported build configuration languages, but Kotlin DSL is preferred. ## AI set up > **Tip:** > > NSDK supports AI-assisted development > > NSDK publishes **Skills**: packaged setup workflows that an AI coding assistant reads > directly. This is a supported way to work with NSDK, and usually the fastest and > smoothest path to a working integration -- the assistant carries out the steps on this > page with you, checking your project as it goes. > > **Download the Kotlin setup Skill** To use an **AI coding assistant** to set up the NSDK for you, do the following: 1. Extract and place the folder into your AI assistant's skills directory. For example: - Claude Code: `.claude/skills/` - Codex: `.agents/skills/` 2. Create or open an existing Android Studio project. If you're creating a new project, use the **Empty Activity** template. 1. Choose a minimum SDK of Android 7.0 / API 24. 2. Set the **Build configuration language** to **Kotlin DSL**. 3. Ask your AI assistant to set up NSDK in your project. It will guide you through the required steps and help configure your project. ## Manual set up To **manually** set up the NSDK in Kotlin, you will need to: 1. [Create an account](#create-an-account-2). 2. [Add GitHub Packages](#add-the-github-packages) for the Maven repository. 3. [Add permissions](#add-permissions) to the Android Manifest. 4. [Configure your app module](#configure-your-app-module). 5. [Set the Kotlin version](#set-kotlin-version). 6. [Create an NSDK session](#create-an-nsdk-session-1). 7. [Provide frame and sensor data](#provide-frame-and-sensor-data-1). 8. [Update the session](#update-the-session). 9. [Review permissions](#permissions-1). Steps to manually set up the NSDK are shown in the following sections. --- ### Create an account Before getting started with the NSDK setup, follow the steps to [create a Scaniverse account](https://www.nianticspatial.com/docs/nsdk/create_account/#create-a-scaniverse-account). You will need an access token to initialize the NSDK session. For development and internal testing, create a developer token in [Scaniverse Web](https://scaniverse.nianticspatial.com/signin) and pass it to NSDK as-is; no exchange step is needed. For production apps, use an access token from your backend. For more information, see [Authorization](https://www.nianticspatial.com/docs/nsdk/auth_getting_started/), [Generate developer tokens](https://www.nianticspatial.com/docs/nsdk/auth_developer_token/), and [Generate access tokens](https://www.nianticspatial.com/docs/nsdk/auth_backend/). ### Add the GitHub packages Add the Niantic Spatial Maven repository so Gradle can resolve the NSDK Android artifacts. In your project's `settings.gradle.kts`, add the Niantic Spatial repository under `dependencyResolutionManagement`: ```kotlin dependencyResolutionManagement { repositories { google() mavenCentral() maven { url = uri("https://raw.githubusercontent.com/nianticspatial/nsdk-library-aar/main") } } } ``` ### Add permissions NSDK uses the camera for AR, location for GPS features, and network access for online services. ARCore also requires camera feature declarations in the app manifest. Add the following entries to your `AndroidManifest.xml` file: ```xml ``` (image: Kotlin Setup Manifest Permissions) Then add the following ARCore entry **inside** the `` element: ```xml ``` ARCore reads the entry from the previous code example when the app starts. If it is missing, ARCore reports an `ARCore init failed` or `Application manifest must include ...` error, the app cannot create an ARCore `Session`, and often shows a black screen. To complete this setup, use `android:value="required"`. Use `optional` only if your app is designed to keep working on devices that do not support AR. ### Configure your app module Add the NSDK dependency, ARCore, Play Services Location, and Kotlin serialization dependencies to your app module. Add the following to your app module `build.gradle.kts` or `build.gradle`. If your project uses a version catalog, keep your existing Android and Kotlin plugin entries and add the following: ```kotlin plugins { alias(libs.plugins.kotlin.serialization) } android { defaultConfig { minSdk = 24 } } dependencies { implementation("com.nianticspatial:nsdk:") implementation("com.google.ar:core:1.47.0") implementation(libs.play.services.location) implementation(libs.kotlinx.serialization.json) } ``` If your project does not use a version catalog, keep your existing Android and Kotlin plugin entries and add the following: ```kotlin plugins { id("org.jetbrains.kotlin.plugin.serialization") } android { defaultConfig { minSdk = 24 } } dependencies { implementation("com.nianticspatial:nsdk:") implementation("com.google.ar:core:1.47.0") implementation("com.google.android.gms:play-services-location:21.3.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.9.0") } ``` Replace `` with the latest published NSDK version. It is the `` value in the repository's [maven-metadata.xml](https://raw.githubusercontent.com/nianticspatial/nsdk-library-aar/main/com/nianticspatial/nsdk/maven-metadata.xml), or browse the [NSDK Android library repository](https://github.com/nianticspatial/nsdk-library-aar) to find it. If your app reads VPS Debugger events (`Vps2Session.debuggerEvents`), also add `com.google.protobuf:protobuf-java` and `com.google.protobuf:protobuf-java-util` version `3.25.3`. ### Set Kotlin version Update your Kotlin and related Gradle entries so they meet the minimum versions required by the NSDK dependencies. Ensure your Kotlin version is set to a minimum of **2.0.0** and playServicesLocation is set to **21.3.0**. Lastly, under Libraries add the following line: `play-services-location = { module = "com.google.android.gms:play-services-location", version.ref = "playServicesLocation" }` 1. Ensure the Kotlin version in `gradle/libs.versions.toml` is at least **2.0.0**. 2. Add the following entries to `gradle/libs.versions.toml`: ```toml [versions] playServicesLocation = "21.3.0" kotlinxSerializationJson = "1.9.0" [libraries] play-services-location = { module = "com.google.android.gms:play-services-location", version.ref = "playServicesLocation" } kotlinx-serialization-json = { module = "org.jetbrains.kotlinx:kotlinx-serialization-json", version.ref = "kotlinxSerializationJson" } [plugins] kotlin-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", version.ref = "kotlin" } ``` (image: Kotlin Setup Kotlin Version) 3. In your app module `app/build.gradle.kts`, add the serialization plugin to the `plugins` block and the dependencies to the `dependencies` block: ```kotlin plugins { alias(libs.plugins.kotlin.serialization) } dependencies { implementation(libs.play.services.location) implementation(libs.kotlinx.serialization.json) } ``` If your project does not use a version catalog, use the following steps: 1. Ensure your Kotlin version is at least **2.0.0** in your project-level `build.gradle` or `build.gradle.kts`. 2. In your app module `build.gradle` or `build.gradle.kts`, add the Kotlin serialization plugin if it is not already present: ```kotlin plugins { id("org.jetbrains.kotlin.plugin.serialization") } ``` 3. Add the dependencies directly: ```kotlin implementation("com.google.android.gms:play-services-location:21.3.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.9.0") ``` In Android Studio, go to **File** -> **Sync Project with Gradle Files** to apply the changes for either setup path. ### Create an NSDK session Create an `NSDKSession` so your app can initialize NSDK features with an access token before you start using live AR data. #### Set up ARCore Before NSDK can use live AR data, your app must already create and run an [ARCore Session](https://developers.google.com/ar/reference/java/com/google/ar/core/Session). NSDK reads frames from that session. It does not create, own, or render it for you. A new project such as a blank Compose app does not include that ARCore session yet. Before you continue, make sure your app is already set up as a working ARCore host app and does the following: - Checks ARCore availability and prompts for installation with `ArCoreApk.requestInstall(...)`. - Requests the `CAMERA` permission. - Creates, resumes, pauses, and closes an ARCore `Session` as your activity moves through its lifecycle. - Runs a render loop, typically a `GLSurfaceView.Renderer`, that calls `session.update()` each frame to get the latest `Frame` and binds the ARCore camera texture. Without a running ARCore session producing frames, NSDK does not receive camera or pose data. This ARCore and GL setup is standard ARCore code, not NSDK-specific code, and it is a substantial part of the app. If you are starting from a blank app, the quickest path is to begin with the [Kotlin sample app](https://github.com/nianticspatial/nsdk-samples-kotlin). For an overview of the sample app, see [Kotlin sample projects](https://www.nianticspatial.com/docs/nsdk/sample_projects/). The sample app already has the session and render loop wired up, so you can add the NSDK steps below instead of building the camera and GL stack from scratch. #### Initialize the session In your application, create an `NSDKSession` object to access NSDK features. You will need a valid access token from your Niantic Spatial account. Most applications will only need to set the configurations exposed in the following simple initializer which creates an NSDKSession instance using your access token to initialize and configure access to NSDK features: ```kotlin val nsdkSession = NSDKSession( accessToken = "CURRENT ACCESS TOKEN HERE", useLidar = false ) ``` Call `close()` when the session is no longer needed to release resources. NSDK supports one active `NSDKSession` per process. Create one session and reuse it for the lifetime of your app. Do not create a second `NSDKSession` while the first one is still active. If you need a new session, for example to change the configuration, call `close()` on the existing session and let it finish closing before you create another one. Use the initializer in the previous code example for a standard setup. If your app needs a different `NSDKSession` setup, see the [NSDKSession API reference](https://www.nianticspatial.com/docs/api/kotlin/com.nianticspatial.nsdk.NSDKSession) and the Kotlin examples in [Client code integration](https://www.nianticspatial.com/docs/nsdk/auth_client/) and [Getting logs](https://www.nianticspatial.com/docs/nsdk/getting_logs/). ### Provide frame and sensor data Before using features that rely on live AR or device data, configure a data source for the session. Use `createLiveDataSource` to supply frame data including camera image, pose, tracking state, GPS, and compass information to the NSDK from ARCore and device sensors. The following code example creates a data source that supplies ARCore frames and device orientation to the NSDK and assigns it to the session so it can use live camera and sensor data: ```kotlin import com.google.ar.core.Frame import com.nianticspatial.nsdk.Orientation import com.nianticspatial.nsdk.helpers.createLiveDataSource var currentFrame: Frame? = null val dataSource = createLiveDataSource( context = this, frameProvider = { currentFrame }, orientationProvider = { Orientation.fromSurfaceRotation(ContextCompat.getDisplayOrDefault(this).rotation) } ) nsdkSession.dataSource = dataSource ``` In the previous code example, `frameProvider` returns the latest ARCore `Frame`, and `orientationProvider` returns the current screen orientation as an `Orientation`. Get the ARCore `Frame` from your existing ARCore update loop. For `orientationProvider`, read the current screen rotation from your activity or window manager, then convert it with `Orientation.fromSurfaceRotation(...)` so NSDK receives an `Orientation`. Configure the data source once for each `NSDKSession`. This step is required for features that use live AR or device data. ### Update the session Each time ARCore produces a frame, your app must pass that frame data to NSDK. Choose one of the following approaches: - Use [update with a data source](#use-update-with-a-data-source) if you created a data source with `createLiveDataSource` and want `NSDKSession` to read the requested values from that data source. - Use [sendFrame to send frame data directly](#use-sendframe-to-send-frame-data-directly) if you want your app to control exactly when the camera image is acquired and exactly which values are sent to NSDK. #### Use update with a data source The `NsdkSessionDataSource` created by `createLiveDataSource` makes the latest ARCore data available to NSDK, but it does not send frames to NSDK by itself. Keep the `dataSource` reference returned by `createLiveDataSource` so you can use it in your frame loop. For each frame: 1. Call `dataSource.prepareFrame()` to capture the current ARCore frame for NSDK. 2. Check the return value. `prepareFrame()` returns `true` when a frame was captured and `false` when no frame was ready. 3. Call `nsdkSession.update()` only when `prepareFrame()` returns `true`. If it returns `false`, skip that frame and wait for the next one. The following code example saves the latest ARCore frame, asks the data source to prepare that frame for the NSDK, and updates the NSDK only if a frame was successfully prepared: ```kotlin import com.google.ar.core.Frame import com.google.ar.core.Session import kotlinx.coroutines.runBlocking fun onSessionUpdated(session: Session, frame: Frame) { currentFrame = frame // prepareFrame() snapshots the current ARCore frame; update() then processes it. val frameReady = runBlocking { dataSource.prepareFrame() } if (frameReady) nsdkSession.update() } ``` Always call `prepareFrame()` before `update()`. If you call `nsdkSession.update()` first, NSDK has no captured frame to read from the data source. In that case, `latestCameraSample()` remains null and NSDK logs `CAMERA_IMAGE requested but cameraSample is null, skipping`. Features that depend on camera data, such as localization, do not run. Also, do not start a second `prepareFrame()` call before the previous `prepareFrame()` and `update()` cycle finishes. `prepareFrame()` is a `suspend` function. You can call it only from a coroutine or from another `suspend` function. A draw callback such as `GLSurfaceView.Renderer.onDrawFrame` is a normal function, so it cannot call `prepareFrame()` directly. If you try, Kotlin reports that the suspend function can be called only from a coroutine or another suspend function. In the previous code example, `runBlocking` is used as a simple bridge so `prepareFrame()` and `update()` run in order on the same thread. If your app draws the camera view every frame, use a coroutine setup that runs `prepareFrame()` and `update()` on one background thread and allows only one frame update at a time. The Kotlin sample app shows this setup. If you use `runBlocking`, make sure your project includes the `org.jetbrains.kotlinx:kotlinx-coroutines-android` library. `nsdkSession.update()` reads from `dataSource` and only acquires the data types currently requested by active features, so costly resources such as the camera image are not transferred on every frame unnecessarily. #### Use sendFrame to send frame data directly With `sendFrame(...)`, your app creates a `FrameData` object for each ARCore frame and passes it directly to `nsdkSession.sendFrame(...)`. `sendFrame(...)` does not read frame data from `nsdkSession.dataSource`. The Kotlin sample app uses `sendFrame(...)`. Before you create `FrameData`, call `getRequestedDataFormats()`. That call returns the inputs NSDK's active features need for the current frame. Then fill in only those parts of `FrameData`. ```kotlin import android.media.Image import com.google.ar.core.Frame import com.google.ar.core.exceptions.NotYetAvailableException import com.nianticspatial.nsdk.CameraIntrinsicsFromARCore import com.nianticspatial.nsdk.FrameData import com.nianticspatial.nsdk.InputDataFlags fun onSessionUpdated(frame: Frame) { val requested = nsdkSession.getRequestedDataFormats() if (InputDataFlags.NONE.Is(requested)) return val data = FrameData(frame.timestamp / 1_000_000, frame.timestamp) var image: Image? = null if (InputDataFlags.CAMERA_IMAGE.Within(requested)) { try { image = frame.acquireCameraImage() data.cameraImagePlanes = image.planes data.cameraImageWidth = image.width data.cameraImageHeight = image.height data.cameraImageFormat = image.format data.cameraIntrinsics = CameraIntrinsicsFromARCore(frame.camera.imageIntrinsics) } catch (ignored: NotYetAvailableException) { // The camera image isn't ready for this frame yet -- skip it. return } } if (InputDataFlags.POSE.Within(requested)) data.cameraPose = frame.camera.pose if (InputDataFlags.TRACKING_STATE.Within(requested)) data.trackingState = frame.camera.trackingState // Set DEVICE_ORIENTATION, GPS_LOCATION, and COMPASS the same way when requested. nsdkSession.sendFrame(data) image?.close() } ``` In the previous code example, the app asks NSDK which inputs it needs, creates a `FrameData` object for the current ARCore frame, fills in only the requested values, and sends that data to NSDK. Using `sendFrame(...)` can also help if logcat repeatedly shows `update: CAMERA_IMAGE requested but cameraSample is null, skipping`. With `update()`, that message means NSDK asked the data source for a camera image, but no image was available for that frame. This usually happens when the `frameProvider` given to `createLiveDataSource` returns `null` or returns a frame whose camera image is not ready yet. The app keeps running, but NSDK does not receive camera frames, so localization and other camera based features do not run. With `sendFrame(...)`, your app acquires the camera image directly with `frame.acquireCameraImage()`. If the image is not ready yet, `acquireCameraImage()` throws `NotYetAvailableException` and the app skips only that frame. ### Permissions NSDK uses the **camera** for AR and **location** for GPS and visual localization. The manifest entries in [Add permissions](#add-permissions) are required, but they do not grant access on their own. On Android 6.0 and later, your app must request `CAMERA` and location permissions at runtime with `ActivityCompat.requestPermissions(...)` and handle the user's response before you use features that depend on them. If you do not request these permissions, NSDK cannot use those device signals. For example, GPS and visual localization do not run, and Android does not grant the permission from the manifest entry alone. Refer to the Kotlin [sample projects](https://www.nianticspatial.com/docs/nsdk/sample_projects/) for examples of a `Utils` class that handles checking and requesting permissions. ### Show the camera feed At this point, NSDK can receive ARCore frames, but your app may still show a black screen. That usually means ARCore is providing camera frames, but your app is not drawing the camera image to the screen. NSDK reads the ARCore camera texture, but it does not render the camera background for you. To show the live camera feed, add the standard ARCore camera background renderer to your app. This code belongs to your ARCore setup, not to NSDK. In that renderer, do the following: 1. Draw a full screen quad that displays the camera texture. 2. Use `frame.transformCoordinates2d(...)` to get the correct texture coordinates for that camera image. 3. Call `session.setDisplayGeometry(...)` when the view size or screen rotation changes. The [Kotlin sample app](https://github.com/nianticspatial/nsdk-samples-kotlin) includes camera background rendering code that you can reuse. For an overview of the sample app, see [Kotlin sample projects](https://www.nianticspatial.com/docs/nsdk/sample_projects/). When you can see the live camera feed, you know your ARCore rendering setup is working and NSDK is receiving frames. Confirm that before you move on to NSDK features. ## Next steps After your project is set up, continue to [Sample Projects](https://www.nianticspatial.com/docs/nsdk/sample_projects/) to explore working examples of NSDK features.