SwiftyNsdk
Type Aliases
| Name | Type | Summary |
|---|---|---|
| NetworkRequestId = ARDK_NetworkRequestId | ARDK_NetworkRequestId | - |
| NsdkHandle = ARDK_Handle | ARDK_Handle | A type alias for the native NSDK handle used to interface with the underlying C API. |
| NsdkVpsAnchorId = String | String | A type alias for VPS anchor identifiers. VPS anchors are identified by unique string identifiers that can be used to track and manage anchors throughout their lifecycle. |
Associated Types
| Name | Type | Summary |
|---|---|---|
| Configuration | Configuration | The type of configuration used by this session |
Classes
| Name | Type | Summary |
|---|---|---|
| AssetResult | AssetResult | Contains all the AssetInfo objects returned by a query to the Sites Manager service. |
| AwarenessImageResult | AwarenessImageResult | Image-based awareness result containing a raw image. |
| AwarenessResult | AwarenessResult | Base class for awarness results such as depth and segmentation. Provides common properties like frame ID, timestamp, camera pose, and intrinsics. |
| BundlePlaybackDatasetLoader | BundlePlaybackDatasetLoader | A loader that retrieves playback dataset data from the app bundle. This is the default implementation for loading datasets from Bundle.main.Frame images and depth data are loaded on-demand when requested. |
| DataResourceOwner | DataResourceOwner | - |
| DepthResult | DepthResult | Contains depth estimation results from the NSDK depth processing system. ## Overview Depth results are generated by the depth processing system and include: - Disparity maps for depth estimation. Unlike direct depth maps that provide distance values in meters, disparity maps contain pixel offset values that represent relative depth differences. These values must be converted to actual depth using camera intrinsic parameters and baseline information. - Camera pose and orientation information - Camera intrinsic parameters for coordinate transformations - Error status and metadata |
| MeshData | MeshData | Contains 3D mesh data for rendering and visualization. MeshData provides access to 3D mesh geometry including vertices, indices,normals, and texture coordinates. ## Overview MeshData includes: - Vertices: 3D position data for mesh geometry - Indices: The triangles that make up the mesh - Normals: Surface normal vectors for the vertices, commonly used for lighting calculations (only available for live meshing) - UVs: Texture coordinates for the vertices, used for mapping textures to the mesh (only available for mesh downloader) ## Example Usage ## Memory Management Mesh data is backed by native memory that is automatically managed. The data remains valid as long as the MeshData instance exists. |
| final MeshDownloaderResults | MeshDownloaderResults | Contains the downloaded mesh geometry data for a VPS location. This object holds an array of mesh results, where each result includes mesh geometry, texture data (if requested), and the transform matrix that positions the mesh in world space. |
| final NsdkDepthSession | NsdkDepthSession | Depth feature session for NSDK. Provides control over depth sensing capabilities. Upon starting the depth session, NSDK will begin processing AR frames to generate depth data. The latest depth data can be retrieved using latestDepth(), and latestImageParams()provides information to synchronize the depth image with camera frame. |
| final NsdkDeviceMappingSession | NsdkDeviceMappingSession | A session for creating VPS maps from AR data on the local device. The device mapping feature provides capabilities for locally building persistent maps that can be used for Visual Positioning System (VPS) localization. These maps capture the visual features and spatial structure of an environment. |
| final NsdkMapStorage | NsdkMapStorage | A storage system for managing device-generated maps. The map storage feature provides capabilities for capturing, storing, and managing map data from AR sessions. This data can be persisted and used for Visual Positioning System (VPS) localization and map updates. |
| final NsdkMeshDownloader | NsdkMeshDownloader | A session-scoped utility for downloading mesh geometry associated with VPS locations. |
| final NsdkMeshingSession | NsdkMeshingSession | A session for real-time 3D mesh generation from AR camera frames. The meshing feature provides capabilities for processing AR session data and generating a triangle mesh representation of the physical environment in real-time. The mesh is divided into chunks that can be individually queried and updated as the environment is scanned. |
| final NsdkObjectDetectionSession | NsdkObjectDetectionSession | - |
| final NsdkRecordingExporter | NsdkRecordingExporter | A session for exporting scan recordings to various formats. NsdkRecordingExporter provides capabilities for converting saved scan datainto recorderV2 format for use in Unity Playback or activating VPS. |
| final NsdkScanningSession | NsdkScanningSession | A session for 3D scanning and visualization functionality. The scanning feature provides capabilities for capturing, processing, and exporting 3D scan data from AR sessions. Scans of a location can be processed by the Visual Positioning System's (VPS's) cloud services to enable VPS localization. |
| final NsdkSemanticsSession | NsdkSemanticsSession | A session for semantic segmentation and environmental understanding. NsdkSemanticsSession provides capabilities for understanding the semantic structureof the environment by classifying pixels into different object categories. This enables applications to make intelligent decisions based on environmental context. ## Overview Semantics features include: - Real-time semantic segmentation of camera images - Multiple semantic categories (sky, ground, buildings, people, etc.) - Confidence maps for semantic classifications - Packed channel data for efficient processing - Suppression masks for filtering unwanted areas ## Usage Pattern |
| final NsdkSession | NsdkSession | The main entry point for the NSDK (Native SDK) framework. NsdkSession provides the core functionality for AR applications, managing the lifecycleof NSDK features and serving as a factory for specialized sessions like VPS, WPS, scanning, and mapping. This class handles frame data processing, configuration management, and resource cleanup. ## Overview Use NsdkSession to:- Initialize the NSDK with your API key and configuration - Send camera frame data for processing - Create specialized feature sessions (VPS, WPS, Scanning, Mapping) - Query required input data formats - Manage the lifecycle of NSDK resources ## Example Usage |
| final NsdkSitesSession | NsdkSitesSession | - |
| @MainActor NsdkView | NsdkView | - |
| final NsdkVps2Session | NsdkVps2Session | - |
| final NsdkVpsCoverageSession | NsdkVpsCoverageSession | A session object for querying Visual Positioning System (VPS) coverage data related resources. |
| final NsdkVpsSession | NsdkVpsSession | A session for Visual Positioning System (VPS) functionality. NsdkVpsSession provides capabilities for precise localization using visual features.VPS can determine device position and orientation relative to a pre-mapped environment, enabling persistent AR experiences that maintain accuracy across sessions. ## Example Usage |
| final NsdkWpsSession | NsdkWpsSession | A session for World Positioning System (WPS) functionality. WPS provides the 3D position and orientation of the device in geographic coordinates as an alternative to using device GPS and compass heading data. WPS provides greater accuracy and frame-to-frame stability than standard GPS positioning, making it more suitable for AR applications. As the user moves around, WPS maintains the device's position, making it suitable for continuous use over long periods of time and long distances. WPS will work in any location where the phone has a GPS signal, but the accuracy will vary depending on GPS accuracy. |
| final ObjectDetectionClassNamesBuffer | ObjectDetectionClassNamesBuffer | A read-only container for the list of class names supported by the object detection model. |
| final ObjectDetectionImageParams | ObjectDetectionImageParams | A read-only container for the image params of object detection |
| final ObjectDetectionMetadata | ObjectDetectionMetadata | A read-only container for the metadata of an object detection frame. |
| final ObjectDetectionResult | ObjectDetectionResult | A read-only container for the results of a single, successful object detection frame. |
| OrganizationResult | OrganizationResult | Contains all the OrganizationInfo objects returned by a query to the Sites Manager service. |
| @MainActor PlaybackBackgroundRenderer | PlaybackBackgroundRenderer | - |
| PlaybackDataset | PlaybackDataset | A dataset loaded from a capture JSON file containing frame metadata. This class uses on-demand loading for frame images and depth data. Only the currently requested frame is loaded into memory, reducing memory pressure for large datasets. |
| PlaybackDatasetLoader | PlaybackDatasetLoader | Base class for loading playback dataset data from various sources. This class provides a base implementation that must be subclassed. Subclasses must override loadCaptureJSON(), loadImage(imageName:), loadDepthData(depthFileName:), andloadDepthConfidence(confidenceFileName:) to provide concrete implementations.The loader uses on-demand loading - only the capture JSON is loaded upfront, and frame images/depth data are loaded when requested by PlaybackDataset.- Note: This class acts as an abstract base class. Do not instantiate directly. |
| PlaybackSession | PlaybackSession | A session that plays back frame metadata and images from a loaded dataset. Playback runs on a background queue and continuously loops through frames at the framerate specified in the dataset. The session notifies its delegate of each frame update. |
| @MainActor PlaybackView | PlaybackView | - |
| RaycastBuffer | RaycastBuffer | A read-only container for the raycast buffer information generated during scanning. |
| SemanticsResult | SemanticsResult | Contains semantic segmentation results from the NSDK semantics processing system. SemanticsResult provides semantic understanding of the environment by classifyingpixels in camera images into different object categories (e.g., sky, ground, buildings, people, vehicles). This enables applications to understand the scene structure and make intelligent decisions based on environmental context. ## Overview Semantic segmentation results include: - Confidence maps: Per-pixel confidence scores for semantic classifications - Packed channels: Multiple semantic categories encoded in a single image - Suppression masks: Masks indicating areas to be ignored or suppressed - Metadata: Frame information, timestamps, and error status ## Example Usage |
| SiteResult | SiteResult | Contains all the SiteInfo objects returned by a query to the Sites Manager service. |
| SitesResult | SitesResult | Base class for results returned by queries to the Sites Manager service. |
| UserResult | UserResult | Contains the user information returned by a query to the Sites Manager service. |
| VoxelBuffer | VoxelBuffer | A read-only container for the voxel buffer information generated during scanning. |
Protocols
| Name | Type | Summary |
|---|---|---|
| NsdkFeatureSession | NsdkFeatureSession | A protocol that defines the common lifecycle and configuration interface for NSDK feature sessions. |
| NsdkLogCallback : AnyObject | AnyObject | Protocol for receiving log messages from NSDK. Implement this protocol to receive NSDK log messages in your application. The callback will be invoked on background threads, so ensure your implementation is thread-safe. ## Example Usage |
| NsdkSessionDelegate : ARSessionDelegate, PlaybackSessionDelegate | ARSessionDelegate, PlaybackSessionDelegate | - |
| PlaybackDatasetSource | PlaybackDatasetSource | Protocol for loading playback dataset data from various sources. This protocol abstracts data retrieval, allowing for different implementations such as bundle loading, file system loading, remote loading, or mock data for testing. Implementations are used for on-demand frame loading - the loader is passed to PlaybackDataset which calls these methods when frames are requested. |
| @MainActor PlaybackRenderer | PlaybackRenderer | - |
| PlaybackSessionDelegate : Sendable | Sendable | Delegate protocol for receiving frame updates during playback. |
| ResourceOwner : AnyObject | AnyObject | - |
Structs
| Name | Type | Summary |
|---|---|---|
| AreaTarget | AreaTarget | Contains a CoverageArea and its associated LocalizationTarget. |
| AreaTargetResult | AreaTargetResult | - |
| ARUtils | ARUtils | Utility functions for AR and device capability detection. ARUtils provides helper methods for detecting device capabilitiesand AR features that are relevant to NSDK functionality. |
| AssetInfo | AssetInfo | Represents asset information from the Sites Manager service. Maps to proto messages AssetRecord, AssetData, and AssetComputedValues. |
| AssetMeshData | AssetMeshData | Mesh-specific asset data. Maps to proto message AssetMeshData. |
| AssetSplatData | AssetSplatData | Splat-specific asset data. Maps to proto message AssetSplatData. |
| AssetVpsData | AssetVpsData | VPS-specific asset data. Maps to proto message AssetVpsData. |
| AuthInfo | AuthInfo | Authentication information containing token claims. Contains parsed JWT claims including token string, expiration, user information, and other standard JWT fields. |
| AwarenessImageParams | AwarenessImageParams | - |
| CoverageArea | CoverageArea | Represents a geographic area where VPS localization is possible |
| CoverageAreaResult | CoverageAreaResult | Contains all the CoverageArea objects returned by a query to the VPS Coverage service. |
| GeolocationData | GeolocationData | Struct representing geolocation data including latitude, longitude, altitude, heading, and orientation. |
| HintImageResult | HintImageResult | Image data returned by a query to a VPS hint image URL. |
| ImageMath | ImageMath | Provides affine transformation utilities for image processing. All affine matrices returned by this class operate in normalized coordinates, where image space is mapped to the [0, 1] range in both axes with origin at top-left. |
| LatLng | LatLng | Struct representing a geographical coordinate with latitude and longitude in degrees. |
| LocalizationTarget | LocalizationTarget | Represents a real-world point of interest that is a VPS localization target. VPS localization is more likely to succeed when a localization target is in camera view. |
| LocalizationTargetResult | LocalizationTargetResult | Contains all the LocalizationTarget objects returned by a query to the VPS Coverage service. |
| MapMetadata | MapMetadata | Structure representing the metadata of a device map for visualization and processing. |
| MeshDownloaderResult | MeshDownloaderResult | Represents a single mesh result with geometry, texture, and transform data. |
| MeshUpdateInfo | MeshUpdateInfo | Information about mesh chunk updates from live meshing operations. Returned by NsdkMeshingSession.updatedMeshInfos() to indicate whichmesh chunks have been modified or removed since the last update. |
| NsdkBuffer | NsdkBuffer | A buffer containing binary data for NSDK operations. NsdkBuffer provides a safe wrapper around binary data buffers used byvarious NSDK features. It handles memory management and provides convenient access to buffer data. ## Overview NSDK buffers are used for: - Image data transfer - Mesh data storage - Configuration data - Any binary data that needs to be passed between Swift and the native NSDK layer ## Example Usage ## Memory Management NsdkBuffer automatically manages memory allocation and deallocation.When created from Swift Data, it maintains a reference to prevent premature deallocation. |
| NsdkFeatureStatus | NsdkFeatureStatus | Status flags for NSDK features indicating their current operational state. NsdkFeatureStatus is an option set that represents various status conditionsfor NSDK features like VPS, WPS, scanning, and mapping. Multiple status flags can be active simultaneously to provide detailed status information. ## Overview Use this to monitor the health and state of NSDK features: - Check for errors that need attention - Monitor initialization progress - Verify configuration and API key validity - Ensure features are ready for operation ## Example Usage |
| NsdkFrameData | NsdkFrameData | A complete frame of data captured from an AR session. NsdkFrameData encapsulates all the sensor data, images, and tracking informationfrom a single AR frame. This includes camera images, depth data, device pose, GPS location, compass heading, and camera intrinsics. ## Overview Frame data is the primary input to NSDK for all AR processing tasks including: - Visual positioning and localization - 3D scanning and reconstruction - Map building and tracking - AR location positioning ## Example Usage |
| NsdkInputDataFlags | NsdkInputDataFlags | Flags indicating which types of input data are required by NSDK. NsdkInputDataFlags is an option set that specifies which data typesshould be included in frames sent to NSDK. Use getRequestedDataInputs()to determine which data is currently needed, then include only the requested data types in your frame data for optimal performance. ## Overview NSDK features dynamically request different types of input data based on: - Which features are active (VPS, WPS, scanning, mapping) - Current processing state and requirements - Device capabilities and available sensors ## Example Usage |
| NsdkPathConfig | NsdkPathConfig | - |
| NsdkPlaybackFrame | NsdkPlaybackFrame | A frame of data from a playback session, bundling together frame metadata and associated buffers. |
| NsdkUtils | NsdkUtils | Utility functions for NSDK string management and memory handling. NsdkUtils provides helper methods for safely managing C string conversionsand memory allocation when working with the NSDK C API. ## Overview The utilities in this struct help manage the complexity of converting between Swift strings and C strings while ensuring proper memory cleanup and avoiding memory leaks. |
| OrganizationInfo | OrganizationInfo | Represents organization information from the Sites Manager service. |
| RawImage | RawImage | A raw image containing pixel data for NSDK operations. RawImage provides access to image data in various formats (RGB, grayscale,depth, etc.) used by NSDK features like depth processing, semantic segmentation, and image analysis. ## Overview Raw images are used throughout NSDK for: - Camera frame processing - Depth map representation - Semantic segmentation results - Image format conversions - Computer vision operations ## Example Usage ## Memory Management This object's pointers are only valid when the NSDK objects that holds it are still in scope. |
| SemanticsChannels | SemanticsChannels | A set of semantic channels represented as a bitmask. This OptionSet allows for efficient bitwise operations on channel sets. |
| SiteInfo | SiteInfo | Represents site information from the Sites Manager service. |
| TextureUtils | TextureUtils | - |
| TimeoutError | TimeoutError | - |
| UserInfo | UserInfo | Represents user information from the Sites Manager service. |
| Vps2GeolocationData | Vps2GeolocationData | GPS and heading data calculated by VPS2. |
| Vps2NetworkRequestRecord | Vps2NetworkRequestRecord | - |
| Vps2Pose | Vps2Pose | Pose in AR coordinate space calculated by VPS2 from a geolocation. |
| Vps2Transformer | Vps2Transformer | - |
| VpsAnchorUpdate | VpsAnchorUpdate | Contains the latest tracking information for a VPS anchor. Anchor updates are retrieved via getAnchorUpdate(anchorId:) and provide themost current information about an anchor's position, orientation, and tracking status. This is a snapshot of the anchor, and the latest anchor update should be used every frame. |
| WpsLocation | WpsLocation | Contains world positioning data from the WPS (World Positioning System). WpsLocation provides global positioning information that combines GPS/GNSSdata with visual positioning for enhanced accuracy and reliability. ## Overview WPS location data includes: - Reference GPS coordinates (latitude, longitude, altitude) - Transformation matrix for coordinate conversions - Status information about positioning quality ## Example Usage |
Enums
| Name | Type | Summary |
|---|---|---|
| AgeLevel | AgeLevel | Codes describing the age level of the user. This enum represents the age classification for users of the NSDK |
| AssetDeploymentType | AssetDeploymentType | Asset deployment type. Maps to proto enum AssetDeploymentType. |
| AssetPipelineJobStatus | AssetPipelineJobStatus | Asset pipeline job status. Maps to proto enum AssetPipelineJobStatus. |
| AssetStatusType | AssetStatusType | Asset status. Maps to proto enum AssetStatusType. |
| AssetType | AssetType | Asset type - determines which typed asset data is present. Maps to proto enum AssetType. |
| AwarenessError | AwarenessError | - |
| ExportResolution | ExportResolution | Resolution option for exported scan images. When exporting a recording, this controls which image resolutions are included in the payload. |
| ImageType | ImageType | Enum representing various image types supported by NSDK. |
| NsdkAsyncState<Value, Error> where Error : Error | Error | Reports the state of an asynchronous NSDK operation. |
| NsdkError | NsdkError | Errors thrown by the NSDK API. NsdkError a subset of all the ARDK_Statuscodes returned by the C API,containing just those that can occur in the Swift environment. |
| NsdkLogLevel | NsdkLogLevel | Defines the available logging levels for NSDK. NsdkLogLevel controls the verbosity of logging output from the NSDK system.Logging can be configured separately for stdout, files, and callback functions. ## Overview Log levels follow a hierarchical structure where higher levels include all messages from lower levels. For example, setting the level to .warn will include warning,error, and fatal messages, but exclude debug and info messages. |
| NsdkNetworkError | NsdkNetworkError | Possible errors from network operations. |
| NsdkNetworkRequestStatus | NsdkNetworkRequestStatus | Status of a network request. |
| PlaybackDatasetConstants | PlaybackDatasetConstants | Constants for playback dataset file names and extensions. |
| SemanticsChannelName | SemanticsChannelName | - |
| TypedAssetData | TypedAssetData | Discriminated union for typed asset data. One of mesh, splat, or vps will be set based on the asset type. |
| Vps2NetworkRequestType | Vps2NetworkRequestType | - |
| Vps2TrackingState | Vps2TrackingState | - |
| VpsGraphOperationError | VpsGraphOperationError | - |
| WpsError | WpsError | Represents the current error status of the WPS (World Positioning System) feature. |
Methods
| Name | Type | Summary |
|---|---|---|
| configure | void | - |
| featureStatus | NsdkFeatureStatus | Gets the current status of the feature. This method reports any errors or warnings that have occurred within the feature system. Check this periodically to monitor the health of operations. Once an error is flagged, it will remain flagged until the problematic process runs again and completes successfully. - Returns: Feature status flags indicating current state and any issues |
| info | String | Returns details about the loader configuration. - Returns: A string describing the loader configuration |
| loadCaptureJSON | Data? | Loads the capture JSON data from the data source. - Returns: The JSON data as Data, or nil if loading fails |
| loadDepthConfidence | Data? | Loads depth confidence data from the data source. This method is called on-demand when confidence data is requested. - Parameter confidenceFileName: The filename of the confidence data file (e.g., "confidence_00000000.bin") - Returns: The confidence data as Data, or nil if loading fails or confidence data is not available |
| loadDepthData | Data? | Loads depth data from the data source. This method is called on-demand when depth data is requested. - Parameter depthFileName: The filename of the depth data file (e.g., "depth_00000000.bin") - Returns: The depth data as Data, or nil if loading fails or depth data is not available |
| loadImage | CGImage? | Loads an image from the data source. This method is called on-demand when a frame image is requested. - Parameter imageName: The filename of the image (e.g., "frame_00000000.jpg") - Returns: The image as a CGImage, or nil if loading fails |
| nsdkSession | void | This is called when a new frame has been sent to the underlying nsdk @param session The nsdk session being run. @param frame The nsdk frame that was just sent to the underlying native NSDK subsystems |
| onLog | void | Called when NSDK generates a log message. - Parameters: - level: The severity level of the log message - message: The log message content - fileName: Optional source file name where the log was generated - fileLine: Line number in the source file - funcName: Optional function name where the log was generated |
| playbackSession | void | Called each time a new frame is ready during playback. This method is called on the playback queue, so any UI updates should be dispatched to the main queue. - Parameters: - session: The playback session that generated the update - frame: The frame data including metadata, image, and depth buffers |
| @MainActor renderFrame | void | - |
| start | void | Starts the feature session. After starting, the session will begin processing incoming frame data according to its configured behavior. The session must be configured before starting. |
| stop | void | Stops the feature session. This halts all processing. The session can be reconfigured and restarted after stopping. |