Basic Concepts

Cameras

Every camera has a stereoscopic mode, also called a HMD (Head-Mounted Display) mode, that allows for the apps to be used within headsets.

ViewAR SDK offers three camera types: perspective, VR and AR.

Perspective Camera offers a bird’s eye view of the scene in a VR environment. Its main feature is an ability to change its pose (moving, rotating, dollying, etc.) by the user interaction with the viewport, either with touches or a mouse.

VR Camera offers a first-person view of the scene, similar to cameras of first-person shooters. It reads gyroscope input for rotation, but its movement needs to be passed programmatically. Movement can be implemented by on-screen joysticks for handheld devices, keyboard input for browser platforms, and bluetooth joysticks when using the HMD mode.

AR Camera takes input from device’s camera and gyroscope. Combined with state-of-the-art environment tracking systems, it provides an immersive AR experience. It is currently not available on the browser platform.

Freeze Frames

Since it is currently extremely impractical to faithfully scan and store real-world environments, capturing and restoring of an AR experience is practically impossible. ViewAR SDK offers a compromise in the way of freeze frames.

AR camera supports freezing and unfreezing of the view. While frozen, camera feed and pose no longer update as the user moves the device around. This allows for easy saving and loading of particular scene angles for later use as a freeze frame. These freeze frames can be switched between while preserving the spatial relations between scene objects as expected. They can also be uploaded to the cloud storage and shared between users of the app.

Models

A model is a collection of data, resources, and assets that fully describes a virtual 3D object. In the ViewAR ecosystem, models are created externally by 3D designers and uploaded to the ViewAR model repository, making them available to apps built using ViewAR SDK. A model uploaded to the ViewAR model repository is theoretically usable by any app built using ViewAR SDK, even those belonging other users. After upload, models are assigned to model catalogs belonging to ViewAR user accounts. Model catalogs are shared by all of user’s apps.

Models often contain semantic information used to describe them to users, such as names and textual descriptions of a product, gallery images, translations, semantic tags, etc.. Semantic information is stored in JSON format.

Additionally, models may have arbitrary app-specific data attached to them. This data can be used by the app to provide further functionality beyond what’s offered by the ViewAR SDK. App-specific data has no meaning outside the app itself.

Due to their often large size, models are usually not bundled with the app, and therefore need to be downloaded by the app itself before they can be used. Fortunately, ViewAR SDK encapsulates the resource management almost entirely, meaning that you don’t have to worry about the status of any model’s resources - ViewAR SDK downloads and caches any required resources on demand. This also means that the app is required to be online most of the time. If this is an issue, complete download of a model catalog can be handled via a certain API call.

Scene, Scene Objects and Containers

The collection of all virtual objects rendered by the app at any time is called a Scene.

Scene objects are visible parts of the scene. They are virtual 3D objects that a user can interact with. Scene objects usually have various kinds of properties that can be manipulated programmatically, changing their appearance in some way. Properties can affect object’s materials (e.g. upholstery of a sofa), geometry (e.g. width of a table), or even transitions between states (e.g. opening and closing of an awning). These properties are defined by models.

Containers are organisational units of the scene. There are two kinds of containers.

Grouped Containers encapsulate their children and make them behave as a single object with respect to interaction. This feature is used for creating complex objects using other scene objects as parts.

Ungrouped Containers do not encapsulate their children, allowing app users to interact with them separately. This is useful for logical grouping of scene objects. Scene itself is an ungrouped container. Ungrouped containers cannot be interacted with, but they can still be manipulated programmatically by changing their pose, visibility, etc..

Tracking

Tracking systems try to estimate world pose (position and orientation) of devices (e.g. head mounted displays, tablets) in real-time. An example use case could be the placement of furniture in an Augmented Reality space.

The tracking systems integrated by ViewAR are using inside-out tracking, which is done entirely on the device, in contrast to outside-in where external sensors are used to estimate the devices pose.

Marker-based Tracking uses fixed marker images, which are recognised by the tracking systems in order to get the pose of devices. Markers can be QR-Codes, Magazines, Photos or any kind of 2D images. Markers have to be known by the tracking system before they can be recognised. Inaccuracies occur as soon as the marker is not visible in the camera frame, because pose information can not be updated. Therefore, extended tracking tries to estimate the pose of a marker even if it is not visible using additional methods.

Marker-less Tracking uses camera input for detecting unique retrievable patterns (i.e. feature points) in the surrounding environment, for estimating the camera pose.

Furthermore, image processing algorithms are used for calculating 3D environmental information with the use of motion. These software-based approaches are less accurate than dedicated 3D sensors, so hardware-based solutions are used for more accuracy.

With 3D object recognition and tracking, a real world object can be recognised and its pose - estimated. Similarly to markers, the 3D target objects need to be known by the tracker beforehand.

Differences between tracking systems integrated by ViewAR
Tracker Target device SLAM Point cloud Mesh generation Marker tracking 3DObject recognition Additional hardware needed
ARKit iOS Yes Yes (in progress) Yes (ViewAR adds QR-Markers) No No
ARCore Android Yes Yes No No No No
Wikitude iOS, Android Yes No No YES (but not used) No (Yes for Wikitude beta) No
Kudan iOS, Android Yes (but not used) No No Yes No No
Vuforia iOS, Android, Windows Yes (but not used) No No Yes No No
Structure.io iOS Yes Yes Yes combination with marker enabled tracker No Yes
HoloLens Windows Yes Yes (in progress) (third party solution) (third party solution) Included
VisionLib iOS, Android No No No No Yes No
VisionLib +ARKit iOS Yes Yes No Yes (ViewAR adds QR-Markers) Yes No

Power economy

3D rendering and environment tracking are very CPU/GPU-intensive processes and can drain device’s battery rather quickly. Certain steps can be taken to significantly reduce the power consumption of the app:

  1. Pause rendering whenever showing a fullscreen overlay or view

  2. Activate tracking system only when needed

  3. Keep the number of scene objects to a minimum