*your mileage may vary

In this tutorial, I’ll walk through how to start building an Augmented Reality Android application using Google’s new Sceneform SDK. First off, I’ll cover some quick concepts and then jump into project setup and some code. If you just want to dive in straight away, you can check out my GitHub repo below!

cgathergood/Your-First-AR-App-with-Sceneform

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over…

github.com

The screenshot below shows what we’ll build today!

The whole world in your hands

Core Concepts

ARCore is Google’s platform for building Augmented Reality experiences. It uses your mobile device for motion tracking, environmental understanding, and light estimation. With these three capabilities, ARCore can build an understanding of the real world around the device. With the mobile device’s position, we as developers can then augment the real world with information, 3D objects, or anything else!

You can read more about the Fundamental Concepts of ARCore here

Sceneform is a new SDK built to empower developers to work with ARCore without having to learn the intricacies of 3D graphics programming and OpenGL. ARCore was initially released with OpenGL samples, and as someone with very basic graphics programming knowledge, I found the learning curve to be super steep. With Sceneform, you can easily write in Java (or Kotlin) just like you would when building a normal Android app. The official samples for ARCore and Sceneform further illustrate this.

68% less lines of code with Sceneform!

Getting Started

To get started, you’ll need Android Studio 3.1 or higher. We want to create a new project with Kotlin support targeting API 24: Android 7.0, so let’s start with an empty activity for now.

Android Studio Project Setup

In our app/build.gradle file, we can add the dependency for Sceneform— remember to click “Sync Now” to update the project. Although we’ll be writing this tutorial in Kotlin, Sceneform uses some language constructs from Java 8, so we’ll need to explicitly add support since our minimum API is less than 26.

android {  
   ...  
   compileOptions {  
    sourceCompatibility JavaVersion.VERSION_1_8  
    targetCompatibility JavaVersion.VERSION_1_8  
   }  
}dependencies {  
   ...  
   implementation “com.google.ar.sceneform.ux:sceneform-ux:1.4.0”  
}

Since this is an augmented reality app, we’re going to have to ask the user for access to the camera. We also need an AR permission, which designates that this app requires ARCore. Finally, there’s a specific AR meta tag used by the Play Store to filter out applications for users who do not have an ARCore-enabled device.

<xml version="1.0" encoding="utf-8"?>

<manifest xmlns:android="http://schemas.android.com/apk/res/android"

package="com.gathergood.calum.ardemo">

<uses-permission android:name="android.permission.CAMERA" />

<uses-feature

android:name="android.hardware.camera.ar"

android:required="true" />

<application

android:allowBackup="true"

android:icon="@mipmap/ic_launcher"

android:label="@string/app_name"

android:roundIcon="@mipmap/ic_launcher_round"

android:supportsRtl="true"

android:theme="@style/AppTheme">

<meta-data

android:name="com.google.ar.core"

android:value="required" />

<activity android:name=".MainActivity">

<intent-filter>

<action android:name="android.intent.action.MAIN" />

<category android:name="android.intent.category.LAUNCHER" />

</intent-filter>

</activity>

</application>

</manifest>

Permissions at lines 5 to 8. Meta tag at lines 18 to 20

Adding our Model

To make life easier, and to allow us to import our own 3D assets, we’ll also include the Sceneform plugin for Android Studio. You can install this by selecting preferences, plugins, browse repositories, and search for Google Sceneform Tools (Beta)

This plugin lets you do some seriously cool stuff!

The first thing we want to do is get a 3D model to use—if you don’t want to make your own, you can head on over to Poly and grab one for free (just remember to credit the creator!). The Sceneform plugin supports OBJ, FBX and glTF file formats. Then, it’s just a case of creating a sampledata/ directory within your app and adding your object file.

Note that the purpose of this folder is for design time only, and these resources won’t be added to the final APK. The next step is right clicking your downloaded object and selecting “Import Sceneform Assert”. For this app, I’m going to pick the Earth asset created by Google.

After this gradle task has finished, you’ll find a reference to your object in the app level gradle file and a .sfb file inside the assets directory. This Sceneform Binary asset is what we’ll use inside our app. The Sceneform Android Studio plugin lets us view a realtime preview of our newly created asset.

Realtime rendered preview of our asset

Time to Code

The next step is to add the Sceneform fragment to our layout. You can just replace the default textview with this element. We’ll also add a Floating Action Button (FAB) to enable us to add our model to the real word.

activity_main.xml layout file

<?xml version="1.0" encoding="utf-8"?>

<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"

xmlns:app="http://schemas.android.com/apk/res-auto"

xmlns:tools="http://schemas.android.com/tools"

android:layout_width="match_parent"

android:layout_height="match_parent"

tools:context=".MainActivity">

<fragment

android:id="@+id/sceneform_fragment"

android:name="com.google.ar.sceneform.ux.ArFragment"

android:layout_width="match_parent"

android:layout_height="match_parent"

app:layout_constraintBottom_toBottomOf="parent"

app:layout_constraintEnd_toEndOf="parent"

app:layout_constraintStart_toStartOf="parent"

app:layout_constraintTop_toTopOf="parent" />

<android.support.design.widget.FloatingActionButton

android:id="@+id/floatingActionButton"

android:layout_width="wrap_content"

android:layout_height="wrap_content"

android:layout_marginBottom="8dp"

android:layout_marginEnd="8dp"

android:layout_marginStart="8dp"

app:layout_constraintBottom_toBottomOf="@+id/sceneform_fragment"

app:layout_constraintEnd_toEndOf="parent"

app:layout_constraintStart_toStartOf="parent"

app:srcCompat="@drawable/ic_explore_white_24dp" />

</android.support.constraint.ConstraintLayout>

We want our users to be able to add a model only when ARCore has detected a scene. In order to do this, we’ll only display the FAB when a suitable plane has been detected. In the official Google Codelab, they achieve something similar by drawing a custom Pointer on the screen when a model can be added to the scene.

The structure of our main activity is as follows:

1. Detect a suitable plane using ARCore onUpdate
2. Show the user the Floating Action Button (FAB)showFab
3. On click of the button load in our model addObject
4. Use the .sfb file to create a renderable object placeModel
5. Attach our renderable to the scene and display to the user addNodeToScene

MainActivity.kt

class MainActivity : AppCompatActivity() {

private lateinit var arFragment: ArFragment

private var isTracking: Boolean = false

private var isHitting: Boolean = false

override fun onCreate(savedInstanceState: Bundle?) {

super.onCreate(savedInstanceState)

setContentView(R.layout.activity_main)

arFragment = sceneform_fragment as ArFragment

// Adds a listener to the ARSceneView

// Called before processing each frame

arFragment.arSceneView.scene.addOnUpdateListener { frameTime ->

arFragment.onUpdate(frameTime)

onUpdate()

}

// Set the onclick lister for our button

// Change this string to point to the .sfb file of your choice :)

floatingActionButton.setOnClickListener { addObject(Uri.parse("NOVELO_EARTH.sfb")) }

showFab(false)

}

// Simple function to show/hide our FAB

private fun showFab(enabled: Boolean) {

if (enabled) {

floatingActionButton.isEnabled = true

floatingActionButton.visibility = View.VISIBLE

} else {

floatingActionButton.isEnabled = false

floatingActionButton.visibility = View.GONE

}

}

// Updates the tracking state

private fun onUpdate() {

updateTracking()

// Check if the devices gaze is hitting a plane detected by ARCore

if (isTracking) {

val hitTestChanged = updateHitTest()

if (hitTestChanged) {

showFab(isHitting)

}

}

}

// Performs frame.HitTest and returns if a hit is detected

private fun updateHitTest(): Boolean {

val frame = arFragment.arSceneView.arFrame

val point = getScreenCenter()

val hits: List<HitResult>

val wasHitting = isHitting

isHitting = false

if (frame != null) {

hits = frame.hitTest(point.x.toFloat(), point.y.toFloat())

for (hit in hits) {

val trackable = hit.trackable

if (trackable is Plane && trackable.isPoseInPolygon(hit.hitPose)) {

isHitting = true

break

}

}

}

return wasHitting != isHitting

}

// Makes use of ARCore's camera state and returns true if the tracking state has changed

private fun updateTracking(): Boolean {

val frame = arFragment.arSceneView.arFrame

val wasTracking = isTracking

isTracking = frame.camera.trackingState == TrackingState.TRACKING

return isTracking != wasTracking

}

// Simply returns the center of the screen

private fun getScreenCenter(): Point {

val view = findViewById<View>(android.R.id.content)

return Point(view.width / 2, view.height / 2)

}

/**

* @param model The Uri of our 3D sfb file

*

* This method takes in our 3D model and performs a hit test to determine where to place it

*/

private fun addObject(model: Uri) {

val frame = arFragment.arSceneView.arFrame

val point = getScreenCenter()

if (frame != null) {

val hits = frame.hitTest(point.x.toFloat(), point.y.toFloat())

for (hit in hits) {

val trackable = hit.trackable

if (trackable is Plane && trackable.isPoseInPolygon(hit.hitPose)) {

placeObject(arFragment, hit.createAnchor(), model)

break

}

}

}

}

/**

* @param fragment our fragment

* @param anchor ARCore anchor from the hit test

* @param model our 3D model of choice

*

* Uses the ARCore anchor from the hitTest result and builds the Sceneform nodes.

* It starts the asynchronous loading of the 3D model using the ModelRenderable builder.

*/

private fun placeObject(fragment: ArFragment, anchor: Anchor, model: Uri) {

ModelRenderable.builder()

.setSource(fragment.context, model)

.build()

.thenAccept {

addNodeToScene(fragment, anchor, it)

}

.exceptionally {

Toast.makeText(this@MainActivity, "Error", Toast.LENGTH_SHORT).show()

return@exceptionally null

}

}

/**

* @param fragment our fragment

* @param anchor ARCore anchor

* @param renderable our model created as a Sceneform Renderable

*

* This method builds two nodes and attaches them to our scene

* The Anchor nodes is positioned based on the pose of an ARCore Anchor. They stay positioned in the sample place relative to the real world.

* The Transformable node is our Model

* Once the nodes are connected we select the TransformableNode so it is available for interactions

*/

private fun addNodeToScene(fragment: ArFragment, anchor: Anchor, renderable: ModelRenderable) {

val anchorNode = AnchorNode(anchor)

// TransformableNode means the user to move, scale and rotate the model

val transformableNode = TransformableNode(fragment.transformationSystem)

transformableNode.renderable = renderable

transformableNode.setParent(anchorNode)

fragment.arSceneView.scene.addChild(anchorNode)

transformableNode.select()

}

}

Final Product

https://www.youtube.com/watch?v=LWnXcbchp4Y

And that’s it! In less than 150 lines of code, we have our very first augmented reality app. I’ve added comments where applicable in the MainActivity.kt. Please feel free to get in touch with me if you have any questions whatsoever. Hopefully you can use this as a starting point for your AR experience on Android. Thanks for reading!

cgathergood/Your-First-AR-App-with-Sceneform

GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over…

github.com