Augmented Images with ARCore and Sceneform

My tweet

I was really excited when Sceneform was announced at I/O earlier this year, and couldn’t wait to get stuck into building AR apps on Android. Essentially Sceneform is an SDK which empowers you to work with ARCore without having to learn 3D graphics and write OpenGL. This is great news if you’ve always been interested in Augmented Reality but don’t know an awful lot about cartesian coordinates and matrix multiplication! (which is my situation for sure!)

ARCore was released in March but now developers don’t need to write OpenGL thanks to Sceneform!

Augmented Images interact with real world images and uses ARCore to create 3D experiences. In this post I’ll cover how to create a Augmented Images Android app using Sceneform (you can find a OpenGL example in the ARCore repo). I recommend checking out the Sceneform Codelab to get an understanding of the library. Also, be sure to take a look at the repo for Sceneform SDK which is where I got the inspiration for this post.

What we’ll build

Take a look at my twitter post to see the an example using planet earth, in this example I’ll go back to the future and render the classic Delorean.

Great Scott!

You can read up on the official developer guide which is where I started or just dive in and checkout my github with all of the code used in this demo!

cgathergood/Sceneform-Augmented-Images

Contribute to Sceneform-Augmented-Images development by creating an account on GitHub.

github.com

Loading Reference Images

To allow our application to identify our movie poster we need to create an image database. There are a few guidelines on image size and quality that’s worth reviewing in this guide, reference images should ideally contain many unique reference points in order to be easily identifiable. For the purposes of this demo I’ll just add our reference image at runtime but for large sets of images it’s recommended to add an image on a worker thread, or even better create a database file at compile time using the arcoreimg tool (this tool also gives you a handy “quality score”, anything 75 and over is ideal for a good reference image).

Back to the Future Universal Pictures

Importing 3D Models

The Sceneform Android Studio plugin lets you import and preview 3D models pretty easily. To get started head on over to Poly and grab a model of your choice (be sure to credit the creator). So far I’ve just been downloading OBJ files but FBX, and glTF formats are also supported.

Ensure you have a sampledata/ folder in your project directory and copy your files into here. Now it’s simply a case of right clicking on your .OBJ file and selecting Import Sceneform Asset. This creates a Scenform Binary Asset (SFB) and adds it to the app gradle file. That’s it, rinse and repeat for as many models as you desire. You will see that when selecting the SFB file the preview window should display the model, note that you can directly edit this which comes in particularly useful when deciding what size you want your rendered model to be (just be sure to re-run gradle build as the models are generated at compile time).

Notice the model is now defined in app.gradle

Identifying the reference image

Now we have our reference image and model setup we can start to code. Adding the image to the database is pretty straightforward, I followed the official guide and added in some null checks.

Remember generating a database on runtime is less performant than loading in a .imgdb file

Next we have to figure out if the device is actually looking at our image. This is done during onUpdateFrame of the ARCore tracking session. ARCore does the heavy lifting here and checks the images from the database against the camera, all we need to do is check for a match then display our model. This code can be easily extended so that you can have a gallery of reference images and models but to keep it simple I’m justing adding one.

Loading the Model

I’ve created a custom node to abstract some of the logic away here and called it AugmentedImageNode. The SFB files that were generated earlier are rendered using the ModelRenderable class. When you build a Renderable, Sceneform loads its resources in the background while returning a CompletableFuture. (This is a Java8 language feature, and I reckon you could achieve something similar with Kotlin Coroutines).

You can easily change the xyz position of the model as well as the rotation

Here we are waiting for the future to complete, then anchoring it to the center of the image. For now I haven’t tampered much with the position but I recommend giving this guide a read if you want to animate your objects, change the lighting or detect gestures. There’s a ton more to explore within Sceneform and Augmented Images but hopefully this can give you a good starting point!

Wrapping Up

First of all a big thanks to Steve P on Poly for creating the Delorean asset, it is a truly awesome piece of work! You can find the complete code base on github here, please let me know if you have any issues or suggestions (I’d love to convert this Kotlin eventually).

Where we’re going we don’t need VR headsets…

It’s early days for Sceneform and I’m eager to see what updates the team brings to the SDK in the coming months. Until next time!

Useful Resources

ARCore Developer Docs

Scenform SDK Repo

I/O 2018 Recap — There’s a ton of great talks on AR and Scenform from this years conference!