After Google Gave Up on Sceneform, This Developer Built a Better Way to Bring AR Faces to Life

When Sceneform was discontinued and archived back in 2020, it made creating ARCore apps on Android more difficult. Inspired by the original Sceneform APIs and based on the ARCore samples, I have created AugmentedFaceFragment and AugmentedFaceListener interface to be able to easily create ARCore AugmentedFaces features on Android.

I will be releasing 3 articles building different AugmentedFaces features using AugmentedFaceFragment and AugmentedFaceListener interface.

What you will build

Part 1 of the series will cover the basics of the AugmentedFaceFragment and AugmentedFaceListener as well as the overview of all the helper classes. At the end of this article you will build a simple demo by writing a few lines of code.

Project setup

Download the Starter Project

Clone the repository:

git clone https://github.com/droid-girl/arfaces_labs.git

What’s our starting point?

Our starting point is a modified version of ARCore SDK sample for Augmented Faces. The code has been modified in a way that we can now easily add different textures and objects to a face object.

Project structure

• Helpers - original Java files from sample code
• Rendering - original Java files from sample code that handles rendering of AR objects and background

Additional files for the repository:

• AugmentedFaceFragment.kt - handles rendering and tracking of Augmented Faces
• AugmentedFaceListener.kt - handles Add and Update Augmented Faces callbacks
• AugmentedFaceNode.kt - handles the rendering of a face including a texture and associated 3D models
• AugmentedFaceRenderer.kt - renders face texture
• FaceRegion.kt - renders 3D model located on a defined FaceLandmark

MainActivity setup

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <androidx.fragment.app.FragmentContainerView android:name="com.ar.arfaces.arface.AugmentedFaceFragment"
        android:id="@+id/face_view"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="top" />

</LinearLayout>

class MainActivity : AppCompatActivity(), AugmentedFaceListener {

    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        binding.faceView.getFragment<AugmentedFaceFragment>().setAugmentedFaceListener(this)
        setContentView(binding.root)
    }

    override fun onFaceAdded(face: AugmentedFaceNode) {}

    override fun onFaceUpdate(face: AugmentedFaceNode) {}

}

In activity_main.xml we add AugmentedFaceFragment as a main view and to be able to receive events from this fragment, we need to define and set AugmentedFaceListener to our fragment.

onFaceAdded method will be called when ARCore detects a new face and onFaceUpdate will be called on each frame update.

Let’s add a face texture as our next step.

Add face texture

We will use ARCore sample assets for our first face mask. You can find assets/models folder in your project. Let’s add freckles.png as face texture:

In MainActivity.kt modify onFaceAdded method as follows:

override fun onFaceAdded(face: AugmentedFaceNode) {
   face.setFaceMeshTexture("models/freckles.png")
}

Add 3D models

Introduction to AugmentedFaceNode

AugmentedFace class uses the face mesh and center pose to identify face regions. These regions are:

• Left forehead (LEFT_FOREHEAD)
• Right forehead (RIGHT_FOREHEAD)
• Tip of the node (NOSE_TIP)

This project includes the AugmentedFaceNode class. Inspired by Sceneform, it is a node that will render visual effects on a detected face using ARCore. AugmentedFaceNode defines same face regions as AugmentedFace in a companion object:

companion object {
   enum class FaceLandmark {
       FOREHEAD_RIGHT,
       FOREHEAD_LEFT,
       NOSE_TIP
   }
}

Later in this tutorial, we will extend FaceLandmark enum class and add our own face regions.

AugmentedFaceNode includes faceLandmarks HashMapthat connects FaceLandmark with a 3D model.

The source code for this project can be found here.