hz-unity-face-tracking

name	hz-unity-face-tracking
license	Apache-2.0
description	Drive ARKit-blendshape-rigged head/face models in Unity with the wearer's facial expressions on Meta Quest via Meta Movement SDK (face tracking + A2E). Use when a user has an FBX with the 52 ARKit blendshapes (any prefix, _L/_R suffixes) and wants it to animate from face tracking on Quest Pro / Quest 3 / Quest 3S.

Unity Face Tracking for ARKit-Rigged Models (Meta Movement SDK)

End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Quest. Uses the public OVRCustomFace extension hook — no OVR_INTERNAL_CODE, ships to 3P.

The model's blendshape names must follow the ARKit naming convention (camelCase, _L/_R suffixes, e.g. eyeBlink_L, jawOpen, mouthSmile_R). An optional prefix like blendShape2.eyeBlink_L is automatically stripped.

When to use

• User has an FBX/GLB/mesh with ARKit-named blendshapes and wants it driven by Quest face tracking.
• User asks: "animate this head with my face", "drive these blendshapes from face tracking", "use Movement SDK A2E with my model", "wire ARKit shapes to Quest".
• Target device: Quest Pro, Quest 3, Quest 3S (Quest 2 is no-op — no face cameras).

Prerequisites checklist

1. Quest face tracking-capable headset (Pro / 3 / 3S).
2. Packages in Packages/manifest.json:
   • com.meta.xr.sdk.core (Meta XR Core — provides OVRFaceExpressions, OVRCustomFace)
   • com.meta.xr.sdk.movement (Meta Movement SDK — A2E + retargeting helpers)
3. Assets/Oculus/OculusProjectConfig.asset (verify via Project Settings → Meta XR):
   • faceTrackingSupport: 1 (Supported) or 2 (Required)
   • eyeTrackingSupport: 1 if the rig has gaze
4. Android manifest (Assets/Plugins/Android/AndroidManifest.xml) permissions:
   • <uses-feature android:name="oculus.software.face_tracking" android:required="false" />
   • <uses-permission android:name="com.oculus.permission.FACE_TRACKING" />
   • <uses-permission android:name="android.permission.RECORD_AUDIO" /> (required for A2E)
   • For eye gaze: oculus.software.eye_tracking + com.oculus.permission.EYE_TRACKING
5. OVRCameraRig in the scene with OVRManager.FaceTrackingDataSources including Audio (A2E) — if you skip Audio, mouth motion is visual-only.

After any change to OculusProjectConfig, call meta_update_android_manifest to regenerate the manifest.

Approach (high level)

1. Drop the ARKitOVRCustomFace script (in references/ARKitOVRCustomFace.cs) into the project.
2. Add an OVRFaceExpressions component to the OVRCameraRig (or anywhere in the scene).
3. On the GameObject that has the model's SkinnedMeshRenderer, add ARKitOVRCustomFace. Adding the component triggers Reset(), which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names.
4. Wire the component's FaceExpressions field to the OVRFaceExpressions instance.
5. For eye gaze: drop ARKitEyeGazeBlendshapeDriver in (see Eye tracking below).
6. Press Play with Meta XR Link, or build APK and side-load. On first launch, accept the Face Tracking / Eye Tracking / Microphone permission prompts.

That's the whole flow. Details below.

Eye tracking (DO THIS — face expressions alone don't work)

Don't rely on OVRFaceExpressions.EyesLook* for eye gaze. Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is OVREyeGaze, which taps the eye tracker directly.

For ARKit rigs with eyeLook* blendshapes (no eye bones), this skill ships ARKitEyeGazeBlendshapeDriver:

• Reads gaze rotation from two OVREyeGaze components (one per eye, TrackingMode = HeadSpace).
• Computes rotation relative to a head reference (e.g. CenterEyeAnchor).
• Decomposes pitch → eyeLookUp/Down_{L,R}, yaw → eyeLookIn/Out_{L,R} (with the ARKit "_In = toward nose" convention).
• Writes weights in LateUpdate, so it overrides whatever ARKitOVRCustomFace wrote in Update.

For rigs with eye bones (no eyeLook* blendshapes), skip the blendshape decomposition and just parent OVREyeGaze to each eye bone with ApplyRotation = true — the component will rotate the bone directly.

Eye gaze setup (blendshape rigs)

1. Create two empty GameObjects under CenterEyeAnchor (or your head transform): LeftEyeGaze, RightEyeGaze.
2. Add OVREyeGaze to each. Set Eye = Left / Right, TrackingMode = HeadSpace, ApplyPosition = false, ApplyRotation = true, ConfidenceThreshold = 0.5.
3. Add ARKitEyeGazeBlendshapeDriver to the head's SkinnedMeshRenderer GameObject (alongside ARKitOVRCustomFace). Wire leftEye, rightEye, and referenceFrame (= CenterEyeAnchor).
4. Tweak maxAngleDeg (default 30°) and smoothing (default 0.4) to taste.

Step-by-step

1. Install the script

Copy references/ARKitOVRCustomFace.cs into Assets/Scripts/ARKitOVRCustomFace.cs. It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a MapBlendshapes() method that scans SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i), strips any prefix before the last ., lowercases, and matches against the table. Unmatched mesh blendshapes are set to OVRFaceExpressions.FaceExpression.Max (sentinel — skipped at runtime).

2. Enable Movement SDK + permissions

Project settings:

Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)

Then regenerate manifest:

• Unity MCP: meta_update_android_manifest
• Or Editor menu: Meta → Tools → Update AndroidManifest.xml

3. Scene setup

Scene Hierarchy
├── OVRCameraRig                         (from meta_add_camerarig)
│   └── (add) OVRFaceExpressions         component
└── YourHeadModel
    └── ...SkinnedMeshRenderer GO...
        ├── SkinnedMeshRenderer          (existing)
        └── (add) ARKitOVRCustomFace     component
             └── FaceExpressions = the OVRFaceExpressions ref
             └── retargetingType = Custom (set automatically by base when overriding)
             └── Mappings[] = auto-filled on Reset()
             └── BlendShapeStrengthMultiplier = 100 (default; OVR weights are 0–1, mesh wants 0–100)

If the head has multiple SkinnedMeshRenderers (e.g. separate teeth/tongue meshes), add ARKitOVRCustomFace to each one.

4. OVRManager — enable Audio as a data source (A2E)

On OVRCameraRig's OVRManager component:

• Face Tracking Data Sources → check Visual AND Audio (Audio = A2E; produces mouth shapes from microphone when the visual face cameras can't see something — talking, occlusion, etc.).

5. Trigger the mapping if the component already existed

MapBlendshapes() runs automatically when the component is added (Reset()) and from OnValidate() when Mappings is empty. If you need to remap manually (e.g. after re-importing the FBX), use the component's inspector context menu → Map Blendshapes, or from a script:

go.GetComponent<ARKitOVRCustomFace>().MapBlendshapes();

From Unity MCP Unity_RunCommand, the cleanest invocation (the OVR/MSDK types aren't referenced in the MCP dynamic assembly, so use SendMessage to avoid reflection):

GameObject.Find("YourHeadModel")
    .GetComponentInChildren<SkinnedMeshRenderer>().gameObject
    .SendMessage("MapBlendshapes", SendMessageOptions.RequireReceiver);

6. Verify

• In Editor, with Meta XR Link / Quest Link, enter Play mode and make faces. The model should mirror them.
• On device, build APK, side-load, grant Face Tracking + Microphone permissions on first launch.
• Check the mapping at edit time: inspect the ARKitOVRCustomFace component. Mappings.Length should equal SkinnedMeshRenderer.sharedMesh.blendShapeCount. The Console log from MapBlendshapes() reports mapped X/N blendshapes — X should be 50 (or 52 if the FBX has all of them).

Troubleshooting

Symptom	Cause / fix
mapped 0/N in Console	Mesh blendshape names don't follow ARKit convention. Verify with mesh.GetBlendShapeName(i). Names must match e.g. eyeBlink_L, jawOpen — case-insensitive, prefix before last . is stripped.
Face is frozen	OVRFaceExpressions not assigned, or scene has no OVRCameraRig/OVRManager with face tracking enabled. Check OVRFaceExpressions.FaceTrackingEnabled and ValidExpressions at runtime.
Mouth doesn't move when speaking	A2E disabled. Enable Audio under OVRManager → Face Tracking Data Sources, and ensure RECORD_AUDIO permission is granted on device.
Eyes don't blink	Mesh's eyelid shapes aren't named eyeBlink_L/_R. Either rename, or add a custom row to ARKitTable.
Eyes don't move (look around)	You're relying on OVRFaceExpressions.EyesLook* instead of OVREyeGaze — switch to ARKitEyeGazeBlendshapeDriver.
Eye gaze in wrong direction	Note the deliberate ARKit ↔ OVR swap: ARKit's eyeLookIn_L (eye looking nose-ward, i.e. right) maps to OVR EyesLookRightL. The table already does this — don't "fix" it.
Weights look half-strength or clamped	BlendShapeStrengthMultiplier defaults to 100 because Unity blendshapes are 0–100 while OVR is 0–1. Don't lower this unless intentional.
cannot change access modifiers when overriding compile error	Base method is protected internal in another assembly. The override must use protected (not protected internal) — already correct in the supplied script.
_mappings out of sync with shared mesh assertion at Start	Mesh changed since mapping was generated. Re-run MapBlendshapes() via the component context menu.

ARKit ↔ OVR FaceExpression mapping (reference)

The 52 ARKit shapes don't 1:1 a FACS-based OVR enum. Notable choices baked into ARKitTable:

• Centrally-named ARKit shapes that have L+R OVR pairs (browInnerUp, cheekPuff, mouthFunnel, mouthPucker, mouthRollLower, mouthRollUpper) pick only the L-side OVR expression. If your model has obviously asymmetric mouth/brow when the user uses these expressions and you want true symmetric drive, subclass and additively sum L+R in a custom MapBlendshapes (use SlothARKitFaceDriver-style per-blendshape sum). For most heads the L-only choice is fine because the face is roughly symmetric and the asymmetry is below visual threshold.
• eyeLookIn/Out are deliberately swapped per side relative to OVR's left/right semantics (see Troubleshooting).
• mouthClose → LipsToward (closest FACS analogue).
• tongueOut → TongueOut (requires HorizonOS ≥ 65 + tongue tracking; otherwise stays at 0).

Files in this skill

• SKILL.md — this file
• references/ARKitOVRCustomFace.cs — drop-in OVRCustomFace subclass
• references/ARKitEyeGazeBlendshapeDriver.cs — OVREyeGaze → eyeLook* blendshape driver (LateUpdate)

Why this approach (vs. alternatives)

• Custom MonoBehaviour driver that reads OVRFaceExpressions[expr] and writes blendshape weights directly: works, but doesn't integrate with MSDK's correctives, eye constraints, or future retargeting upgrades. Use only if you can't extend OVRCustomFace.
• OVRCustomFace + RetargetingType.ARKitBlendshapes: clean, but guarded behind #if OVR_INTERNAL_CODE in the public Oculus Integration package — not 3P-shippable.
• OVRCustomFace + RetargetingType.Custom + GetCustomBlendShapeNameAndExpressionPairs override (this skill): fully public API, shippable, and gets all base-class behavior (data validity gating, weight scaling, mesh assertion).