Script Coexistence¶
As any actor knows, a performance lives on its timing. In FaceCue that is true twice over: in the performance a character gives, and in when its machinery runs each frame.
This page gives you what you need to work with a character FaceCue drives, so your own scripts land where they should. It covers calling FaceCue, and reading or writing a bone or a blendshape FaceCue drives. FaceCue calculates each character late in the frame, on the faster settings on worker threads, in parallel with the main thread. So what your script sees, and when what it changes reaches the character, depends on where in the frame it runs.
Best Practices¶
Two Places to Work With FaceCue¶
There are two good places for a script to work with FaceCue, and both hold on every setting.
In Update. The usual place. Everything you ask of FaceCue there happens that frame: a change reaches this frame's character, and a read returns at once. A change does not show the moment you make it, though. FaceCue applies it to the character later in the frame. In the same way, what you read there comes from the character as it was last drawn: the character you see from Update is the one drawn last frame, its bones, its blendshapes, and what FaceCue reports about it. For most work that makes no difference.
In an OnPostResolve handler. For a script that needs this frame's finished character to decide what to do. Read and act there: every read is this frame's, and free. Two things change. A change you make there reaches the next frame's character. And the handler runs at a moment that depends on the settings, anywhere from the brain's own LateUpdate to just before skinning, once each time FaceCue recalculates the character, which on a character with a lower output rate is not every frame.
The ordinary LateUpdate is the place to avoid. On Worker Threads the character is still being calculated there. Interacting with FaceCue has the detail.
Moving Bones and Blendshapes With FaceCue¶
To move a bone FaceCue's breathing also moves, a shoulder or the chest, push it with PushBone from Update, every frame you want the motion. The bones the gaze turns don't take a push: aim the eyes with a look target, and turn the head with PushHeadGesture. To set a blendshape FaceCue also drives, share it through a Sampled channel and set the weights in Update. Setting a bone or a blendshape FaceCue drives directly puts your script and FaceCue in a contest your script loses. Writing a Bone and Writing a Blendshape have the detail.
The Solve Schedule¶
Each character's brain has two settings that decide where the character is calculated and when the result is written to the rig. They sit in the Solve Schedule group under Optimization. The character looks the same whichever you pick. What changes is how much work leaves the main thread, and when in the frame the rig holds the finished result.
Solve¶
Where the character is calculated.
| Solve | What it does |
|---|---|
| Auto | FaceCue picks. It picks Worker Threads wherever the platform has them. |
| Main Thread | Inside the character's own LateUpdate, on the main thread. |
| Worker Threads | On worker threads, in parallel with the main thread. Apply At says when the result is written. |
Outside play mode every character runs as Main Thread, so a preview in the editor never shows the later timings.
Apply At¶
When the result is written to the rig, for a character on Worker Threads.
| Apply At | What it does |
|---|---|
| Auto | FaceCue picks. It picks Before Skinning. |
| At Character LateUpdate | Inside the character's own LateUpdate, the same moment as Main Thread. |
| After Every LateUpdate | Once every other script's LateUpdate has run. |
| Before Skinning | Just before Unity skins the meshes, the last moment that still reaches the frame. This gives the main thread the most time back. |
One Frame, Step by Step¶
Unity runs every script's Update, then the Animator, then every script's LateUpdate, then skins the meshes and draws the frame. Within each of those passes, scripts run in their execution order. FaceCue's brain runs at execution order -150 in both passes, ahead of every script left at the default order of 0. Steps 0 to 4 are the same on every setting. From step 5 on, the path depends on Solve.
0. Updates Ordered Before the Brain¶
The Update of every script with an execution order below -150. FaceCue does nothing here.
1. The Brain's Update¶
FaceCue moves its clock forward and gets the frame ready. It does not touch the rig yet.
2. Every Other Update¶
The Update of every script ordered after the brain, the default order included. FaceCue does nothing here.
3. The Animator¶
The Animator poses the character, exactly as it would without FaceCue. FaceCue does nothing here.
4. LateUpdates Ordered Before the Brain¶
The LateUpdate of every script with an execution order below -150. The rig holds the Animator's pose. FaceCue does nothing here.
On Main Thread¶
5. The Brain's LateUpdate¶
FaceCue reads the rig, calculates the character, writes the result to the rig and raises OnPostResolve, all before the brain's LateUpdate ends.
6. Every Other LateUpdate¶
Every script ordered after the brain finds this frame's finished character on the rig. FaceCue does nothing here.
7. After Every LateUpdate¶
FaceCue does nothing here.
8. Before Skinning¶
FaceCue does nothing here.
9. Skinning¶
Unity skins the meshes and draws the frame.
On Worker Threads¶
5. The Brain's LateUpdate¶
FaceCue reads the rig and hands the character to the worker threads. If Apply At is At Character LateUpdate, FaceCue takes the finished result back, writes it to the rig and raises OnPostResolve before the brain's LateUpdate ends, and the rest of the frame runs as on Main Thread. Otherwise the brain's LateUpdate ends, and the main thread moves on while the worker threads work.
6. Every Other LateUpdate¶
On the two later Apply At values, the character is still being calculated here, and a script does not yet find this frame's result on the rig. This is the gap the rest of the page is about.
7. After Every LateUpdate¶
If Apply At names this moment, FaceCue takes the finished result from the worker threads, writes it to the rig and raises OnPostResolve. If it names a different moment, nothing happens here.
8. Before Skinning¶
If Apply At names this moment, FaceCue takes the finished result from the worker threads, writes it to the rig and raises OnPostResolve. If it names a different moment, nothing happens here.
9. Skinning¶
Unity skins the meshes and draws the frame.
The sections below name these steps by number, and their tables have a column for each path. On Worker Threads there means Apply At on After Every LateUpdate or Before Skinning. On At Character LateUpdate a character follows On Main Thread, as step 5 says.
Interacting with FaceCue¶
Calling the methods and properties of FaceCue's components: the FaceCue Brain, the Resolver, the Eyes, Speech, Emotion, Prosody and Breath drivers, the Speaker Context and the Listener Context.
The Recommended Way¶
Where to call depends on whether the call depends on the character's state.
- A call that does not depend on it: make it in
Update, in step 0 or step 2. A change reaches this frame's character, and a read returns at once. - A call that depends on what the character is doing now: read and act in an
OnPostResolvehandler. Every read there is this frame's and free, and a change reaches the next frame's character, the same frame a change from the nextUpdatewould reach. The Resolver raisesOnPostResolveonce for each resolve, so on a character whose output rate is below the frame rate some frames have none, and the handler runs at the rate the character is recalculated.
Both hold on every setting.
resolver.OnPostResolve += () =>
{
// reads here see this frame's character, and nothing waits
if (!eyes.IsLookingAway && wantsToThink)
eyes.LookAwayThinking();
};
At Other Steps¶
| When you call | On Main Thread | On Worker Threads |
|---|---|---|
Step 4, a LateUpdate ordered before the brain |
A change reaches this frame's character. A read returns at once. | Same. |
Step 6, every other LateUpdate |
A change acts at once and reaches the next frame's character. A read returns at once. | See below. |
Calls in Step 6 on Worker Threads¶
In step 6 on Worker Threads the character is still being calculated. A write is set aside and applied when the result is written, so it reaches the next frame's character, and nothing waits. A read is always accurate, but can make the main thread wait for the worker threads, which costs you the main-thread time that frame would have saved. What counts as which:
| Kind of call | Examples | Counts as |
|---|---|---|
| A write that returns nothing | Setting a property, PushBone, SetLookTarget |
A write. |
| A write that hands back a handle | EmotionDriver.Play, SpeakerContext.PlayPhonemes |
A write. The handle comes back at once, and what it starts begins when the result is written. Through the handle, a change like Release is a write, and a read like IsActive is a read. |
A write that reports through an out argument |
LookAwayThinking(out LookAwayResult result), LookBack(out bool wasLookingAway) |
A read. Each also has a form without the out argument, which is a plain write. |
| Any read | Getting a property, PeekResolvedBasePose |
A read. It includes the writes your script set aside earlier in the frame. |
If a call of yours makes the main thread wait, the console says so the first time, naming the script, so a read in the wrong place does not go unnoticed. Report Conflicts, in the Solve Schedule section, turns the warning off. It only ever appears in the editor and in development builds, so we advise leaving it on.
Finishing the Character on the Spot¶
For the rare script that has to read the character in step 6 and cannot move, such as a camera move or an attachment that runs for a moment in a LateUpdate of its own, FaceCue can finish a character on the spot. This suspends the worker threads' gain for that frame instead of turning Worker Threads off altogether. Reading in the next frame's Update usually serves just as well, so keep it for the case that really needs this frame.
void ForceApplyNow()
static void ForceApplyAllNow()
| Member | What it does |
|---|---|
void ForceApplyNow() |
On the brain. Waits for the worker threads to finish this character, writes the result to the rig, and applies the writes set aside so far. After it, the rig holds this frame's finished character and every call on the character acts at once, for the rest of the frame. Does nothing when the character is not being calculated. |
static void ForceApplyAllNow() |
On FaceCueBrain. The same for every character still being calculated. |
Both make the main thread wait for the worker threads, which costs you the main-thread time that frame would have saved.
void LateUpdate()
{
if (!framingTheFace) return;
brain.ForceApplyNow();
// the rig holds this frame's finished character
cameraTarget.position = headBone.position;
}
Assets the Characters Share¶
Changing an asset the characters use, a rig map, a settings asset, cue data or an emotion library, is not a call on a character, so nothing sets it aside, and on Worker Threads FaceCue may be calculating a character from it at that moment. Make such changes in step 0 or step 2, or call FaceCueBrain.ForceApplyAllNow() first.
Reading a Bone Directly¶
Reading a bone directly means reading its Transform the way Unity holds it: headBone.rotation, headBone.position and the like. FaceCue also has read calls of its own, such as PeekResolvedBasePose. Those are calls on FaceCue, covered under Interacting with FaceCue, and they follow its rules instead.
The Recommended Way¶
Read it in an OnPostResolve handler. The Resolver raises OnPostResolve right after it writes the result, in step 5, 7 or 8 depending on the setting, so inside a handler the bones and blendshapes hold this frame's finished character. It is the same on every setting, it costs nothing, and the handler never waits for anything.
event Action OnPostResolve
| Member | What it does |
|---|---|
event Action OnPostResolve |
On the Resolver. Raised after each resolve, once the character's bones and blendshapes hold FaceCue's result for the frame. A frame that resolves more than once raises it once per resolve, and a frame with no resolve raises none. |
resolver.OnPostResolve += () =>
{
// the head bone holds FaceCue's result here, whatever the setting
Vector3 forward = headBone.forward;
};
At Other Steps¶
| Where you read | On Main Thread | On Worker Threads |
|---|---|---|
Step 0 or step 2, an Update |
The bone as it was drawn last frame. Free, but one frame old. Fine for anything that can be a frame behind. | Same. |
Step 4, a LateUpdate ordered before the brain |
This frame's animation, before FaceCue has added anything. A bone nothing animates still holds last frame's pose. | Same. |
Step 6, every other LateUpdate |
This frame's finished pose, free. | Stale. The bone as it was before FaceCue's work this frame: this frame's animation without FaceCue's motion, or last frame's pose on a bone nothing animates. |
Reading a Blendshape Directly¶
Reading a blendshape directly means asking its renderer, SkinnedMeshRenderer.GetBlendShapeWeight. FaceCue's own reads of its blend weights are calls on FaceCue, covered under Interacting with FaceCue.
The Recommended Way¶
Read it in an OnPostResolve handler, where the renderer holds this frame's finished weights, on every setting and for free. The handler is the one described under Reading a Bone Directly.
At Other Steps¶
| Where you read | On Main Thread | On Worker Threads |
|---|---|---|
Step 0 or step 2, an Update |
The weight as it was drawn last frame. | Same. |
Step 4, a LateUpdate ordered before the brain |
The weight drawn last frame, or the animation's, for a shape an animation keys. | Same. |
Step 6, every other LateUpdate |
This frame's finished weight. | Stale: the weight drawn last frame. |
Writing a Bone¶
FaceCue moves some of a character's bones itself. The gaze turns the eyes, the head and the neck, and the body on a wide look. Breathing lifts the chest and the shoulders. If your script moves one of those bones as well, setting its rotation yourself puts the two of you in a fight your script loses. On a breathing bone, pass your rotation to FaceCue instead, and it combines it with its own motion. The bones the gaze turns have their own way in, below.
Bridging Another System or Asset?
If what moves these bones is an existing system or asset, an IK package, a procedural animation tool or a plugin of your own, it may be a good candidate for a FaceCue integration. Suggest it to us.
The Recommended Way¶
Push the rotation with PushBone from Update, in step 0 or step 2, every frame you want the motion. From Update a push reaches this frame's character, on every setting.
void PushBone(Transform bone, Quaternion rotation, BoneChannelMode mode, float weight, int sourceId)
| Parameter | What it does |
|---|---|
bone |
The bone to move. A null bone is ignored. |
rotation |
A local rotation, read the way mode says. |
mode |
How your rotation combines with the bone's, one of the two below. |
weight |
How much of it to apply, 0 to 1. Most callers pass 1. |
sourceId |
A stable number naming your system, so contributions can be told apart. |
BoneChannelMode |
What it does |
|---|---|
Delta |
Adds your rotation on top of what the bone already has this frame: the animation's pose and FaceCue's motion so far. Several Delta pushes stack in the order they arrive. For an offset, a nod, a sway. |
Target |
Sets the bone to your rotation outright, replacing what came before it in the frame. Delta pushes after it still add on top, breathing's among them. For an aim you calculate yourself. |
The Bones the Gaze Turns¶
Don't push the bones set up in the Eyes driver's Setup: the eyes, the head, and the bones listed under its Neck and Body sections. The gaze turns those bones itself and replaces a push on one whenever it moves it, so a push there shows only some of the time. In the editor and in development builds, the console warns the first time a script pushes one, once per session for each character.
Each has its own way in instead. Aim the eyes with a look target. Turn the head with the Eyes driver's PushHeadGesture, described under Driving the Head Yourself, which lays your gesture over the look.
Push Every Frame¶
A push counts for one frame, and FaceCue keeps none of it. Each frame it starts the bone from a fresh pose, the one the Animator gave it this frame, or the pose the bone had before FaceCue moved it when nothing animates it, and adds your push to that.
Two things follow:
- A Delta never adds up over frames. Pushing the same 10° every frame gives 10°, not a growing turn.
- Anything you want to hold has to be pushed again every frame, a constant offset or a constant replacement alike. On a frame where FaceCue does not recalculate the character, a push is ignored rather than saved for later, so pushing every frame is exactly right.
A bone FaceCue does not otherwise drive works the same way: it joins FaceCue's rig at its first push.
When You Stop Pushing¶
What happens next depends on whether FaceCue moves the bone too:
| The bone | When your pushes stop |
|---|---|
| One FaceCue also moves, a shoulder or the chest | Your motion is gone from the next frame. |
| One only your pushes move | The bone keeps your last pose for a moment, then eases back over the Handback Time, to the animation, or to the pose it had before you pushed it when nothing animates it. |
The Handback Time is on the brain, under Animator Coexistence, and the same ease ends a look.
For a soft stop on a bone FaceCue also moves, bring the push's weight down to nothing over a few frames before you stop. It works in both modes: a Delta fades out, and a Target lets the bone drift back to the pose it would have without it. A script that stops without fading, because it is disabled or destroyed, takes its motion off such a bone at once.
A fade keeps pushing while the weight falls, and measures the fall in seconds, so it takes the same time at any frame rate:
void Update()
{
if (stopping) weight = Mathf.MoveTowards(weight, 0f, Time.deltaTime / fadeSeconds);
if (weight <= 0f) return;
resolver.PushBone(shoulder, offset, BoneChannelMode.Delta, weight, MySourceId);
}
At Other Steps¶
| When you push | On Main Thread | On Worker Threads |
|---|---|---|
Step 4, a LateUpdate ordered before the brain |
This frame's character. | Same. |
Step 6, every other LateUpdate, or an OnPostResolve handler |
The next frame's character. This frame's is already calculated. | The next frame's character. The push is set aside until the result is written, and nothing waits. |
Setting the Rotation Yourself¶
On a bone FaceCue moves, one set up in the Eyes driver's or the Breath driver's Setup, don't set the rotation yourself. It does not hold at any step, and that includes after FaceCue is done applying: in an OnPostResolve handler, or in a LateUpdate ordered after the brain's, your rotation still does not stay. FaceCue lays its own motion over the bone again as late in the frame as it can, so that an animation posing the bone late cannot wipe that motion out, and to FaceCue a script setting the bone looks the same as such an animation. Push a breathing bone with PushBone, aim the eyes with a look target, and turn the head with PushHeadGesture.
On a bone FaceCue does not move, a rotation you set is what shows, on every setting:
| Where you set it | What shows |
|---|---|
Step 0 or step 2, an Update |
Yours, unless the Animator animates the bone. Then the Animator's pose replaces yours in step 3. |
Step 4 or later, an OnPostResolve handler included |
Yours. |
After you stop, the bone keeps your rotation until something writes it again, which on an animated bone is the next frame's animation.
With Report Conflicts on, the console warns the first time a script moves a bone behind FaceCue in a session, naming the script, and the Solve Schedule section shows the count. The Check Wiring button finds scripts in the open scenes that hold a reference to a bone FaceCue moves. It cannot see a script that only reads a bone and keeps no serialized reference to it.
Writing a Blendshape¶
On a blendshape FaceCue drives, FaceCue's value is what shows. When another script or plugin, a lip sync or an expression system of your own, has to set some of those shapes too, register a Sampled channel naming them. Before each calculation FaceCue then reads what the other writer put on those shapes and blends it with its own, the way it blends its own drivers. A character with a Sampled channel does its facial work on the main thread whatever Solve says, so it always follows the On Main Thread steps.
Bridging Another System or Asset?
If what sets these shapes is an existing system or asset, a lip sync, a facial capture tool or an expression system, it may be a good candidate for a FaceCue integration. Suggest it to us.
The Recommended Way¶
Register a Sampled channel, and have the other writer set the shared weights in Update, in step 0 or step 2, every frame they should count. FaceCue blends them into this frame's face.
void RegisterSampledChannel(IBlendChannel channel)
void UnregisterSampledChannel(IBlendChannel channel)
ScopedSampledChannel(string id, Func<ChannelRegionProfile> profileGetter, IEnumerable<(SkinnedMeshRenderer smr, int blendIndex)> scope)
| Member | What it does |
|---|---|
void RegisterSampledChannel(IBlendChannel channel) |
On the Resolver. FaceCue starts reading the shapes the channel covers, from the next calculation. |
void UnregisterSampledChannel(IBlendChannel channel) |
On the Resolver. FaceCue stops reading them, and its own value shows on them again. |
ScopedSampledChannel(string id, Func<ChannelRegionProfile> profileGetter, IEnumerable<(SkinnedMeshRenderer smr, int blendIndex)> scope) |
A channel over the shapes in scope, each a renderer and a blendshape index. profileGetter returns how the other writer's weights share each part of the face with FaceCue's, usually ChannelRegionProfile.Sampled. id names the channel in logs. |
var face = GetComponentInChildren<SkinnedMeshRenderer>();
int jawOpen = face.sharedMesh.GetBlendShapeIndex("JawOpen");
var channel = new ScopedSampledChannel("My Lip Sync",
() => ChannelRegionProfile.Sampled(0f, 1f, 0f, 1f, 0f, 1f),
new[] { (face, jawOpen) });
GetComponentInChildren<FaceCueResolver>().RegisterSampledChannel(channel);
FaceCue only takes a weight the other writer changed. What FaceCue itself last left on the shape, and a weight the shape already held when FaceCue first read it, such as one saved on the character, are never taken as the other writer's.
At Other Steps¶
| Where the other writer sets its weight | What happens |
|---|---|
Step 4, a LateUpdate ordered before the brain |
Blended into this frame's face. |
Step 6, every other LateUpdate |
Blended into the next frame's face, while its raw value is what shows for this frame. |
Setting a Weight Without a Channel¶
A shape FaceCue drives, even one it only holds still, is written by FaceCue every time it recalculates the character, so a weight you set with SetBlendShapeWeight only shows if you set it after that, and only until FaceCue writes the shape again on the next frame:
| Where you set it | On Main Thread | On Worker Threads |
|---|---|---|
| Step 0, step 2 or step 4 | Replaced by FaceCue's. | Same. |
Step 6, every other LateUpdate |
Yours, for that frame only. | Replaced by FaceCue's. |
An OnPostResolve handler |
Yours, for that frame only. | Same. |
A shape FaceCue never drives keeps the weight you set, at any step, until you change it.
Changing the Rig While It Runs¶
When a bone FaceCue reads or moves gets a new parent while the character is running, or any transform between such a bone and the top of the character does, tell the brain:
void RefreshBoneHierarchy()
| Member | What it does |
|---|---|
void RefreshBoneHierarchy() |
On the brain. FaceCue reads each bone's parent again. Until it is called, FaceCue works from the hierarchy it found when the character started, and the poses it writes can land in the wrong place. Safe to call when nothing changed. |
In play mode FaceCue does this on its own when a transform above the FaceCue Brain's object gets a new parent, a character put on a vehicle for example. It does not notice in edit mode, or when the bones hang somewhere the brain's object is not under. In the editor and in development builds, an error names a bone that moved without it.