Cues, Not Keyframes

What FaceCue actually stores when it bakes a clip, and why that makes the same line come out differently on different characters.

Sheet Music

Most lip sync tools produce animation. You give them a recording, they give you back a clip of blendshape keys, and that clip is the performance. Change the character and you need a new one.

FaceCue does not do that. A bake listens to the recording once, in the editor, and writes down what is in the dialogue. Only what is in the dialogue. Which phonemes were spoken, when each one started and stopped, and how it was said. Nothing about a face, nothing about a rig, nothing about a pose.

That file is sheet music. It says what was played. It does not say what it sounds like on your instrument.

The playing happens at runtime, on the character, every time the line runs.

Why It Is Built This Way

A bake is expensive and a performance is cheap, so the expensive half happens once, offline, and a baked line then costs almost nothing to play.

The guarantee is not that a line plays back identically. It is that what the recording contained, the words and the way they were spoken, is established once from the whole clip and never re-derived at play time. Everything above that stays free to vary, so two characters can read the same line differently and a face keeps its own motion, without either departing from what was actually said.

What a Cue Holds

A cue is one timed instruction. A phoneme cue carries:

• Which phoneme, one of the units of speech its language is built from.
• When it starts and ends, in seconds from the top of the clip.
• What the language does with it, such as whether it carries the stress in its word, or sits at the core of a syllable.
• How it was delivered, a handful of measurements read off the recording itself.

The last two are what make a performance read as meaning something instead of as a mouth moving. The language side says what the sentence is doing, the delivery side says what the actor did with it, and Emphasis reads both.

The baker finds each phoneme as it was actually spoken, its phone sound, and times it into a cue. A few are split into two cues, where the mouth clearly moves twice on one phoneme.

Cues are not only about the mouth. If you wrote emotion or cognitive tags into the transcript, the bake writes those out as cues too, on the same timeline. See Transcript Tags.

The Performance Happens Live

At runtime the character reads its cues and forms each shape as it goes.

It does not step from one pose to the next. Real speech does not either. Every sound is pulled out of shape by the sounds on either side of it, and FaceCue reproduces that, so shapes overlap and run into each other the way they do in a real mouth.

Blending alone would not be enough. Left to itself it smooths away the very phonemes that make speech readable, so a layer of rules sits over the top, protecting what has to land and easing back what should not compete. Those rules are tuned per language, because languages do not behave the same way.

None of that is in the cue file. The cue file said which phoneme and when. Everything about what it looks like is decided on the character, at the moment it plays.

Why Two Characters Differ on One Sheet

Point the same bake at two characters and you will get two different performances. Three things sit between the cues and the face.

The maps. A map is a character's translation layer. It says which shapes on this particular rig stand for which target, and how much of each. Two rigs almost never name their shapes the same way, or even have the same ones, so each carries its own.

You mostly do not build these by hand. The OneClick Wizard detects which family a character's rig belongs to and builds every map for you. The Face Rig Designer is where you tune the result, and where a character nobody has seen before is mapped from scratch.

Tuning covers two different jobs. One is correctness, since rigs differ even inside one family and a shape may not land quite where the map assumed. The other is character, making one person hold a word differently from everyone else, or pull to one side. Maps are assets, so if your characters are close enough you may only need one between them.

The drivers. Each part of the performance is a driver on the character: speech, emphasis, emotion, gaze, breath. Every one is optional and every one is tuned per character. A character with the emphasis driver switched off gets a still head over a moving mouth, from exactly the same cues.

The character's own randomness. A resting face is never still, and the small motion is not the same on two characters. Each one can take its own randomness, so a crowd does not blink in unison.

None of that touches the bake. You can re-tune a character for a week and never re-bake a line.

What Lives Outside the Cues

The mouth is the only part of the performance the cues describe directly, and even there they describe the speech rather than the shape.

Everything else runs on its own and answers to what the voice is doing:

• Emphasis reads the measurements in the cues and carries them continuously into the head, the brows and the breath, so the delivery moves the whole face rather than sitting under a still one.
• Emotion is authored, never inferred. You place it as a tag in the transcript, in the Cue Clip Composer, on a Timeline track, or from your own code, and it can also sit underneath everything as a standing mood. FaceCue does not guess how a character feels from the sound of their voice.
• Gaze follows a target you give it, from your own code or from the scene. The recording has no say in where a character looks. What FaceCue owns is how the eyes, head and body get there.
• Breath runs as its own cycle, shaped by speech and exertion. While the character speaks the breath holds low, since speech rides mostly on the exhale, and a pause afterwards draws a refill breath.

That separation is the reason a line you baked a year ago still improves when you tune the character.

The Mouth Is One Layer

Worth saying plainly, because it is the thing people are most often surprised by.

Cues drive the mouth. Nothing else in the performance depends on them existing.

A character can also form its mouth without a bake at all, straight from audio as it plays, which is what happens for a microphone, for voice chat, or for a clip that reached the game after the bake pass. That path is less accurate, because it is guessing from the sound rather than reading what a bake worked out with the whole recording in front of it. It is always available.

What does not change is everything above the mouth. Emphasis, emotion, gaze, head motion and breath run the same way whether the mouth is reading cues or reading audio. So an unbaked line degrades in one place instead of falling apart, and a character you set up once behaves consistently either way.

The Speech driver covers how a character chooses between those sources, and Live Speech covers what the unbaked paths need and what they cost.