Baked

A baked line knows which phoneme is happening and when, before it plays. Everything on this page is what Baked Phoneme does with that knowledge: the shapes running into each other, the mouth setting up ahead of the voice, the character's own way of speaking, and the rules that keep speech readable.

Coarticulation

Every phoneme is shaped by the ones around it. The mouth is already moving toward the next while it makes this one.

FaceCue reproduces that instead of stepping between poses. Shapes overlap and run into each other, and the mouth is almost never sitting on one clean shape. That is the difference between speech and a puppet cycling through positions.

Each shape is one shape, and FaceCue moves it as one. A viseme is the whole mouth in one position, taken whole from the character's rig map, and it is blended whole against the visemes around it. When a phoneme rounds the mouth, the whole shape rounds. When one closes it, the whole shape closes. What changes from one moment to the next is how much of a shape is showing and for how long, and that is all that changes.

You can see it directly in the Cue Clip Composer, where the rounding mark on a cue trails back over the cues in front of the rounded vowel.

Anticipation

The mouth runs slightly ahead of the voice.

Real speakers set a shape up before the sound arrives, because the tongue and lips have to be in place for the sound to be made at all. A mouth that moves exactly on the audio reads as slightly late, which is one of the things that makes lip sync feel mechanical without anyone being able to say why.

A baked line can do this because it knows what is coming. How far ahead it runs is tunable, and it is one of the few settings worth touching early if a character reads as lagging.

Speaker Delivery

How a character speaks, as distinct from what it says. This is the section that gives two characters different mouths on the same line, and it is the first place to go when a character sounds right and looks wrong.

Style is the quick way in. Five shipped styles, Neutral, Casual, Articulate, Intimate and Projected, each a whole way of speaking painted onto the character in one pick, from a relaxed mumble to a voice carrying across a room. A style is a starting point: everything it sets is yours to move afterwards, and the row says Drifted once you have, so a character never silently claims a style it no longer matches.

Under it, the levers the styles are made of:

• Coarticulation Smearing is how far each shape runs into its neighbours in time. More is a looser, more casual mouth. Less is more deliberate and more precisely articulated.
• Enunciation is how strongly each shape holds its own form while blended. A shape that yields easily gives a softer read. One that holds its ground gives a crisper one.
• Articulation Colour is the speaker's habit rather than any one phoneme. Warmth carries the whole mouth forward or spreads it wide, rounder and darker one way, flatter and brighter the other. Boldness is how much the speaker commits to every word. Raise it for a projected, emphatic, dramatic read, and lower it for a guarded one that keeps the words close. Phonemes that close the lips still close at any setting.

None of these is a correctness setting. They change how a line is spoken, never which phonemes are in it.

The Styles Are Experimental

The levers are what the styles are made of. The named styles have had less testing than the rest of the speech settings, which is why they read Experimental in the picker, so a style is a good first move rather than a finished character, and its values may shift a little between versions. Styles of your own are on the roadmap.

The Rules Underneath

Blending alone would smooth speech into mush, so a layer of rules sits over the top.

Some phonemes have to land. Where a word needs the lips to close, they close, even while an expression is competing for the mouth. Without that, the word stops reading no matter how good the rest is.

Some shapes ease back when they have to. Not every phoneme reaches its full shape in ordinary speech, and forcing each one to is what makes a mouth look like it is reciting.

These rules are tuned per language, because languages do not behave the same way. See Language Support.

Loudness & Prominence

Two separate readings shape how far each phoneme goes, and they are not the same thing.

Loudness is how loud the phoneme was. It sets how far the mouth opens, and it is the reading everyone expects.

Prominence is how much the phoneme stood out from the line around it. One can be quiet and still be the one being leant on. FaceCue measures that at bake time, relative to the speaker's own line, and the mouth sharpens its shape on the phonemes that stand out.

Both readings are made inside the clip, against the clip. A phoneme is loud, long or standing out because of how it compares with the others in its own line, and that comparison is the whole of it. Nothing outside the clip takes part: no other recording, no profile of the speaker, no level set somewhere else. The reading is complete the moment the bake is, and it is the same wherever the clip plays.

One Clip Is One Yardstick

The other side of that is that the clip decides what counts as ordinary. A recording whose delivery changes inside it, a whisper that becomes a shout, has its quiet half read as quiet, and the mouth plays it smaller. If a line looks underplayed and you know its audio is uneven, that is the first thing to suspect.

Splitting the recording, tuning the reading under Advanced, or leaving it as the more truthful performance are all reasonable answers. The fuller account is under Emphasis & Prosody.

The two are separate dials, so a character can be loud and slack or quiet and precise. Prominence is not the same thing as the head and brow motion on emphasised words. That belongs to the Prosody driver, covered under Emphasis, which reads the same baked measurements by its own route, so the two agree about which phonemes stood out without one driving the other.

Base Pose

Speech rarely plays a viseme at full. How much of a shape shows is set moment to moment by loudness and prominence, above, so a quiet or unstressed phoneme drives its viseme weakly, and near silence a shape can fall almost to nothing. Most shapes survive that. Some stop reading as their phoneme, and look like a different one rather than a softer one.

The base pose is the least of a viseme that speech will let it show. Each viseme carries its own, shipped with the viseme library and the same on every character, so the phonemes that lose their meaning as they fade keep enough of their shape to still read.

It is a floor, not an addition. While the base pose is in use, a viseme shows whichever is larger, the amount the voice drives or its base pose. A phoneme already driven past the base is left exactly as it was. Only one that would fall under it is lifted to it. So the base pose never makes speech louder or bigger, it only stops the quiet moments from going slack.

It is off unless something turns it on. On the Speech driver, under Articulation, the Base Pose section holds the switch:

• Use Minimum Spoken Pose applies the base pose under every viseme while the character speaks. It is off by default, and the Intimate delivery style turns it on, for the close, soft read that style is built for.
• Minimum Spoken Pose Scale takes a share of each viseme's base pose rather than all of it, for a fainter floor.

The mouth also rests on the base pose between words and around a line, as Between the Words describes. Residual Base Pose Scale sets how much of it those moments use.

To see a viseme's base pose on your own character, open it in the Face Rig Designer. Its speech pad marks the base pose as a line on the bar and can park the shape there, as The Base Pose on the Speech Channel page shows.

Organic Variation

The same phoneme twice in a row should not be identical twice in a row, because a real mouth never is.

FaceCue varies each one slightly, drawing from the character's own randomness so two copies of the same character do not vary in unison. It is a small effect and it is the sort of thing nobody notices until it is switched off.

After the Line Ends

A mouth does not go slack the instant audio stops. The clip ending is not the same moment as the speaker finishing, and the face should read as the second. How it winds down, and what it does between lines, is on Between the Words.

Laughs

A laugh written in a transcript plays on the mouth as laughter, as Laugh Tags on the Transcript Tags page describes. A laugh is often much quieter than the words around it, so it keeps a minimum opening of its own, apart from speech: holding a laugh open never makes the character's words any bigger. Its first pulse opens with its sound, and the pulses after it settle into a steadier rhythm that rides through the dips between its bursts.

Most characters never need to change any of this. The settings for it are in the inspector, under Non-Speech Sounds, in Laughs.

Lip Sync Without Audio

Everything above assumes a line has audio, because that is what a face follows. Some games do not want the audio: a silent interactive novel where the text is read and the character mouths it, a line mouthed under music, a scene with no voice track. FaceCue can play a set of baked cues on their own, with no Audio Source involved, and the face evaluates them exactly as it would a playing clip.

Two ways in. On a Timeline, the Phoneme Cues track plays a cue file or a registry clip while the playhead is inside it. From code, the brain plays a cue file in one call:

public IDriveHandle PlayPhonemes(TextAsset cueFile, TextAsset emotionFile = null,
    InterruptedLineMode? interruption = null)
public IDriveHandle PlayPhonemesOver(TextAsset cueFile, float seconds, TextAsset emotionFile = null,
    InterruptedLineMode? interruption = null)
public IDriveHandle PlayPhonemesAtSpeed(TextAsset cueFile, float speed, TextAsset emotionFile = null,
    InterruptedLineMode? interruption = null)

Parameter	What it does
cueFile	The phoneme cue file the baker wrote for the line
emotionFile	The line's emotion cue file, to play its emotions alongside. Leave it out for the mouth alone
seconds	For PlayPhonemesOver: how long the line should take, in seconds
speed	For PlayPhonemesAtSpeed: how fast it plays, as a multiple of the pace it was baked at
interruption	What happens to a line this call cuts short: one still speaking when the call starts. InterruptedLineMode.WindDown relaxes it the way a line that finishes does, InterruptedLineMode.Clear lets it go at speaking speed for a clean break. It counts for this call only: it replaces any choice made with SetNextInterruption, and is used or dropped as the call starts, so it never carries over to a later line. Leave it out, or pass null, to follow what is already chosen, a pending SetNextInterruption or else the character's own choice. See When a Line Is Cut Short

var brain = character.GetComponent<FaceCueBrain>();

// as baked: runs from zero and lets go by itself when the last cue ends
brain.PlayPhonemes(cueFile);

// fitted to a duration, for a mouth that keeps time with a text reveal
brain.PlayPhonemesOver(cueFile, 3.5f);

// faster or slower than it was baked
brain.PlayPhonemesAtSpeed(cueFile, 1.25f);

// a skip: whatever the character was still saying breaks off cleanly
brain.PlayPhonemes(nextCueFile, interruption: InterruptedLineMode.Clear);

Each returns the line's handle, or null when the file could not be read for this character:

Member	What it does
bool IsActive { get; }	Whether the line is still playing. False once it has ended, or from the moment you release it
event Action<IDriveHandle> Finished	Raised once, when the line has handed the mouth back, whatever ended it. Emotions played alongside may still be falling away
void Release()	Ends the line early, handing the mouth back with the same settle a finished line gets

When something else owns the playback position outright, the Speaker Context's own PlayPhonemes follows a time source instead:

public IDriveHandle PlayPhonemes(
    PhonemeCueData phonemeCues,
    EmotionCueData emotionCues = null,
    IAudioTimeSource timeSource = null,
    float speed = 1f,
    InterruptedLineMode? interruption = null)

Parameter	What it does
phonemeCues	The line's phoneme cues
emotionCues	Its emotion cues, to play alongside. Leave it out for the mouth alone
timeSource	Where the line gets its time from. Leave it out and it plays from the moment you call and ends on its own. Pass a SettableAudioTimeSource and set its time every frame to decide where the line is yourself, and release the line when you are done with it. A time left unset for half a second is taken to mean the line was forgotten, and the line ends
speed	How fast a line with no time source plays, as a multiple of the pace it was baked at. Ignored with a time source
interruption	What happens to a line this call cuts short, exactly as for the brain's calls above: WindDown or Clear for this call only, or null to follow what is already chosen

Baking Without Audio

The cues still have to come from somewhere, and today the baker needs a recording to hear. Until it can bake a transcript on its own, which is on the roadmap, the way through is to let it make the recording: type the transcript, press Generate Audio & Bake, and discard the synthesized audio afterwards. The cues it leaves behind are timed to a real reading of the line, which is what makes them play well without one. We know the round trip is tedious, which is why the transcript-only bake is coming.