Speaking & Listening

How FaceCue knows where a character sits in a conversation, what changes when it does, and how you tell it.

This is a small amount of machinery with a large effect, because a great deal of what the character does keys off one value.

Two Components, Deliberately Symmetric

The speaker side watches audio playback. It is what knows a clip is playing, which clip it is, and how loud it is at this instant, and it is where cues are resolved from a registry. If a character speaks, it needs this.

The listener side watches the character's own state instead. It is the route by which something that is not audio, a game event or a change in the world, reaches the face.

They are built as mirrors of each other on purpose. One sources what the character is doing from sound, the other from state, and everything downstream takes them the same way.

Both are required peers of the brain, and Unity adds them with it. Even a character that will never be spoken to carries the listener side, and it sits idle until something drives it.

Conversant State

Everything above resolves to one value: Talking, Listening, or Idle.

That single value is read right across FaceCue, and it changes more than you would expect.

The eyes carry most of it. How often the character blinks, how often its gaze breaks away and how far it travels when it does, and how much the head sways underneath. Breath shapes its cycle differently while listening.

The gaze part is the one worth understanding, because it is not a single dial turned up and down.

An idle character makes small movements very often, the constant scanning of eyes with nothing in particular to hold. A character in a conversation makes larger movements much more rarely, breaking away from a face and coming back. So idle eyes move most and travel least, and a talking character breaks away noticeably more often than a listening one.

Listening Is Not Quiet Talking

The three states are genuinely different behaviours, not three amounts of one behaviour.

Blink rate is the clearest case. Talking raises it. Listening deliberately leaves it alone, because paying attention does not change how often people blink.

What listening does change is stillness. People hold noticeably still when they are paying attention, so the head settles and the eyes break away far less. A listening character reads as attentive instead of as one waiting for its turn.

Connecting Your Own Game

FaceCue ships the interface, not the game state. It has no idea who is talking to whom, and no opinion about how your dialogue system works.

What it offers is a small interface to implement on whatever you already have. Tell it who is speaking and who is being spoken to, and the state above resolves itself.

Conversation is one of four things your game can tell a character, alongside what it is looking at, how hard it is working, and how bright the scene is. All of them work the same way, and they are set out in World Inputs.

When your game knows which FaceCue character is speaking, it can say so as well, and the character keeps it: whom it is listening to, and whom it is waiting on. Dialogue State has the calls.

Until you wire any of it up, a character that plays audio still resolves to Talking on its own. That is enough for a great deal, and it is why a character set up with the wizard already behaves sensibly before you have written any integration code. What you gain by wiring it up is Listening, which is the state a character cannot infer from its own audio, because nothing is playing.

Where the Value Goes

Worth knowing, because it explains behaviour that otherwise looks like it came from nowhere.

• Gaze takes almost all of it: how often the eyes break away and how far, how often the character blinks, and how much the head sways.
• Breath builds a different cycle while the character is listening.

None of those are things you switch on. They follow from the state being right, which is why wiring the interface up is worth doing even though the character works without it.

Listening Gestures

The small nods a listener gives while someone else talks. A character that holds perfectly still while it is being spoken to reads as absent, even when its eyes are alive, and these nods are the plainest sign that somebody is being listened to.

What a Nod Looks Like

A nod's shape is set on the Eyes driver, because the head is what moves: how deep it dips, how long it takes, and whether it comes alone or as a short run that narrows as it goes. It is one shape for every nod, whether the character gives it on its own or your code asks for it, so tuning it once tunes both. The same rows show on the listener side as well, under Nodding, so you can shape the nods where you switch them on, and a change on either side shows on both. They grey out while Nodding is None, since the character then only nods when asked, and you tune those nods on the Eyes driver. See Listening Gestures on the At Rest page.

Ambient Listening Nods

Nodding decides whether the character nods by itself. At None it only nods when your own code asks it to. At Ambient it also nods on its own while listening, without answering anything the other speaker said in particular.

Ambient nods only come while the character is listening. One while it talks or idles would read as a tic rather than attention, so the character never gives one then.

Typical Gap (s) is the average time between them. Judge it against how long your characters actually listen for. A nod has to land inside the line being spoken to read as a response to it, so a gap longer than your typical line means most lines pass without one. Gap Unevenness (%) sets how irregular the spacing around that average is, which is what keeps the nods from arriving on a metronome.

Asking for a Nod

Your game will often know something the character cannot work out for itself, that it has just been asked a question, or that it agrees with what it heard. Ask the Eyes driver for a nod and the character shapes it: you get the runs, the narrowing across a run, the settle and the recovery without writing any of it, and a nod your game asks for looks like one the character gives on its own.

It works whatever Nodding is set to, so a game that drives every nod itself can leave the character's own nodding off.

public class DialogueNods : MonoBehaviour
{
    [SerializeField] EyesDriver eyes;

    // the character agrees with what it just heard
    public void Agree() => eyes.NodListening();

    // or a single emphatic one, overriding the character's usual depth
    public void AgreeStrongly() => eyes.NodListening(depthDeg: 6f, cycles: 1);
}

The call is on the Eyes driver, in two forms. The second also says how the request went:

public void NodListening(float depthDeg = 0f, float durationMs = 0f, int cycles = 0)
public void NodListening(out NodResult result, float depthDeg = 0f, float durationMs = 0f, int cycles = 0)

Parameter	What it does
result	In the second form, whether the nod played, and why not when it was refused.
depthDeg	How far the head dips, in degrees.
durationMs	How long one dip and return takes, in milliseconds.
cycles	How many nods in the run.

Leave any of them out, or at zero, and it comes from the character's own Nod Shape, variation included. One you pass is used exactly as given.

NodResult reads the same way as a look-away's result:

Member	What it tells you
NodOutcome Outcome { get; }	Started, Restarted when a nod was already playing and this one replaced it, or Refused.
NodRefusal Refusal { get; }	Why it was refused: Inactive when the Eyes driver is disabled or not set to Active, NoHead when the character has no head bone. None otherwise.
float StartTime { get; }	When the nod started, on the character's own clock, in seconds.
bool Accepted { get; }	True when the nod plays.

Two more members on the Eyes driver say when the character last nodded, whether your code asked or the character nodded on its own:

Member	What it tells you
float LastNodStartTime { get; }	When the most recent nod started, on the character's own clock. Negative before the first.
float LastNodEndTime { get; }	When the most recent nod ended. Earlier than the start time while a nod is playing.

The Nodding setting itself is on the listener side, ListenerContext, and can be read or set from code:

public ListenerContext.ListeningNodMode ListeningNods { get; set; }

A nod you ask for does not wait. It plays at once, whatever the character is doing, because a caller asking for one has a reason FaceCue cannot see. Asking again while one is playing restarts it rather than queueing. It is refused, and never plays later, while the Eyes driver is not driving the head.

The form with an out argument reads how the request went, so it counts as a read. Called from an ordinary LateUpdate on a character calculated on worker threads, it finishes that character first, which costs main-thread time. The form without one never does. See Calling FaceCue.

For a gesture that is not a nod at all, the Eyes driver takes a rotation you animate yourself, frame by frame. See Listening Gestures.

Cognitive Gestures

What a character does while it works something out, rather than while it listens: today, the look-away a person gives while reaching for a word.

Asking for a Look-Away

A character marked as thinking in its lines looks away on its own (see Mental State). When your game knows the character is searching for a word at a moment no line marks, ask the Eyes driver for the same look. The eyes glance away from whatever the character is looking at and come back, without letting go of it, and the head joins in a little the way it does for a marked moment.

The call is on the Eyes driver, in two forms. The second also says how the request went:

public void LookAwayThinking(
    LookAwayVertical vertical = LookAwayVertical.Drawn, float? verticalDeg = null,
    LookAwaySide side = LookAwaySide.Drawn, float? horizontalDeg = null,
    float? durationSec = null, float? headShare = null, float? eyeStretch = null,
    float? awayPaceMs = null, float? backPaceMs = null)
public void LookAwayThinking(out LookAwayResult result,
    LookAwayVertical vertical = LookAwayVertical.Drawn, float? verticalDeg = null,
    LookAwaySide side = LookAwaySide.Drawn, float? horizontalDeg = null,
    float? durationSec = null, float? headShare = null, float? eyeStretch = null,
    float? awayPaceMs = null, float? backPaceMs = null)

Parameter	What it does
result	In the second form, whether the look-away played, and why not when it was refused.
vertical	Up, down, or Drawn to let the character choose.
verticalDeg	How far up or down the eyes go, in degrees.
side	Left, right, Centre for no sideways part, or Drawn to let the character choose.
horizontalDeg	How far to the side the eyes go, in degrees.
durationSec	How long the eyes stay away, in seconds.
headShare	How much the head turns along with the eyes, from 0 to 1.
eyeStretch	How far the eyes go before the head takes the rest of the look, from 0 (their comfortable range) to 1 (their full range).
awayPaceMs	How quickly the eyes move away, in milliseconds. Lower is quicker.
backPaceMs	How quickly the eyes come back, in milliseconds. Lower is quicker.

Leave any of them out and it comes from the character's own Thinking look-away settings, with the variety a marked moment has. One you pass is used exactly as given. The same request can be built once and passed as a struct, LookAwayRequest, through LookAwayThinking(in LookAwayRequest request) or LookAwayThinking(in LookAwayRequest request, out LookAwayResult result).

public class DialogueThinking : MonoBehaviour
{
    [SerializeField] EyesDriver eyes;

    // the character searches for an answer, in its own style
    public void Ponder() => eyes.LookAwayThinking();

    // a longer look up and to the left
    public void RecallSomething() => eyes.LookAwayThinking(LookAwayVertical.Up, side: LookAwaySide.Left, durationSec: 2f);
}

Member	What it does
void LookBack()	Ends a look-away now and brings the eyes back.
void LookBack(out bool wasLookingAway)	The same, and wasLookingAway says whether one was playing.
bool IsLookingAway { get; }	True while the eyes are away.
bool ThinkingLookAwaysEnabled { get; set; }	The Thinking group's Enable. Off, the call is refused and marked lines leave the eyes where they are.

A look-away you ask for plays at once, rather than at a moment the character picks, and asking again while one is playing replaces it. It is refused when Thinking look-aways are switched off on the character, when the Eyes driver is not driving the eyes, and on a character with no eyes. A new look target, or the target being cleared, ends a look-away early.

The forms with an out argument read how the request went, so they count as reads. Called from an ordinary LateUpdate on a character calculated on worker threads, they finish that character first, which costs main-thread time. The forms without one never do. See Calling FaceCue.

Reacting to a Demeanor

While a character listens, or waits for the player to pick a reply, its face drifts through a demeanor: it picks an emotion from a palette, holds it a while, and moves on to the next. Demeanor Palettes covers what they hold. The listener side announces each of those moments, so something outside FaceCue can react in time with the face. The idle demeanor belongs to the Emotion driver and is not announced here.

The case it is made for is the rest of the body. FaceCue moves the face and the head, and a listener whose shoulders and weight answer the same moment reads as far more present than one whose face moves on its own. Head nods are not what this is for, the character makes those itself. See Listening Gestures below.

public class ListenerPosture : MonoBehaviour
{
    [SerializeField] ListenerContext listener;
    [SerializeField] Animator body;

    void OnEnable()  => listener.TriggerEmotionDrawn += OnDrawn;
    void OnDisable() => listener.TriggerEmotionDrawn -= OnDrawn;

    // the body answering what the face is already showing
    void OnDrawn(TriggerEmotionDraw draw)
    {
        if (draw.Emotion == "Sympathetic") body.SetTrigger("LeanIn");
        else if (draw.Emotion == "Curious") body.SetTrigger("ShiftWeight");
    }
}

What the listener side, the ListenerContext component, offers:

Member	What it tells you
TriggerActivated	A demeanor started, or switched to another palette.
TriggerEmotionDrawn	The demeanor picked an emotion. Carries the emotion's name as Emotion, its Intensity, and Duration, the seconds until the next pick.
TriggerDeactivated	The demeanor is over, its fade out included.
TriggerStrength	How far the demeanor has faded in, from 0 to 1. Read it each frame and multiply it by the pick's intensity for how strongly the demeanor is playing.
TriggerPaused	True while the character's own line has paused the demeanor.

Treat each pick as what the face shows now, not as a schedule. Duration only holds if nothing interrupts. A line pauses the demeanor, and when the line ends it picks again straight away, so a new pick can arrive before the last one's time is up.

Keep the handlers light. The events fire while FaceCue updates the character, so a slow handler costs the frame. Change dialogue state from your own code's update rather than from inside a handler.

This May Change

These events are how a demeanor is announced today. The Dialogue System Integration on the roadmap works in the same area, and it may change what is announced or when, so expect to revisit anything built on these when it lands.