Baking from Code¶

Baking, transcribing and synthesising from a script, with the windows' own settings.

Everything the Cue Baker and the Speech Production Tools do by hand, a script can do too. Bake a folder after an import, bake a line again after its transcript changed, speak a line again after its text changed, or transcribe a folder of recordings overnight. The calls live on one static class, FaceCueBaking, in the Firecrest.Systems.FaceCue.Editor namespace.

Each tool has one call for each of its window's two modes, so a script reads like the Mode row it stands in for:

Tool	Single	Batch
The Cue Baker	`Bake`	`BakeBatch`
Recognize Speech	`Transcribe`	`TranscribeBatch`
Synthesize Speech	`Synthesize`	`SynthesizeBatch`

How a Call Works¶

The window's saved settings are the starting point. Every call reads what its window last saved: a bake from the Cue Baker, a transcription from the Recognize Speech tab, a synthesis from the Synthesize Speech tab. An options object changes any of those for that one call, and a setting you leave unset keeps the saved value. So the usual pattern is to tune the window once, by hand, and let the script do the repetition. The right-hand settings panes are not part of the options: a call uses what they have saved, as the window does.

A call writes what the window writes. The calls run the same code the windows run, so a call and a window run with the same settings produce the same files.

The calls are for the editor. They run on the main thread and return when the work is done, with no progress bar. Put the script in an Editor folder, or an assembly that only builds for the editor.

Nothing is downloaded. A call that needs a model you have not installed stops at once with an InvalidOperationException, and the message names what to download in the Cue Baker's Setup. The same goes for anything else the call cannot work with, a missing audio file, a language the strategy does not read, a voice that does not exist: the exception comes before any work is done, and its message says what to change.

A model stays loaded after a call, so the next call skips the load, which takes several seconds for the larger models. See Freeing the Models.

Choices That Carry Their Settings¶

In the windows, some settings sit beneath another one. Picking a strategy in the Cue Baker brings up its language and its recognizer. Picking a CSV index as the transcript source brings up a CSV picker. Turning Output Beside Source off brings up a folder.

The options follow the same shape. A choice like that is not a plain value: it is built by a method named after the option you would pick in the window, and the settings beneath it are that method's arguments.

Strategy = CueBakeStrategy.Accurate("es", RecognizerKind.TranscriptDecoder, dialect: "River Plate"),
Transcript = TranscriptSource.CsvIndex("Assets/Dialogue/lines.csv"),
Output = OutputLocation.Folder("Assets/Cues/Spanish"),

Two things follow from that:

A setting goes only where it is read. There is nowhere to give the Universal strategy a language, or a .txt transcript a CSV path, because neither reads one. Your editor's autocomplete lists exactly the settings each choice has.
Whatever a choice leaves out keeps the saved value, the same rule as an option left unset. CueBakeStrategy.Accurate() bakes with the Accurate strategy in the language, recognizer and dialect the Cue Baker saved, and CueBakeStrategy.Accurate("fr") changes only the language. That is what the window does when you switch strategy: each one comes back with its own last settings.

Choice	Built with	In the window
`CueBakeStrategy`	`Accurate(language, recognizer, dialect)`, `MultilingualFreeDecode()`, `MultilingualTranscript(language)`, `Universal()`	The Cue Baker's Strategy and the rows beneath it
`TranscriptSource`	`PlainText(text)`, `TxtSidecar()`, `CsvIndex(csvPath)`	The Cue Baker's transcript Source
`TranscribeStrategy`	`Specialized()`, `Multilingual(language)`	Recognize Speech's Strategy and Language
`TranscriptOutput`	`TxtSidecar(location)`, `CsvIndex(location, existingIndex)`	Recognize Speech's Format and the output beneath it
`SpeechLine`, `SpeechLines`	See Synthesising	Synthesize Speech's Source
`LanguageCoaching`	`None()`, `Standard()`, `Voice(strength)`	Synthesize Speech's Language Coach and Coaching
`OutputLocation`	`BesideSource()`, `Folder(path)`, `File(path)`	Output Beside Source and the path under it

Existing Files¶

Every call takes an ExistingFiles argument, and it has no default. A cue file or a transcript can hold edits made by hand, so whether a run may replace one is a choice the script states every time, rather than one it inherits.

Value	What it does
`ExistingFiles.Skip`	Keeps the file that is there and writes nothing in its place
`ExistingFiles.Overwrite`	Replaces it. A cue file or a transcript is first copied into a backup folder beside it. Audio is replaced without a backup
`ExistingFiles.KeepAndIterate`	Keeps the file and saves the new one beside it under a numbered name. Only `Synthesize` does this. Every other call treats it as `Skip`

Baking¶

public static CueBakeResult Bake(string audioFilePath, ExistingFiles existingFiles,
    BakeOptions options = null)
public static IReadOnlyList<CueBakeResult> BakeBatch(string audioFolderPath, ExistingFiles existingFiles,
    BakeOptions options = null)

Parameter	What it does
`audioFilePath`	For `Bake`: the clip's audio file, WAV, OGG, MP3 or FLAC, as a path under `Assets/` or an absolute path
`audioFolderPath`	For `BakeBatch`: the folder holding the audio. Only its top level is read
`existingFiles`	What to do with a clip that already has a cue file. See Existing Files
`options`	Settings that replace the Cue Baker's saved ones for this call. Leave it out to bake with the saved settings

BakeBatch pairs clips with their transcripts the way Batch Baking does, from a TXT file beside each recording or from a CSV index. One clip's failure does not stop the rest.

// Single mode: one clip, with the Cue Baker's saved settings
FaceCueBaking.Bake("Assets/Audio/intro_01.wav", ExistingFiles.Overwrite);

// Batch mode: every clip in a folder, its transcripts read from the TXT files beside the audio
FaceCueBaking.BakeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip,
    new BakeOptions { Transcript = TranscriptSource.TxtSidecar() });

Bake Options¶

Member	What it does
`CueBakeStrategy Strategy { get; set; }`	How the cues are produced, with the settings beneath it. See Strategy
`TranscriptSource Transcript { get; set; }`	Where each clip's transcript is read from. See Transcript
`LanguageSpans? LanguageSpans { get; set; }`	How parts of a transcript tagged as another language are read: `Ignore`, `SinglePass` or `PerLanguage`
`bool? UseTranscript { get; set; }`	Whether a strategy that reads the audio alone also reads the transcript, for its tags. `false` leaves it unread. A transcript decoder always reads it, so `false` with one is refused
`BakingBackend? Backend { get; set; }`	`Cpu`, `DirectML` or `Cuda`. It must be one installed in this editor. See Backends
`OutputLocation Output { get; set; }`	`OutputLocation.BesideSource()` writes each cue file beside its audio, `OutputLocation.Folder(path)` into one folder
`string ClipName { get; set; }`	The name the cue file is written under. `BakeBatch` does not accept it
`int? BackupVersions { get; set; }`	How many earlier backups an overwrite keeps. 0 keeps every one

Strategy¶

The strategies are described in Recognizers.

Built with	What it bakes with
`CueBakeStrategy.Accurate(language, recognizer, dialect)`	The recognizer built for the language. `language` is one of the ten tuned languages, `recognizer` is `RecognizerKind.TranscriptDecoder` or `RecognizerKind.FreeDecoder`, and `dialect` is one of the language's dialects, or `""` for its default. The language's first recognizer of that kind in the Recognizer dropdown is used. A language with none of that kind uses its first recognizer instead, and the Console says so
`CueBakeStrategy.MultilingualFreeDecode()`	One model for many languages, from the audio alone
`CueBakeStrategy.MultilingualTranscript(language)`	One model for many languages, following the transcript. `language` is any language this mode reads, or `FaceCueBaking.AutoDetect` to detect each clip's language from its transcript
`CueBakeStrategy.Universal()`	The audio alone, with no language. It needs the OVRLipSync integration

Every argument can be left out, and then keeps what the Cue Baker saved. The codes and dialects are listed under Language & Dialect Codes.

Transcript¶

Built with	Where the transcript comes from
`TranscriptSource.PlainText(text)`	The text you give, used as it is. `BakeBatch` does not accept it, since a folder bake reads each clip's own transcript
`TranscriptSource.TxtSidecar()`	A TXT file beside each recording, with the clip's name
`TranscriptSource.CsvIndex(csvPath)`	A row of a CSV index. `csvPath` names the CSV, or is `FaceCueBaking.AutoDetect` for the one found beside the audio, the CSV picker's Auto. Left out, it keeps the CSV the Cue Baker saved

A bake has no row picker. Each clip reads the row whose clip id is its own file name, in single mode as in batch, so baking one clip from a CSV index is a matter of naming the CSV.

Every option set, for one Spanish clip:

var options = new BakeOptions
{
    Strategy = CueBakeStrategy.Accurate("es", RecognizerKind.TranscriptDecoder, dialect: "River Plate"),
    Transcript = TranscriptSource.PlainText("Lo encontré ahí en la calle."),
    LanguageSpans = LanguageSpans.PerLanguage,
    Backend = BakingBackend.Cuda,
    Output = OutputLocation.Folder("Assets/Cues/Spanish"),
    ClipName = "street_01",
    BackupVersions = 5,
};
FaceCueBaking.Bake("Assets/Audio/es_street_01.wav", ExistingFiles.Overwrite, options);

And for a folder whose clips are in several languages, each detected from its row of a CSV index:

var options = new BakeOptions
{
    Strategy = CueBakeStrategy.MultilingualTranscript(FaceCueBaking.AutoDetect),
    Transcript = TranscriptSource.CsvIndex("Assets/Dialogue/lines.csv"),
    Backend = BakingBackend.Cuda,
    Output = OutputLocation.BesideSource(),
    BackupVersions = 5,
};
FaceCueBaking.BakeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip, options);

Bake Results¶

Each CueBakeResult names the clip's AudioPath, the CueFilePath written or kept, a Status and, when the clip was not baked, a Message saying why. Status is one of Baked, KeptExisting, NoTranscript (the recognizer reads against a transcript and the clip has none) or Failed. ToString() puts all of it on one line, ready for a log.

CueBakeResult result = FaceCueBaking.Bake("Assets/Audio/intro_01.wav", ExistingFiles.Skip);
Debug.Log(result.CueFilePath);

foreach (CueBakeResult r in FaceCueBaking.BakeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip))
{
    if (r.Status != CueBakeStatus.Baked) Debug.LogWarning(r);
}

Transcribing¶

public static TranscribeResult Transcribe(string audioFilePath, ExistingFiles existingFiles,
    TranscribeOptions options = null)
public static IReadOnlyList<TranscribeResult> TranscribeBatch(string audioFolderPath, ExistingFiles existingFiles,
    TranscribeOptions options = null)

Parameter	What it does
`audioFilePath`	For `Transcribe`: the clip's audio file
`audioFolderPath`	For `TranscribeBatch`: the folder holding the audio. Only its top level is read
`existingFiles`	What to do with a clip that already has a transcript. See Existing Files
`options`	Settings that replace the Recognize Speech tab's saved ones for this call

The transcript is written where the tab would write it, a TXT file or a row in a CSV index, and the result also hands it back as text. TranscribeBatch reads several clips at once, which on a graphics card is faster than transcribing them one by one.

// Single mode: one clip, with the tab's saved settings
FaceCueBaking.Transcribe("Assets/Audio/intro_01.wav", ExistingFiles.Skip);

// Batch mode: every clip in a folder, each language detected from its audio
FaceCueBaking.TranscribeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip,
    new TranscribeOptions { Strategy = TranscribeStrategy.Multilingual(FaceCueBaking.AutoDetect) });

Transcribe Options¶

Member	What it does
`TranscribeStrategy Strategy { get; set; }`	Which model reads the audio, with its language. See Strategy and Language
`TranscriptOutput Output { get; set; }`	What each transcript is written as, and where. See Output
`BakingBackend? Backend { get; set; }`	The hardware the model runs on
`int? BackupVersions { get; set; }`	How many earlier backups an overwrite keeps. 0 keeps every one
`int? BatchSize { get; set; }`	How many clips are read at once, 1 to 16. Higher is faster and uses more graphics memory. Only `TranscribeBatch` accepts it

Strategy and Language¶

Built with	Which model reads the audio
`TranscribeStrategy.Specialized()`	The English model: English only, about twice as fast, with the same English transcripts. See Recognize Speech
`TranscribeStrategy.Multilingual(language)`	The model for every supported language. `language` is a code the model reads, or `FaceCueBaking.AutoDetect` to detect each clip's language. Left out, it keeps the tab's saved language

Output¶

Built with What is written

TranscriptOutput.TxtSidecar(location) One TXT file per clip, named after it. location is OutputLocation.BesideSource() or OutputLocation.Folder(path)

TranscriptOutput.CsvIndex(location, existingIndex) One row per clip in a CSV index. location is OutputLocation.BesideSource(), which writes a CSV named after the audio's folder, or OutputLocation.File(path). A CSV that does not exist yet is created. existingIndex is ExistingIndex.Merge, which keeps the rows already there, or ExistingIndex.Replace, which writes the index from this run alone. Only TranscribeBatch accepts it, since one clip always merges its own row

Either argument can be left out, and then keeps what the tab saved.

Every option set, for one French clip:

var options = new TranscribeOptions
{
    Strategy = TranscribeStrategy.Multilingual("fr"),
    Output = TranscriptOutput.TxtSidecar(OutputLocation.Folder("Assets/Transcripts/French")),
    Backend = BakingBackend.Cuda,
    BackupVersions = 5,
};
FaceCueBaking.Transcribe("Assets/Audio/fr_intro_01.wav", ExistingFiles.Overwrite, options);

And for a folder written into one CSV index:

var options = new TranscribeOptions
{
    Strategy = TranscribeStrategy.Multilingual(FaceCueBaking.AutoDetect),
    Output = TranscriptOutput.CsvIndex(OutputLocation.File("Assets/Dialogue/lines.csv"), ExistingIndex.Merge),
    Backend = BakingBackend.Cuda,
    BackupVersions = 5,
    BatchSize = 8,
};
FaceCueBaking.TranscribeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip, options);

Transcribe Results¶

Each TranscribeResult carries the clip's AudioPath, the TranscriptPath that holds its transcript, the Text itself, a Status and a Message. Status is Transcribed, KeptExisting, NoTranscript (nothing usable was heard) or Failed.

TranscribeResult line = FaceCueBaking.Transcribe("Assets/Audio/intro_01.wav", ExistingFiles.Skip);
Debug.Log(line.Text);

foreach (TranscribeResult r in FaceCueBaking.TranscribeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip))
{
    if (r.Status == TranscribeStatus.NoTranscript) Debug.LogWarning(r);
}

Synthesising¶

public static SynthesizeResult Synthesize(SpeechLine line, ExistingFiles existingFiles,
    SynthesizeOptions options = null)
public static IReadOnlyList<SynthesizeResult> SynthesizeBatch(SpeechLines lines, ExistingFiles existingFiles,
    SynthesizeOptions options = null)

Parameter	What it does
`line`	For `Synthesize`: the one line to speak. See below
`lines`	For `SynthesizeBatch`: the lines to speak. See below
`existingFiles`	What to do when a line's audio is already there. `Synthesize` can keep it and number the new take. See Existing Files
`options`	Settings that replace the Synthesize Speech tab's saved ones for this call

The line is built the way the tab's Source row picks it:

Built with	What is spoken	The wav is named after
`SpeechLine.PlainText(text, fileName)`	The text you give. Transcript tags are read as the tab reads them	The first words of the text
`SpeechLine.TxtFile(path, fileName)`	The line in a TXT file	The TXT file
`SpeechLine.CsvRow(csvPath, clipId, fileName)`	The line in the CSV row with that clip id	The clip id
`SpeechLines.TxtFolder(path)`	Every TXT file in a folder, one line per file	Each TXT file
`SpeechLines.CsvIndex(csvPath)`	Every row of a CSV index	Each row's clip id

fileName is optional, and names the wav instead. A batch names each file after its line, so the audio pairs with its transcript for a bake straight away.

// Single mode: a line given as text, the line in a TXT file, or one row of a CSV index
FaceCueBaking.Synthesize(SpeechLine.PlainText("Keep the door closed until I come back."), ExistingFiles.KeepAndIterate);
FaceCueBaking.Synthesize(SpeechLine.TxtFile("Assets/Dialogue/captain_01.txt"), ExistingFiles.Overwrite);
FaceCueBaking.Synthesize(SpeechLine.CsvRow("Assets/Dialogue/scene_03.csv", "captain_03"), ExistingFiles.Overwrite);

// Batch mode: every line in a folder of TXT files, or every row of a CSV index
FaceCueBaking.SynthesizeBatch(SpeechLines.CsvIndex("Assets/Dialogue/scene_03.csv"), ExistingFiles.Skip);

Voices¶

A call names its voice with Voice, in one of three ways:

`Voice`	Which voice speaks
`"Captain #1"`	One of your registered voices, by its name exactly as Registered Voices lists it. That name is the voice file's name without its extension
`"Male Sample"`	The voice FaceCue comes with
`"Assets/Voices/Extras/Old Guard.fcv1"`	A voice file anywhere on disk, by its path

Leave Voice out to use the voice the tab has selected. A name that matches no voice stops the call, and the message lists the voices there are.

Male Sample Is for Trying Things Out

A call that speaks with Male Sample writes a warning to the Console, the same one the tab shows before it synthesises. Turning the warning off in the Speech Production Tools turns it off here too.

Synthesize Options¶

Member	What it does
`string Voice { get; set; }`	The voice that speaks. See Voices
`string Language { get; set; }`	A language code, or `FaceCueBaking.AutoDetect` to detect it from the text. See Language & Dialect Codes
`LanguageCoaching LanguageCoach { get; set; }`	`LanguageCoaching.None()`, `LanguageCoaching.Standard()` or `LanguageCoaching.Voice(strength)`, where `strength` is `CoachingStrength.Light`, `Firm` or `Full`, and left out keeps the tab's Coaching. See Language Coach
`int? Seed { get; set; }`	The number that fixes the random choices in a take. The same seed, voice and text give the same audio on the same machine
`BakingBackend? Backend { get; set; }`	The hardware the model runs on
`OutputLocation Output { get; set; }`	`OutputLocation.BesideSource()` writes each wav beside the file its line was read from, `OutputLocation.Folder(path)` into one folder. A line given as text has no file to sit beside, so it takes a folder

Every option set, for one row of a CSV index:

var options = new SynthesizeOptions
{
    Voice = "Captain #1",
    Language = "en",
    LanguageCoach = LanguageCoaching.Voice(CoachingStrength.Firm),
    Seed = 7,
    Backend = BakingBackend.Cuda,
    Output = OutputLocation.Folder("Assets/Audio/Scene3"),
};
FaceCueBaking.Synthesize(SpeechLine.CsvRow("Assets/Dialogue/scene_03.csv", "captain_03", fileName: "captain_03_take2"),
    ExistingFiles.KeepAndIterate, options);

And for a folder of TXT files, each wav written beside its line:

var options = new SynthesizeOptions
{
    Voice = "Captain #1",
    Language = FaceCueBaking.AutoDetect,
    LanguageCoach = LanguageCoaching.Standard(),
    Seed = 7,
    Backend = BakingBackend.Cuda,
    Output = OutputLocation.BesideSource(),
};
FaceCueBaking.SynthesizeBatch(SpeechLines.TxtFolder("Assets/Dialogue/Scene3"), ExistingFiles.Skip, options);

Synthesize Results¶

Each SynthesizeResult carries the line's Name, the AudioPath written or kept, a Status, Synthesized, KeptExisting or Failed, and a Message when it failed.

SynthesizeResult take = FaceCueBaking.Synthesize(SpeechLine.PlainText("Hold the gate."), ExistingFiles.KeepAndIterate);
Debug.Log(take.AudioPath);

foreach (SynthesizeResult r in FaceCueBaking.SynthesizeBatch(SpeechLines.CsvIndex("Assets/Dialogue/scene_03.csv"), ExistingFiles.Skip))
{
    if (r.Status == SynthesizeStatus.Failed) Debug.LogWarning(r);
}

Language & Dialect Codes¶

Scripts name a language by its code, shown beside each language below. Search for a language to find its code, or for a code to find its language. The same checker is on Language Support, which explains what each layer means.

The chips say which calls take the code.

Chip	The calls that take the code
Accurate	`Bake` and `BakeBatch`, in `CueBakeStrategy.Accurate` and in `CueBakeStrategy.MultilingualTranscript`
Multilingual	`Bake` and `BakeBatch`, in `CueBakeStrategy.MultilingualTranscript`
Universal	None. `CueBakeStrategy.MultilingualFreeDecode` and `CueBakeStrategy.Universal` read any language, so they take no code
Transcript	`Transcribe` and `TranscribeBatch`, in `TranscribeStrategy.Multilingual`
Synthesis	`Synthesize` and `SynthesizeBatch`, in `SynthesizeOptions.Language`

A call given a code it does not take stops before any work, and the message lists the codes it does. Norwegian has two codes, "nb" and "no", and every call takes either.

Dialects belong to the Accurate strategy, and the checker lists them under their language. CueBakeStrategy.Accurate takes the name as shown there as its dialect, such as "River Plate".

Freeing the Models¶

public static int ClearModelsInMemory()

Frees every model the Cue Baker and the Speech Production Tools hold in memory, what Clear All does in the Models In Memory window, and returns how many were held. Call it when a script has finished a run and the memory is wanted back. The next call that needs a model loads it again.

FaceCueBaking.BakeBatch("Assets/Audio/Chapter1", ExistingFiles.Skip);
FaceCueBaking.ClearModelsInMemory();