--- version: "5.3" language: "en" --- # SG Com ## Welcome to SG Com The SG Com SDK provides access to Speech Graphics' real-time audio-driven facial animation technology, enabling developers to build interactive applications with avatars driven by voice. * [SG Com Introduction](https://docs.speech-graphics.com/en/sg-com/5.3/introduction-to-sg-com.md) * [SG Com Concepts](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-concepts.md) * [SG Com API](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-c-api.md) * [SG Com Unreal Engine Plugin](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-unreal-engine-plugin.md) * [SG Com Unity Package](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-unity-package.md) --- version: "5.3" language: "en" --- # SG Com API Reference Jump to [SG Com API Reference](http://sgcom-docs.speech-graphics.com/SG_Com_5.3/html/index.html). --- version: "5.3" language: "en" --- # Engines and Players SG Com has two main components: Engines and Players. The **Engine** does the main processing work: it converts an incoming audio stream into synchronous facial animation. Output animation includes lip sync, full-face emotional expressions, blinks, eye darts, and even head movement, all driven by the speaker's voice through Speech Graphics algorithms. There is a small processing latency (50 milliseconds) between audio input and animation output. An Engine has one audio source and generates animation for one character. See [SG Com Input](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-input.md) regarding input audio. Engine output must be streamed to a **Player**for buffering, decoding and playback. The character's face is animated by incrementing the current time in the Player and updating the scene based on the animation values obtained. Animation values are automatically interpolated based on the times requested. ## Networked vs local use cases The input voice may be used to drive an animated character either locally or remotely over a network. In the local use case, both Engine and Player are on the same machine: ![EnginePlayerLocal.png](https://docs.speech-graphics.com/__attachments/a_7fd7a630bed8e73b343a5fa5d27d16684d3773994d0d4c14bd58004a892427fd/EnginePlayerLocal.png?cb=32db86488e8cc6f41ef52f7eb04e3bb1) In the remote use case, one may run audio through the Engine locally, but then send the Engine output (and audio) to remote Players for buffering and output. Below is a depiction of a two-way conversation over a network, in which each side has a local Engine for processing input audio, and a Player for output of the other participant's animation. ![EnginesPlayersNetworked.png](https://docs.speech-graphics.com/__attachments/a_146efaab32cc9e21ae325419e6c0a16104762243dc16ba55d57c0bfd50a1041e/EnginesPlayersNetworked.png?cb=44898f842656140768801653fa2263c6) SG Com does not perform network operations. Data must be transmitted by the application. To get the benefit of SG Com within a fully networked, end-to-end service, see [Rapport.cloud](https://www.rapport.cloud/). ## Engine idle SG Com does not require sound in order to continue generating behavior on the target character. There can be breaks in the audio input, and the engine can continue turning over, generating nonverbal behavior and responding to behavior controls. This is called idling. Note that idling adds output that has no corresponding audio input, which must be taken into account when synchronizing output of audio and animation . There are two ways to generate idle behavior: by running an engine with no audio input, or by inputting zeros for the audio. See [Idle mode](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Idle-mode) in the SG Com API Tutorial. ## Synchronizing output In order to synchronize audio and animation output, it's important to keep in mind the following: * **SG Com latency**. After an audio sample is input into the Engine, there is a 50 millisecond delay until the corresponding output packet of animation is produced. This represents a perceptible asynchrony. Therefore, audio received from the microphone or other source cannot be immediately output at the same time it is input to the Engine. It must be output in synchrony with the animation. * **Time is audio-based**. The animation playback loop involves passing time values to the Player to retrieve animation frames. The time domain of the Player is based on the audio that went into the Engine (plus any idle ticks). Therefore, to keep animation synchronized with audio, you must know the time stamp of the audio, correcting for any lost audio data. Time 0 corresponds to the very first frame of audio that was input into the Engine. ## Metadata updates In addition to generating real-time animation, the SG Com Engine also generates updates from its internal algorithms, which can be used externally to inform other processes. At present, SG Com generates updates for these kinds of changes: * Change in detected auto-mode (positive, negative, laugh, effort, etc) * Change in behavior mode * Change in facial expression * Change in voice detection (voice or not) * Change in breath direction (inhale vs exhale) In each case, the notification is accompanied by a text payload describing the new state (e.g., the new behavior mode). --- version: "5.3" language: "en" --- # SG Com Introduction * [What is SG Com?](https://docs.speech-graphics.com/en/sg-com/5.3/what-is-sg-com.md) * [What's new in SG Com 5.3?](https://docs.speech-graphics.com/en/sg-com/5.3/what-s-new-in-sg-com-5-1.md) * [SG Com License](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-license.md) * [SG Com Files](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-files.md) * [SG Com Release Notes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-release-notes.md) * [SG Com Platform Support](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-platform-support.md) * [SG Com Compute Resource Usage](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-compute-resource-usage.md) --- version: "5.3" language: "en" --- # SG Com API Tutorial ## Initialization, shutdown and logging Before the library can be used it must be initialized using [SG_COM_Initialize](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aeddb74e1b4c12a8d6d53baadea41f46d), and when it will no longer be used it should be shut down using [SG_COM_Shutdown](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#af761920a3c8022f5c64eea0d9f9ba5a8). Calls to these functions should be the first and last interactions with the API. [SG_COM_Initialize](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aeddb74e1b4c12a8d6d53baadea41f46d) takes three main arguments: the logging level ([SG_LoggingLevel](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#acbfe2b9afcea20b7116b7cf6f6ebf128)), an optional callback to receive log messages ([SG_LoggingCallback](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#a02a6982c31b19fe2ee5a7eab9fd03bcb)), and license data. The license data will be either the path of the license file, or in the case of a non-networked license, the license string loaded into memory. ## Creating and destroying Engines and Players Animating a character with SG Com requires an Engine and a corresponding Player (see [Engines and Players](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md)). To create an Engine, use [SG_COM_CreateEngine](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a9c529aaf08063e4a45ca157faabe70e0), which requires an [Engine configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-configuration). To create a Player, use [SG_COM_CreatePlayer](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#acc790120a7e32f097046fe568a129223), which requires an [Player configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Player-configuration). To destroy Engines and Players, use [SG_COM_DestroyEngine](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aba12fc104bd56ea949a020e625e4d188) and [SG_COM_DestroyPlayer](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aba12fc104bd56ea949a020e625e4d188), respectively. ## Engine configuration An [SG_COM_EngineConfig](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___c_o_m___engine_config.html) structure includes the following: * The [Character Control File](https://docs.speech-graphics.com/sgx?contextKey=character-setup-process&version=latest&language=default#Character-Control-File), loaded as bytes into memory. * The sample rate and sample type of the audio bytes that will be input (see [Engine input](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-input)). * The input buffer length, expressed as the maximum audio duration (in seconds) that the input buffer can hold (see [Engine input](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-input)). This should be as long as the longest audio chunk. * Optionally, a handle to the corresponding Player if it is local (see [Engine to Player transfer](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-to-Player-transfer)). * An [SG_COM_EngineBroadcastCallback](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a0f3eedb7d916f1f609a941841c27ba7b) function, which will be invoked whenever an Engine has an output packet ready. Not needed for local Players (see [Engine to Player transfer](#)). * An [SG_COM_EngineStatusCallback](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a6c086e7b02ed8c7492e312e891835c21) function, which will be invoked whenever there is an Engine status update (see [Metadata updates](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Metadata-updates)). ## Player configuration An [SG_COM_PlayerConfig](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___c_o_m___player_config.html) structure includes the following: * The [Character Control File](https://docs.speech-graphics.com/sgx?contextKey=character-setup-process&version=latest&language=default#Character-Control-File), loaded as bytes into memory. This must be the same file that is used in the corresponding Engine. * The output buffer length, expressed as the maximum duration of animation (in seconds) that the output buffer can hold. This should be at least as along as the Engine's input buffer length. It can also be longer to avoid losing older animation if playback is delayed. ## Engine input The function [SG_COM_InputAudio](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aff18c6193260c284ef0b85819fd8e528) feeds audio into the Engine to drive facial animation. For each call to [SG_COM_InputAudio](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aff18c6193260c284ef0b85819fd8e528), you will provide an array of bytes containing some number of audio samples. For low-latency applications, we recommend inputting audio as soon as it is available. The ideal input size is 10 milliseconds since that is the internal processing frame size. However, the size of each input package can vary. The upper limit is determined by the buffer size you specify in [SG_COM_EngineConfig.buffer_sec](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___c_o_m___engine_config.html#af814d7e67f1b423e4bffd6044c2dc08e). Input audio is required to be raw, linear PCM, mono (see [SG Com Input](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-input.md)), and the sampling must agree with the [SG_COM_EngineConfig](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___c_o_m___engine_config.html) that was used to create the Engine, specifically [SG_COM_EngineConfig.audio_sample_type](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___c_o_m___engine_config.html#a295e47a8cc12535d3c9bd45dfff8c7e4) and [SG_COM_EngineConfig.audio_sample_rate](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___c_o_m___engine_config.html#a295e47a8cc12535d3c9bd45dfff8c7e4). In some customized use cases, you might also have need for [SG_COM_InputAuxData](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#ae7c3750ba4e2f9dd6354e378eb131ea8), which is used to input auxiliary (non-audio) data. Note auxiliary data is always assumed to have a sample rate of 100 Hz. ## Engine processing tick When you input data using [SG_COM_InputAudio](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aff18c6193260c284ef0b85819fd8e528) or [SG_COM_InputAuxData](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#ae7c3750ba4e2f9dd6354e378eb131ea8), no actual processing happens; the input data is buffered. In order to actually process the input data into animation, you must call [SG_COM_ProcessTick](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a16e893aaa805f815d6703df1c385fa08). This function executes the processing necessary to advance one frame of input (10 ms) through the Engine pipeline. The Engine tick function must be called at least once for every 10 ms of input data. After an initial algorithmic delay of 50 ms, each call to [SG_COM_ProcessTick](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a16e893aaa805f815d6703df1c385fa08) will generate one output packet. Depending on the [Engine configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-configuration), this packet will be delivered either to the [SG_COM_EngineBroadcastCallback](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a0f3eedb7d916f1f609a941841c27ba7b) or directly into the local Player (see [Engine to Player transfer](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-to-Player-transfer)). [SG_COM_ProcessTick](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a16e893aaa805f815d6703df1c385fa08) is designed to be called on a background thread or threadpool. ### Idle mode Normally the Engine requires input in order to produce output. However it can also generate [idle behavior](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Engine-idle) -- blinks, eye darts, and nonverbal expressions -- in the absence of input. To enable idle behavior, use the flag [SG_COM_EngineConfigFlag.SG_COM_ENGINE_CONFIG_ENABLE_IDLE](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#ae555b30cbc7b2c511e836b62a5b6d33aae13047448f9cc3cf2d0bd0cc66d4c1c8) in your [Engine configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-configuration). When idle mode is enabled, it means that even when there is no input, the processing tick will continue to generate output frames, replacing data-driven behavior with idle behavior. An equivalent way to generate idle behavior without this flag is to use [SG_COM_InputAudio](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aff18c6193260c284ef0b85819fd8e528) with all zeros in the input bytes. Idle behavior is smoothly blended with other output. There will be no breaks or sudden changes in the animation when the character enters or leaves idle mode. ## Engine to Player transfer Output packets from the Engine must be delivered into the Player. There are two ways of doing this, depending on whether the Player is deployed locally or across a network (see [Networked vs local use cases](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Networked-vs-local-use-cases)): * For the local case, provide the Player handle in the [Engine configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-configuration). Output packets from the Engine will automatically be input into the Player, without the need for additional data handling. * For the networked case: 1. obtain output packets from the [SG_COM_EngineBroadcastCallback](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a0f3eedb7d916f1f609a941841c27ba7b) 2. transfer them over the network 3. input them into the remote Player using [SG_COM_ReceivePacket](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#af21561a05c9cce0d9b1c4d2d0efe2b87) ## Animation nodes In order to play the output, it's necessary to first get the animation nodes from the Player using [SG_COM_GetAnimationNodes](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#aef83b22a26679b6b689fe0e9296e5657). This provides an array of nodes. Each [SG_AnimationNode](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html) represents an object in the character rig, as defined in SGX Studio (see [Editing Animation Targets](https://docs.speech-graphics.com/sgx?contextKey=editing-animation-targets-maya&version=latest&language=default)). The [SG_AnimationNode](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html) structure contains: * [SG_AnimationNode.name](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html#af6a8c6b633930b44f89026c269fec2d4) The name of the node * [SG_AnimationNode.type](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html#a2b8147e86d09573e281aad9050bc0b7c) The type of the node ([SG_JOINT](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#a7b95efe8facb27c3e0af84b6cf400f35acc6c45fdfde72e790f6374cf0d8964f8), [SG_BLENDSHAPE](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#a7b95efe8facb27c3e0af84b6cf400f35afb21bf6e6ff2b40ff9318772d47e6ccf) or [SG_CONTROL](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#a7b95efe8facb27c3e0af84b6cf400f35afb21bf6e6ff2b40ff9318772d47e6ccf)) * [SG_AnimationNode.num_channels](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html#a8de5957d969019c082dfb1d6ee73e525) The number of animation channels * [SG_AnimationNode.channel_names](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html#a6393b0099c2cbb28b7c482ff8dc9a007) An array containing the names of the animation channels * [SG_AnimationNode.channel_values](http://sgcom-docs.speech-graphics.com/SG_Com/html/struct_s_g___animation_node.html#a3e71022b5a95e3f980c99999a16dc553) An array containing the current values of the animation channels Below is an example showing how to obtain the animation nodes. C SG_AnimationNode* animation_nodes; sg_size num_animation_nodes; SG_COM_GetAnimationNodes(player_handle, &animation_nodes, &num_animation_nodes); After retrieving the animation nodes, the animation channels in the nodes must be mapped to the corresponding attributes in the character rig, so that values can be quickly copied from the nodes to the rig during animation. This mapping only needs to be established once after constructing the Player, since the animation nodes do not change. ## Animation playback To output animation from the Player, call [SG_COM_UpdateAnimation](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a75f66459ef5da889b09e44e8950d7c1e) at regular intervals within your application's animation update loop. The function takes a time value as input; see [Synchronizing output](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#%255BinlineExtension%255DSynchronizing-output) on keeping these time values synched to audio output. Given a time value, the function samples and decodes the buffered motion data in the Player to produce a frame of animation, which is stored in the channels of the [Animation nodes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Animation-nodes). These values can then be copied from the animation nodes into the corresponding attributes of the character rig in your scene. C // Enter the real-time playback loop float *animation_frame; for (;;) { // Update the animation. err = SG_COM_UpdateAnimation(player_handle, time_ms, ¤t_time_ms); // Get the current value for each channel of each node for (size_t i = 0; i See [Using Behavior Modes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Using-Behavior-Modes) for how to set behavior modes explicitly in SG Com API. ### Auto Modes In addition to setting behavior modes explicitly, you can also let SG Com determine the behavior mode from the input audio. SG Com uses AI to detect several vocal qualities, triggering automatic behavior modes called **auto modes**. The auto modes may be defined as any of the behavior modes you have designed for the character. There are currently five auto modes in SG Com: * **Positive**- Happy or positive emotion is detected in the voice. * **Negative**- Negative or unhappy emotion is detected in the voice. This encompasses a range from sadness to anger. * **Laugh**- Laughter is detected in the voice. * **Effort**- A grunt or effort sound is detected in the voice. This may happen when the voice actor is portraying activities of physical exertion or bodily impact, typically involving a tightening of the vocal chords to make guttural non-speech sounds. * **Acknowledge** - An auto mode triggered by entering the listening role (see [SG Com Roles](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-roles.md)). Auto modes may be mapped to the behavior modes of your character, as shown in the image below. This mapping may be changed at any time. ![SGX_AutoModes.png](https://docs.speech-graphics.com/__attachments/a_bd8ea658b1cdcfbee2570bc53ea8e83c701b51efadeb7ed09221679c23adb108/SGX_AutoModes.png?cb=7e7fcfa20d408d1c6f1001796eaac7ce) The auto modes system is optional. While activated, auto modes will trigger whenever their vocal conditions are detected; otherwise, the character remains in the default mode. It is up to the user which auto modes to map. If an auto mode does not have a defined mapping, it will not trigger a change in behavior. See [Using Auto Modes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Using-Auto-Modes) for how to configure and activate auto modes in SG Com API. ## Behavior Modifiers Behavior modifiers are numerical parameters that can be used to scale particular dimensions of the performance, such as magnitude, speed and frequency of events. Like behavior modes, these values can be changed at any time while SG Com is running, and the effects are character-specific. |--------------------------|-------------------------------------------------|-------------------|----------------------|-------------| | **Modifier** | **Description** | **Minimum** | **Default (normal)** | **Maximum** | | **Speech Magnitude** | Adjusts magnitude of all speech movements | 0.0 (no movement) | 1.0 | 2.0 | | **Nonverbal Magnitude** | Adjusts magnitude of all nonverbal movements | 0.0 (no movement) | 1.0 | 2.0 | | **Breath Magnitude** | Adjusts magitude of all breaths | 0.0 (no breath) | 1.0 | 2.0 | | **Nonverbal Speed** | Adjusts speed of all nonverbal movements | 0.0 (exclusive) | 1.0 | 2.0 | | **Breath Speed** | Adjusts speed and frequency of breath movements | 0.0 (exclusive) | 1.0 | 2.0 | | **Expression Frequency** | Adjusts frequency of expression changes | 0.0 (exclusive) | 1.0 | 2.0 | | **Blink Frequency** | Adjusts frequency of blinks | 0.0 (no blinks) | 1.0 | 2.0 | | **Dart Frequency** | Adjusts frequency of eye microdarts | 0.0 (no darts) | 1.0 | 2.0 | See [Using Behavior Modifiers](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Using-Behavior-Modifiers) for how to adjust behavior modifiers through SG Com API. --- version: "5.3" language: "en" --- # SG Com API * [SG Com API Reference](https://docs.speech-graphics.com/en/sg-com/5.3/api-reference-1.md) * [SG Com API Tutorial](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md) * [Simple Code Example](https://docs.speech-graphics.com/en/sg-com/5.3/simple-code-example.md) --- version: "5.3" language: "en" --- # SG Com Compute Resource Usage SG Com runs on the CPU; it has no impact on the GPU. Below are measurements for memory, CPU and power consumption. ## Memory A typical chat use case involves a single Engine to process local microphone input, and one or more Players to calculate animation targets for local or remote characters (see [Engines and Players](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md)). Heap allocations and deallocations are only performed during construction and destruction of Engines and Players, and the heap is not touched by other functions. |----------|-----------------------|------------------------------------------------------------------|-----------| | **Code** | **Heap (per Engine)** | **Heap (per Player)** | **Stack** | | \~12 MB | \~12 MB | \~1.5 MB + (44 KB \* b) where b = Player buffer length (seconds) | \< 10 KB | ## CPU We provide speed measurements of the two high-frequency functions for Engine and Player: * [SG_COM_ProcessTick](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a16e893aaa805f815d6703df1c385fa08) (Engine). This function is called 100 times per second to process audio input in 10 ms blocks, and the measurement below is the time taken to process 1 block. * [SG_COM_UpdateAnimation](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a75f66459ef5da889b09e44e8950d7c1e) (Player). This function is called at the application frame rate to calculate values of the output animation channels, and the measurement below is the time taken to process 1 frame. For [SG_COM_UpdateAnimation](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a75f66459ef5da889b09e44e8950d7c1e), the processing time depends on the complexity and type of character rig. Two sample rigs are considered: * Rig 1 is a bone rig with 82 targeted bones * Rig 2 is an Epic MetaHuman with 252 targeted scalar channels (and a few bones) |-------------------------|--------------------|------------------------|--------------------|-------------------|------------|-----------| | | **Windows** | **Linux** | **MacOS** | **iOS/Android** | **XBox X** | **PS5** | | **Processor** | Intel Xeon E5-4620 | Intel Xeon Silver 4114 | Intel Xeon E5-1650 | ARM v8 Cortex-A72 | AMD Zen 2 | AMD Zen 2 | | **Speed (GHz)** | 2.2 | 2.2 | 3.5 | 1.8 | 3.66 | 3.5 | | **Process Tick (ms)** | 0.695 | 0.673 | 0.374 | 0.816 | 0.357 | 0.171 | | **Update - Rig 1 (ms)** | 0.187 | 0.183 | 0.0884 | 0.522 | 0.0981 | 0.0797 | | **Update - Rig 2 (ms)** | 0.0923 | 0.0823 | 0.0422 | 0.251 | 0.0357 | 0.0202 | Memory to CPU bandwidth is usually the limiting factor on performance, affected by cache size and patterns of memory use in the whole system. For best results the thread calling [SG_COM_ProcessTick](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g___com_8h.html#a16e893aaa805f815d6703df1c385fa08) is given fixed core affinity wherever possible. ## Mobile power consumption SG Com is optimized for mobile power consumption by using integer arithmetic for most calculations. The current consumption estimate is based on the time taken to discharge the device to 90% of its specified capacity versus an idle control case. Note this does not include the cost of 3D rendering. |----------------------------|-------------------------| | | **Galaxy S20 Ultra 5G** | | **Power consumption (mA)** | \~115 | --- version: "5.3" language: "en" --- # SG Com Concepts * [SG Com Input](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-input.md) * [Engines and Players](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md) * [SG Com Behavior Controls](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-behavior-controls.md) * [SG Com Roles](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-roles.md) --- version: "5.3" language: "en" --- # SG Com Files These are the files required to use SG Com: * The SG Com library headers and binaries * Your license key * For each character you want to drive with SG Com, a corresponding [Character Control File](https://docs.speech-graphics.com/sgx?contextKey=character-setup-process&version=latest&language=default#Character-Control-File) These files will be provided by Speech Graphics when you license or evaluate the software. If using the SG Com Unreal Engine Plugin, there is an Unreal package to download. See [SG Com UE Download and Installation](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-installation.md). --- version: "5.3" language: "en" --- # SG Com Input SG Com accepts only single channel raw audio. See accepted [sample rates](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#aad39d85514c66f5d255c00012ba0d91e) and [sample types](http://sgcom-docs.speech-graphics.com/SG_Com/html/_s_g_8h.html#a2f6de1d0c897a61984551b13486b4800). Uncompressed audio will give the best results. SG Com works for any spoken language, even fictional ones. It uses a universal model of the human vocal tract to map whatever sounds a person can make to corresponding articulation. It works under a variety of recording conditions, and with both natural speech and synthesized speech (text-to-speech). For custom applications, SG Com can accept other kinds of data through an auxiliary input stream. --- version: "5.3" language: "en" --- # SG Com License SG Com is distributed under a pre-agreed Software License Agreement between Speech Graphics Ltd and the licensee. Your license entitles you to a specific number of concurrent SG Com Engines running at any given time. With an evaluation or purchase of SG Com, you will receive a license file (`.lic`) that provides credentials to our cloud-based license server. It requires an active internet connection to contact the license server. You will also have to open two network ports, which will be specified when you receive the license file. For product deployments, non-networked license options are also available. --- version: "5.3" language: "en" --- # SG Com Platform Support SG Com 5 is built and tested on the following platforms: * Windows (64 bit) * Linux (64 bit) * MacOS (Intel, ARM) * Android (ARMv8) * iOS (16.4 or higher) * PlayStation 5 * Xbox Series X * Nintendo Switch --- version: "5.3" language: "en" --- # SG Com Release Notes ## SG Com SG Com 5.3.1 * Prevention of muscle order mismatch between Engine and Player * Fix to engine reset, so that output is again fully deterministic when using a fixed-seed engine * Refinement of blink frequency and dynamics * Refinement of microdart movement patterns SG Com 5.3.0 * See [What's new in SG Com 5.3](https://docs.speech-graphics.com/en/sg-com/5.3/what-s-new-in-sg-com-5-1.md). SG Com 5.2.0 * SG Com 5.2 is compatible with character control files produced in SGX Studio 4.5. Character setup is training-free, and character control files are now smaller, due to the absence of the neural network. See [++What's new in SGX 4.5++](https://docs.speech-graphics.com/en/sgx/4.5/what-s-new-in-sgx-4-5). * In the SG_COM_Error enum, two new error types have been added: * `SG_COM_ERROR_CONTEXT_NOT_INITIALIZED` (The SG Com context is not initialized. Call SG_COM_Initialize() first) * `SG_COM_ERROR_CONTEXT_ALREADY_INITIALIZED` (The SG Com context is already initialized. Call SG_COM_Shutdown() first) * The `SG_COM_Status` message`SG_COM_STATUS_AUTO_MODE_CHANGED` is now emitted regardless of whether any auto modes are activated. This is helpful for developers that want to use information from SG Com automatic event detection but do not want behavior modes to be triggered automatically; e.g., they would rather combine the acoustic event detection with other sources of information to decide the behavior mode. SG Com 5.1.2 * Fixed crash for Unity3D builds on iOS SG Com 5.1.1 * Fix for SG_COM_SetModifier returning "Unrecognized modifier code" when setting SG_COM_MOD_BREATH_SPEED to 0. * Fix for SG_COM_SetModifier not returning error when setting SG_COM_MOD_EXPRESSION_FREQ or SG_COM_MOD_NONVERBAL_SPEED to 0. SG Com 5.1.0 See [What's new in SG Com 5.1?](https://docs.speech-graphics.com/en/sg-com/5.3/what-s-new-in-sg-com-5-1.md) SG Com 5.0.9 * In SG.h, the deprecated enum `SG_COM_AnimationType` has been removed. * The `SG_COM_PlayerConfiguration` struct no longer has a member for `SG_COM_AnimationType` SG Com 5.0.8 * Fix for Intel Mac compatibility SG Com 5.0.7 * `SG_OTHER_ANIMATION_NODE` is now called `SG_CONTROL` * Universal Mac build SG Com 5.0.6 * Fix for crash with empty behavior mode SG Com 5.0.5 * Bug fix for `SG_COM_Initialize()` function throwing exception if given license filepath from Unreal Engine SG Com 5.0.2 * CPU cost of solving bones in SG_COM_Player reduced by two thirds * Choice of key-based and cloud-based licensing. Cloud-based licensing is now available for Windows and Linux. * New `SG_COM_Status` notification type, `SG_COM_STATUS_EXPRESSION_CHANGED` * Enhanced auto-mode controls * New auto mode: Effort detection ## SG Com Player SG Com Player 5.3.0 * See [What's new in SG Com 5.3](https://docs.speech-graphics.com/en/sg-com/5.3/what-s-new-in-sg-com-5-1.md). ## SG Com UE Plugin The SG Com UE Plugin is distributed in an [example application](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-sample-usage.md). SGComUE5.3-5.2.0 * Update to SG Com 5.2.0 SGComUE5.3-5.1.5 * Added UE5.6 support SGComUE5.3-5.1.4 * Update for SG Com API 5.1.2 SGComUE5.3-5.1.1 * Update to SG Com API 5.1.0 * Add compatibility with UE5.2 SGComUE5.3-5.0.0-beta.9 * Add `InputSound`blueprint node for getting raw PCM data out of a SoundWave. SGComUE5.3-5.0.0-beta.8 * Fixed an issue preventing shipping builds * Added `OnVoiceActivityChanged` delegate SGComUE5.3-5.0.0-beta.7 * `SGComInstaller.exe` utility for easy installation of SG Com binaries and license * Update to SG_COM_APL.xml * Ability to load MetaHuman Character Control Files baked from Maya without renaming joints ## SG Com Unity Package SG_Com_Unity-5.2.2 * Reset bones to neutral pose on enable. Ensures bones return to neutral pose when animation component is enabled. * Restore animator state before playing graph. Animator state is now properly restored prior to graph playback * Update Unity version of sample project to 6. Sample project now targets Unity 6. SG_Com_Unity-5.2.1 * Preserve animator pose and initialize bone data. Improves animator state management during initialization. SG_Com_Unity-5.2.0 * Update to SG Com 5.2.0 SG_Com_Unity-5.1.7 * Improved blending of bone animation between SG Com and Animation clips * New method `SetAudioClip` allows switching between audio clips and microphone input at runtime SG_Com_Unity-5.1.6 * Update for SG Com API 5.1.2 SG_Com_Unity-5.1.3 * Plugin is now compatible with Animator SG_Com_Unity-5.1.2 * No longer activate auto modes on voice activity SG_Com_Unity-5.1.1 * Asset update SG_Com_Unity-5.1.0 * Update to SG Com API 5.1.0 SG_Com_Unity-5.0.1 * Simplified AudioInput script. * Audio playback time drives UpdateAnimation. --- version: "5.3" language: "en" --- # SG Com Roles Normally the audio going into SG Com is used to generate lipsync animation, to make it appear that the character being animated is producing the speech. Alternatively, however, you can generate animation that makes it appear that the character is *hearing* the speech, and reacting to it emotionally in real time. This is similar to [idle behavior](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Engine-idle) in that there is no lip sync, but with the added benefit of nonverbal reactions to the speech. These two possibilities are called **roles** , and the two roles are **speaker** and **listener**. The switching of a character's role can happen many times over the course of a conversation, whenever the character goes from speaking to listening or back to speaking. This requires feeding multiple alternating voices into the same Engine. As you switch the role of a character, you can also manipulate the mapping of its [auto modes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-behavior-controls.md#Auto-Modes) to reflect the emotional interaction between speaker and listener. For example, if speaker and listener are friendly, then they might be aligned in their auto modes: e.g. both happy when the speaker's voice is Positive, or both sad when the speaker's voice is Negative. This is empathetic mirroring. But if there is aggression and intimidation going on, then when the voice is Positive, the speaker might be in some manic state while the listener is not amused, and when the voice is Negative, the speaker might be angry while the listener is afraid. --- version: "5.3" language: "en" --- # SG Com UE Configuration The **SG Com Session** component has several configurable settings (mostly corresponding to [Engine configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Engine-configuration) and [Player configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-api-tutorial.md#Player-configuration) in SG Com API) which can be found under the component's *Config*category. Their values can also be set via blueprint. ![Screenshot 2023-12-12 152108-20231212-152142.png](https://docs.speech-graphics.com/__attachments/a_5bf811241a9f4248249412c280131dd731b1bb46904ea95e8addb961f174ce0f/Screenshot%202023-12-12%20152108-20231212-152142.png?cb=ae812e4cfa90852eaa5b5378dbdd6297) |--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Setting** | **Description** | | Character File Path | Path to the [Character Control File](https://docs.speech-graphics.com/sgx?contextKey=character-setup-process&version=latest&language=default#Character-Control-File) (\*.k). | | Audio Sample Type | The type of incoming audio samples. This can be 16- bit integer, 32 bit integer or 32 bit float. | | Audio Sample Rate | The sample rate of the incoming audio data. | | Audio Buffer Seconds | The amount of audio in seconds that the input buffer can hold. This should be at least as large as the largest audio chunk. | | Animation Buffer Seconds | The amount of animation data in seconds that the output buffer can hold. This should be at least as large as the audio buffer. | | Enable Idle | Whether idle animation will be played when no audio data is being input. | | Fixed Random Seed | Whether a fixed seed will be used in random number generators for deterministic output. | --- version: "5.3" language: "en" --- # SG Com UE Download and Installation ## Download The following are available for [download](https://downloads.speech-graphics.cloud/): * The [SG Com UE Demo Application](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-sample-usage.md), which contains the SG Com UE Plugin. Start with this to learn how to integrate the package into your own application. * The SG Com UE Plugin by itself. ## Installation In order to use the plugin, it must be located in one of the search paths for Plugins, either in your project, or in the Engine itself: * Engine: `[Unreal Engine Root Directory]/Engine/Plugins/` * Game plugins: `[Project Root Directory]/Plugins/` You will also need to place your SG Com binaries into your SGComUE installation. The exact location is SGComUE/Source/SGCom/Binaries/\. Where \ is your target platform, e.g. * Win64 * Android/arm64-v8a * PS5 The license file path, log directory path and logging level are set per project through the UE Editor under **Edit** \> **Project Settings** \> **Plugins** \> **SG Com** and saved in DefaultEngine.ini. ![Screenshot 2023-12-05 152952.png](https://docs.speech-graphics.com/__attachments/a_7eafdb1d07566f7326e2a2ea8604cd0eef616282c0839fa205f7bf0201df75e5/Screenshot%202023-12-05%20152952.png?cb=09061cc7690c0e3162cd7b65bc3348b2) You can easily install the license and binaries by running the **SGComInstaller.exe** utility, located in the root directory of the project. ![image-20240227-132354.png](https://docs.speech-graphics.com/__attachments/a_fe215186557d10660bbb4f7ea093dd2a22c0cf12727fcf401e59f322d8c00ba6/image-20240227-132354.png?cb=86141c053a7e91fbcec776397b2ac14c) ![image-20240827-104532.png](https://docs.speech-graphics.com/__attachments/a_653b924cc414a35c7e1fdc7d0d5883ffba41cbbad78384e8ddc7fe584e901d48/image-20240827-104532.png?cb=9be756d3e7707f16b2b84e96629b8be5) You need to provide the path to the Unreal Engine project folder (the folder containing the .uproject file) and the path to the SGComUE plugin folder (the folder containing the .uplugin file). The license file will be copied into the "SGComExampleUE\\Content\\Resources\\License" directory (after deleting any existing directory) and the project will be configured accordingly. --- version: "5.3" language: "en" --- # SG Com UE Overview The SG Com plugin for the Unreal Engine exposes [SG Com API](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-c-api.md) features to the UE Editor and Blueprints. The plugin comes pre-packaged inside an [Unreal Engine 5 demo application](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-sample-usage.md) to help you get started with integration into your own application. --- version: "5.3" language: "en" --- # SG Com UE Demo Application ## Introduction This article provides an overview of the UE5 demo application with which the SG Com UE Plugin is shipped. The demo makes some simplifying assumptions: * There is a single character driven by a single audio stream. * Animation is local -- i.e., Engine and corresponding Player are on the same machine (see [Networked vs local use cases](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Networked-vs-local-use-cases)). * There are two ways to input audio: from an audio file or from a microphone. For more complex use cases, the developer will need to adapt the demo. This demo application is a guide as to how you might utilize the SG Com UE plugin. Note that this is an example and not every step here has to be followed exactly as described. ## Enabling the plugin Ensure that the plugin has been correctly installed in either the engine or project and that it has been enabled in the Unreal Engine editor. After enabling the plugin you will have to restart the Unreal editor ## Configuring the plugin Navigate to *Project Settings* in the editor and find the *SG Com* section under *Plugins*. Here you will find a few different settings to configure: * Set the *License File Path* to the location of your SG Com license. If this path is incorrect or the license that it points to is invalid, the plugin will not function. * *Log Directory Path* is where SG Com logs will be generated. Set this to any convenient location, for example, in your project under a new directory such as *Saved/SGComLogs.* * *Logging Level* determines the amount of detail provided by SG Com in its logs, with *None* of course being the lowest and *Debug* being the highest. ## File Example For this example we are putting our logic in the level blueprint, which allows us to initialize and perform any setup when the level is started, then shut down when the level is ended. ### Adding and configuring the SG Com Session component * Add to the scene a MetaHuman that we want to animate using SG Com. ![image-20231206-163145.png](https://docs.speech-graphics.com/__attachments/a_4814fa332b68508db5ce2a50b4fe2fd35a121335a9341b688c9f6b27e9b78ab5/image-20231206-163145.png?cb=3478820a2517edaf762db27e8d8b0ece) * Select the actor in the scene outliner and then clicking the *Add* option at the top of the actor's details panel. Select *SGCom Session.* ![Screenshot 2023-12-07 094843-20231207-095352.png](https://docs.speech-graphics.com/__attachments/a_2b078dd4b56c82ffe16a052b47279d6b06aaa1866c4a5bfce168715b95b1c3a1/Screenshot%202023-12-07%20094843-20231207-095352.png?cb=0909f4a2874037d1f348eb738a730d80) * Configure the SG Com Session. Guidance on how to do this can be found in [SG Com UE Configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-configuration.md). ### Getting the SG Com Session component * Obtain and then store a reference to the SG Com Session component. This will be used to call any necessary session functions in the level blueprint. ![Screenshot 2023-12-11 144314 - Copy.png](https://docs.speech-graphics.com/__attachments/a_cf929d8b2aa5a820655c80ac3a70e4c4f4680e780e3966033d52334167fc62ba/Screenshot%202023-12-11%20144314%20-%20Copy.png?cb=4e7b1730ec6137a9536682c7645fc944) ### Initializing SG Com * Initialize SG Com by calling the *Initialize SGCom* function. ![Screenshot 2023-12-11 144314.png](https://docs.speech-graphics.com/__attachments/a_79ab43223e2a06513f71c9a7365a3acebaf706b3f75b32c7e50866984889a4ed/Screenshot%202023-12-11%20144314.png?cb=4603341cb411829a9112ed9de3012086) ### Starting the session * Start the session by calling *Start Session* using the component reference we stored earlier. ![Screenshot 2023-12-11 150616-20231211-150635.png](https://docs.speech-graphics.com/__attachments/a_558a78df73da0df1e0e0434c24ce433c6d9742c63fd477e15465c46bfa9832e7/Screenshot%202023-12-11%20150616-20231211-150635.png?cb=9af117b83bd5b6ee2517b074bdb5af8e) ### Auto mode detection * Map the *Positive* auto mode to the "positive" behavior mode by calling *Set Auto Mode* , selecting the Auto Mode from the dropdown list and then typing the desired behavior mode name in the *Mode Name* text box. ![Screenshot 2023-12-11 155206-20231211-155240.png](https://docs.speech-graphics.com/__attachments/a_078956608cbcafefd5166e479156a8c321e49ab45d977ef485d0c4cd46d8c9f8/Screenshot%202023-12-11%20155206-20231211-155240.png?cb=75b4d0508acb57f27a0d1fd0462abff8) * Repeat this process for the *Negative* , *Effort* and *Acknowledge* auto modes, Mapping them to the "negative"*, "*effort" and "acknowledge" behavior modes, respectively. ![Screenshot 2023-12-11 164623-20231211-164715.png](https://docs.speech-graphics.com/__attachments/a_0aa05f25993476503d1cb6c905fd08a811fb484054f34294fe91556866e2233f/Screenshot%202023-12-11%20164623-20231211-164715.png?cb=f90cde55c5cff5bf939b3b8a218d965b) * Activate the auto modes by calling *Activate Auto Modes* on the SG Com Session component. ![Screenshot 2023-12-11 170011-20231211-170036.png](https://docs.speech-graphics.com/__attachments/a_98da9d57d0c9fb3996f4b068f924f028ed76aa290d558e03136b41d4a42acb9f/Screenshot%202023-12-11%20170011-20231211-170036.png?cb=45b562ad8d5ec2be6efa0c76bdc949da) ### Audio input * Use the *Input* *SoundWave*node to feed a SoundWave to SG Com ![image-20241007-162343.png](https://docs.speech-graphics.com/__attachments/a_c8bfcc0df9ec07646ea05183af9614e1ce1c9d03c3e77ec29c186689502f9118/image-20241007-162343.png?cb=c1bae0889f380a326f566df87ce47594) ### File playback The playback state of the SoundWave drives the animation update. ![image-20241007-163101.png](https://docs.speech-graphics.com/__attachments/a_2cdebeb03a2ef2884f6b3b75daffecd1a19f8b0cb45512d47864dc4fe3048324/image-20241007-163101.png?cb=6f2e9493bf0181e0d79907b80e2572e5) ### Idle We want the character to continue animating after the file has been played in its entirety, or if we decided to pause the audio file partway through. In order to do this we'll need to feed empty audio data to SG Com whenever there is no file currently playing. (See [Engine idle](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Engine-idle)). The *Input Silence*node proveides a convenient way of doing this. ![image-20241007-162813.png](https://docs.speech-graphics.com/__attachments/a_34524714a8b60810146e740f9af59c96e69d7bfdd7377a38cddbb3dba652763a/image-20241007-162813.png?cb=58da350b7bbb6a1f47e3d2c6c25c0847) ### Setting up callbacks * Create custom events for both *On Mode Changed* and *On Expression Changed* callbacks. These custom events should take a string as input, so make sure to add this. ![Screenshot 2023-12-12 150539-20231212-150621.png](https://docs.speech-graphics.com/__attachments/a_5622ec7ea89aefec72b09e521f7dc878a8498dc2f45890dc603f015149ff72cc/Screenshot%202023-12-12%20150539-20231212-150621.png?cb=ce68142c44b0d8759194f7191be4a640) ![Screenshot 2023-12-12 150958-20231212-151044.png](https://docs.speech-graphics.com/__attachments/a_88c0d3be599918433733025ec3dbfa31f3886833a82c8018785ae2c9c121984c/Screenshot%202023-12-12%20150958-20231212-151044.png?cb=1370068983a2e7c0565b0c53ee05c9e5) * Bind these events. ![Screenshot 2023-12-12 151202-20231212-151249.png](https://docs.speech-graphics.com/__attachments/a_6dfc1c693d3698546c0604ee73f8e6a8ea6bca8c8b97a3b4c073fb45da8aa70b/Screenshot%202023-12-12%20151202-20231212-151249.png?cb=87ef806267974cd8324c922fad1e8e5c) * Set the events to print their output, for debugging purposes. ![Screenshot 2023-12-12 151625-20231212-151658.png](https://docs.speech-graphics.com/__attachments/a_3d0d65bdfc7a5cb95e3dc524ec358843a45297271ce2651ecee6f0fd46c64dbe/Screenshot%202023-12-12%20151625-20231212-151658.png?cb=0347a0926974a85e71baed3983244427) ### Updating animation Here we make use of the custom event *Playback Time Changed* which we created earlier when setting up the file playback. * When the event is triggered, get the new playback time. * Calculate the delta since the last playback time change. * Use this delta to update the animation in SG Com by the necessary amount. ![Screenshot 2023-12-12 163651-20231212-163721.png](https://docs.speech-graphics.com/__attachments/a_5eb0c8d5c1db8898a208e68d990d006ad429c1e51a93fe16ba53c45a9be667c9/Screenshot%202023-12-12%20163651-20231212-163721.png?cb=6be90d972324a311fc257680ededcbb6) ### Ending the session and shutting down SG Com * End the active session by calling *End Session.* * Call *Shutdown SGCom.* ![Screenshot 2023-12-11 152949-20231211-153012.png](https://docs.speech-graphics.com/__attachments/a_5f16fdfe5198ae05192f66b01db6ec5e4840ed03bad190b59daae568ad266d36/Screenshot%202023-12-11%20152949-20231211-153012.png?cb=a4f2b4d9673bac7a9a9bfc5c7ae74e81) ## Microphone Example For this example we are putting our logic in the level blueprint, which allows us to initialize and perform any setup when the level is started, then shut down when the level is ended. ### Adding and configuring the SG Com Session component * Add to the scene a MetaHuman that we want to animate using SG Com. ![image-20231206-163145.png](https://docs.speech-graphics.com/__attachments/a_4814fa332b68508db5ce2a50b4fe2fd35a121335a9341b688c9f6b27e9b78ab5/image-20231206-163145.png?cb=3478820a2517edaf762db27e8d8b0ece) * Select the actor in the scene outliner and then click the *Add* option at the top of the actor's details panel. Select *SGCom Session.* ![Screenshot 2023-12-07 094843-20231207-095352.png](https://docs.speech-graphics.com/__attachments/a_2b078dd4b56c82ffe16a052b47279d6b06aaa1866c4a5bfce168715b95b1c3a1/Screenshot%202023-12-07%20094843-20231207-095352.png?cb=0909f4a2874037d1f348eb738a730d80) * Configure the SG Com Session. Guidance on how to do this can be found in [SG Com UE Configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-configuration.md). * Add an *SGMicrophoneAudio* Actor to your scene. ![image-20250416-085957.png](https://docs.speech-graphics.com/__attachments/a_bfabc11326e85ad1957f026332b3b10f3f9adb8f67a32a33e3835ecfde988d89/image-20250416-085957.png?cb=f190609ff47917d2113e0711d1d27cf0) ### Getting the SG Com Session component * Obtain and then store a reference to the SG Com Session component. This will be used to call any necessary session functions in the level blueprint ![Screenshot 2023-12-11 144314 - Copy.png](https://docs.speech-graphics.com/__attachments/a_cf929d8b2aa5a820655c80ac3a70e4c4f4680e780e3966033d52334167fc62ba/Screenshot%202023-12-11%20144314%20-%20Copy.png?cb=4e7b1730ec6137a9536682c7645fc944) ### Initializing SG Com * Initialize SG Com by calling the *Initialize SGCom* function. ![Screenshot 2023-12-11 144314.png](https://docs.speech-graphics.com/__attachments/a_79ab43223e2a06513f71c9a7365a3acebaf706b3f75b32c7e50866984889a4ed/Screenshot%202023-12-11%20144314.png?cb=4603341cb411829a9112ed9de3012086) ### Starting the session * Start the session by calling *Start Session* using the component reference we stored earlier. ![Screenshot 2023-12-11 150616-20231211-150635.png](https://docs.speech-graphics.com/__attachments/a_558a78df73da0df1e0e0434c24ce433c6d9742c63fd477e15465c46bfa9832e7/Screenshot%202023-12-11%20150616-20231211-150635.png?cb=9af117b83bd5b6ee2517b074bdb5af8e) ### Auto mode detection * Map the *Positive* auto mode to the "positive" behavior mode by calling *Set Auto Mode* , selecting the Auto Mode from the dropdown list and then typing the desired behavior mode name in the *Mode Name* text box. ![Screenshot 2023-12-11 155206-20231211-155240.png](https://docs.speech-graphics.com/__attachments/a_078956608cbcafefd5166e479156a8c321e49ab45d977ef485d0c4cd46d8c9f8/Screenshot%202023-12-11%20155206-20231211-155240.png?cb=75b4d0508acb57f27a0d1fd0462abff8) * Repeat this process for the *Negative* , *Effort* and *Acknowledge* auto modes, Mapping them to the "negative"*, "*effort" and "acknowledge" behavior modes, respectively. ![Screenshot 2023-12-11 164623-20231211-164715.png](https://docs.speech-graphics.com/__attachments/a_0aa05f25993476503d1cb6c905fd08a811fb484054f34294fe91556866e2233f/Screenshot%202023-12-11%20164623-20231211-164715.png?cb=f90cde55c5cff5bf939b3b8a218d965b) * Activate the auto modes by calling *Activate Auto Modes* on the SG Com Session component. ![Screenshot 2023-12-11 170011-20231211-170036.png](https://docs.speech-graphics.com/__attachments/a_98da9d57d0c9fb3996f4b068f924f028ed76aa290d558e03136b41d4a42acb9f/Screenshot%202023-12-11%20170011-20231211-170036.png?cb=45b562ad8d5ec2be6efa0c76bdc949da) ### Audio input and playback * Call *Start Microphone* on the *SGMicrophoneAudio* Actor and pass through the SG Com Session component reference. Audio data will now continuously be fed to SG Com from the microphone for processing. * This will also handle the audio playback (by configuring and spawning and Audio Component) and animation update. * If you want direct control over audio playback by geting the audio component (*Get Audio Component* ) from the Actor. ![image-20250416-085714.png](https://docs.speech-graphics.com/__attachments/a_c6367b79357a33f57d3aa048b1adc6d7afc84d007767dca289fd5576af151e51/image-20250416-085714.png?cb=f4b25870df0daf25ba6d8ca176ee2802) ### Setting up callbacks * Create custom events for both *On Mode Changed* and *On Expression Changed* callbacks. These custom events should take a string as input, so make sure to add this. ![Screenshot 2023-12-12 150539-20231212-150621.png](https://docs.speech-graphics.com/__attachments/a_5622ec7ea89aefec72b09e521f7dc878a8498dc2f45890dc603f015149ff72cc/Screenshot%202023-12-12%20150539-20231212-150621.png?cb=ce68142c44b0d8759194f7191be4a640) ![Screenshot 2023-12-12 150958-20231212-151044.png](https://docs.speech-graphics.com/__attachments/a_88c0d3be599918433733025ec3dbfa31f3886833a82c8018785ae2c9c121984c/Screenshot%202023-12-12%20150958-20231212-151044.png?cb=1370068983a2e7c0565b0c53ee05c9e5) * Bind these events. ![Screenshot 2023-12-12 151202-20231212-151249.png](https://docs.speech-graphics.com/__attachments/a_6dfc1c693d3698546c0604ee73f8e6a8ea6bca8c8b97a3b4c073fb45da8aa70b/Screenshot%202023-12-12%20151202-20231212-151249.png?cb=87ef806267974cd8324c922fad1e8e5c) * Set the events to print their output, for debugging purposes. ![Screenshot 2023-12-12 151625-20231212-151658.png](https://docs.speech-graphics.com/__attachments/a_3d0d65bdfc7a5cb95e3dc524ec358843a45297271ce2651ecee6f0fd46c64dbe/Screenshot%202023-12-12%20151625-20231212-151658.png?cb=0347a0926974a85e71baed3983244427) ### Ending the session and shutting down SG Com * Stop the microphone. * End the active session by calling *End Session.* * Call *Shutdown SGCom.* ![image-20250416-101729.png](https://docs.speech-graphics.com/__attachments/a_6c8dc54c515fa847dd2f72eced235259f64caea1fa14d48baee3c14d14a84a34/image-20250416-101729.png?cb=5f64a55f6af6f5ac289430f33dbbfaef) ## Animation Blueprint To apply the SG Com animation data to the MetaHuman actor, we need to use the *SG Com Pose* animation node. For this example, we will keep it simple and just make a new, very basic animation blueprint. * Create the animation blueprint asset targeting the MetaHuman's body skeletal mesh (the MetaHuman setup includes both face and body targets). * Add an *SG Com Pose* node. * Feed the output pose from the SG Com node to the final output pose of the blueprint. ![Screenshot 2023-12-13 113427-20231213-113450.png](https://docs.speech-graphics.com/__attachments/a_e79543809d37c8413f6430865cc7196ce52c454ff4decf83f83a27b8017a8a12/Screenshot%202023-12-13%20113427-20231213-113450.png?cb=0ce614b6d9a8bf9c626c3ef15477aee5) * Set the body mesh component of the MetaHuman actor to use this new animation blueprint. ![Screenshot 2023-12-13 114042-20231213-114154.png](https://docs.speech-graphics.com/__attachments/a_1724869531e52aed167ef4cb3e94b5e99888e5fa82ddf7474364047e43500d84/Screenshot%202023-12-13%20114042-20231213-114154.png?cb=762229da9d0359879a160689702d62fa) --- version: "5.3" language: "en" --- # SG Com Unity Package * [SG Com Unity Package Overview](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-unity-package-overview.md) * [SG Com Unity Package Download and Installation](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-unity-package-download-and-installation.md) * [SG Com Unity Package Scripts](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-unity-package-scripts.md) --- version: "5.3" language: "en" --- # SG Com Unity Package Download and Installation ## Download The following are available for [download](https://downloads.speech-graphics.cloud/): * The SG Com Unity Example, which contains the SG Com Unity Package. Start with this to learn how to integrate the package into your own application. * The SG Com Unity Package by itself. ## Package installation In order to install the SG Com Unity Package into your own project, place it in the **Packages** folder of your project. The Package Manager will automatically interpret it as an **Embedded package**. The package includes a sample scene which you can import to your project. ![image-20240502-134017.png](https://docs.speech-graphics.com/__attachments/a_6e31eba4c5bd0ef48f8ec7b2bc4640d32283b4a8a08e9192f0a98227b4901058/image-20240502-134017.png?cb=5a6d19f0d6df96dee0f11436f5c72f0a) ## Library and license installation Whether using your own project or the example project, you must install the SG Com binaries and license file, as follows: * Place your SG Com binaries in **Packages/SGCom/Runtime/Plugins**. * Place your license file in **Assets/StreamingAssets/SpeechGraphics/License**. You can also install the license and binaries by running the **SGComInstaller.exe** utility, located in the root directory of the project. ![image-20241118-232359.png](https://docs.speech-graphics.com/__attachments/a_944cd214082968244bf0b89f443c9f594089f018127e6b588dd1f9444044e2b1/image-20241118-232359.png?cb=eb5d487a7c7f0e2a82e4ac2ad886ab79) --- version: "5.3" language: "en" --- # SG Com Unity Package Overview The SG Com Package for Unity provides [SG Com API](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-c-api.md) C# bindings and scripts to facilitate its integration to your project. The package comes with a sample project to help you get started with integration into your own application. --- version: "5.3" language: "en" --- # SG Com Unity Package Scripts ## SG Com Global Responsible for initialising and cleaning up shared SG Com resources. Ensure that there is only one instance of this component per scene. ### Inspector Properties * `License File Name`: The name of the license file (including the file extension) in "StreamingAssets/SpeechGraphics/License". * `Logging Level`: Sets the verbosity of the SG Com log. ![image-20240502-155708.png](https://docs.speech-graphics.com/__attachments/a_c7d3b8ae3bfbf140cbc9dee15113aceab9235986aec108d8306547cca449c6fa/image-20240502-155708.png?cb=3f856437049bad38a98bb6d839753215) ## SG Com Session Manages a single session of SG Com. Add the script to the GameObject you wish to animate. ### Inspector Properties * `Character Control File Name`: Name of control file (including the file extension) in "StreamingAssets/SpeechGraphics". * `Audio Sample Rate`: The sample rate of the audio. * `Audio Sample Type`: The sample type of the audio. * `Audio Buffer Seconds`: The size of the audio buffer in seconds. * `Animation Buffer Seconds`: The size of the animation buffer in seconds. * `Enable Idle`: A Boolean flag to enable or disable idle animation. * `Use Fixed Seed`: A Boolean flag to use a fixed seed for the random number generator. ![image-20240502-155611.png](https://docs.speech-graphics.com/__attachments/a_9d859f85377af9eb5a50de2383bcd5b076453e83bfd0da5591dd82629c83e305/image-20240502-155611.png?cb=98a0a6f16a2c108953b780b1a8422824) ## SG Animation Instance Handles the playback of animation. Add the script to the GameObject you wish to animate. Requires the **SG Com Session** script. ### Inspector Properties * `Avatar Root`: A GameObject that acts as the root of the avatar. * `Rotation Factor`: Converts between Maya and Unity coordinate systems for rotation. * `Translation Factor`: Converts between Maya and Unity coordinate systems for translation. * `Blendshape Factor`: Adjusts SG Com blendshape values (0-1) to Unity blendshape values (0-100). ![image-20240502-155547.png](https://docs.speech-graphics.com/__attachments/a_754aa95128c0496bad56d9befa05efe10cd145d184ce77ed4e997cdd934ab81c/image-20240502-155547.png?cb=27acb8b33e2027cb488fd27cba6924a7) ## SG Audio Input Handles microphone or file input for real-time facial animation. Add the script to the GameObject you wish to animate. Requires the **SG Com Session** and **Audio Source**scripts to be added to the same GameObject. If no audio file is attached to the **Audio Source**script it will use the microphone input. If an audio file is attached, it will stream the audio file. --- version: "5.3" language: "en" --- # SG Com Unreal Engine Plugin * [SG Com UE Overview](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-overview.md) * [SG Com UE Download and Installation](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-installation.md) * [SG Com UE Configuration](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-configuration.md) * [SG Com UE Functions](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md) * [SG Com UE Demo Application](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-ue-sample-usage.md) --- version: "5.3" language: "en" --- # Simple Code Example [SG_Com_simple_example.cpp](https://docs.speech-graphics.com/__attachments/a_0e3e2ef4924f20bc267445031ef9d2fb13951ff36aefe2d09f7b4caae6907ca3/SG_Com_simple_example.cpp.md?cb=2b50b9494076e93be0f399abefef2da0) C++ // // SG_Com_simple_example.cpp // 2019/07/05 // Simple example use of SG Com in C++ (mostly C). // // We construct an Engine and a Player to illustrate generation // and playback of animation. There is an option to make the Player "local" // (connected directly to the Engine) or "remote" (with a callback function to "send" // packets to the Player). This example does not illustrate audio sync, idle mode, // or control features such as mode switching. // // Copyright (c) 2019 Speech Graphics Ltd. All rights reserved. // #ifndef _CRT_SECURE_NO_WARNINGS #define _CRT_SECURE_NO_WARNINGS #endif #include "../src/SG_Com.h" #include #include #include #include #include // memcpy #include const char* help_text = "Usage: SG_Com_Simple_Example.exe license_string character_file audio_file audio_sample_rate audio_sample_type output_file\n\n" "license_string : License string (directory path for RLM builds|license string for other builds)\n" "character_file : Path to character .k file.\n" "audio_file : Path to input audio file containing speech. Must be a single channel wav file.\n" "output_file : Path to the file to save the output animation in.\n"; // Example mode. Set to true to treat the Player as a "remote" player. Otherwise it will be treated as local. static bool remote_player = true; // Global objects static SG_COM_EngineHandle engine_handle; static SG_COM_PlayerHandle player_handle; // Input settings const static float input_buffer_sec = 1.0; // maximum duration (seconds) of the input buffer const static int input_chunk_ms = 10; // duration of audio to be input on each process tick // Output settings const static float output_buffer_sec = 1.0; // maximum duration (seconds) of the output buffer const static int output_fps = 30; struct WAVFile { char* audio_data; sg_size data_size; SG_AudioSampleRate sample_rate; SG_AudioSampleType sample_type; }; enum WAV_Format { WAV_PCM_FORMAT = 0x01, WAV_IEEE_FORMAT = 0x03 }; // Convert an int to SG_AudioSampleRate SG_AudioSampleRate int_to_AudioSampleRate(unsigned int sample_frequency) { switch (sample_frequency) { case 8000: return SG_AUDIO_8_KHZ; case 12000: return SG_AUDIO_12_KHZ; case 16000: return SG_AUDIO_16_KHZ; case 24000: return SG_AUDIO_24_KHZ; case 32000: return SG_AUDIO_32_KHZ; case 44100: return SG_AUDIO_44_1_KHZ; case 48000: return SG_AUDIO_48_KHZ; default: fprintf(stderr, "Audio file has an unsupported sample rate."); exit(1); } } // Get SG_AudioSampleType based on the bit depth and wav sample format SG_AudioSampleType get_AudioSampleType(unsigned int bit_depth, unsigned short wav_format) { if (bit_depth == 16 && wav_format == WAV_PCM_FORMAT) { return SG_AUDIO_INT_16; } else if (bit_depth == 32 && wav_format == WAV_PCM_FORMAT) { return SG_AUDIO_INT_32; } else if (bit_depth == 32 && wav_format == WAV_IEEE_FORMAT) { return SG_AUDIO_FLOAT_32; } else { fprintf(stderr, "Audio file has an unsupported bit depth and sample format combination."); exit(1); } } // Load a file into a byte buffer void load_file(const char *file_in, char **bytes_out, sg_size *buffersize) { FILE *file = fopen(file_in, "rb"); fseek(file, 0, SEEK_END); *buffersize = ftell(file); rewind(file); *bytes_out = (char*) malloc(*buffersize); fread(*bytes_out, 1, *buffersize, file); fclose(file); } // Load a WAV file detecting the sample rate and sample type WAVFile load_wav_file(const char* file_in) { // Byte offsets into the WAV file header const int format_offset = 0x14; const int channel_count_offset = 0x16; const int sample_rate_offset = 0x18; const int bit_depth_offset = 0x22; const int data_offset = 0x2C; char* file = NULL; sg_size buffersize = 0; load_file(file_in, &file, &buffersize); WAVFile audiofile; // Check the wav file is monophonic unsigned short channel_count = *((unsigned short*) &file[channel_count_offset]); if (channel_count != 1) { fprintf(stderr, "Only single channel audio files are supported."); exit(1); } // Get the sample rate audiofile.sample_rate = int_to_AudioSampleRate(*((unsigned int*) &file[sample_rate_offset])); // Get the sample type WAV_Format wav_format = (WAV_Format) *((unsigned short*) &file[format_offset]); if (wav_format != WAV_PCM_FORMAT && wav_format != WAV_IEEE_FORMAT) { fprintf(stderr, "Audio file format must be signed PCM or float."); exit(1); } unsigned short bit_depth = *((unsigned short*) &file[bit_depth_offset]); if (bit_depth != 16 && bit_depth != 32) { fprintf(stderr, "Audio file must have a bit depth of 16 or 32."); exit(1); } audiofile.sample_type = get_AudioSampleType(bit_depth, wav_format); // Copy the audio data audiofile.data_size = buffersize - data_offset; audiofile.audio_data = (char*) malloc(audiofile.data_size); memcpy(audiofile.audio_data, file, audiofile.data_size); free(file); return audiofile; } // Error handler void check_error(SG_COM_Error error) { if (error != SG_COM_ERROR_OK) { fprintf(stderr, "Exiting with error code %d", error); std::exit(error); } } // Simulated network send method void __send__(char *packet, sg_size packet_size) { // Receive a data packet into the remote Player SG_COM_Error err = SG_COM_ReceivePacket(player_handle, packet, packet_size); check_error(err); } // Engine broadcast callback void engine_broadcast_callback( SG_COM_EngineHandle engine, char *packet, sg_size packet_size, void *custom_data) { __send__(packet, packet_size); } // Engine status callback void engine_status_callback( SG_COM_EngineHandle engine, SG_COM_Status status, const char *message, void *custom_data) { if (status == SG_COM_STATUS_MODE_CHANGED) { fprintf(stderr, "Mode changed: %s\n", message); } if (status == SG_COM_STATUS_EXPRESSION_CHANGED) { fprintf(stderr, "Expression changed: %s\n", message); } if (status == SG_COM_STATUS_VOICE_ACTIVITY_CHANGED) { fprintf(stderr, "Voice activity changed: %s\n", message); } } // Logging callback void sg_com_log_messages(const char *message) { printf(message); } // Get samples per second values size_t get_audio_samples_per_second(const SG_AudioSampleRate sample_rate) { switch (sample_rate) { case SG_AUDIO_8_KHZ: return 8000; case SG_AUDIO_12_KHZ: return 12000; case SG_AUDIO_16_KHZ: return 16000; case SG_AUDIO_24_KHZ: return 24000; case SG_AUDIO_32_KHZ: return 32000; case SG_AUDIO_44_1_KHZ: return 44100; case SG_AUDIO_48_KHZ: return 48000; default: return 16000; } } // Get sample type sizes size_t get_audio_bytes_per_sample(const SG_AudioSampleType sample_type) { switch (sample_type) { case SG_AUDIO_INT_16: return 2; case SG_AUDIO_INT_32: case SG_AUDIO_FLOAT_32: return 4; default: return 0; } } // Write the saved animation output to a file in Speech Graphics .rts file format void write_output_file( const char *output_file_path, const sg_size num_animation_nodes, SG_AnimationNode *animation_nodes, float *results_ptr, float *results, int num_output_frames, const sg_size num_aggregated_output_channels) { FILE *file = fopen(output_file_path, "w"); // Write frame rate line fprintf(file, "%d\n", output_fps); // Write channel names for (size_t i = 0; i < num_animation_nodes; i++) { for (size_t j = 0; j < animation_nodes[i].num_channels; j++) { fprintf(file, "%s.%s", animation_nodes[i].name, animation_nodes[i].channel_names[j]); if (i < num_animation_nodes - 1 || j < animation_nodes[i].num_channels - 1) { fprintf(file, ","); } else { fprintf(file, "\n"); } } } // Write frame data results_ptr = results; for (size_t i = 0; i < num_output_frames; i++) { for (size_t j = 0; j < num_aggregated_output_channels; j++, results_ptr++) { fprintf(file, "%f", *results_ptr); if (j < num_aggregated_output_channels - 1) { fprintf(file, ","); } else { fprintf(file, "\n"); } } } fclose(file); } void create_player(const char *character_file_in_memory, sg_size character_file_bytes) { SG_COM_Error err; SG_COM_PlayerConfig player_config; player_config.character_file_bytes = character_file_bytes; player_config.character_file_in_memory = (sg_byte*)character_file_in_memory;; player_config.buffer_sec = output_buffer_sec; err = SG_COM_CreatePlayer(&player_config, &player_handle); check_error(err); } void create_engine( const char *character_file_in_memory, sg_size character_file_bytes, SG_AudioSampleRate audio_sample_rate, SG_AudioSampleType audio_sample_type) { SG_COM_Error err; SG_COM_EngineConfig engine_config; engine_config.audio_sample_type = audio_sample_type; engine_config.audio_sample_rate = audio_sample_rate; engine_config.buffer_sec = input_buffer_sec; engine_config.character_file_bytes = character_file_bytes; engine_config.character_file_in_memory = (sg_byte*)character_file_in_memory; engine_config.local_player = remote_player ? nullptr : player_handle; // SG_Com will automatically handle passing packets to the local player engine_config.engine_broadcast_callback = remote_player ? engine_broadcast_callback : nullptr; engine_config.engine_status_callback = engine_status_callback; engine_config.flag = SG_COM_ENGINE_CONFIG_NONE; engine_config.custom_engine_data = nullptr; err = SG_COM_CreateEngine(&engine_config, &engine_handle); check_error(err); SG_COM_SetAutoMode(engine_handle, SG_COM_POSITIVE_MODE, "positive"); SG_COM_SetAutoMode(engine_handle, SG_COM_NEGATIVE_MODE, "negative"); SG_COM_SetAutoMode(engine_handle, SG_COM_EFFORT_MODE, "effort"); SG_COM_SetAutoMode(engine_handle, SG_COM_ACKNOWLEDGE_MODE, "acknowledge"); SG_COM_ActivateAutoModes(engine_handle); } // Input audio into the Engine void audio_input( SG_AudioSampleRate audio_sample_rate, SG_AudioSampleType audio_sample_type, const sg_size *audio_file_bytes, const char *audio_file_in_memory) { SG_COM_Error err; // Feed audio into the Engine in chunks of input_chunk_ms length, simulating real-time input from a microphone. int samples_per_ms = (int)get_audio_samples_per_second(audio_sample_rate) / 1000; int input_chunk_samples = input_chunk_ms * samples_per_ms; int input_chunk_bytes = input_chunk_samples * (int)get_audio_bytes_per_sample(audio_sample_type); int num_input_chunks = *audio_file_bytes / input_chunk_bytes; uint8_t *input_chunk = (uint8_t*)malloc(input_chunk_bytes); for (size_t i = 0; i < num_input_chunks; i++) { auto start_time = std::chrono::high_resolution_clock::now(); memcpy(input_chunk, audio_file_in_memory + (i * input_chunk_bytes), input_chunk_bytes); err = SG_COM_InputAudio(engine_handle, input_chunk, input_chunk_bytes); check_error(err); // ProcessTick is called from here for simplicity // It is recommended this call be offloaded to a background thread, which is then notified using a mechanism like std::condition_variable. // This will prevent the processing from blocking the real-time audio thread. err = SG_COM_ProcessTick(engine_handle, nullptr, nullptr); check_error(err); // Sleep thread until the duration of the loop equals input_chunk_ms to simulate real-time audio input auto duration = std::chrono::duration_cast(std::chrono::high_resolution_clock::now() - start_time); std::this_thread::sleep_for(std::chrono::milliseconds(input_chunk_ms - duration.count())); } free(input_chunk); } // Playback process void play_output(const char *output_file_path, float output_sec) { // Get the animation nodes. sg_size num_animation_nodes = 0; sg_size num_aggregated_output_channels = 0; SG_AnimationNode *animation_nodes = nullptr; SG_COM_Error err = SG_COM_GetAnimationNodes(player_handle, &animation_nodes, &num_animation_nodes); // Count the aggregated output channels for (sg_size i = 0; i < num_animation_nodes; i++) { num_aggregated_output_channels += animation_nodes[i].num_channels; } // Prepare output storage. (Normally you would play the output in real time. // But in this example we will save it to a file.) int num_output_frames = (int)roundf(output_sec * output_fps); float *results = (float*)malloc(sizeof(float) * num_aggregated_output_channels * num_output_frames); float *results_ptr = results; //Wait until some animation is in the buffer double min_time, max_time, current_time; bool bufferred = false; while (!bufferred) { SG_COM_GetPlayableRange(player_handle, &min_time, &max_time); bufferred = (max_time - min_time > 0); std::this_thread::sleep_for(std::chrono::milliseconds(10)); } // Enter the real-time playback loop auto delta_time = std::chrono::microseconds(1000000 / output_fps); auto start_time = std::chrono::high_resolution_clock::now(); auto frame_time = start_time; for (size_t i = 0; i < num_output_frames; i++) { std::chrono::duration time_elapsed_ms = frame_time - start_time; SG_COM_Error err = SG_COM_UpdateAnimation(player_handle, time_elapsed_ms.count(), ¤t_time); if (err != SG_COM_ERROR_OK) { break; } for (size_t j = 0; j < num_animation_nodes; j++) { // Copy the animation values out of the node // This is where animation would be applied to the character memcpy(results_ptr, animation_nodes[j].channel_values, animation_nodes[j].num_channels * sizeof(float)); results_ptr += animation_nodes[j].num_channels; } // Wait for delta to elapse std::this_thread::sleep_for(delta_time); frame_time = std::chrono::high_resolution_clock::now(); } write_output_file( output_file_path, num_animation_nodes, animation_nodes, results_ptr, results, num_output_frames, num_aggregated_output_channels); // Clean up free(results); } // Run the example void run( const char *license_string, const char *character_file_path, const char *audio_file_path, const char *output_file_path) { SG_COM_Error err = SG_COM_Initialize(SG_LOGLEVEL_ERROR, sg_com_log_messages, license_string, nullptr, nullptr); check_error(err); printf("SG_Com Version: %s\n", SG_COM_GetVersionString()); // Load files into memory char *character_file_in_memory; sg_size character_file_bytes; load_file(character_file_path, &character_file_in_memory, &character_file_bytes); WAVFile audio_file = load_wav_file(audio_file_path); create_player(character_file_in_memory, character_file_bytes); create_engine( character_file_in_memory, character_file_bytes, audio_file.sample_rate, audio_file.sample_type); // Start the audio thread std::thread input_thread(audio_input, audio_file.sample_rate, audio_file.sample_type, &audio_file.data_size, audio_file.audio_data); // "Play" the animation float audio_sec = (float)audio_file.data_size / (get_audio_samples_per_second(audio_file.sample_rate) * get_audio_bytes_per_sample(audio_file.sample_type)); play_output(output_file_path, audio_sec - 0.1f); // Clean up input_thread.join(); free(character_file_in_memory); free(audio_file.audio_data); // Make sure asynchronous audio input has been shut down before destroying the Engine. // In this example audio delivery will complete by itself when it reaches the end of the audio file. SG_COM_DestroyEngine(engine_handle); SG_COM_DestroyPlayer(player_handle); SG_COM_Shutdown(); } // Main function int main( int argc, char *argv []) { if (argc != 5) { printf("Incorrect number of arguments\n%s", help_text); exit(0); } run(argv[1], argv[2], argv[3], argv[4]); exit(0); } --- version: "5.3" language: "en" --- # SG Com UE Functions ## SGComBlueprintFunctionLibrary These functions can be called from anywhere, but the Initialize and Shutdown functions should generally be used in a game mode or level blueprint, for example. This is in order to ensure that they can be called at the start and end of the game, before any [SGComSessionComponent](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md#SGComSessionComponent) calls are made, and after all necessary [SGComSessionComponent](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md#SGComSessionComponent) calls have been completed. **Initialize SGCom** : Initializes the [SG Com API](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-c-api.md). This should be called once, at the start of the game, before using any other SG Com functions. ![Screenshot 2023-12-06 120735.png](https://docs.speech-graphics.com/__attachments/a_e7fee4fa4d016b6e8a068291a082f84aae58b83d353afcd783b142f0a325e2e1/Screenshot%202023-12-06%20120735.png?cb=f6c07b5b9c375f7a75d8e23cc62c366c) **Shutdown SGCom** : Shuts down the SG Com API. This should be called once, at the end of the game, after all SG Com Sessions have been ended through **EndSession()**. ![Screenshot 2023-12-06 120911.png](https://docs.speech-graphics.com/__attachments/a_09c144198f93886881ba83ffa83421522739035265e224b0d6832b237e26deb1/Screenshot%202023-12-06%20120911.png?cb=ad12f440d30c2744df9d3e07dcbee34a) **Decompress SoundWave (Editor Only)** : Helper function. Decompresses SoundWave audio to Raw PCM data that can be passed to SG Com. *Loading Behavior Override* of the SoundWave needs to be set to *Force Inline* or the function will fail in packaged builds due to Audio Streaming Caching ![Screenshot 2023-12-06 121032.png](https://docs.speech-graphics.com/__attachments/a_b73af4a835ce91b94cb41ae9df4d5be6720eac20414d5a1cdb36281443a2523e/Screenshot%202023-12-06%20121032.png?cb=3ab1e5bc7311ab0cb69b3d093cd27b5b) **Input SoundWave** : Helper function. Suitable for packaged builds. Inputs a SoundWave to the [SGComSessionComponent](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md#SGComSessionComponent) for processing and attaches FSourceEffectSGCom to the audio source. The SoundWave will be decompressed and the raw PCM data will be sent to the [SGComSessionComponent](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md#SGComSessionComponent). Calls **InputAudio()** on the [SGComSessionComponent](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md#SGComSessionComponent) internally. **IMPORTANT:**When using SourceComponents Unreal Engine provides raw audio in 32 bit float and a sample rate that depends on the current platform (in practice, it defaults to 48000Hz). You need to set your SG Com configuration accordingly, regardless of the audio file you are using. ![image-20240322-150349.png](https://docs.speech-graphics.com/__attachments/a_d13fddc009cbc75b0e8e706e55fa7a192ad24ff567039dfe9ca3819723ed4446/image-20240322-150349.png?cb=bfe85ef65510110bd14ba79881d4cf8a) ## SGComSessionComponent A "session" comprises a combination of SG Com Engine and Player for a single character (see [Engines and Players](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md)). This component covers the majority of SG Com functionality, with the rest being part of the [SGComBlueprintFunctionLibrary](https://docs.speech-graphics.com/en/sg-com/5.3/using-sg-com-ue.md#SGComBlueprintFunctionLibrary). A session component should be attached to any character in the scene that you intend to animate locally using SG Com. ### Session management Functions for managing individual SG Com sessions. **Start Session**: Starts the SG Com session based on the provided configuration. ![Screenshot 2023-12-06 121210.png](https://docs.speech-graphics.com/__attachments/a_9c2b35a2282bb3c482dc0a01eedf6c75bb3731b6c497178f6869f48736531ac5/Screenshot%202023-12-06%20121210.png?cb=0052ac84729e5954ab47d4c20e12f9f7) **End Session**: Ends the SG Com session. ![Screenshot 2023-12-06 121343.png](https://docs.speech-graphics.com/__attachments/a_03ce51392bb4a802468153fad8b80fd73f00f6228e5fa3e32b0eed4dab6731fb/Screenshot%202023-12-06%20121343.png?cb=97da1de301ff033495137e340958ee56) **Start Player-Only Session**: Starts an SG Com session where the engine is remote. ![Screenshot 2023-12-06 121548.png](https://docs.speech-graphics.com/__attachments/a_9f7dd670e9bfe6b27fcc073ed03e50ac83c1ef8a85ca5676e99f996fba0e0e15/Screenshot%202023-12-06%20121548.png?cb=699f069baebc20b65d01dd9c65918688) **End Player-Only Session**: Ends the player-only SG Com session. ![Screenshot 2023-12-06 121707.png](https://docs.speech-graphics.com/__attachments/a_23b87f99d79744500cfd330761256d424355b91a967fbffed35e7baa70fa4aec/Screenshot%202023-12-06%20121707.png?cb=ec96710d5c26870dc87e7b85f281984a) ### Audio input Functions for managing [SG Com Input](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-input.md). **Input Audio**: Inputs an array of raw PCM audio data to the SGCom engine. ![Screenshot 2023-12-06 121845.png](https://docs.speech-graphics.com/__attachments/a_a21a99d4e1327e4e2433b1ff5b4b42a07f040aa94fa98dbee09632c2edbc6be7/Screenshot%202023-12-06%20121845.png?cb=2d9a628d8abb445332d3c95694153bc3) ### Animation control Functions for managing animation generation and playback. **Reset**: Resets the SG Com Engine, discarding all animation. ![Screenshot 2023-12-06 142234-20231206-142311.png](https://docs.speech-graphics.com/__attachments/a_3f5cda6d226e4c350461409f96a671cc855718616d0c92670842097c7a993fd4/Screenshot%202023-12-06%20142234-20231206-142311.png?cb=77bea6acb88cc2fffb9f0dfd93c885a5) **Set Greedy Process Enabled**: Enable/Disable greedy processing of audio. If enabled, the Engine will immediately process all available audio. Enabled by default. ![Screenshot 2023-12-06 142403-20231206-142439.png](https://docs.speech-graphics.com/__attachments/a_4bc13572be5483c38df776e4eb0222b17fd9b1f8caf34fae75714bd3c89b3ce9/Screenshot%202023-12-06%20142403-20231206-142439.png?cb=7df5b07016c6224b239b5367fe557ee8) **Set Current Animation Time**: Sets the current animation time in seconds. ![Screenshot 2023-12-06 143706-20231206-143740.png](https://docs.speech-graphics.com/__attachments/a_d02587b017301ee4eef4cb1025caedf517eecc70263924839672ef189c5df459/Screenshot%202023-12-06%20143706-20231206-143740.png?cb=6757a3f0d342bb397d56cef648183788) **Get Animation Range**: Retrieves the time range of the available animation in seconds. ![Screenshot 2023-12-06 143814-20231206-143842.png](https://docs.speech-graphics.com/__attachments/a_bc2040415b2e15a83fcab95f3b16d8bbc9d2b5e8e09599c4529a7cd3e5fe713e/Screenshot%202023-12-06%20143814-20231206-143842.png?cb=29bc3c408ff0473fdcab19988a258799) **SG Com Pose**: Animation node that produces a pose based on the current SG Com Engine values. The values are applied to the Base Pose of the character ![Screenshot 2023-12-06 150714-20231206-150745.png](https://docs.speech-graphics.com/__attachments/a_ec5dd34ce8b30f008e9edd3065e41a142df3d4560c7f61c8b86b06022496530a/Screenshot%202023-12-06%20150714-20231206-150745.png?cb=a9438baaab7296ce9c81417fbb23212d) ### Behavior Modes Functions for using [Behavior Modes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-behavior-controls.md#Behavior-Modes). **Set Mode**: Sets the behavior mode of the character. The input must be one of the available mode names. ![Screenshot 2023-12-06 122814.png](https://docs.speech-graphics.com/__attachments/a_30b3270f8075054a5a3b916002a9030fc1505a55a78509b0148ea97517f9a23e/Screenshot%202023-12-06%20122814.png?cb=de759817a31571371bb4ba6415c1ab3f) **Get Mode**: Returns the current behavior mode of the character. ![Screenshot 2023-12-06 123055.png](https://docs.speech-graphics.com/__attachments/a_52f30f02048d16c560bd0ba4f199b7490ab42f818ed7690bd553c928eb9fe51c/Screenshot%202023-12-06%20123055.png?cb=461b4eb76fa4520af427c1f651232d34) **Get Mode Names**: Returns an array of strings containing the available behavior modes of the character. ![Screenshot 2023-12-06 124914.png](https://docs.speech-graphics.com/__attachments/a_64555ad9378652fff511b27701c9f54d1fdaf5dc6c5c000973ada562c17ffbfa/Screenshot%202023-12-06%20124914.png?cb=590ade46390424f1cb66ab894d7c91a0) ### Auto Modes Functions for using [Auto modes](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-behavior-controls.md#Auto-Modes). **Set Default Mode**: Changes the default mode, which is the behavior mode that will be active if no other mode is automatically or explicitly set. ![Screenshot 2023-12-06 125102-20231206-125142.png](https://docs.speech-graphics.com/__attachments/a_325e817f7a9e49b134100a9807af67ca7331f3b659f974d430e0e4ef79b99b20/Screenshot%202023-12-06%20125102-20231206-125142.png?cb=976381bd110f4d0ecf5c0d78129152da) **Set Auto Mode**: Maps an auto mode to a specific behavior mode of the character. If not specified, the auto mode will not trigger any change in behavior. ![Screenshot 2023-12-06 125220-20231206-125257.png](https://docs.speech-graphics.com/__attachments/a_9fe42cda6d8ddbe0d5afcd01bac41656332bc0d0602f7f35a94e348d82a1d111/Screenshot%202023-12-06%20125220-20231206-125257.png?cb=6db89705def3a2db54ce9cdbdfdea4e9) **Unset Auto Modes**: Clears the auto mode mappings. ![Screenshot 2023-12-06 125606-20231206-125637.png](https://docs.speech-graphics.com/__attachments/a_ec8be56ee41f6688acc6ed039f94ab02085a29318dd275b3a16d3d6d9ac64b1a/Screenshot%202023-12-06%20125606-20231206-125637.png?cb=cf54279af4f951aa1cf55acb1eef5063) **Activate Auto Modes** : Activates the auto modes system. Note that using **Set Mode** to explicitly set the behavior mode will automatically deactivate auto modes. Over the course of dectivation and reactivation, the previously set auto-mode mappings are preserved. ![Screenshot 2023-12-06 125755-20231206-125821.png](https://docs.speech-graphics.com/__attachments/a_40656a227914b130753cec4569e6a142db5191691130aed4f5a33fac030d3509/Screenshot%202023-12-06%20125755-20231206-125821.png?cb=ea01f32c92c95da2ad0100d5cbccbc50) **Insert Expression** : Insert an expression into the Engine. *ExpressionName* : The path of the expression in the character's behavior mode library. Ifempty, an expression will be automatically selected from the current behavior mode. *DurationMs* : The desired duration of the expression in milliseconds. If 0, the duration will be automatically determined. *Scale*: The desired scale of the expression (0 to 1). If 0, the scale will be automatically determined. ![image-20240827-104110.png](https://docs.speech-graphics.com/__attachments/a_4ac082713f8649c03bca87c5e5f5d1f50851b223ed0607b9ac5ff5ce26a2da82/image-20240827-104110.png?cb=08ef588a501cf6ce2549e8c9093afbb5) ### Behavior Modifiers Functions for using [Behavior Modifiers](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-behavior-controls.md#Behavior-Modifiers). **Set Speech Magnitude**: Adjust magnitude of speech muscle motion. Default is 1.0. ![image-20240827-101527.png](https://docs.speech-graphics.com/__attachments/a_213454356bd1b186aa97d34b6adaaf37c6f9b25c97ac89165b7bffa872180294/image-20240827-101527.png?cb=c43780808b9c837ace67817cc2e679bb) **Get Speech Magnitude**: Get the magnitude of speech muscle motion. ![image-20240827-101649.png](https://docs.speech-graphics.com/__attachments/a_23006cdab3f0227f4e617cc400b974c45bef3dedfd050034837d2020745ed233/image-20240827-101649.png?cb=c2791de62e98d03f575f378c6b63b896) **Set Nonverbal Magnitude**: Adjust magnitude of nonverbal muscle motion. Default is 1.0. ![image-20240827-101831.png](https://docs.speech-graphics.com/__attachments/a_d877dd6ade28ecbf5120e410c946f0870d7c98e13e88a36018ec14b0e8e847c5/image-20240827-101831.png?cb=0d3a972b25d5d2caea350bdca29907fa) **Get Nonverbal Magnitude**: Get the magnitude of nonverbal muscle motion. ![image-20240827-101852.png](https://docs.speech-graphics.com/__attachments/a_e367ee0dc9ceed1f6d765149f2d996402214a4f74b4e75ad543dc639d28529be/image-20240827-101852.png?cb=96728b0cf9501237f4a5d515b74d3c69) **Set Nonverbal Speed**: Adjust the speed of nonverbal muscle motion. Default is 1.0. ![image-20240827-102107.png](https://docs.speech-graphics.com/__attachments/a_7a2ee03a92e2098fcb07af63a0c918aaee5ae9ead0cb446d1d9f89e5d774a09f/image-20240827-102107.png?cb=a5f9b783c89cd1cadaf4d5fb3b8f5dfd) **Get Nonverbal Speed**: Get the speed of nonverbal muscle motion. ![image-20240827-102121.png](https://docs.speech-graphics.com/__attachments/a_7fb401f71eb7bc2583e7890a75629f5e8591de746749c703ca0ceda01949be3f/image-20240827-102121.png?cb=e9b1c64de4d05a6d30e5d43530f6f581) **Set Breath Magnitude**: Adjust the magnitude of breath. Default is 1.0. ![image-20240827-102342.png](https://docs.speech-graphics.com/__attachments/a_ac1b5743539c1a799f49fed3708305a985c60f0cf00ce5cc110d49d7765ddad8/image-20240827-102342.png?cb=fa52559d76adcd94953200a49ea1fa8f) **Get Breath Magnitude**: Get the magnitude of breath. ![image-20240827-102359.png](https://docs.speech-graphics.com/__attachments/a_3de770646490b17613b25edc5acc0bbe6776e62f71f7a9436903a3f4c233c315/image-20240827-102359.png?cb=efa8955849ec451dc311286f85dd76a4) **Set Breath Speed**: Adjust the speed of breath. Default is 1.0. ![image-20240827-102550.png](https://docs.speech-graphics.com/__attachments/a_7b36e2e5c81c8bf47a3fceb649d7fd573808152ca6926a946fda5e5b772ae79b/image-20240827-102550.png?cb=cd67a16e32914387e0f36d09d3bb1f41) **Get Breath Speed**: Get the speed of breath. ![image-20240827-102604.png](https://docs.speech-graphics.com/__attachments/a_efcb4ff6568ac77ea6954f393f0f5f061cec6f8300be67fe5664f5dee7e340d5/image-20240827-102604.png?cb=b1979536b20b77280e724ae45464bea1) **Set Expression Frequency**: Adjust the frequency of expression. Default is 1.0. ![Screenshot 2023-12-06 145513-20231206-145541.png](https://docs.speech-graphics.com/__attachments/a_ca45b3f2bdcc3b755cc9f728fcc888b9045f6bbfac1e268039bddfead7270d1b/Screenshot%202023-12-06%20145513-20231206-145541.png?cb=b63472d572f860e762023a32bb8cc9a1) **Get Expression Frequency**: Get the frequency of expression. ![Screenshot 2023-12-06 145621-20231206-145648.png](https://docs.speech-graphics.com/__attachments/a_0656a727f383c6bd682c394c71e65f8c9fca8ca6333b1774c03aebdcfc555d45/Screenshot%202023-12-06%20145621-20231206-145648.png?cb=93657f718858f758f80b7827e6f2482f) **Set Blink Frequency**: Adjust the frequency of blinking. Default is 1.0. ![image-20240827-102810.png](https://docs.speech-graphics.com/__attachments/a_a87e9c7c1b052b2a3e33932055a5f51f906daea89d86f374601ffedc39a1d81f/image-20240827-102810.png?cb=dfb1e42d58482e729eaad0eb50ed39da) **Get Blink Frequency**: Get the frequency of blinking. ![image-20240827-102822.png](https://docs.speech-graphics.com/__attachments/a_951be257c821298df7870732470714815582ff83b7df8ca6c640bd9c63638af0/image-20240827-102822.png?cb=2c1490fa847dfbc3541014e19be3de63) **Set Microdart Frequency**: Adjust the frequency of microdarts. Default is 1.0. ![image-20240827-102931.png](https://docs.speech-graphics.com/__attachments/a_07fa8d0c54bc958045b6f8e30a207cdddc38d6dc8f7d15fe17ec43975b867b52/image-20240827-102931.png?cb=5b69ac91525b14b868485922a9243785) **Get Microdart Frequency**: Get the frequency of microdarts. ![image-20240827-102946.png](https://docs.speech-graphics.com/__attachments/a_62cd515d805512ccbaad3e313490dc420b91afa515ce04049ca29a6db0ccdab6/image-20240827-102946.png?cb=e544e8c61e957795493164bfa83b5cbe) ### Role Functions for using [SG Com Roles](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-roles.md). **Set Role**: Sets the session role (Speaking/Listening). ![Screenshot 2023-12-06 150436-20231206-150513.png](https://docs.speech-graphics.com/__attachments/a_5f2dc0099cdd865036384efd7bdb08ec28a12ea47655a6d89404ddbe13819019/Screenshot%202023-12-06%20150436-20231206-150513.png?cb=fc070ee26ed668ef5b400633124aa9eb) **Get Role**: Gets the session role. ![Screenshot 2023-12-06 150557-20231206-150623.png](https://docs.speech-graphics.com/__attachments/a_befacd74a12b95c493b06f9f513df4352fa6a4634b16ccdb9a4fdf1d60fd7cfe/Screenshot%202023-12-06%20150557-20231206-150623.png?cb=285f771438f960e5c7aeebeccbbf6401) ### Metadata updates Event callbacks that can be be used for tracking or reacting to specific changes in SG Com algorithms. See [Metadata updates](https://docs.speech-graphics.com/en/sg-com/5.3/engines-and-players.md#Metadata-updates). **On Mode Changed**: Event callback that is triggered when the character's mode changes. ![Screenshot 2023-12-06 151555-20231206-151629.png](https://docs.speech-graphics.com/__attachments/a_296cdc1ffc7db6d1532f2838569023361b25bfb5041880a8b73f46b7941c32bd/Screenshot%202023-12-06%20151555-20231206-151629.png?cb=e24ed29259fb9b2a7dfc3729f7f1b674) ![image-20240827-091755.png](https://docs.speech-graphics.com/__attachments/a_da24400c4a92e127cdc101d4702729049f352a6bdadd678fca0e5b86171baee4/image-20240827-091755.png?cb=39604133fa9b24e1fe91907bc595fa55) **On Auto Mode Changed**: Event callback that is triggered when the character's Auto mode changes. ![image-20240827-091458.png](https://docs.speech-graphics.com/__attachments/a_984609050c1f252759f8ba44d650f272e95b139e6ac2f40c97ac4dc0c76a5777/image-20240827-091458.png?cb=df6f46109a901d17b2deac35c54f2147) ![image-20240827-091928.png](https://docs.speech-graphics.com/__attachments/a_6c722bd6435664395f6c88e73f77ab2f7d3fcd660e64dc88c1f433ce89c82346/image-20240827-091928.png?cb=b6d693fcb118fe46677c8fcbe3a8cd72) **On Expression Changed**: Event callback that is triggered when the character's expression changes. ![Screenshot 2023-12-06 151712-20231206-151752.png](https://docs.speech-graphics.com/__attachments/a_0c5ce8bf7b11daac9abae4a93948059ef4efa084b9bf3c2e397ca6e60402fdaf/Screenshot%202023-12-06%20151712-20231206-151752.png?cb=cefb22f6dbc7fded83cd80270e6965d2) ![image-20240827-092031.png](https://docs.speech-graphics.com/__attachments/a_6d31908d182c116029f96189e5ff9af15f0cc8bbfbe0ecd9db123ad1c2ecb9f0/image-20240827-092031.png?cb=8ccbbaa086763debb0777cb3ea193b2e) **On Voice Activity Changed**: Event callback that is triggered when voice activity change is detected. Returns "true" for voice and "false" for silence. ![image-20240311-193625.png](https://docs.speech-graphics.com/__attachments/a_54d78fd7e6c495f43b3d72d92d0c07dd1f626dc912082d4705d1ac9b4059933d/image-20240311-193625.png?cb=af7d4d7a72a1f68678bf2ab69cc68ae5) ![image-20240827-092243.png](https://docs.speech-graphics.com/__attachments/a_30066c7b597d9a2bdcac708c1c1ff988f5a70177250d3e2ebff465a96639755b/image-20240827-092243.png?cb=54c4c85bfafccd51577e1c90319a31a0) **On Breath Changed**: Event dispatcher that is triggered when breath switches either from inhale to exhale or from exhale to inhale. Returns "inhale" or "exhale" indicating the new direction of breath. ![image-20240829-161848.png](https://docs.speech-graphics.com/__attachments/a_39bc527cda8729be27daa67d388ab1b10b10625e7a78019870ec7d6bc4dc5da8/image-20240829-161848.png?cb=476717307cbfbbf5bd0accff8e502936) ![image-20240829-162013.png](https://docs.speech-graphics.com/__attachments/a_2823b6b4b40ec54fcafe948eae95faa1f5293fba70da23db071ff231076ceba5/image-20240829-162013.png?cb=3a673d70e1e4177e1df9b45e31f82082) ## SGMicrophoneAudio Actor The *SGMicrophoneAudio* actor is responsible for capturing microphone input and streaming audio to *SGComSessionComponents*. It also provides playback functionality for microphone audio. **Start Microphone:** Starts capturing microphone input and streams the audio to the specified *SGComSessionComponents*. Also sets up an Audio Component for playback. ![image-20250416-102230.png](https://docs.speech-graphics.com/__attachments/a_6dbf9ae53797f561d8db43d16461bb33383d4d1d6730db81eeccb8095b12e317/image-20250416-102230.png?cb=b00b9360fba0fc2284a506d95b27d9cb) **Stop Microphone:**Stops microphone input. Destroys the Audio Component. ![image-20250416-102631.png](https://docs.speech-graphics.com/__attachments/a_774171f76316005f0cbf1d4ea6fc23fb8248c05fc0de17529cbfa7dd559a7baa/image-20250416-102631.png?cb=4920d977b43a3d04e650a00489aae59e) **Get Audio Component:** Returns the Audio component created by *Start Microphone.* ![image-20250416-102800.png](https://docs.speech-graphics.com/__attachments/a_75b17cfa344d4487eef684d0a9b8de7858ff9b5227f754b1316590cb6913106d/image-20250416-102800.png?cb=19fe71484392cb7b3b1a5f59419206e4) --- version: "5.3" language: "en" --- # What is SG Com? The SG Com SDK provides access to Speech Graphics' real-time audio-driven facial animation technology, enabling developers to build applications with avatars driven by voice. SG Com converts an audio stream into high-fidelity facial animation which may be streamed to other endpoints. The processing delay between input and output is **50 milliseconds**. The SDK includes: * [SG Com API](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-c-api.md), a C API * [SG Com Player API](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-c-api.md), a subset of the SG Com API including only the Player functions * [SG Com Unreal Engine Plugin](https://docs.speech-graphics.com/en/sg-com/5.3/sg-com-unreal-engine-plugin.md), for easy integration into the Unreal Engine C# bindings and sample Unity integrations are also available on request. ## SG Com features * Accurate lip sync in any languagefor any character * Automatic detection of emotion (positive, negative, neutral) * Automatic detection of nonverbal vocalizations such as laughter and grunts * Automatic detection of audible breaths * Full-face expressions, driven by detected emotion or vocalizations * Head motion appropriate for the current speech * Chest breath motion that is timed with the speech and audible breath * Blinks and eye microdarts * Live lookat system with coordinated response of eyes, head and torso muscles * Fully live and interruptable behavior * Idle behavior during quiet periods * Roles for both speaking and listening (reacting to speech) * Variety of live behavior controls with which to direct the behavior mode, facial expression, and characteristics such as intensity and speed of movements or frequency of changes * Notifications for changes in behavior mode, expression, voice activity, breath, and speech beats --- version: "5.3" language: "en" --- # What's new in SG Com 5.3? SG Com 5.3 includes a number of brand-new features as well as several improved features. ## Lookat The lookat feature moves the character's muscles so that its gaze tracks a specific target in 3D space. This is useful for maintaining eye contact with the camera, or to look at a specific object in the scene. The lookat movements incorporate the eyeballs, head and torso. Since the system operates through the dynamic muscles, it is immediately responsive to changes in target. Most advanced lookat systems require solving an inverse kinematic (IK) problem over a chain of bones, so the solutions tend to be rig-specific. The SG Com lookat system by contrast is agnostic about the rig, which means it can be used for any rig. Instead of solving for joint rotations, it solves for muscle activations. This requires first learning how the character's muscles affect gaze direction. For each character, enabling the lookat feature requires a simple configuration step in SGX Studio Maya (4.6 or later). During configuration, the user selects 3 joints in the character's head: left eye, right eye, and eye root. The current model also requires these 7 muscles to be defined: * eyeball_pitch * eyeball_yaw * head_pitch * head_roll * head_yaw * torso_pitch * torso_roll At runtime, the lookat system operates in coordination with nonverbal expressions. Each expression has a gaze direction invoving the above muscles. The muscle activations will be modified to achieve the desired gaze direction with minimal alteration to the expression. The Lookat system includes several new additions to the SG Com API: * A new structure containing 3D positional coordinates. typedef struct SG_COM_Position { float x; float y; float z; } SG_COM_Position; * A function to set the activation of the Lookat system. The activation value is the probability that the character will focus on the target during each new expression. The range is \[0,1\]. At 0, the character ignores the target; at 1, it will always be fixated on the target. Use in-between values to sometimes look at the target and sometimes ignore it. SG_COM_Error SG_COM_SetLookatActivation( SG_COM_EngineHandle engine_handle, float activation ); * A function to set the lookat target. Its arguments include the target position, as well as the base positions of the three head joints (left eye, right eye, and eye root). The 'base positions' of the head joints means their positions *prior* to any SG Com animation being applied. The head joint positions inform the system of the character's orientation relative to the target. Call this function whenever the target moves or whenever the character's head moves due to non-SG Com animation (e.g. idle animation). It can be called every frame if necessary. SG_COM_Error SG_COM_SetLookatTarget( SG_COM_EngineHandle engine_handle, SG_COM_Position* target_position, SG_COM_Position* left_eye_base_position, SG_COM_Position* right_eye_base_position, SG_COM_Position* eye_root_base_position ); * A function to retrieve the names of the character's three head joints so they can be looked up in the scene. This should only be called once when loading the character. SG_COM_Error SG_COM_GetLookatHeadJoints( SG_COM_EngineHandle engine_handle, char* left_eye_joint, char* right_eye_joint, char* eye_root_joint, sg_size buffersize ); ## Beat detection A 'beat' in speech is an emphasis, a stressed syllable, that stands out as being stronger than the speech around it, and indicates a likely point for a rhythmic nonverbal gesture to occur. SG Com now animates beats, thus providing a timing framework for gesture animation. It does not go so far as to animate the gestures yet, but the timing is useful for external systems to trigger gestures appropriately. To signal a beat, `SG_COM_EngineStatusCallback` will emit the new status type `SG_COM_STATUS_SPEECH_BEAT`. The `message` string in the callback will contain the normalized intensity of the beat as a floating-point integer. In principal the average intensity is 1.0, and most beats will be higher than this. The intensity has no strict bounds but most beats will be on the scale \[1.0, 2.0\]. ![Screenshot 2026-06-09 194931.png](https://docs.speech-graphics.com/__attachments/a_40fc5a69e890f683c5b996596b64b1a811a85655a6c87a4fb37ba23419ae7b24/Screenshot%202026-06-09%20194931.png?cb=4d12ea7072ab40c6d6a51ec2f039f582) Record of detected beats in SG Com ## Player-only library SG Com 5.3 includes a second smaller library, **SG Com Player**, which excludes Engine-related functions and focuses solely on the Player. This is useful for lightweight implementations of SG Com clients, e.g. in Web Assembly, where Engine-side functions would be handled in the cloud or elsewhere. The main differences between SG Com Player and the full SG Com library are the following: * Removal of all Engine-related identifiers including `SG_COM_EngineHandle`, `SG_COM_EngineConfig`, `SG_COM_EngineConfigFlag`, `SG_COM_Status`, `SG_COM_Modifier`, `SG_COM_Role`,` SG_COM_AutoMode`and `SG_COM_Position` * Removal all functions having an `SG_COM_EngineHandle`argument * The replacement of the `SG_COM` prefix with `SG_COM_PLAYER` -- for example `SG_COM_Initialize()` becomes `SG_COM_PLAYER_Initialize()` * Removal of "Player" from identifiers since it's in the prefix. For example: |------------------------|-------------------------| | **SG Com** | **SG Com Player** | | `SG_COM_PlayerConfig` | `SG_COM_PLAYER_Config` | | `SG_COM_PlayerHandle` | `SG_COM_PLAYER_Handle` | | `SG_COM_CreatePlayer` | `SG_COM_PLAYER_Create` | | `SG_COM_DestroyPlayer` | `SG_COM_PLAYER_Destroy` | ## SGX event playback A new function has been added: `SG_COM_ReceiveEvent()` -- or `SG_COM_PLAYER_ReceiveEvent()` in the SG Com Player API. SG_COM_Error SG_COM_ReceiveEvent( SG_COM_PlayerHandle player_handle, const char* event, sg_size event_bytes ); This function inserts an SGX event file (`.event`), containing a pre-generated muscle sequence, into the Player. This is an alternative to inserting real-time packets emitted by the Engine, and is equivalent to adding a series of packets at once. If the animation buffer is not long enough to hold the new animation, it will be clipped. At present, there is no blending of animation at the beginning and end of the inserted sequence, so some discontinuities may occur. SGX events must be generated using SGX Production Tools. ## Improved features ### Nonverbal muscles The dynamic muscle models used for nonverbal behavior have been completely revamped. * Bidirectional and unidirectional muscles now have the same physical modeling * Acceleration is more controlled so that muscles more closely adhere to the dynamic settings in SGX (e.g. so that head muscles no longer look faster in SG Com than in SGX). * Instead of arcing, muscles now exhibit an overshoot and settling behavior. This has several advantages over arcing: * It is closer to the way real muscles behave. * It reduces the apparent latency of expressions because the peak of the overshoot occurs earlier in the expression than the peak of a long arc. * It avoids the pitfalls of arcing such as the fact that the intensity of the arc grew with the duration of the expression. * Glitching in the muscle models has been virtually eliminated. ![image-20260803-145057.png](https://docs.speech-graphics.com/__attachments/a_c8751f0d3821d3dcf697e34de93ca0890740b82d7a7d9c98c35b5febdf355ed3/image-20260803-145057.png?cb=c8754ebba9d915b8064d376c0de55c7b) Overshoot and settling ### Blinks The SG Com blink system has been improved to match the SGX blink system in terms of quality of blink predictions. Formerly, blinks were too rare during speech and overabundant during pauses. ### Microdarts Similarly to blinks, the microdart system has been improved to match the SGX microdart system in terms of quality of microdart predictions. In addition, microdarts are now more varied; rather than simply alternating between the far left and far right sides of the range, they cover the middle ground. And the microdart system has been refined to work harmoniously with the new lookat system. ### Lip sync The universal lip sync system has been improved, and now has the same quality level as character-specific lip sync models. This implies that character-specific training is no longer needed except in the case of highly custom muscle systems. --- version: "SGX 4.6" language: "en" --- # SGX ## Welcome to SGX This is a user guide for the Speech Graphics Audio-driven Animation System, SGX™, which converts audio into synchronized high-quality facial animation. * [Speech Graphics](https://docs.speech-graphics.com/en/sgx/4.6/speech-graphics.md) * [SGX Introduction](https://docs.speech-graphics.com/en/sgx/4.6/sgx-introduction.md) * [SGX Concepts](https://docs.speech-graphics.com/en/sgx/4.6/sgx-concepts.md) * [SGX Product Guides](https://docs.speech-graphics.com/en/sgx/4.6/sgx-product-guides.md) --- version: "SGX 4.6" language: "en" --- # Anatomical Regions Muscles are grouped into 11 **anatomical regions**: * Brows * Cheeks * Eyeballs * Eyelids * Head * Jaw * Nose * Throat * Tongue * Torso The main significance of anatomical regions -- aside from providing a convenient organizational structure for sorting muscles -- is that they carry default [dynamic parameters](https://docs.speech-graphics.com/en/sgx/4.6/muscle-dynamics.md). For example, displacements of the tongue generally happen much faster than those of the head due to their relative sizes and distances traveled, and these differences are reflected in the default dynamics of the two regions. New muscles added to each region automatically inherit the default dynamics of that region. When adding new muscles, it is important to add them to the closest matching anatomical region, so that they inherit dynamic properties suitable for the muscle. Although some of the properties can be edited after adding them, it helps to have a good base. There are also unexposed dynamic properties that cannot be edited and so are entirely dependent on the anatomical region. If you are adding muscles for non-human anatomy, add them to the anatomical region that has most in common with the new muscle in terms of location on the body and expected velocity. --- version: "SGX 4.6" language: "en" --- # Animation Style The way a character transitions between poses can dramatically shape the tone and style of a performance. From the smooth, lifelike flow of a cinematic close-up to the snappy, exaggerated cuts seen in anime, the timing and dynamics of motion are core tools in an animator's kit. To give creators more control over these stylistic choices, SGX provides the Animation Style feature, which provides a choice between three different ways of interpolating between successive poses: *Ease* , *Linear* and *Stepped* . The default style is *Ease*, which provides the most physically natural dynamics. All three Animation Styles are built on the same SGX muscle-dynamic simulation, the difference being that the underlying muscle movements are processed into animation differently in each case. Thus all three styles have the same benefits: precise lip-sync matching the pronunciation, intensity and rhythm of the speaker's voice, and believable emotional expressions. All styles can be used for any language and on any character, and they are rig agnostic, making them flexible and compatible with your needs. [comparison.mp4](https://docs.speech-graphics.com/__attachments/a_0726a1c9c8649f66af75c8072d3d7f178c071cbb87c2262bdca2da28ecc02264/comparison.mp4.md) Comparison of the three animation styles ## Ease **Ease**, the default style, delivers fluid motion with acceleration and deceleration - ideal for realistic performances. [ease.mp4](https://docs.speech-graphics.com/__attachments/a_0e64f6531f79d35f537a51ef614020d76ed64c33fae327ba5e24f3c972afb5e5/ease.mp4.md) Ease Animation Style ## Linear The **Linear** style uses constant velocities, creating robotic transitions with a uniform rate of movement. Note that Linear processing require a [language module](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Language) to be selected; it will not work with the Universal lip sync model. [linear.mp4](https://docs.speech-graphics.com/__attachments/a_289a9e13993baa3015f115bd88a22866297a10d8bb1e812dce09690fd6241cd1/linear.mp4.md) Linear Animation Style ## Stepped For artists looking to capture the sharp, stylized timing seen in anime or stop-motion, the **Stepped** option eliminates in-betweening almost entirely - cutting straight from one pose to the next for a bold, punchy effect. Stepped animation is computer animation but with a hand-crafted appeal, emulating traditional anime as well as the kind of stop-motion clay animation seen in *Wallace and Gromit* or *Chicken Run* . Some famous recent examples of stepped animation include *The Lego Movie* and the classic 2D animation, *South Park*. Note that Stepped processing require a [language module](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Language) to be selected; it will not work with the Universal lip sync model. [stepped.mp4](https://docs.speech-graphics.com/__attachments/a_aa74ac94052714dfd0f5cda652bc2cf9fb282f718f1ee934a305b26b0ef58bee/stepped.mp4.md) Stepped Animation Style --- version: "SGX 4.6" language: "en" --- # Animation Targets The very first step in the character setup process is to select the rig parameters that Speech Graphics software will control. These are called the **animation targets** .Normally you will simply provide a list of these to character setup services. In a Maya rig, animation targets will be attributes on scene nodes. These may include: * weights on blendshape nodes * translation, rotation, or scale attributes on joints or any other transform nodes * custom attributes on any node Animation targets may thus support any kind of rigging, whether using blendshapes, bones/joints, some higher-level animator UI handles, or any mixture of these. Once animation targets are set, we may proceed to define the [neutral pose](https://docs.speech-graphics.com/en/sgx/4.6/neutral-pose.md) and [muscle poses](https://docs.speech-graphics.com/en/sgx/4.6/muscles.md#Setting-muscle-poses) of the character in terms of displacements of those parameters. ## Reference pose The animation targets must be selected while the character is in the *reference pose*, which is the original pose of the character in the scene. The reference pose values for the animation targets are recorded in the character control file at the same time that the animation targets are selected. It is important to have the right reference pose because it helps determine displacements for muscle definitions, and is the basis for absolute animation. ## Transforms Note that bones and other transforms have 9 attributes that can be targetted: * translation x * translation y * translation z * rotation x * rotation y * rotation z * scale x * scale y * scale z You may choose some subset of these 9 attributes to target. However, note the following rule: if the given targets include any components of *rotation* or *scale*, then all 9 components of the transform will automatically be included in the final target set. ## Changes to animation targets Any changes to the deformational effects of your rig's animation targets -- such as bone skinning changes -- may require adjustments to the muscle poses or neutral pose. Furthermore, any changes to the *names* of the animation targets in the scene may cause the muscles using them to stop working properly because the targets are not found. If naming changes occur, you can can remap animation targets in [SGX Studio](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-plugins.md), or contact our support staff to make the change for you. Similiarly, any discrepancies in animation targets between the rig on which the character was set up, and the rig that the animation is being applied to, can cause problems with the animation of muscles using those targets. Contact our support staff if you have different versions of the rig for different purposes. ## Animation levels For more complex rigs with animator-friendly interfaces, there may be two levels of targets: high-level UI controls and low-level controls (such as bones and blendshapes), where the latter are driven by the former. In SGX these two sets of targets are called *animation levels* -- namely, the *control level* , and the *baked level*. The animation targets at the baked level receive their values via flow of information from the control level. In typical use cases, the control level is better suited for manually setting up the muscle poses, but the baked level is used for animation in the game engine. SGX allows you to define both animation levels in the same character control file by simply selecting two sets of animation targets. Muscles can then be *baked*, meaning the muscle poses defined on the control level are automatically re-created on the baked level. ![SGX_Baking.png](https://docs.speech-graphics.com/__attachments/a_3f3ad677fc824b3bccd981ae4ae221e45f22b20a97e5e5b1c9ed8cdb83c660b9/SGX_Baking.png?cb=825d53c537aff83315d3aec893b61750) --- version: "SGX 4.6" language: "en" --- # Audio Guidelines ![image-20230426-131339.png](https://docs.speech-graphics.com/__attachments/a_0ebec620dd5d81687a7c522a92ef0d5b7dab850f9a63151885de503e369bad06/image-20230426-131339.png?cb=e562fe7c476038a255881bf9498fdf35) SGX generates animation based upon the audio files you provide. Making sure that your audio adheres to the guidelines below is critical for creating high-quality animation with SGX. SGX supports a wide variety of PCM-based audio formats at a number of sample rates and bit depths. ## Supported file formats * wav * mp3 * ogg * aiff * au ## Supported sample types * Mono audio only. Stereo recordings are *not* supported and will not be processed. * Minimum 16bit depth * Minimum 16Khz sample rate ## Audio production best practices When you provide SGX with high-quality audio input it will generate better animation output. Below are some best practices for producing audio for use with SGX. * **Minimize background noise** - A clean, isolated voice will produce the best results. Even low-level ambient sounds and background music can potentially affect the quality of the animation. * **Use raw, unprocessed voice** - Reverb and other audio effects can negatively impact animation quality. * **Use uncompressed audio** - Compression reduces the amount of information available in the audio for SGX to use, which could impact output quality. * **Avoid recording in echoic spaces** - Like reverb post-effects, audible echoes from the recording venue can diminish the quality of the resulting animation. * **Single speaker** - SGX is designed to only animate one speaker for a given audio file. * **Padding** - If the audio file is cut very close to the beginning or end of the audible speech, the animation might then start or end with the face in an active state, since there will be no time for a transition into or out of the speech. To avoid this consider padding the beginning and end of the clip with a bit of silence; alternatively use the [Pre-Roll and Post-Roll](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Pre-Roll-and-Post-Roll) options in SGX. --- version: "SGX 4.6" language: "en" --- # Audio-to-Motion Processing The central function of SGX is to convert audio (and optional transcripts) into synchronous animation. This happens through a combination of audio analyses, neural networks and muscle dynamic simulations. When an audio file is processed, the first thing that happens is that it is imported, along with its corresponding transcript, into an [event](https://docs.speech-graphics.com/en/sgx/4.6/events.md). ![image-20251014-161746.png](https://docs.speech-graphics.com/__attachments/a_b5ab90dd5c83e4a24c5aee7ee451212d181adf2c44219c09c7a38e1a4e514d5e/image-20251014-161746.png?cb=565e7de109275c3e2ae8f0b8b8ba1eab) Then, SGX algorithms analyze the input audio and transcript, and generate corresponding motion. In the course of this analysis, a number of [metadata](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md)representations are also produced on the audio timeline. All of these sequences are appended to the event. Then the event file is saved. ![image-20251014-161626.png](https://docs.speech-graphics.com/__attachments/a_8cc028758ed4465de27309ee00a607fa1bcad585271d98a1e11af0c27e52f0a1/image-20251014-161626.png?cb=87eec5599791895a03137918c6c680aa) Event processing may be performed either interactively in the GUI in [SGX Director](https://docs.speech-graphics.com/en/sgx/4.6/processing-events.md), or as a batch process in [SGX Producer](https://docs.speech-graphics.com/en/sgx/4.6/sgx-producer.md). ## Phonetic and Acoustic lip sync processing SGX processing combines two different lip sync systems, which we call **phonetic** and **acoustic**. The phonetic lip sync system utilizes the transcript as a supplement to the audio, whereas the acoustic lip sync system does not. The phonetic system can usually achieve higher quality lip sync than the acoustic system because of the additional information provided by the transcript. There are three conditions under which the acoustic system is automatically triggered: 1. When a transcript is not used 2. When a language module is not used 3. When there is low confidence in the phonetic analysis Under the third condition the acoustic analysis takes over to fix problems in the phonetic analysis. This helps make the phonetic analysis very robust. Because of these auto-corrections, the lip sync system being used may change over the course of an event. These changes should be seamless. However, the choice of lip sync system is visible in the [metadata](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#%255BinlineExtension%255DLip-sync)and can be edited on the timeline in SGX Director (see [Editing Events](https://docs.speech-graphics.com/en/sgx/4.6/inspecting-and-editing-sequences.md)). Events have a measure called **phonetic analysis score**, which refers to the proportion of the event during which the phonetic system was deemed correct and was therefore used. --- version: "SGX 4.6" language: "en" --- # Audio View The audio view shows the waveform. Optionally, it can also include a spectrogram (grayscale or color); see [Display settings](https://docs.speech-graphics.com/en/sgx/4.6/settings.md#Display-settings). ![image-20240108-144001.png](https://docs.speech-graphics.com/__attachments/a_47e4f7508e1fe4d83706b3b418529ea4acc7b88b4d1141aa61c6a36d4cb4cef2/image-20240108-144001.png?cb=d4983b45c94a5f879dd728ba5c2622f6) When an Audio sequence is selected, the *Sequence* display in the [Inspector](https://docs.speech-graphics.com/en/sgx/4.6/navigating-events.md#Inspector) will show this layout: ![image-20240108-171159.png](https://docs.speech-graphics.com/__attachments/a_6ee7a66b0f03fea1c85d0fc7f08546a9ed5e5ef270c07269809c36bbd953a174/image-20240108-171159.png?cb=bca9255058fb9c40cb9af5be7305d4c1) which lists the following properties: |--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Property** | **Description** | | Sample Rate | The sample rate of the audio file. | | Channels | The number of channels. Note that SGX processing requires single-channel audio (see [Supported sample types](https://docs.speech-graphics.com/en/sgx/4.6/audio-guidelines-1.md#Supported-Sample-Types)). | | Bit depth | The bit depth of the audio. | | Format | The audio file format. | The speaker icon at the bottom of the Audio properties ![image-20240107-214444.png](https://docs.speech-graphics.com/__attachments/a_f2b091ad1e44df353573b979e18e61f509c90e72db3d59b1c52378c6bcf5d129/image-20240107-214444.png?cb=21ab1a6071e7ce5dd8e6402b19097a38) may be clicked to mute/unmute the audio for playback. --- version: "SGX 4.6" language: "en" --- # Batch Input Formats There are two ways to specify input audio files and (optional) text transcripts for batch processing: 1. Directory input 2. Spreadsheet input (`.csv` file) ## Directory Input Using this approach, all of the input files for the batch must be in one directory. If you are using transcripts, each of the transcript files must also be in a separate (`.txt`) file. ![dir_input.png](https://docs.speech-graphics.com/__attachments/a_7b816c1012c0b3e8d050513525b3b5ee4314e8cd5c8ff47dae0383a86da34656/dir_input.png?cb=2bbbe1a4292d1db5e17bbae1580063f6) Each transcript (.txt) file in the input folder contains the transcript of the audio file with the same base name. For example, `p364_025.txt` contains the transcript for audio file `p364_025.wav`. ## Spreadsheet Input (.csv file) Another approach is to provide a spreadsheet in .csv format. The first column contains the paths of the input audio files, and the second column contains the corresponding text transcripts. The transcripts column is optional. D:\Voiceover_01\audio\p364_025.wav,Hey point that thing at someone else D:\Voiceover_01\audio\p364_026.wav,I just heard from the scout there is a convoy approaching D:\Voiceover_01\audio\p364_027.wav,"So I just walked up the beach, and there was this great thing coming out of the water so I ran I hope it's not following me" Note that you may also use relative paths to refer to the audio files: p364_025.wav,Hey point that thing at someone else p364_026.wav,I just heard from the scout there is a convoy approaching p364_027.wav,"So I just walked up the beach, and there was this great thing coming out of the water so I ran I hope it's not following me" The relative paths will be resolved using the location of the .csv file as the base directory. For example, if the if the above file is `D:\Voiceover_01\batch_01.csv`, then each of the relative paths will be expanded based on the directory `D:\Voiceover_01\`. --- version: "SGX 4.6" language: "en" --- # Batch Processing See [SGX Batch Processing](https://docs.speech-graphics.com/en/sgx/4.6/sgx-batch-processing.md) for a general introduction to batch processing. Batch processing may be launched in two ways: 1. From SGX Director, which launches an instance of SGX Producer 2. By running `SGX_Producer.exe` directly on the command line The first method, using the SGX Director launcher, is discussed here. For further information on running SGX Producer directly, see the [SGX Producer product guide](https://docs.speech-graphics.com/en/sgx/4.6/sgx-producer.md). ## The processing context Prior to launching a batch process, the following must be staged: * Prepare input files - i.e. audio files and optional transcripts (see [Preparing Input](https://docs.speech-graphics.com/en/sgx/4.6/preparing-input-1.md)). * Make any desired changes to the [Processing settings](https://docs.speech-graphics.com/en/sgx/4.6/settings.md#Processing-settings). * Configure processing resources in the [Resources](https://docs.speech-graphics.com/en/sgx/4.6/resources-1.md) tab, including the target character, the language or dialect heard in the audio, and global behavior settings. ## The Batch tab SGX Director provides a user-friendly interface for launching batches -- the Batch tab. Open a new Batch tab by selecting **File** \> **New** \> **Batch** from the application menu, or alternatively, use the shortcut **Ctrl+B**. ![image-20260914-165613.png](https://docs.speech-graphics.com/__attachments/a_facbf62eaf1a77358fd47a5140fd11ff2739dabb8a4e31425f47e9f4b6610f3c/image-20260914-165613.png?cb=cd721144d6f2eebf5e5fa2bf43324f58) ## Batch settings Following are the settings for configuring a batch: ### 1. Processing options There are two processing modes to choose from: process new events, or process existing events (see [SGX Batch Processing](https://docs.speech-graphics.com/en/sgx/4.6/sgx-batch-processing.md) for an overview of the difference). Note, this selection affects the remaining options. ![Screenshot 2026-09-14 175608 - Copy.png](https://docs.speech-graphics.com/__attachments/a_6f91993bfe25181c82d0c9809105ac567938a0919087acf706f2ab39d3cf5a7d/Screenshot%202026-09-14%20175608%20-%20Copy.png?cb=75fbaa777f8eb8f2f373669eac26237d) ### 2. Audio location (only with *process new events* ) Specify the *Audio Location*, where input audio and optional accompanying transcripts are found: ![select_audio.png](https://docs.speech-graphics.com/__attachments/a_2945bdf39ab3345797b393de03dbadd31415586949836ed2f0074f2e0dea5449/select_audio.png?cb=93831fa215803899979b13870c04a85f) As described in [Batch Input Formats](https://docs.speech-graphics.com/en/sgx/4.6/batch-input-formats.md), there are two choices for the audio input: * a directory of input files; * a spreadsheet file containing audio file paths and transcripts. Choose the input format ("directory or "file") from the dropdown menu: ![select_dir.png](https://docs.speech-graphics.com/__attachments/a_c3a32fd362d494d0219d846c4fc9318c2a595ba5cfa40c18acbfe026f148b400/select_dir.png?cb=b2655526d5d4127dbeff4768a5dcd72d) Then click the file icon to browse for the path of the directory or spreadsheet containing the input. ### 3. Events location The *Events Location* is the directory where you want the newly created events to be saved, **or** where the events to be processed are already located, depending on the processing mode. ![Screenshot 2026-09-14 175608 - Copy (2).png](https://docs.speech-graphics.com/__attachments/a_4f758a9cf08f8c3785a70c3ff999dae8e918b9bee8884299d0b11b340f3255eb/Screenshot%202026-09-14%20175608%20-%20Copy%20(2).png?cb=96db59911973dc025d371d891ba9bbf7) Browse for the path by clicking on the file icon. Note that when *Process existing events* is selected, the Events Location may be either: * a directory of event files; * a file containing a list of event file paths ![select_events.png](https://docs.speech-graphics.com/__attachments/a_5588e0c637eacfe94635cae8365d1873277404989c513560f8171d1c4602603f/select_events.png?cb=1ae8fdf1270bf68a088980f2dcaf91a8) ### 4. Overwrite (only with *process new events* ) The *Overwrite*option determines whether newly created events can overwrite existing events with the same name in the events directory. If overwrite is not enabled, any duplicate events will not be saved and those items will be reported as uncreated. ![overwrite.png](https://docs.speech-graphics.com/__attachments/a_a81e7b8fa257fd4dccce23f31e90748ea09460467bc5894d0d312946400bec19/overwrite.png?cb=824c48631906affe545e67c797a2becd) ### 5. Export animation As described in [Exporting](https://docs.speech-graphics.com/en/sgx/4.6/exporting.md), there are multiple ways to get animation out of SGX events. The primary approach is to use [SGX Studio plugins](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-plugins.md) (in Maya, Unreal, etc.). However, another approach is to export animation in a plain-text format called [RegularTime Series](https://docs.speech-graphics.com/en/sgx/4.6/time-series-types.md#Regular-Time-Series) (`.rts`), which can be imported into any format using a custom script. One can add RTS export to the batch process by ticking the *Export Animation* checkbox. ![export_animation.png](https://docs.speech-graphics.com/__attachments/a_bd5055db105b1c7bbc8a1b68406a94a438a1d875161ac6c2fe9cb267d145aff8/export_animation.png?cb=95426afe4211dbb79068baaf2ec5513f) This outputs RTS files next to the event files, with matching base names. ![12b50bff-11ce-46a1-9344-2b780c7be1cc.png](https://docs.speech-graphics.com/__attachments/a_2355b40565745f9e0b62286ad9425a9d28b17c1d1e8811e022d3cf167ca01f55/12b50bff-11ce-46a1-9344-2b780c7be1cc.png?cb=62d84e542671635c13d661bce2c8d7ab) ### 6. Export metadata Tick the *Export Metadata* option to export metadata from events; see [Exporting Metadata](https://docs.speech-graphics.com/en/sgx/4.6/exporting.md#metadata). This is an extended feature on your [license](https://docs.speech-graphics.com/en/sgx/4.6/sgx-license-1.md). Contact [customer support](https://docs.speech-graphics.com/en/sgx/4.6/speech-graphics-customer-support.md) for more information. ![export_metadata.png](https://docs.speech-graphics.com/__attachments/a_40409f43a699e0fe0acef14e879c6ff6b7d7ce1361b87577e5d93c1489f7570c/export_metadata.png?cb=7d29ffa144c8e647c8fb567ea5db8fe6) ### 7. Scoped Log Tick the *Scoped Log* option to inject parseable tags into batch processing logs. These tags enable automated log parsers to isolate specific processing stages, trace errors to distinct source assets and aggregate performance metrics. See [Scoped Logging](https://docs.speech-graphics.com/en/sgx/4.6/scoped-logging.md). ![Screenshot 2026-09-15 101144.png](https://docs.speech-graphics.com/__attachments/a_5495227e914a5ca181a2e0d59ba7dcd9d39e4c900c9f62c79ebee09593b9a052/Screenshot%202026-09-15%20101144.png?cb=1a9e97c08f27425587a87ba1d1c697aa) ### 8. Custom parameters This field accepts a list of parameters for custom or experimental features, in the form `xxx;xxx;xxx`. Each `xxx` can be either a single term or a key-value pair in the form `key,value`. This may come up if you are participating in a beta test, for example. ## Running the batch Once the processing context and batch configuration are set, you can run the batch. The batch tab must be the current content tab. Then, to launch batch processing, do one of the following: * Select **Process** \> **Run** from the menu * Use the keyboard shortcut **Ctrl+R** * Press the *Run Batch* button: ![Screenshot 2026-09-14 204309.png](https://docs.speech-graphics.com/__attachments/a_dd30d78ed89c88fe65dc20044a1d275ee3532a39205aa9580cadfdfcd42f6511/Screenshot%202026-09-14%20204309.png?cb=ebfb8bdaa0fe28ba293370a0bfd05940) This launches an instance of [SGX Producer](https://docs.speech-graphics.com/en/sgx/4.6/sgx-producer.md) which runs in a console window. This is a separate process and will continue running after SGX Director is closed. ![image-20260914-193656.png](https://docs.speech-graphics.com/__attachments/a_81106cb966c2d0931bada507c88bf18d4383c1c8fe2832380d7ab059ab06262c/image-20260914-193656.png?cb=1ffd5d5631cb558416800be93d38ff19) Another way to run the batch is to print an SGX Producer command corresponding to the current processing configuration, and copy the command into the console or a script. To print the command, press *Print Command:* ![print_command.png](https://docs.speech-graphics.com/__attachments/a_43921dc082c6d5fde7f6e766ab2f2b4326cdf6dc17bfb9c435e33e57b9f9c58f/print_command.png?cb=4abbd97844de097c75f524d4804aeba3) The SGX Producer command will be printed to the command window, where it can be selected and copied for pasting into the console or somewhere else: ![printed_command.png](https://docs.speech-graphics.com/__attachments/a_169ed1dd50166f5ed42dfd704d8a2bd7808a703a1e4c19acec30155080d51de9/printed_command.png?cb=32db0a2a01434a17a5e742e4526dfe29) --- version: "SGX 4.6" language: "en" --- # Behavior Controls In contrast with facial motion capture solutions, which copy motion from a real actor, SGX is a procedural solution, which means it *generates*behavior. Because it is generative, it can offer more than just automation; it also allows creative control, by giving animators an opportunity to direct the performance of the character. With motion capture, the actor is in control -- with SGX, the animator is in control. Behavior can be directed at various points in the SGX workflow: * **Before processing** : via markup in transcripts (see [Behavior Markup](https://docs.speech-graphics.com/en/sgx/4.6/markup-1.md)) * **During processing** : via processing parameters applied to a whole batch or a single event (see [SGX Director](https://docs.speech-graphics.com/en/sgx/4.6/resources-1.md#%255BinlineExtension%255DBehavior-controls) and [SGX Producer](https://docs.speech-graphics.com/en/sgx/4.6/sgx-producer.md#Process-command) ) * **After processing** : via edits to behavior metadata (see [Editing Events](https://docs.speech-graphics.com/en/sgx/4.6/inspecting-and-editing-sequences.md)) There are two types of behavior controls: **modes** and **modifiers**. These are described below. ## Behavior Modes Behavior modes make it easy to determine an entire behavior pattern with a single label. At any point in time during animation, your character is in some behavior mode. Behavior mode definitions are part of the character setup -- see [Behavior Modes](https://docs.speech-graphics.com/en/sgx/4.6/behavior-modes.md) for an overview. In short, a behavior mode contains a variety of expressions reflecting that mode. For example, if you have a behavior mode "Happy", then you would populate that mode with a variety of happy-looking expressions. When the character is in a behavior mode, it draws on expressions from that mode and deploys them automatically with timing and dynamics determined by SGX algorithms. Behavior modes may correspond to emotional or physical states, like "happy", "angry", "tired" or "struggling". The set of a character's behavior modes is open-ended: you can define as many as you wish. Every character has one default behavior mode that is active if none has been specified. Here is an example of an animation with changing behavior modes. The current behavior mode is printed above the character as she speaks (as a "mood"). ### Auto Modes While SGX allows creative direction, the goal is also to generate good animation at scale. For this reason, SGX uses AI to detect several vocal qualities, triggering automatic behavior modes called **auto modes**. The auto modes may be defined as any of the behavior modes you have designed for the character. There are currently three auto modes in SGX, with more on the way. * **Positive**- Happy or positive emotion is detected in the voice. * **Negative**- Negative or unhappy emotion is detected in the voice. This encompasses a range from sadness to anger. * **Effort**- A grunt or effort sound is detected in the voice. This may happen when the voice actor is portraying activities of physical exertion or bodily impact, typically involving a tightening of the vocal chords to make guttural non-speech sounds. * **Laugh** - Laughter is detected in the voice. Auto modes may be mapped to the behavior modes of your character, as shown in the image below. This mapping may be changed at any time. ![SGX_Mappings.png](https://docs.speech-graphics.com/__attachments/a_0c5ab3bc495d7c45bb09d7d22ae5d82fd6c3898635a24a8ca7ed861e4b32d4b9/SGX_Mappings.png?cb=7c4bd4109ad89c2cc150faafa2dcea70) It is up to the user which auto modes to activate. If an auto mode is not active, it will not trigger a change in behavior mode. ## Behavior Modifiers Behavior modifiers are numerical parameters that can be used to adjust particular dimensions of the performance, such as magnitude, speed and frequency of movements. Like behavior modes, these values can be applied to an entire batch of events, to an individual event, or to any sub-interval of an event. Behavior modifiers serve as "tweaks" to correct any issues seen in the automatic output. You should not normally need to use them unless there is something to correct. An overview of the behavior modifiers is presented in the following table, and each modifier is discussed in greater detail below. |---------------------------------------------------------------------------------|--------------------------------------------------------------|-------------------------|----------------------|--------------------| | **Modifier** | **Description** | **Minimum** | **Default (normal)** | **Maximum** | | [**Speech Magnitude**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Speech-Magnitude) | Scale factor applied to magnitude of all speech movements | 0.0 (no movement) | 1.0 | 2.0 | | [**Speech Speed**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Speech-Speed) | Scale factor applied to speed of all speech movements | 0.0 (no change) | 1.0 | 2.0 | | [**Hyperarticulation**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Hyperarticulation) | Boost articulatory effort | 0.0 (normal) | 0.0 | 1.0 | | [**Jaw Limit**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Jaw-Limit) | Upper limit on degree of jaw opening | 0.0 (jaw always closed) | 1.0 | 1.0 (unrestricted) | | [**Nonverbal Magnitude**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Nonverbal-Magnitude) | Scale factor applied to magnitude of all nonverbal movements | 0.0 (no movement) | 1.0 | 2.0 | | [**Nonverbal Speed**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Nonverbal-Speed) | Scale factor applied to speed of all nonverbal movements | 0.0 (no movement) | 1.0 | 2.0 | | [**Breath Magnitude**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Breath-Magnitude) | Scale factor applied to magnitude of all breath movements. | 0.0 (no movement) | 1.0 | 2.0 | | [**Breath Speed**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Breath-Speed) | Scale factor applied to speed of all breath movements. | 0.0 (no movement) | 1.0 | 2.0 | | [**Blink Frequency**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Blink-Frequency) | Scale factor applied to frequency of blinks | 0.0 (no blinks) | 1.0 | 2.0 | | [**Dart Frequency**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Dart-Frequency) | Scale factor applied to frequency of eye microdarts | 0.0 (no darts) | 1.0 | 2.0 | The modifiers **Speech Speed** and **Hyperarticulation** are effective only for phonetic processing, not acoustic processing. (See [Phonetic and Acoustic lip sync processing](https://docs.speech-graphics.com/en/sgx/4.6/audio-to-motion-processing.md#Phonetic-and-Acoustic-lip-sync-processing).) ### Speech Magnitude The **Speech Magnitude**modifier affects the amplitude of all speech movements. It's like a volume knob for lip sync. Generally, SGX attempts to match the magnitude of muscle movements to the intensity of the speech in the audio, syllable by syllable. However, if the animated movements are not emphatic enough (or a particular style is desired), you can use the Speech Magnitude modifier to amplify the speech; conversely if the movements seem too exaggerated given the audio, you can use this modifier to dial the speech down. Setting Speech Magnitude to 0 will effectively turn off lip sync. Note that increasing Speech Magnitude will not *always* result in a proportional increase in the amplitude of movements. The movements of speech are governed by a complex muscle-dynamic model, which requires each movement to be physically natural according to the model's limits on velocity and acceleration. This means that if you amplify the target magnitudes, the muscles might not have enough time to reach the desired degree of displacement given other things it is trying to accomplish on the timeline. One way around this limitation is to also increase [Speech Speed](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Speech-Speed) or [Hyperarticulation](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Hyperarticulation). Below are examples of Speech Magnitude set to three different values: * 1.0 (default) * 0.3 (=30% magnitude) * 2.0 (=200% magnitude) ### Speech Speed The speed of muscles is controlled by the muscle-dynamic model. Rarely, you might want to make them move faster, perhaps to accentuate a fast-talking character (though note that rapid speech is detected by the system and movements should be commensurate with the detected speech rate). The **Speech Speed** modifier is a global multiplier on the basic speed of speech movements. Below are examples of Speech Speed set to three different values: * 1.0 (default) * 0.3 (=30% magnitude) * 2.0 (=200% magnitude) ### Hyperarticulation *Hyperarticulation* describes a phenomenon in which the speaker is putting extra energy into making the articulatory organs move faster and farther in order to make the speech very clear. The **hyperarticulation**modifier is a corrective tweak that can be used when the animation does not correctly match the hyperarticulation heard in the audio. This may be warranted particularly in very rapid but still clear speech, where the speaker is putting extra effort into hitting articulatory targets in the shorter time available rather than letting the consonants and vowels be slurred. Generally speaking, hyperarticulation should be used only sparingly if at all; small adjustments go a long way. Too much hyperarticulation can result in jittery animation. Most of the modifiers have default 1.0 and range \[0.0, 2.0\]. However the default setting for hyperarticulation is 0.0 and its range is \[0.0, 1.0\]. Below are examples of Hyperarticulation set to four different values: * 0.0 (default) * 0.2 * 0.5 * 1.0 ### Jaw Limit The **Jaw Limit** modifierallows you to restrict the degree of jaw opening permitted. For example, this could be used to keep the teeth clenched. The lower the value, the less the jaw is allowed to open. At 1.0 there is no restriction, and at 0.0 the jaw will stay completely closed. Jaw Limit is designed to impede speech movement, so consider this when applying it to normal speech. Below are examples of Jaw Limit set to three different values: * 1.0 (default) * 0.7 * 0.1 ### Nonverbal Magnitude The **Nonverbal Magnitude** modifier affects the amplitude of all nonverbal movements. It is thus complementary to [Speech Magnitude](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Speech-Magnitude), which affects only speech muscles. Generally, SGX matches the magnitude of nonverbal expressions to the intensity of the speech in the audio, phrase by phrase. However if the animated movements are not emphatic enough, you can increase the Nonverbal Magnitude to amplify the speech; conversely if the movements seem too exaggerated given the audio, you can decrease the modifier to dial the nonverbal movements down. Setting Nonverbal Magnitude to 0 will effectively turn off nonverbal animation. We recommend Nonverbal Magnitude and Speech Magnitude are modified together. In the examples below, *both*of these modifiers are set to the following values: * 1.0 (default) * 0.3 * 2.0 ### Nonverbal Speed **Nonverbal Speed** is a global modifier affecting the basic speed of nonverbal muscle movements. It is analogous to the [Speech Speed](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Speech-Speed) modifier, which operates only on speech muscles. Generally, the speed of muscles is controlled by the muscle-dynamic model. Rarely, you might want to make them move faster, e.g. in the case of a particularly sharp reaction that isn't captured in the automatic animation. This modifier will achieve that. ### Breath Magnitude The **Breath Magnitude**modifier affects the intensity of breath. Adjust this if you want to have deeper or more shallow breathing than what is automatically generated. ### Breath Speed The **Breath Speed** modifier affects the speed of breath movements, as well as the rate of breathing during idles (faster breath leads to more breaths per minute). Increase this to make the character appear more out of breath; decrease it to make the character appear calmer. ### Blink Frequency Blinks are automatically generated in SGX using a model of human blink patterns. The probability of a blink at any time is conditioned by audio properties and other activity in the face. However, you can alter the base frequency of blinks using the **Blink Frequency**modifier. If you'd like the character to blink less frequently, set the value to below 1.0; if you'd like the character to blink more frequently, set it to above 1.0. Setting Blink Frequency to 0 will effectively turn off blinks. ### Dart Frequency Eye microdarts in SGX refer to small horizontal saccades (rapid eye movements) that simulate looking between the two eyes of an interlocutor. This behavior supports the lifelikeness of our character animation. Like blinks, microdarts are automatically generated in SGX using a model of human eye movement patterns. The probability of a microdart at any time is conditioned by audio properties and other activity in the face. However, you can alter the base frequency of darts using the **Dart Frequency**modifier. For example, if you feel the character's eyes are darting too frequently, set the value to below 1.0. If you feel the character's gaze is too steady, set it to above 1.0. Setting Dart Frequency to 0 will effectively turn off eye microdarts. --- version: "SGX 4.6" language: "en" --- # Behavior Modes Behavior modes make it easy to determine an entire behavior pattern with a single label. At any point in time during animation, your character is in some behavior mode or other. Behavior modes are a powerful creative tool for designing behaviors of the character at setup time. Behavior modes may correspond to emotional or physical states, like "happy", "angry", "tired" or "struggling". The set of a character's behavior modes is entirely user-defined and open-ended: you can create as many as you wish (see [SGX Studio](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-plugins.md) product guides on how to set up behavior modes). Your initial character control file will include a set of default behavior modes, which correspond to the automatically detected modes of the system: Positive, Negative, Effort, and Neutral (See [SGX Auto Modes](https://docs.speech-graphics.com/en/sgx/4.6/behavior-controls.md#Auto-Modes).) Here is an example of an animation with changing behavior modes. The current behavior mode is printed above the character as she speaks (as a "mood"). ## Behavior mode definitions Essentially, a behavior mode definition contains a variety of nonverbal poses -- which we call "expressions" -- that are exemplary of that mode. For example, if you have a behavior mode "Happy", then you would populate that mode with a variety of happy-looking expressions. When the character is in a behavior mode, it draws on expressions from that mode and deploys them automatically with timing and dynamics determined by Speech Graphics algorithms. By this process the face is continuously changing, autonomous and alive. When constructing a behavior mode it is important to populate it with a number of different expressions. One may be tempted to use a single prototypical expression to represent the entire emotion or state. However, this can result in repetetive and predictable behavior, which is highly unnatural. Using a sufficent number of alternate expressions ensures that the character appears organic rather than mechanical. Keep in mind that real people never make the exact same facial expression twice. For example, below is a set of expressions created for the behavior mode "Effort". ![image-20230712-172946.png](https://docs.speech-graphics.com/__attachments/a_3a58eda1019997ca8c17b825444cf3c00bac9ab7284b6b567a457aa6da8412e4/image-20230712-172946.png?cb=2f49313c7ebfccf98d11f8913421fba6) ![image-20230712-173046.png](https://docs.speech-graphics.com/__attachments/a_e2a0332e5c66eaf4e3590f143a6e0fd8e27ad694ffd889c4846cba3641b6477e/image-20230712-173046.png?cb=342c8cc6a04d853542ad59ff275278aa) ![image-20230712-173222.png](https://docs.speech-graphics.com/__attachments/a_14226211da380151c631efff5c261c20592154e5514fe8fff7921f6624a7bfc5/image-20230712-173222.png?cb=886f1bb2926464d2fdf6d7bd286535a0) ![image-20230712-172814.png](https://docs.speech-graphics.com/__attachments/a_6a6bc061cf9e15b93b4f9ebae0f5a7c83dc544aab3853ec761c404168d64a1fb/image-20230712-172814.png?cb=1f30b44b68b24ae4984e13bb09ee40f1) ![image-20230712-173009.png](https://docs.speech-graphics.com/__attachments/a_d017cfaa215e5389a7fb3b599231798cef0b43f9013b4dfaee930b32595594d8/image-20230712-173009.png?cb=391e6d416088288b38ad08849a9cf83d) ![image-20230712-173058.png](https://docs.speech-graphics.com/__attachments/a_5d7a36f8b0ea7098649211f05b097828c143f42c1ece167a13f0d8f6c06feaa9/image-20230712-173058.png?cb=acdc848a6f7fa1c9da1cccccba6943a9) ![image-20230712-172828.png](https://docs.speech-graphics.com/__attachments/a_d47315f68d01bbff8aaa8850a02a94169e2585c0b878fbc85267c02b6a96a7f2/image-20230712-172828.png?cb=74ef8a0d6ba3cad7eca28e84fb9e748d) ![image-20230712-173028.png](https://docs.speech-graphics.com/__attachments/a_fa12a8e1a63bb362c505a85e236d8cbd2e79b6a100165e013282cecf3fd60a5e/image-20230712-173028.png?cb=a2d0005adc6d592cc1241de6691503ba) ## Expression definitions Each individual expression is defined as a pose of the character's [muscles](https://docs.speech-graphics.com/en/sgx/4.6/muscles.md). For example, the following expression consists of the muscle activations outlined in the table below. ![image-20230711-145736.png](https://docs.speech-graphics.com/__attachments/a_c78039183e5e644cc398a083925909110f67f42b1b19fd790d216917a85a0b9c/image-20230711-145736.png?cb=67a2b5dc4f78ac895c5cf76d2fa8f6c3) |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------| | brow_lower_L brow_lower_R brows_in cheek_raise_L cheek_raise_R cheek_raise_outer_L cheek_raise_outer_R eye_squint_L eye_squint_R eyeball_pitch eyeball_roll head_pitch head_roll head_yaw inner_brow_lower_L | 0.94 0.99 1.00 0.20 0.24 0.60 0.55 0.56 0.54 -0.06 1.00 -0.25 -0.21 0.26 0.48 | inner_brow_lower_R jaw_shift_lateral larynx_raise lip_corner_depress_L lip_corner_depress_R lower_eyelid_flex_L lower_eyelid_flex_R neck_tense_R nose_wrinkle_L nose_wrinkle_R nostril_compress nostril_flare outer_brow_raise_R retraction_L retraction_R | 0.35 0.04 0.55 0.30 0.28 0.33 0.36 0.17 0.28 0.09 0.09 0.32 0.22 0.03 0.05 | Thus whereas each muscle is formed by setting the rig's animation targets, each expression is formed by setting the muscles. This hierarchical relationship is shown in the diagram below: ![hierarchy.png](https://docs.speech-graphics.com/__attachments/a_b64bf5eefd595d5f9e0438eb8652747eb34821714860c9a248bfda42929a3301/hierarchy.png?cb=16220102d2d0451c3a2b35b0a8b8cec4) Expressions should involve the entire face, as this keeps the face alive and prevents any "dead" areas. Furthermore, expressions are not restricted to the face. They can include head motion and other parts of the body as well. Using the full musculature of the character in defining expressions keeps the character feeling alive. During animation, expressions are automatically selected, timed and blended by the muscle-dynamic system. The expressions are also scaled according to the intensity of the speech. For example, the expression from above is shown here at different scales. ![image-20230712-171311.png](https://docs.speech-graphics.com/__attachments/a_a6ceb73c021be0e7ffecc7e133ec366d4cd4b7ab75963f82586065f520a88d64/image-20230712-171311.png?cb=e219f7cb39ff4f83dc6589fa42e73b4a) 0% ![image-20230712-171032.png](https://docs.speech-graphics.com/__attachments/a_ae2c82a68902e2af4cc7a244f00a62912315f6ade9e654eeac0fadc8c6101b54/image-20230712-171032.png?cb=c10988fb6c9778f9f3f6017390e0024f) 20% ![image-20230712-171058.png](https://docs.speech-graphics.com/__attachments/a_fbb851f6d185c7bd43c025789e8de4872dacd8ad853992fdc0e343ae6277bf83/image-20230712-171058.png?cb=a412d4a0112825856cd8c39d5d3bb30d) 40% ![image-20230712-171115.png](https://docs.speech-graphics.com/__attachments/a_652b14a9faea0214cb4cf97cbe2923c98a57494342b2d36ed3e359ff2693d6ac/image-20230712-171115.png?cb=2ffc61c621c4fd857bd3265189a73ad4) 60% ![image-20230712-171135.png](https://docs.speech-graphics.com/__attachments/a_d85cab845cba64230918c9b61b380884ecb3d679794b957d7ad59e346cd1ef3a/image-20230712-171135.png?cb=c709c31ddf72c7d678c498f6e3086312) 80% ![image-20230711-145736.png](https://docs.speech-graphics.com/__attachments/a_c78039183e5e644cc398a083925909110f67f42b1b19fd790d216917a85a0b9c/image-20230711-145736.png?cb=67a2b5dc4f78ac895c5cf76d2fa8f6c3) 100% However, when defining a new expression, it is essential to pose it at the maximum intensity conceivable for that expression. If expressions are posed too weakly, then they will result in not very expressive animation. Note that in actual animation you will probably never encounter an expression at 100%. But if the maximum pose is not truly maximum, then by extension all lower scales of the expression will appear weaker than the system intends, and there will be degrees of intensity the character cannot reach when desired. When defining an expression, pose it at its maximum intensity. Expressions are a good way to achieve idiosyncratic behavior in the character. Assymetries and other quirks can be accomplished easily using expressions. ### Importing expressions Expressions consist of poses of the Speech Graphics muscles. However, at times you may have poses of a character in terms of the rig parameters themselves -- for example as a result of motion capture or hand animation -- which you want to use as expressions. To do this, you may import the poses into the muscle space: [SGX Studio](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-plugins.md) provides a tool which finds the best fit of the muscle activations to a current pose of the [animation targets](https://docs.speech-graphics.com/en/sgx/4.6/animation-targets.md) on the rig. Note, this does require that the controls you use to create the pose are included among the animation targets. ## Default behavior mode Every character has one *default behavior mode* that is active if no other mode has been specified or automatically triggered. Normally, the default mode should be one in which the emotion is neutral. A neutral mode can still have expressions, but should not express any intense mood or physical state. ## Bi-tonal behavior modes Behavior modes may be either bi-tonal or uni-tonal. Uni-tonal modes contain a single pool of expressions. Bi-tonal modes contain two pools: **high-tone** expressions and **low-tone** expressions. The distinction is based on speech intonation patterns. During processing, high-tone expressions are triggered during high or rising pitch, and low-tone expressions are triggered during low or falling pitch. The expressions in the two groups should have contrasting characteristics. The following characteristics are recommended: |------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | **High-tone expressions**⬆️ | **Low-tone expressions**⬇️ | | Energetic Face opening Brows raising Eyes widening Head lifting Eyes lifting | Subdued Face contracting Brows lowering Eyes narrowing Head lowering Eyes lowering | By causing expressions to change quality as the voice rises and falls, bi-tonal modes make the facial animation more responsive to vocal changes and more closely aligned with speech patterns. This gives a more a convincing, coherent and expressive performance overall. ## Behavior mode best practices Here is a summary of the best practices with respect to authoring behavior modes: * *Do* make behavior modes bi-tonal, especially modes that are expected to be used during speech. * Pose each expression with the maximum intensity the expression could conceivably have on a real person's face. * Use a good number of expressions per mode to support variation and avoid repetitive behavior. A good rule of thumb is 7 or more expressions per tone group. * Move various parts of the musculature in the expressions to keep the character alive. No part of the character should be continuously still. Include head muscles in expressions to generate head motion. Also use expressions to drive non-human anatomy like anntennae. * Add assymetries and other idiosyncratic quirks to expressions. * When using assymetry, avoid randomly switching sides. For example, if the character raises the left eyebrow sometimes and the right eyebrow other times, this may cause odd-looking alternations. Keep assymetries natural and one-sided. * When creating behavior modes that will be used during speech, avoid extreme displacements of the lip muscles, which may interfere with speech; avoid jaw opening altogether. --- version: "SGX 4.6" language: "en" --- # Blueprint Nodes (Runtime UE) The Runtime Unreal Engine 5 Plugin provides several Blueprint nodes, which are described here. All Blueprint nodes require a **FaceFX Component** to be wired to their respective **Target** slots. The Setup Blueprint node requires the **Skel Mesh Comp** slot to be wired with the **Skeletal Mesh Component** to animate. The **Skel Mesh Comp** is optional for the other Blueprint nodes (this is so a single Blueprint can drive multiple characters). If it is not supplied, the first one found is used. ## Setup ![FaceFXSetupBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_a016e27028193a0718153defd23f4eacdba364b70343364686bb7379be0e8809/FaceFXSetupBlueprintNode.png?cb=0e113b7204b11b68c36368ded08a1142) The Setup Blueprint node initializes the linked **FaceFX Component** with the linked **Skeletal Mesh Component** and specified **FaceFXActor** asset. It must be called in the **Construction Script** before any other Blueprint node. * The **Target** slot is required and should be wired to the **FaceFX Component**. * The **Skel Mesh Comp** slot is required and should be wired to the **Sekeltal Mesh Component** to animate. * The **Audio Component** property should be set to the Audio Component used to play the audio. * The **Asset** property is required and should be set to the **FaceFXActor** asset to be used for animation. * The **Compensate For Force Front XAxis** property should only be **checked** if **Force Front XAxis** was **checked** when the Skeletal Mesh was imported. Not keeping these settings in sync will result in animations playing in the wrong coordinate system and can resemble an exploded face. * The **Automatically Play Sound** property is optional. When **checked** , the audio associated with the **FaceFXAnim** asset being played will automatically be started at the appropriate time. * The **Ignore Events** property is optional. When **checked**, events will be ignored. This is irrelevant for SGX. ## Play ![FaceFXPlayBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_74a57e1bf094562b4810d7ce83dd9aa874ae418313f2155984cc5c40f0fd27d8/FaceFXPlayBlueprintNode.png?cb=3750565f66468a3f6d0202a60449c995) The Play Blueprint node plays the linked **FaceFXAnim** asset. * The **Target** slot is required and should be wired to the **FaceFX Component**. * The **Animation** property is required and should be set to the **FaceFXAnim** asset to be played. * The **Skel Mesh Comp** slot is optional. It should be wired to the **Skeletal Mesh Component** to animate. If it is not set, the first **Skeletal Mesh Component** found is used. * The **Loop** property is optional. When set it causes the **FaceFXAnim** asset being played to loop continuously until manually stopped. ## Play by Id ![FaceFXPlayByIdBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_34a4ce62909c78d22024a92b6ccb025d58b6a6b8ba76ba055b2a52ffd308498e/FaceFXPlayByIdBlueprintNode.png?cb=28071e9316ce8f4e0ec27f0198f7c59f) The Play by Id Blueprint node plays a **FaceFXAnim** asset using a string ID created in the form **Group/AnimName** . The **FaceFXAnim** asset *must* be linked to the **FaceFXActor** asset in order for it to be found and played. * The **Target** slot is required and should be wired to the **FaceFX Component**. * The **Group** property is optional. If set, it specifies the animation group that contains the animation to be played. If not set, the first animation linked to the **FaceFXActor** asset found with the same name specified by the **Anim Name** property is played. * The **Anim Name** property is required. It specifies the name of the animation to play. * The **Skel Mesh Comp** slot is optional. It should be wired to the **Skeletal Mesh Component** to animate. If it is not set, the first **Skeletal Mesh Component** found is used. * The **Loop** property is optional. When set it causes the **FaceFXAnim** asset being played to loop continuously until manually stopped. ## Pause ![FaceFXPauseBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_d1609f84d2b44723a43d2398d39557a74c750ee3bb2236ee779e6fb7124d91ab/FaceFXPauseBlueprintNode.png?cb=dc7847b6fa209306a5e780623d3b4c61) The Pause Blueprint node pauses all animations and audio files playing on an actor. * The **Target** slot is required and should be wired to the **FaceFX Component**. * The **Skel Mesh Comp** slot is optional. It should be wired to the **Skeletal Mesh Component** to animate. If it is not set, the first **Skeletal Mesh Component** found is used. ## Resume ![FaceFXResumeBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_dc140fb02d5585bd6b42615e036f81e400c7bd4f0ddd6fe75ccccc16c315a201/FaceFXResumeBlueprintNode.png?cb=ccc5f81926851aee5fadf03e27d4429d) The Resume Blueprint node resumes all paused animations and audio files. * The **Target** slot is required and should be wired to the **FaceFX Component**. * The **Skel Mesh Comp** slot is optional. It should be wired to the **Skeletal Mesh Component** to animate. If it is not set, the first **Skeletal Mesh Component** found is used. ## Stop ![FaceFXStopBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_070dff852e2d688c1632bf66d78224ad0f7da3e03c0f786ea2d3bb1e7de532f4/FaceFXStopBlueprintNode.png?cb=dd09afda4e00021dfc7af4ea53544090) The Stop Blueprint node stops all animations and audio files playing on an actor. * The **Target** slot is required and should be wired to the **FaceFX Component**. * The **Skel Mesh Comp** slot is optional. It should be wired to the **Skeletal Mesh Component** to animate. If it is not set, the first **Skeletal Mesh Component** found is used. ## On Animation Event ![OnAnimationEvent.png](https://docs.speech-graphics.com/__attachments/a_265f40571e8437b4a0e424a4865bba70a036c1effdbb151d4b79ad6a3b4bfaa1/OnAnimationEvent.png?cb=03456f87c8919a48f3a11531e4615d06) The **On Animation Event Node** in the **Event Graph** is created by selecting the **FaceFX Component** from the Blueprint and adding the **On Animation Event** from the **Events** section of the **Details** tab. Use this node to make use of the event payload and other event properties (the Skeletal Mesh Component, Animation Id, Channel Index, Channel Time, and Event Time) in your game. This is irrelevant for SGX. --- version: "SGX 4.6" language: "en" --- # Categorical Time Series View [Categorical Time Series](https://docs.speech-graphics.com/en/sgx/4.6/time-series-types.md#Categorical-Time-Series) are the predominant form of metadata in events. This data content is highly simplified, consisting of just labeled time intervals, which makes it easy to edit. The primary metadata sequences of this type include: * [word alignment](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Word-alignment) * [phone alignment](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Phone-alignment) * [lip sync](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Lip-sync) * [behavior mode](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Behavior-mode) * [expressions](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Expressions) ## Display The display of a [Categorical Time Series](https://docs.speech-graphics.com/en/sgx/4.6/time-series-types.md#Categorical-Time-Series) shows labeled segments with boundaries; for example: ![image-20240108-182215.png](https://docs.speech-graphics.com/__attachments/a_d9e149dda5c9c5d0c52929d6574509a3395d468604e5b0a2206eacb5b5f775e0/image-20240108-182215.png?cb=0e57cbe916b6babaacda7bc0826c2d44) ![image-20240108-175455.png](https://docs.speech-graphics.com/__attachments/a_46dd81296c67aa53dfce05e00c85134054695e04e4df22c80bef3dfe926eb675/image-20240108-175455.png?cb=2362c2e52ec5d65290beb7a31b067f14) The commonly seen caret symbol `^` indicates an absence, such as a pause. When a Categorical Time Series is currently selected, the *Sequence* section of the [Inspector](https://docs.speech-graphics.com/en/sgx/4.6/navigating-events.md#Inspector) shows this information: ![image-20240108-182406.png](https://docs.speech-graphics.com/__attachments/a_f8efa210ce4d2774ac8c9a144638890c167b7ba9b1ab2d158e68f302d13c8f84/image-20240108-182406.png?cb=ae9c5d7c7418ad117a6065e246b53fa2) The properties include: * the **height**, which is the number of tiers * the label of the currently selected segment, if any ## Boundary and segment selection **Selection** in a Categorical Time Series is useful for editing. You can select both boundaries and segments, as shown in the video below: [2024-01-07 20-29-37.mp4](https://docs.speech-graphics.com/__attachments/a_67b185ef68f3f6b61f010295a60acc592fbcd26de660cbc4cc48e89e41bd6d4a/2024-01-07%252020-29-37.mp4.md?cb=a07535a112d44ddc0116872183956f97) Selecting boundaries or segments When a segment is selected, the arrow keys on the keyboard may be used to move the selection to adjacent segments. The left and right arrow keys move the selection to the preceding or following segments on the same tier; the up and down keys move the selection between tiers (if there is more than one tier). This is illustrated below. [2024-01-07 20-32-46.mp4](https://docs.speech-graphics.com/__attachments/a_947f7c266f4a582e6de5552b2c531e43877b2a274c2217ae4a6c26d30dd10672/2024-01-07%252020-32-46.mp4.md?cb=4867ed63205a22c1fc0f8bac7551b765) Moving the segment selection in a Categorical Time Series using arrow keys When a segment is selected, its label is shown in the Inspector: ![image-20240108-182706.png](https://docs.speech-graphics.com/__attachments/a_0279ffd161a70abfec1b6602434aa95a8f652ef05984f42e344b644891a64a01/image-20240108-182706.png?cb=2c6cc628612f251beae48d62dc3f1ab5) ## Moving boundaries In order to move a boundary: 1. [Select](https://docs.speech-graphics.com/en/sgx/4.6/categorical-time-series-view.md#Boundary-and-segment-selection) the boundary. The selected boundary is now red. 2. Hover over the selected boundary. The cursor will change to a two-sided arrow. ![boundary_cursor.png](https://docs.speech-graphics.com/__attachments/a_11200ecba5338eac922086a1f35990ea007aaa3bb5e83806ccd8119d98e72a76/boundary_cursor.png?cb=e36c56480087c5a603c895d8c8e07201) 3. Click and drag the boundary left or right. ## Inserting boundaries A Categorical Time Series boundary may be inserted at a boundary of the current time selection. These are the steps: 1. Select an interval or time point on the timeline (see [Time selection](https://docs.speech-graphics.com/en/sgx/4.6/navigating-events.md#Time-selection)). 2. Within the Categorical Time Series, hover over the boundary of the time selection. The hovered portion will turn green. 3. Double-click the hovered portion to create the boundary. Inserting a boundary inside a segment divides it into two segments, each with the same label as the original. This is typically followed by [relabeling](https://docs.speech-graphics.com/en/sgx/4.6/categorical-time-series-view.md#Relabeling-segments) one of the segments. The video below shows boundary insertions in action. [2024-01-14 22-18-57.mp4](https://docs.speech-graphics.com/__attachments/a_c055c1623df2656fcc0a879c81005e33a90b74534fe163b8528be68f420f24fd/2024-01-14%252022-18-57.mp4.md?cb=57e2bfc3e1e8f13549f7642eca5a2db9) Inserting boundaries ## Deleting boundaries To delete a boundary: 1. [Select](https://docs.speech-graphics.com/en/sgx/4.6/categorical-time-series-view.md#Boundary-and-segment-selection) the boundary. The selected boundary is now red. 2. Press the *delete*key on your keyboard. When a boundary between two segments is deleted, the two segments are merged into one, and take the label of the right segment. ## Deleting segments To delete a segment: 1. [Select](https://docs.speech-graphics.com/en/sgx/4.6/categorical-time-series-view.md#Boundary-and-segment-selection) the segment. 2. Press the *delete*key on your keyboard. When a segment is deleted, it is merged into the segment to its right. ## Relabeling segments Segment labels may be changed using the Inspector. Following are the steps: 1. [Select](https://docs.speech-graphics.com/en/sgx/4.6/categorical-time-series-view.md#Boundary-and-segment-selection) the segment. 2. In the [Inspector](https://docs.speech-graphics.com/en/sgx/4.6/navigating-events.md#Inspector) *Sequence*properties, you will find the label of the currently selected segment on a button. Click the button to select a new label from a list. For [behavior mode](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Behavior-mode) and [expressions](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Expressions), the label lists are populated with options from the currently selected Character Control File (see [Resources](https://docs.speech-graphics.com/en/sgx/4.6/resources-1.md)). In the example below, a behavior mode segment is selected, and the list of options is populated with all the behavior modes of the current character. ![select_mode.png](https://docs.speech-graphics.com/__attachments/a_f8cfb56c4c94b971f8f56c4f5753feb3f4c0efb88eb54138d7781469eb2ab77f/select_mode.png?cb=82e80375e9cb18e26b11ecd9a331ddb5) While editing behavior modes and expressions, make sure that the currently selected [character](https://docs.speech-graphics.com/en/sgx/4.6/resources-1.md#Character)is the one for which the event is intended, so that the label menus are populated with the right options. ## Relabeling expression segments The [Expressions](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Expressions) sequence provides special functionality for efficient editing of expressions. If the event is *bound* to a running instance of SGX Studio (see [Playing Events](https://docs.speech-graphics.com/en/sgx/4.6/playing-events.md) on how to bind events to SGX Studio for visualization), then whenever an expression is selected in the event, the corresponding expression pose is displayed on the character. The video below demonstrates -- using an event bound to [SGX Studio in Unreal Engine](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-unreal-engine.md) -- how to preview expressions in the event, preview potential replacements, and choose the replacement. [2024-01-15 00-34-28.mp4](https://docs.speech-graphics.com/__attachments/a_dd887f66f499769323e5aa89aa32f730cd1fc0978e5b25457e35a6c025095ff4/2024-01-15%252000-34-28.mp4.md) ## Editing alignment In the specific cases of [word alignment](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Word-alignment) and [phone alignment](https://docs.speech-graphics.com/en/sgx/4.6/metadata.md#Phone-alignment), note that editing has some restrictions: insertions, deletions, and label changes are generally not allowed; only moving of boundaries is permitted. This protects adherence of these analyses to the transcript. However, in the word alignment, it is possible to insert and delete *pause*segments. To insert a pause, select a word segment, and in the menu, select either of the following: * **Edit** \> **Insert Pause Before Word** * **Edit** \> **Insert Pause After Word** --- version: "SGX 4.6" language: "en" --- # Character Assets (UE) The SGX Studio UE plugin always requires two assets: * A Speech Graphics [++Character Control File++](https://docs.speech-graphics.com/en/sgx/4.6/character-control-file.md#Character-Control-File) * An Unreal asset -- either a skeletal mesh or blueprint ## Character control file At present, all character control files must be initiated in [++SGX Studio Maya++](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-maya.md), or by Speech Graphics [++Setup Services++](https://docs.speech-graphics.com/en/sgx/4.6/character-control-file.md#Setup-services). ## Unreal asset The Unreal asset will be the target for imported animation sequences, and for character setup manipulation widgets. The plugin accepts either of two asset types: a skeletal mesh or a blueprint. A **skeletal mesh** can be chosen in order to target a single-mesh character or a single mesh from a modular character. **Blueprints** can be selected for modular characters comprising multiple skeletal meshes, such as MetaHumans. When using a blueprint, SGX Studio will find the first skeletal mesh in the blueprint and use it as the target mesh. To change the target mesh, click **File** \>**Select Target Skeletal Mesh**in the plugin, which presents a list of available meshes in the blueprint. For MetaHumans created in Unreal Editor 5.6+, Blueprints must be used. Skeletal meshes are no longer supported. --- version: "SGX 4.6" language: "en" --- # Character Setup Process ## Character Control File Animating a character with Speech Graphics software requires a one-time setup of a unique binary file called the **character control file** (`.k`). This file contains all the information that determines how our animation systems work with your rig, including muscle definitions and behavioral characteristics. Contents of the control file: * [**Animation targets**](https://docs.speech-graphics.com/en/sgx/4.6/animation-targets.md) - A list of Maya node attributes (blendshapes, joints, and/or UI controls) which SG muscle poses will drive. *Note that a character control file can contain two levels of animation targets: controls (for use in Maya) and deformers (for game engines). See* [*Animation Levels*](https://docs.speech-graphics.com/en/sgx/4.6/animation-targets.md#Animation-levels)*for more information.* * [**Neutral pose**](https://docs.speech-graphics.com/en/sgx/4.6/neutral-pose.md) - The pose the character returns to when all muscles are relaxed. * [**Muscles**](https://docs.speech-graphics.com/en/sgx/4.6/muscles.md) - The fundamental facial poses that SGX activates to produce animation. * [**Closures**](https://docs.speech-graphics.com/en/sgx/4.6/training.md)- A set of consonant poses involving direct contact between two parts of the oral anatomy. * [**Behavior modes**](https://docs.speech-graphics.com/en/sgx/4.6/behavior-modes.md)- A library of non-verbal facial expressions. ## Plugin support All stages of the character setup process are carried out in SGX Studio. SGX Studio plugins are available in [Maya](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-maya.md) and the [Unreal Editor](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-unreal-engine.md), and an API is available. A character control file can be created in one SGX Studio plugin and edited in another. Currently, the ability to edit muscles is available only in Maya; for this reason, the character setup process must be started in Maya until the muscles are defined. Rigs can be created in other 3D applications such as 3DS Max, Blender and MotionBuilder, but must be exported into Maya. Transferring rigs from other applications into Maya normally only works for simple rigs without a complex rig UI. For example, a basic bone rig can be easily transferred between 3DS Max and Maya, using FBX. It is also possible to transfer animation between platforms provided skinning information and other elements are the same (see [Integrating Animation](https://docs.speech-graphics.com/en/sgx/4.6/exporting.md#Integrating-animation)). ## Setup services To ensure the best quality, character control files can be initialized by our team of trained animators, technical artists and animation linguists who have extensive knowledge of Speech Graphics' core technology. You can then further edit the setup yourself in the [SGX Studio](https://docs.speech-graphics.com/en/sgx/4.6/sgx-studio-plugins.md) plugin. As your character's rig or persona changes over time, its control file can be repeatedly edited accordingly. In order to set up your characters, Speech Graphics will require access to your character assets, along with other information. ### Rig delivery checklist **A character delivery package for Speech Graphics should include the following:** * [ ] Your [final](https://docs.speech-graphics.com/en/sgx/4.6/character-control-file.md#Final-rig) Maya rig and the Maya version (for more information, see our [Rigging Guidelines](https://docs.speech-graphics.com/en/sgx/4.6/rigging.md)). * [ ] If you plan to use breath animation, include your final body rig as well as the face rig. * [ ] All face textures and shaders pre-mapped with a workspace.mel included. * [ ] Any plug-ins and additional files that are needed for the rig to function. * [ ] A list of [animation targets](https://docs.speech-graphics.com/en/sgx/4.6/animation-targets.md), these can be UI controls, skeleton joints or blendshapes. If possible, provide a selection set in Maya and name it "SG_targets". * [ ] Audio (and corresponding transcripts) that you want the character to be tested with. **If you want to utilize the dual animation levels (needed for game engines), also provide the following:** * [ ] A list of baked level animation targets. If possible, provide a selection set and name it "SG_baked_targets" in your Maya scene. * [ ] If using Unreal Engine or Unity, specify the version for testing, and include your character's FBX imported into the engine. **Additional helpful information:** * [ ] Reference animations or footage showing the animation style and behavioral traits you want SGX to match. **Let us know if you do** ++**not**++ **want us to include any of the following:** * [ ] head and neck motion * [ ] eye darts * [ ] breath animation ### Final rig Character rigs should be final before delivery to Speech Graphics for setup. As with any animation system, if the target rig is changed, then animation applied to the rig controls will look different and may be broken. Changes to the rig may require changes to the character control file, which may incur additional time and cost. The final rig includes the final face mesh, UI controls, skinning, blendshapes, and bind pose that will be used in engine. --- version: "SGX 4.6" language: "en" --- # Character Setup Driving characters with SG * [Character Setup Process](https://docs.speech-graphics.com/en/sgx/4.6/character-control-file.md) * [Animation Targets](https://docs.speech-graphics.com/en/sgx/4.6/animation-targets.md) * [Neutral Pose](https://docs.speech-graphics.com/en/sgx/4.6/neutral-pose.md) * [Muscles](https://docs.speech-graphics.com/en/sgx/4.6/muscles.md) * [Closures](https://docs.speech-graphics.com/en/sgx/4.6/training.md) * [Behavior Modes](https://docs.speech-graphics.com/en/sgx/4.6/behavior-modes.md) * [Rigging Guidelines](https://docs.speech-graphics.com/en/sgx/4.6/rigging.md) --- version: "SGX 4.6" language: "en" --- # Character Setup (Runtime UE) In order for the Runtime Unreal Engine 5 Plugin to control a character, the character first needs to be properly set up with an **Animation Blueprint** and a **Blueprint Class**. * Create an **Animation Blueprint** for the character and add a **Blend FaceFX Animation** node. The **Blend FaceFX Animation** node needs to be connected to the **Final Animation Pose** node via a **Component To Local** node. ![BlendFaceFXAnimationBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_7229b3b2842d847a7f860a29863a08d5db9725c38e72a1680be96f825506b415/BlendFaceFXAnimationBlueprintNode.png?cb=964adce7238771ca864c115b06faad41) * Create a **Blueprint Class** for the character that is based on **SkeletalMeshActor** and references the **Animation Blueprint** created in the previous step. Add a **FaceFX** component and an **Audio** component to the **Blueprint Class**. ![FaceFXAndAudioComponents.png](https://docs.speech-graphics.com/__attachments/a_b6a1f1534e740bd7e9d5e4f03e977be8d8d06e123397ec6fe6fda9622afd6745/FaceFXAndAudioComponents.png?cb=fca5782c1073a3c10b084e8b0c66eec4) * In the **Construction Script** of the **Blueprint Class** created in the previous step, add a **Setup FaceFX** node. Drag the **Skeletal Mesh Component** from the **Components** list into the **Construction Script** and wire it up to the **Setup FaceFX** node's **Skel Mesh Comp** slot. Finally, wire the **Construction Script** node to the **Setup FaceFX** node. ![FaceFXSetupBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_bc84dd441cf2e53244a12e0912cc2aa46e6d8dd661faac98c9f7dd3a8e14a917/FaceFXSetupBlueprintNode.png?cb=0e113b7204b11b68c36368ded08a1142) * Animations can be played with **Play** or **Play By Id** FaceFX Blueprint nodes if they are compatible with the actor. A compatible animation is one that was compiled with the actor or compiled with an actor having an identical Face Graph setup. ![FaceFXPlayBlueprintNode.png](https://docs.speech-graphics.com/__attachments/a_fc72e5e84ca1cacb86824138ad9c850b786f9ada607754453c3a0653b5bd4fff/FaceFXPlayBlueprintNode.png?cb=3750565f66468a3f6d0202a60449c995) --- version: "SGX 4.6" language: "en" --- # Commands ## Help command The **help**command prints version/copyright information and usage information. ### Syntax SGX_Producer.exe help ## About command The **about**command prints version/copyright information, and opens a web page in your default browser listing notices and additional terms for licensed third-party software components included within SGX. ### Syntax SGX_Producer.exe about ## Process command The **process** command executes audio-to-animation processing on a given batch. This command can be launched alternatively from the SGX Director [batch tab](https://docs.speech-graphics.com/en/sgx/4.6/batch-processing.md#The-Batch-tab). As discussed in [SGX Batch Processing](https://docs.speech-graphics.com/en/sgx/4.6/sgx-batch-processing.md), there are two batch processing modes: * process new events * process existing events In SGX Producer, the default mode is *process existing events* . In order to *process new events* , use the `-i` option, which specifies the location from where new audio and transcripts will be imported into new events. ### Syntax SGX_Producer.exe process -e -c [-i ] [-f] [ -L ] [ -l ] [ -b behavior_settings ] [-s ] [-P ] [-Q ] [-A] [-N] [-E] [-M] [-D] [-r ] [-v] [-x ] ### Options Options without arguments are booleans --- true if present, false otherwise. |-----------------------------|------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Option** | **Status** | **Argument** | **Description** | | `-e [ --events ]` | required | Path | If using `--import` option (processing new events), this is the path to the directory where new events will be output. Otherwise (processing existing events), this is the path containing the input events. This may be a directory or a file containing a list of event files paths. | | `-c [ --character ]` | required | Path | The path to the character control file. See [Processing Configuration - Character](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Character). | | `-i [ --import ]` | optional | Path | The path containing input audio and transcript files to import. This may be a directory or spreadsheet (`.csv`) file. See [Batch Input Formats](https://docs.speech-graphics.com/en/sgx/4.6/batch-input-formats.md). | | `-f [ --overwrite ]` | optional | *None* | Whether newly created events can overwrite existing events with the same name in the events directory. Only relevant in *process new events*mode. | | `-L [ --languages ]` | optional | Path | The path of the directory containing the language modules. See [Installing Language Packs](https://docs.speech-graphics.com/en/sgx/4.6/installing-language-packs.md). | | `-l [ --language ]` | optional | String | The language module base name (e.g. `English_US_2.1`). Be sure to specify the language (and dialect) heard in the input audio. If no language is specified, transcripts are not used. See [Input Options](https://docs.speech-graphics.com/en/sgx/4.6/input-options-1.md). | | `-b [ --behavior ]` | optional | String | Behavior settings as a string made up of [markup attributes](https://docs.speech-graphics.com/en/sgx/4.6/markup-1.md). See example below. | | `-s [ --animation_style ]` | optional | String | Animation Style. The argument is a string: "ease", "linear", or "stepped". Default value is "ease". Note that "linear" and "stepped" require the use of a language module. See [Animation Style](https://docs.speech-graphics.com/en/sgx/4.6/animation-style.md). | | `-P [ --pre_roll ]` | optional | Number | Add pre-roll to each animation. Either provide a fixed time value (in milliseconds), or "0" for auto-calculation. See [Pre Roll](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Pre-Roll-and-Post-Roll) . The actual duration of pre-roll added will be reported in the [Batch Report](https://docs.speech-graphics.com/en/sgx/4.6/sgx-batch-processing.md#Batch-reports). | | `-Q [ --post_roll ]` | optional | Number | Add post-roll to each animation. Either provide a fixed time value (in milliseconds), or "0" for auto-calculation. See [Post Roll](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Pre-Roll-and-Post-Roll) . The actual duration of post-roll added will be reported in the [Batch Report](https://docs.speech-graphics.com/en/sgx/4.6/sgx-batch-processing.md#Batch-reports). | | `-r [ --framerate ]` | optional | Number | Frame rate of all output events in frames/second (See [Frame Rate](https://docs.speech-graphics.com/en/sgx/4.6/settings.md#Frame-Rate)). | | `-A [ --active_start ]` | optional | *None* | Active start option (see [Active Start](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Active-Start-and-Neutral-Finish) ). | | `-N [ --neutral_finish ]` | optional | *None* | Neutral finish option (see [Neutral Finish](https://docs.speech-graphics.com/en/sgx/4.6/processing-configuration.md#Active-Start-and-Neutral-Finish)). | | `-E [ --export_animation ]` | optional | *None* | Export animation as plain-text RTS files alongside event files (see [Export animation](https://docs.speech-graphics.com/en/sgx/4.6/batch-processing.md#5.-Export-animation) ). | | `-M [ --export_metadata ]` | optional | *None* | Export metadata from events. | | `-D [ --scoped_log ]` | optional | *None* | Enable [scoped logging](https://speech-graphics.atlassian.net/wiki/spaces/SGKB/pages/2503180332/Scoped+Logs) to tag the stages in batch processing, making the output easier to parse and monitor programmatically | | `-v [ --verbose ]` | optional | *None* | Print additional information to the log for debugging purposes. | | `-x [ --custom ]` | optional | String | List of parameters for custom or experimental features, in the format `xxx;xxx;xxx`. | ### Examples Example processing command including only the required arguments: SGX_Producer.exe process -e "C:/Events" -c "C:/Characters/Vivian.k" Example processing command including optional arguments: SGX_Producer.exe process -e "C:/Events" -c "C:/Characters/Vivian.k" -i "C:/Characters/Audio" -L "C:/Languages" -l English_GB-EAW_1.1 -b "mode=\"Neutral\" auto_modes=\"positive,negative,effort\" speech_magnitude=\"1.050000\" nonverbal_magnitude=\"1.200000\" " -P 0 -Q 0 -N -E -M --- version: "SGX 4.6" language: "en" --- # Compilation SGX Runtime introduces a compilation step that is not required in previous SGX workflows. Before source SGX character and animation data can be used by SGX Runtime, it must be compiled into Runtime data. Compilation is a required step in the SGX Runtime content pipeline. Run the compiler after authoring or exporting source SGX data and before the data is imported, packaged, or otherwise made available to your game engine. You can use the compiler as a standalone executable or integrate it directly into your content pipeline tools through its shared library interface. The SGX Runtime Compiler converts source SGX data---including character `.k` files and `.event` files---into compact Runtime data designed for efficient evaluation. It: 1. Converts the character's `.k` file into the [Face Graph structure](https://docs.speech-graphics.com/en/sgx/4.6/terminology.md) used by SGX Runtime. 2. Extracts animation data from SGX `.event` files and performs key reduction and perceptual compression to reduce memory usage while preserving animation quality during playback. 3. Organizes the resulting data into compact, contiguous memory blocks for efficient, cache-friendly traversal during animation evaluation. The compiled output has a reduced memory footprint and a data layout optimized for efficient runtime playback. For compiler usage, integration details, and reference material, see the [SGX Runtime Compiler documentation](https://docs.speech-graphics.com/en/sgx/4.6/sgx-runtime-compiler.md). --- version: "SGX 4.6" language: "en" --- # Context Menus (Runtime UE) The Runtime Unreal Engine 5 Plugin adds several new options to the right-click context menus for assets in the Unreal Editor. ## FaceFXActor Asset ![FaceFXActorContextMenu.png](https://docs.speech-graphics.com/__attachments/a_8f6ed53119bf0abeffcd3cddd09fb45f68b8dfa886fb8ae69163c338f5961334/FaceFXActorContextMenu.png?cb=cd72be9ae0629c08d956d190b62833d7) ### Open in FaceFX Studio Opens the `.facefx` file in FaceFX Studio (if the path to your FaceFX Studio installation is correct in the **FaceFX.ini** file). **Note:** Only available on Windows and irrelevant to SGX. ### Open FaceFX Studio Asset Folder Opens the folder that contains your `.facefx` file. ### Reimport FaceFX Studio Assets Batch imports the actor and any modified animation via the `.ffxc` folder that is parallel to the `.facefx` file and links the animations to the actor. ### Set new FaceFX Studio Assets Assigns this actor to a different `.facefx` file. Use this to update the source data for the Unreal Engine 5 asset while still preserving all of the Blueprint asset references in UE5. ### Show FaceFX Asset Details Lists the animations linked to this actor, the bone names in the actor, and the size of the actor. ### Link FaceFX Animset Links animations to the actor. ### Unlink FaceFX Animset Unlinks animations from the actor. ## FaceFXAnim Asset ![FaceFXAnimationContextMenu.png](https://docs.speech-graphics.com/__attachments/a_8ee805ea284e0f3faf4438f86cd196c69e792f3224d271866d47f4378c7720fc/FaceFXAnimationContextMenu.png?cb=c763324ff50b8bda65d3b7f5c844fd6e) ### Open FaceFX Studio Asset Folder Opens the folder that contains your `.facefx` file. ### Reimport FaceFX Studio Assets Reimports the selected animation asset only. Does not link or unlink the animation. ### Set new FaceFX Studio Assets Assign the animation to a different `.facefx` file. ### Show FaceFX Asset Details Displays the source FaceFX filename, the animation group, and the animation name. ### Link with FaceFX Actor Links the animation with an actor. ### Unlink From FaceFX Actor Unlinks the animation from an actor. --- version: "SGX 4.6" language: "en" --- # Conventions SGX Runtime follows several FaceFX conventions that differ from previous SGX workflows. Be aware of these conventions when preparing character and animation data for compilation. ## Audio Starts at Time 0 In FaceFX and SGX Runtime, audio always starts at time `0`. Animation does not necessarily start at time `0`. Animation keys may occur at negative times, which FaceFX uses as pre-roll. Negative time animation allows the character to begin moving into position before the audio begins. It also allows speech gestures, such as head nods or eyebrow raises, to begin before the first audio sample. This differs from SGX pre-roll handling. Rather than shifting the audio on the timeline, SGX Runtime keeps the audio start at time `0` and places any pre-roll animation at negative times. By default, the SGX Runtime Compiler shifts animation keys into negative time so that the audio begins at time `0`. Use the `--no-preroll-shift` compiler option to prevent the creation of negative-time animation. Audio still begins at time `0`; however, any required pre-roll must be included in the audio file, typically as leading silence. ## Inactive Reference Pose The character reference pose must not contain active morph targets or controls. The compiler reports an error if it finds an active morph target or control in the reference pose. This usually indicates either a rigging error or a value that should be incorporated into the character's neutral pose instead. ## Audio File Placement The compiler can locate audio files associated with `.event` files and generate an audio import file for use by the game engine. For example, the Unreal Engine plugin uses this generated file to import the associated audio automatically. To enable this behaviour, place each audio file in the same folder as its corresponding `.event` file and give both files the same base name: C:\Game\Hero\Events\Monologue.event C:\Game\Hero\Events\Monologue.wav The compiler can then associate `Monologue.wav` with `Monologue.event`. --- version: "SGX 4.6" language: "en" --- # Loading, Creating and Saving Events ## Open Open events using **File** \> **Open** or the shortcut **Ctrl+O**. ![open_event.png](https://docs.speech-graphics.com/__attachments/a_a6dff3cad7d61329bdde0be58f3c1083f8e1ae70347a54f3af0628b65f0b35a6/open_event.png?cb=e34b43a30790315369519d3682fa9976) Select one or more event files in the explorer and hit "Open". ![open.png](https://docs.speech-graphics.com/__attachments/a_d6498d1a8f59a96b6032c45cefdde2c7598ab9c28118020f00a4b0a735f3928a/open.png?cb=1341cc6bb88b09a73b763a67dd9d989e) The selected events will be opened as [event tabs](https://docs.speech-graphics.com/en/sgx/4.6/introduction-to-sgx-director.md#Tabs): ![image-20240112-224415.png](https://docs.speech-graphics.com/__attachments/a_a57aabf8d2dae0af2ddfae1fbf4b94b5dec43527127efbe8489ceb888b122718/image-20240112-224415.png?cb=25be2031058bccecf157182f6ccbd3c7) Once an event is loaded, hovering on the event tab shows its file path. You can also use **View** \> **Event** \> **Show Event in Explorer** to navigate to that location. ## Open Recent To re-open recent files, use **File** \> **Open Recent** ***.*** ![open_recent.png](https://docs.speech-graphics.com/__attachments/a_9de99b8ae62b7eb72e1c315b3e6d9fac32f298abea5b45835334bcddc772c9aa/open_recent.png?cb=c6f89cb8e2519e921ad6e8fc55128bdc) ## Next/Previous Event You can navigate through events in a directory. Use **File** \> **Next Event** or**Ctrl+Down,** to load the next event, and **File\>Previous Event** or **Ctrl+Up** to load the previous event. The next/previous event will be loaded into the current (highlighted) event tab. ![next_event.png](https://docs.speech-graphics.com/__attachments/a_45b379a88ecb94b7d428627b8d65698fb14479cb8a2095e140dd2c0e1ad5850d/next_event.png?cb=7b80eadc72b480a028658c715d36ff1f) ## New Event Create a new empty event using **File** \> **New** \> **Event** or **Ctrl+N.**The event will contain no sequences. ![image-20240112-230421.png](https://docs.speech-graphics.com/__attachments/a_d2c2883fe3d472fd90c5a0ac2cadf43dc00805348fa7ae72e8acd77d39582482/image-20240112-230421.png?cb=ff54a3148abef72478174d6b60748595) You may then load sequences into the empty event using **File** \> **Open**-- normally an audio file and its corresponding transcript file: ![open_audio_event.png](https://docs.speech-graphics.com/__attachments/a_7e71a426f9cd30ab61230e5e0e0a92d3ec756ce57fbf70b0de35f7073c9e3827/open_audio_event.png?cb=292df2db9fbe0becadd9a4cfac156bd9) This results in an event containing only the input: ![image-20240112-231630.png](https://docs.speech-graphics.com/__attachments/a_92064cd5ecd08619db423e410abc7307de0c59eec401fdc1bc813dc4a1c5e60b/image-20240112-231630.png?cb=81083c0877bb8391d45ebfe363e24c33) This event may be then processed to produce animation and metadata (see [Processing Events](https://docs.speech-graphics.com/en/sgx/4.6/processing-events.md)). ## Reload Event Although SGX Director does not currently have an Undo function for editing, you can revert edits to an event by re-loading it. Selecting **File** \> **Reload** \> **Event** reloads the current event from its known file path. ## Save and Save As After making changes to an event, you can save the current event using **File** \> **Save** or **Ctrl+S** (if the event already has a path) or **File** \> **Save As** to save the current event to a new path. [Next Page](https://docs.speech-graphics.com/llms-full.txt/1)