SG Com API Tutorial

Initialization, shutdown and logging

Before the library can be used it must be initialized using SG_COM_Initialize, and when it will no longer be used it should be shut down using SG_COM_Shutdown. Calls to these functions should be the first and last interactions with the API.

SG_COM_Initialize takes three main arguments: the logging level (SG_LoggingLevel), an optional callback to receive log messages (SG_LoggingCallback), and license data. The license data will be either the path of the license file, or in the case of a non-networked license, the license string loaded into memory.

Creating and destroying Engines and Players

Animating a character with SG Com requires an Engine and a corresponding Player (see Engines and Players).

To create an Engine, use SG_COM_CreateEngine, which requires an Engine configuration. To create a Player, use SG_COM_CreatePlayer, which requires an Player configuration.

To destroy Engines and Players, use SG_COM_DestroyEngine and SG_COM_DestroyPlayer, respectively.

Engine configuration

An SG_COM_EngineConfig structure includes the following:

• The Character Control File, loaded as bytes into memory.

• The sample rate and sample type of the audio bytes that will be input (see Engine input).

• The input buffer length, expressed as the maximum audio duration (in seconds) that the input buffer can hold (see Engine input). This should be as long as the longest audio chunk.

• Optionally, a handle to the corresponding Player if it is local (see Engine to Player transfer).

• An SG_COM_EngineBroadcastCallback function, which will be invoked whenever an Engine has an output packet ready. Not needed for local Players (see Engine to Player transfer).

• An SG_COM_EngineStatusCallback function, which will be invoked whenever there is an Engine status update (see Metadata updates).

Player configuration

An SG_COM_PlayerConfig structure includes the following:

• The Character Control File, loaded as bytes into memory. This must be the same file that is used in the corresponding Engine.

• The output buffer length, expressed as the maximum duration of animation (in seconds) that the output buffer can hold. This should be at least as along as the Engine’s input buffer length. It can also be longer to avoid losing older animation if playback is delayed.

Engine input

The function SG_COM_InputAudio feeds audio into the Engine to drive facial animation. For each call to SG_COM_InputAudio, you will provide an array of bytes containing some number of audio samples. For low-latency applications, we recommend inputting audio as soon as it is available. The ideal input size is 10 milliseconds since that is the internal processing frame size. However, the size of each input package can vary. The upper limit is determined by the buffer size you specify in SG_COM_EngineConfig.buffer_sec.

Input audio is required to be raw, linear PCM, mono (see SG Com Input), and the sampling must agree with the SG_COM_EngineConfig that was used to create the Engine, specifically SG_COM_EngineConfig.audio_sample_type and SG_COM_EngineConfig.audio_sample_rate.

In some customized use cases, you might also have need for SG_COM_InputAuxData, which is used to input auxiliary (non-audio) data. Note auxiliary data is always assumed to have a sample rate of 100 Hz.

Engine processing tick

When you input data using SG_COM_InputAudio or SG_COM_InputAuxData, no actual processing happens; the input data is buffered. In order to actually process the input data into animation, you must call SG_COM_ProcessTick. This function executes the processing necessary to advance one frame of input (10 ms) through the Engine pipeline.

The Engine tick function must be called at least once for every 10 ms of input data. After an initial algorithmic delay of 50 ms, each call to SG_COM_ProcessTick will generate one output packet. Depending on the Engine configuration, this packet will be delivered either to the SG_COM_EngineBroadcastCallback or directly into the local Player (see Engine to Player transfer).

SG_COM_ProcessTick is designed to be called on a background thread or threadpool.

Idle mode

Normally the Engine requires input in order to produce output. However it can also generate idle behavior – blinks, eye darts, and nonverbal expressions – in the absence of input. To enable idle behavior, use the flag SG_COM_EngineConfigFlag.SG_COM_ENGINE_CONFIG_ENABLE_IDLE in your Engine configuration. When idle mode is enabled, it means that even when there is no input, the processing tick will continue to generate output frames, replacing data-driven behavior with idle behavior.

An equivalent way to generate idle behavior without this flag is to use SG_COM_InputAudio with all zeros in the input bytes.

Idle behavior is smoothly blended with other output. There will be no breaks or sudden changes in the animation when the character enters or leaves idle mode.

Engine to Player transfer

Output packets from the Engine must be delivered into the Player. There are two ways of doing this, depending on whether the Player is deployed locally or across a network (see Networked vs local use cases):

• For the local case, provide the Player handle in the Engine configuration. Output packets from the Engine will automatically be input into the Player, without the need for additional data handling.

• For the networked case:

  1. obtain output packets from the SG_COM_EngineBroadcastCallback

  2. transfer them over the network

  3. input them into the remote Player using SG_COM_ReceivePacket

Animation nodes

In order to play the output, it's necessary to first get the animation nodes from the Player using SG_COM_GetAnimationNodes. This provides an array of nodes. Each SG_AnimationNode represents an object in the character rig, as defined in SGX Studio (see Editing Animation Targets). The SG_AnimationNode structure contains:

• SG_AnimationNode.name The name of the node

• SG_AnimationNode.type The type of the node (SG_JOINT, SG_BLENDSHAPE or SG_CONTROL)

• SG_AnimationNode.num_channels The number of animation channels

• SG_AnimationNode.channel_names An array containing the names of the animation channels

• SG_AnimationNode.channel_values An array containing the current values of the animation channels

Below is an example showing how to obtain the animation nodes.

SG_AnimationNode* animation_nodes;
sg_size num_animation_nodes;
SG_COM_GetAnimationNodes(player_handle, &animation_nodes, &num_animation_nodes);

After retrieving the animation nodes, the animation channels in the nodes must be mapped to the corresponding attributes in the character rig, so that values can be quickly copied from the nodes to the rig during animation. This mapping only needs to be established once after constructing the Player, since the animation nodes do not change.

Animation playback

To output animation from the Player, call SG_COM_UpdateAnimation at regular intervals within your application’s animation update loop. The function takes a time value as input; see Synchronizing output on keeping these time values synched to audio output.

Given a time value, the function samples and decodes the buffered motion data in the Player to produce a frame of animation, which is stored in the channels of the Animation nodes. These values can then be copied from the animation nodes into the corresponding attributes of the character rig in your scene.

// Enter the real-time playback loop
float *animation_frame;
for (;;) {

    // Update the animation. 
    err = SG_COM_UpdateAnimation(player_handle, time_ms, &current_time_ms);

    // Get the current value for each channel of each node
    for (size_t i = 0; i<num_animation_nodes; i++) {
        for (size_t j = 0; j<animation_nodes[i].num_channels; j++) {
            float val = animation_nodes[i].channel_values[j];
            // Do something with val.
        }
    }
}

The Player contains only a finite interval of the animation timeline. To check the start and end time of the playable interval, use SG_COM_Get_PlayableRange. The maximum duration of this interval will be determined by the output buffer duration specified in the Player configuration. The playable interval moves forward in time as data is added from the Engine (see Engine to Player transfer). Calls to SG_COM_UpdateAnimation will clip the requested time to this range; thus the actual time value of the retrieved animation frame (saved in the argument current_time_ms) might differ from the requested time.

The animation values obtained from SG_AnimationNode.channel_values are always relative to the reference pose. Therefore, channel values should be applied additively to the base values.

Using Behavior Modes

When you construct a new Engine, the character will be in the behavior mode marked “default” in your Character Control File. To manually change the current behavior mode, use SG_COM_SetMode. For example, this call

SG_COM_SetMode(engine_handle,"angry");

sets the character to the “angry” mode. The character will stay in this mode until it is set again, or until auto modes are switched on.

Behavior modes are character-specific, so to set them you must know what behavior modes is available in the Character Control File. For example, in the case above, the behavior mode “angry” is required to be defined in the character; if it is not, this function will have no effect.

To get a list of available modes for the character, use SG_COM_GetModeList. To find out the currently active behavior mode, use SG_COM_GetMode.

Using Auto Modes

To use the auto modes, define their mappings using SG_COM_SetAutoMode. For example, the code below maps SG_COM_POSITIVE_MODE to “happy”, SG_COM_NEGATIVE_MODE to “angry”, and SG_COM_EFFORT_MODE to “effort”.

SG_COM_SetAutoMode(engine_handle, SG_COM_POSITIVE_MODE, "happy");
SG_COM_SetAutoMode(engine_handle, SG_COM_NEGATIVE_MODE , "angry");
SG_COM_SetAutoMode(engine_handle, SG_COM_EFFORT_MODE , "effort");

Any auto modes that have not been set to an actual behavior mode will have no effect. To clear all current auto mode settings, use SG_COM_UnsetAutoModes.

You can also set the default mode, which is the mode that occurs when no auto mode is detected. Do this using SG_COM_SetDefaultMode.

To activate the entire auto modes system, you must call SG_COM_ActivateAutoModes. Note that calling SG_COM_SetMode to manually set the behavior mode will automatically deactivate the auto modes system. You must call SG_COM_ActivateAutoModes again to reactivate it. Over the course of dectivation and reactivation, the previously set auto-mode mappings are preserved.

Using Behavior Modifiers

When an Engine is created, all of its behavior modifiers are at their default values. To change the current setting of a modifier, use SG_COM_SetModifier. For example, this line sets Nonverbal Magnitude to 1.2:

SG_COM_SetModifier(engine_handle, SG_COM_MOD_NONVERBAL_MAG, 1.2);

To get the current value of a modifier, call SG_COM_GetModifier.

Using Roles

The function SG_COM_SetRole allows you to toggle an Engine between speaking and listening roles, using the enum type SG_COM_Role, with values SG_COM_ROLE_SPEAK and SG_COM_ROLE_LISTEN.

SG_COM_SetRole(engine_handle, SG_COM_ROLE_LISTEN);
SG_COM_SetRole(engine_handle, SG_COM_ROLE_SPEAK);

If this function is never called, SG Com will simply stay in the default role, SG_COM_ROLE_SPEAK.

To obtain the current role, call SG_COM_GetRole.