What is SG Com?

The SG Com SDK provides access to Speech Graphics' real-time audio-driven facial animation technology, enabling developers to build applications with avatars driven by voice.

SG Com's processing Engine converts an incoming audio stream into synchronous, high-fidelity facial animation which may be streamed to other endpoints. Output animation includes lip sync, full-face emotional expressions, blinks, eye darts, and even head movement, all driven by the speaker's voice through proprietary Speech Graphics algorithms. The processing delay between input and output is 50 milliseconds.

SG Com also includes a Player that handles buffering, decoding and synchronized playback of the animation produced by the Engine.

The SDK includes:

• SG Com API, a C API

• SG Com Unreal Engine Plugin, for easy integration into the Unreal Engine

C# bindings and sample Unity integrations are also available on request.

What’s new in SG Com 5?

CPU cost savings

For bone rigs, the CPU cost of SG_COM_UpdateAnimation has been reduced by two thirds (per bone). See SG Com Compute Resource Usage.

Licensing

There are now two licensing options:

1. key-based licensing

   1. Key string is input into license_data argument of SG_COM_Initialize

2. cloud-based licensing

   1. Path to .lic file is input into license_data argument of SG_COM_Initialize

   2. Available on Windows and Linux only

   3. Requires internet connection with two open ports

Expression Changed status type

SG Com emits a new status type for the SG_COM_EngineStatusCallback: SG_COM_STATUS_EXPRESSION_CHANGED. This is triggered whenever SG Com starts a new facial expression. The payload includes the full path of the expression. This can be used to synchronize other animations with the expressions.

Auto Mode Controls

We now use the enum SG_COM_AutoMode for automatically detected states such as emotions or other vocal events.

SG Com 4 relied on specific naming of the character's behavior modes to trigger them automatically, such as "positive", "negative", "acknowledge". Now auto modes can be mapped to any behavior mode at runtime, so there is no longer a naming requirement.

For example:

SG_COM_SetAutoMode(engine_handle, SG_COM_POSITIVE_MODE, "happy");

maps the positive auto mode to the character’s behavior mode called “happy”, whereas

SG_COM_SetAutoMode(engine_handle, SG_COM_NEGATIVE_MODE, "angry");

maps the negative auto mode to the character’s behavior mode called “angry”. Note that an auto mode must be mapped to something in order for it to have an effect on behavior.

To clear the current auto-mode mappings, use

SG_COM_UnsetAutoModes();

The default behavior mode, which is used when no auto mode is detected, can also be changed:

SG_COM_SetDefaultMode(engine_handle, "friendly");

Note that in order to activate automatic mode detection, we now use

SG_COM_ActivateAutoModes(engine_handle);

Whereas to turn off automatic mode detection and instead use a fixed mode, e.g. "happy", use

SG_COM_SetMode(engine_handle, "happy");

Effort Detection

There is a new auto mode, SG_COM_EFFORT_MODE, which is triggered when a grunt or effort sound is detected in the voice. This may happen when one is undergoing physical exertion, and typically involves a tightening of the vocal chords and guttural non-speech sounds. This auto mode can be used to trigger expressions that reflect effort, such as an extreme contraction of the facial muscles and gritting the teeth.

API Changes

There are various API changes for the new nonverbal features, as well as some straight name changes such "moods" now being called "modes", and "controls" now being called "modifiers".