What's new in SG Com 5.3?

SG Com 5.3 includes a number of brand-new features as well as several improved features.

Lookat

The lookat feature moves the character’s muscles so that its gaze tracks a specific target in 3D space. This is useful for maintaining eye contact with the camera, or to look at a specific object in the scene. The lookat movements incorporate the eyeballs, head and torso. Since the system operates through the dynamic muscles, it is immediately responsive to changes in target.

Most advanced lookat systems require solving an inverse kinematic (IK) problem over a chain of bones, so the solutions tend to be rig-specific. The SG Com lookat system by contrast is agnostic about the rig, which means it can be used for any rig. Instead of solving for joint rotations, it solves for muscle activations. This requires first learning how the character’s muscles affect gaze direction.

For each character, enabling the lookat feature requires a simple configuration step in SGX Studio Maya (4.6 or later). During configuration, the user selects 3 joints in the character’s head: left eye, right eye, and eye root. The current model also requires these 7 muscles to be defined:

• eyeball_pitch

• eyeball_yaw

• head_pitch

• head_roll

• head_yaw

• torso_pitch

• torso_roll

At runtime, the lookat system operates in coordination with nonverbal expressions. Each expression has a gaze direction invoving the above muscles. The muscle activations will be modified to achieve the desired gaze direction with minimal alteration to the expression.

The Lookat system includes several new additions to the SG Com API:

• A new structure containing 3D positional coordinates.

typedef struct SG_COM_Position {
		float x;
		float y;
		float z;
	} SG_COM_Position;

• A function to set the activation of the Lookat system. The activation value is the probability that the character will focus on the target during each new expression. The range is [0,1]. At 0, the character ignores the target; at 1, it will always be fixated on the target. Use in-between values to sometimes look at the target and sometimes ignore it.

SG_COM_Error SG_COM_SetLookatActivation(
        SG_COM_EngineHandle engine_handle,
		float activation
    );

• A function to set the lookat target. Its arguments include the target position, as well as the base positions of the three head joints (left eye, right eye, and eye root). The ‘base positions' of the head joints means their positions prior to any SG Com animation being applied. The head joint positions inform the system of the character’s orientation relative to the target. Call this function whenever the target moves or whenever the character’s head moves due to non-SG Com animation (e.g. idle animation). It can be called every frame if necessary.

SG_COM_Error SG_COM_SetLookatTarget(
        SG_COM_EngineHandle engine_handle,
		SG_COM_Position* target_position,
		SG_COM_Position* left_eye_base_position,
		SG_COM_Position* right_eye_base_position,
		SG_COM_Position* eye_root_base_position
	);

• A function to retrieve the names of the character’s three head joints so they can be looked up in the scene. This should only be called once when loading the character.

SG_COM_Error SG_COM_GetLookatHeadJoints(
        SG_COM_EngineHandle engine_handle,
        char* left_eye_joint,
		char* right_eye_joint,
		char* eye_root_joint,
        sg_size buffersize
    );

Beat detection

A ‘beat’ in speech is an emphasis, a stressed syllable, that stands out as being stronger than the speech around it, and indicates a likely point for a rhythmic nonverbal gesture to occur. SG Com now animates beats, thus providing a timing framework for gesture animation. It does not go so far as to animate the gestures yet, but the timing is useful for external systems to trigger gestures appropriately.

To signal a beat, SG_COM_EngineStatusCallback will emit the new status type SG_COM_STATUS_SPEECH_BEAT. The message string in the callback will contain the normalized intensity of the beat as a floating-point integer. In principal the average intensity is 1.0, and most beats will be higher than this. The intensity has no strict bounds but most beats will be on the scale [1.0, 2.0].

Player-only library

SG Com 5.3 includes a second smaller library, SG Com Player, which excludes Engine-related functions and focuses solely on the Player. This is useful for lightweight implementations of SG Com clients, e.g. in Web Assembly, where Engine-side functions would be handled in the cloud or elsewhere.

The main differences between SG Com Player and the full SG Com library are the following:

• Removal of all Engine-related identifiers including SG_COM_EngineHandle, SG_COM_EngineConfig, SG_COM_EngineConfigFlag, SG_COM_Status, SG_COM_Modifier, SG_COM_Role, SG_COM_AutoModeand SG_COM_Position

• Removal all functions having an SG_COM_EngineHandleargument

• The replacement of the SG_COM prefix with SG_COM_PLAYER – for example SG_COM_Initialize() becomes SG_COM_PLAYER_Initialize()

• Removal of “Player” from identifiers since it’s in the prefix. For example:

SGX event playback

A new function has been added: SG_COM_ReceiveEvent() – or SG_COM_PLAYER_ReceiveEvent() in the SG Com Player API.

 SG_COM_Error SG_COM_ReceiveEvent(
        SG_COM_PlayerHandle player_handle,
        const char* event,
        sg_size event_bytes
    );

This function inserts an SGX event file (.event), containing a pre-generated muscle sequence, into the Player. This is an alternative to inserting real-time packets emitted by the Engine, and is equivalent to adding a series of packets at once. If the animation buffer is not long enough to hold the new animation, it will be clipped. At present, there is no blending of animation at the beginning and end of the inserted sequence, so some discontinuities may occur.

SGX events must be generated using SGX Production Tools.

Improved features

Nonverbal muscles

The dynamic muscle models used for nonverbal behavior have been completely revamped.

• Bidirectional and unidirectional muscles now have the same physical modeling

• Acceleration is more controlled so that muscles more closely adhere to the dynamic settings in SGX (e.g. so that head muscles no longer look faster in SG Com than in SGX).

• Instead of arcing, muscles now exhibit an overshoot and settling behavior. This has several advantages over arcing:

  • It is closer to the way real muscles behave.

  • It reduces the apparent latency of expressions because the peak of the overshoot occurs earlier in the expression than the peak of a long arc.

  • It avoids the pitfalls of arcing such as the fact that the intensity of the arc grew with the duration of the expression.

• Glitching in the muscle models has been virtually eliminated.

Blinks

The SG Com blink system has been improved to match the SGX blink system in terms of quality of blink predictions. Formerly, blinks were too rare during speech and overabundant during pauses.

Microdarts

Similarly to blinks, the microdart system has been improved to match the SGX microdart system in terms of quality of microdart predictions. In addition, microdarts are now more varied; rather than simply alternating between the far left and far right sides of the range, they cover the middle ground. And the microdart system has been refined to work harmoniously with the new lookat system.

Lip sync

The universal lip sync system has been improved, and now has the same quality level as character-specific lip sync models. This implies that character-specific training is no longer needed except in the case of highly custom muscle systems.