Behavior Controls

In contrast with facial motion capture solutions, which copy motion from a real actor, SGX is a procedural solution, which means it generates behavior. Because it is generative, it can offer more than just automation; it also allows creative control, by giving animators an opportunity to direct the performance of the character. With motion capture, the actor is in control – with SGX, the animator is in control.

Behavior can be directed at various points in the SGX workflow:

• Before processing: via markup in transcripts (see Behavior Markup)

• During processing: via processing parameters applied to a whole batch or a single event (see SGX Director and SGX Producer )

• After processing: via edits to behavior metadata (see Editing Events)

There are two types of behavior controls: modes and modifiers. These are described below.

Behavior Modes

Behavior modes make it easy to determine an entire behavior pattern with a single label. At any point in time during animation, your character is in some behavior mode. Behavior mode definitions are part of the character setup – see Behavior Modes for an overview. In short, a behavior mode contains a variety of expressions reflecting that mode. For example, if you have a behavior mode “Happy”, then you would populate that mode with a variety of happy-looking expressions. When the character is in a behavior mode, it draws on expressions from that mode and deploys them automatically with timing and dynamics determined by SGX algorithms.

Behavior modes may correspond to emotional or physical states, like “happy”, “angry”, “tired” or “struggling”. The set of a character’s behavior modes is open-ended: you can define as many as you wish.

Every character has one default behavior mode that is active if none has been specified.

Here is an example of an animation with changing behavior modes. The current behavior mode is printed above the character as she speaks (as a “mood”).

Auto Modes

While SGX allows creative direction, the goal is also to generate good animation at scale. For this reason, SGX uses AI to detect several vocal qualities, triggering automatic behavior modes called auto modes. The auto modes may be defined as any of the behavior modes you have designed for the character.

There are currently three auto modes in SGX, with more on the way.

• Positive - Happy or positive emotion is detected in the voice.

• Negative - Negative or unhappy emotion is detected in the voice. This encompasses a range from sadness to anger.

• Effort - A grunt or effort sound is detected in the voice. This may happen when the voice actor is portraying activities of physical exertion or bodily impact, typically involving a tightening of the vocal chords to make guttural non-speech sounds.

Auto modes may be mapped to the behavior modes of your character, as shown in the image below. This mapping may be changed at any time.

It is up to the user which auto modes to activate. If an auto mode is not active, it will not trigger a change in behavior mode.

Behavior Modifiers

Behavior modifiers are numerical parameters that can be used to adjust particular dimensions of the performance, such as magnitude, speed and frequency of movements. Like behavior modes, these values can be applied to an entire batch of events, to an individual event, or to any sub-interval of an event.

Behavior modifiers serve as “tweaks” to correct any issues seen in the automatic output. You should not normally need to use them unless there is something to correct.

An overview of the behavior modifiers is presented in the following table, and each modifier is discussed in greater detail below.

Speech Magnitude

The Speech Magnitude modifier affects the amplitude of all speech movements.  It's like a volume knob for lip sync. Generally, SGX attempts to match the magnitude of muscle movements to the intensity of the speech in the audio, syllable by syllable. However, if the animated movements are not emphatic enough (or a particular style is desired), you can use the Speech Magnitude modifier to amplify the speech; conversely if the movements seem too exaggerated given the audio, you can use this modifier to dial the speech down.

Setting Speech Magnitude to 0 will effectively turn off lip sync.

Note that increasing Speech Magnitude will not always result in a proportional increase in the amplitude of movements. The movements of speech are governed by a complex muscle-dynamic model (see Speech Graphics Technology), which requires each movement to be physically natural according to the model’s limits on velocity and acceleration. This means that if you amplify the target magnitudes, the muscles might not have enough time to reach the desired degree of displacement given other things it is trying to accomplish on the timeline. One way around this limitation is to also increase Speech Speed or Hyperarticulation.

Below are examples of Speech Magnitude set to three different values:

• 1.0 (default)

• 0.3 (=30% magnitude)

• 2.0 (=200% magnitude)

Speech Speed

The speed of muscles is controlled by the muscle-dynamic model. Rarely, you might want to make them move faster, perhaps to accentuate a fast-talking character (though note that rapid speech is detected by the system and movements should be commensurate with the detected speech rate). The Speech Speed modifier is a global multiplier on the basic speed of speech movements.

Below are examples of Speech Speed set to three different values:

• 1.0 (default)

• 0.3 (=30% magnitude)

• 2.0 (=200% magnitude)

Hyperarticulation

Hyperarticulation describes a phenomenon in which the speaker is putting extra energy into making the articulatory organs move faster and farther in order to make the speech very clear. The hyperarticulation modifier is a corrective tweak that can be used when the animation does not correctly match the hyperarticulation heard in the audio. This may be warranted particularly in very rapid but still clear speech, where the speaker is putting extra effort into hitting articulatory targets in the shorter time available rather than letting the consonants and vowels be slurred. Generally speaking, hyperarticulation should be used only sparingly if at all; small adjustments go a long way. Too much hyperarticulation can result in jittery animation.

Most of the modifiers have default 1.0 and range [0.0, 2.0]. However the default setting for hyperarticulation is 0.0 and its range is [0.0, 1.0].

Below are examples of Hyperarticulation set to four different values:

• 0.0 (default)

• 0.2

• 0.5

• 1.0

Jaw Limit

The Jaw Limit modifier allows you to restrict the degree of jaw opening permitted. For example, this could be used to keep the teeth clenched. The lower the value, the less the jaw is allowed to open. At 1.0 there is no restriction, and at 0.0 the jaw will stay completely closed. Jaw Limit is designed to impede speech movement, so consider this when applying it to normal speech.

Below are examples of Jaw Limit set to three different values:

• 1.0 (default)

• 0.7

• 0.1

Nonverbal Magnitude

The Nonverbal Magnitude modifier affects the amplitude of all nonverbal movements. It is thus complementary to Speech Magnitude, which affects only speech muscles. Generally, SGX matches the magnitude of nonverbal expressions to the intensity of the speech in the audio, phrase by phrase. However if the animated movements are not emphatic enough, you can increase the Nonverbal Magnitude to amplify the speech; conversely if the movements seem too exaggerated given the audio, you can decrease the modifier to dial the nonverbal movements down.

Setting Nonverbal Magnitude to 0 will effectively turn off nonverbal animation.

We recommend Nonverbal Magnitude and Speech Magnitude are modified together. In the examples below, both of these modifiers are set to the following values:

• 1.0 (default)

• 0.3

• 2.0

Nonverbal Speed

Nonverbal Speed is a global modifier affecting the basic speed of nonverbal muscle movements. It is analogous to the Speech Speed modifier, which operates only on speech muscles.

Generally, the speed of muscles is controlled by the muscle-dynamic model. Rarely, you might want to make them move faster, e.g. in the case of a particularly sharp reaction that isn’t captured in the automatic animation. This modifier will achieve that.

Blink Frequency

Blinks are automatically generated in SGX using a model of human blink patterns. The probability of a blink at any time is conditioned by audio properties and other activity in the face. However, you can alter the base frequency of blinks using the Blink Frequency modifier. If you’d like the character to blink less frequently, set the value to below 1.0; if you’d like the character to blink more frequently, set it to above 1.0.

Setting Blink Frequency to 0 will effectively turn off blinks.

Dart Frequency

Eye microdarts in SGX refer to small horizontal saccades (rapid eye movements) that simulate looking between the two eyes of an interlocutor. This behavior supports the lifelikeness of our character animation.

Like blinks, microdarts are automatically generated in SGX using a model of human eye movement patterns. The probability of a microdart at any time is conditioned by audio properties and other activity in the face. However, you can alter the base frequency of darts using the Dart Frequency modifier. For example, if you feel the character’s eyes are darting too frequently, set the value to below 1.0. If you feel the character’s gaze is too steady, set it to above 1.0.

Setting Dart Frequency to 0 will effectively turn off eye microdarts.