SG Com UE Demo Application
Introduction
This article provides an overview of the UE5 demo application with which the SG Com UE Plugin is shipped. The demo makes some simplifying assumptions:
There is a single character driven by a single audio stream.
Animation is local – i.e., Engine and corresponding Player are on the same machine (see Networked vs local use cases).
There are two ways to input audio: from an audio file or from a microphone.
For more complex use cases, the developer will need to adapt the demo.
This demo application is a guide as to how you might utilize the SG Com UE plugin. Note that this is an example and not every step here has to be followed exactly as described.
Enabling the plugin
Ensure that the plugin has been correctly installed in either the engine or project and that it has been enabled in the Unreal Engine editor.
After enabling the plugin you will have to restart the Unreal editor
Configuring the plugin
Navigate to Project Settings in the editor and find the SG Com section under Plugins. Here you will find a few different settings to configure:
Set the License File Path to the location of your SG Com license. If this path is incorrect or the license that it points to is invalid, the plugin will not function.
Log Directory Path is where SG Com logs will be generated. Set this to any convenient location, for example, in your project under a new directory such as Saved/SGComLogs.
Logging Level determines the amount of detail provided by SG Com in its logs, with None of course being the lowest and Debug being the highest.
File Example
For this example we are putting our logic in the level blueprint, which allows us to initialize and perform any setup when the level is started, then shut down when the level is ended.
Adding and configuring the SG Com Session component
Add to the scene a MetaHuman that we want to animate using SG Com.
Select the actor in the scene outliner and then clicking the Add option at the top of the actor’s details panel. Select SGCom Session.
Configure the SG Com Session. Guidance on how to do this can be found in SG Com UE Configuration.
Getting the SG Com Session component
Obtain and then store a reference to the SG Com Session component. This will be used to call any necessary session functions in the level blueprint.
Initializing SG Com
Initialize SG Com by calling the Initialize SGCom function.
Starting the session
Start the session by calling Start Session using the component reference we stored earlier.
Auto mode detection
Map the Positive auto mode to the “positive” behavior mode by calling Set Auto Mode, selecting the Auto Mode from the dropdown list and then typing the desired behavior mode name in the Mode Name text box.
Repeat this process for the Negative, Effort and Acknowledge auto modes, Mapping them to the “negative”, “effort” and “acknowledge” behavior modes, respectively.
Activate the auto modes by calling Activate Auto Modes on the SG Com Session component.
Audio input
For Editor Only, you can use the SG Com helper function Decompress Sound Wave to convert a sound wave asset into raw PCM data.
Input this raw audio data into SG Com using the Input Audio function.
File playback
Here we use the MetaSounds audio system. A regular SoundWave could also be used. See the sample project for both implementations.
Create and set up a new MetaSound Source asset using the desired SoundWave.
Back in the level blueprint, add an audio component to the level.
Set the audio component to use the newly created MetaSound.
Create a Watch Output node and bind its On Output Value Changed callback to a custom event Playback Time Changed, which will allow us to keep track of the current playback time of the MetaSound.
Set the Output Name field of the node to “Time”. Then, whenever this value is changed, The Event Playback Time Changed callback will be triggered.
Idle
We want the character to continue animating after the file has been played in its entirety, or if we decided to pause the audio file partway through. In order to do this we’ll need to feed empty audio data to SG Com whenever there is no file currently playing. (See Engine idle).
Setting up callbacks
Create custom events for both On Mode Changed and On Expression Changed callbacks. These custom events should take a string as input, so make sure to add this.
Bind these events.
Set the events to print their output, for debugging purposes.
Updating animation
Here we make use of the custom event Playback Time Changed which we created earlier when setting up the file playback.
When the event is triggered, get the new playback time.
Calculate the delta since the last playback time change.
Use this delta to update the animation in SG Com by the necessary amount.
Ending the session and shutting down SG Com
End the active session by calling End Session.
Call Shutdown SGCom.
Microphone Example
For this example we are putting our logic in the level blueprint, which allows us to initialize and perform any setup when the level is started, then shut down when the level is ended.
Adding and configuring the SG Com Session component
Add to the scene a MetaHuman that we want to animate using SG Com.
Select the actor in the scene outliner and then click the Add option at the top of the actor’s details panel. Select SGCom Session.
Configure the SG Com Session. Guidance on how to do this can be found in SG Com UE Configuration.
Getting the SG Com Session component
Obtain and then store a reference to the SG Com Session component. This will be used to call any necessary session functions in the level blueprint
Initializing SG Com
Initialize SG Com by calling the Initialize SGCom function.
Starting the session
Start the session by calling Start Session using the component reference we stored earlier.
Auto mode detection
Map the Positive auto mode to the “positive” behavior mode by calling Set Auto Mode, selecting the Auto Mode from the dropdown list and then typing the desired behavior mode name in the Mode Name text box.
Repeat this process for the Negative, Effort and Acknowledge auto modes, Mapping them to the “negative”, “effort” and “acknowledge” behavior modes, respectively.
Activate the auto modes by calling Activate Auto Modes on the SG Com Session component.
Audio input
This example is set up to use a custom MicrophoneGameModeBase game mode. This is written in C++ and uses the FAudioManager microphone functionality to feed audio data into SG Com via the SG Com Session component.
Get the game mode and cast it to MicrophoneGameModeBase.
Call Start Microphone on the game mode and pass through the SG Com Session component reference. Audio data will now continuously be fed to SG Com from the microphone for processing.
Microphone playback
Get the USoundWaveProcedural stored in MicrophoneGameModeBase. This contains audio data from the microphone which we want to play back.
Spawn a 2D sound in the scene which uses the procedural sound wave object.
Get the audio component returned by the Spawn Sound 2D function and bind a custom event Playback Time Changed to the component’s On Audio Playback Percent callback.
Play the audio component which will start audio playback.
Setting up callbacks
Create custom events for both On Mode Changed and On Expression Changed callbacks. These custom events should take a string as input, so make sure to add this.
Bind these events.
Set the events to print their output, for debugging purposes.
Updating animation
Here we make use of the custom event Playback Time Changed which we created earlier when setting up the microphone playback.
When the event is triggered, calculate the current playback time.
Calculate the delta since the last update.
Use this delta to update the animation in SG Com by the necessary amount.
Ending the session and shutting down SG Com
End the active session by calling End Session.
Call Shutdown SGCom.
Animation Blueprint
To apply the SG Com animation data to the MetaHuman actor, we need to use the SG Com Pose animation node. For this example, we will keep it simple and just make a new, very basic animation blueprint.
Create the animation blueprint asset targeting the MetaHuman’s face mesh.
Add an SG Com Pose node.
Feed the output pose from the SG Com node to the final output pose of the blueprint.
Set the face mesh component of the MetaHuman actor to use this new animation blueprint.
