# Tutorial — Conversation Client Plugin *Augmented Traveler local tutorial.* Back to [[Unreal — Tutorials]] · the service: [[AI Mind — Process]] · how the figure uses it: [[Tutorial — Authoring a Q&A Experience with Ambient Idle]] How to build `ATConversation`, a small **C++ plugin** that connects a figure in Unreal to the studio's conversation service. It does four jobs: - records the visitor's question from the microphone; - streams it to the service; - receives text, a gesture tag and **streaming voice audio**; - plays the audio so the lip-sync source can drive the face. > **Draft.** The code below is a skeleton showing the Unreal parts. Fix the message protocol with the AI engineer first (section 2). Compile and test it on the kiosk PC. Unreal engineer + AI engineer. ## 1. Why a C++ plugin - **Reuse:** the same client works in every program level and every surface build. - **Performance and threading:** audio and network work happen off the game thread. - **Conversion later:** logic kept in C++ with thin Blueprint calls converts more easily when UE6 arrives ([[Tutorial — Blueprints for Augmented Traveler]]). - **Secrets stay out of Blueprints:** API keys come from the kiosk's secure config, never from content. ## 2. The protocol (agree it first) One **WebSocket** per conversation turn or session, `wss://` in production: | Direction | Message | Payload | |---|---|---| | → service | `start` (JSON) | figure ID, venue ID, session ID, language, audio format (16 kHz mono PCM16) | | → service | audio frames (binary) | 20–40 ms of PCM each | | → service | `end_of_speech` (JSON) | — (push-to-talk released, or end of speech detected) | | ← service | `transcript` (JSON) | recognised question (shown briefly as a caption) | | ← service | `answer_meta` (JSON) | answer ID, authored-or-generated flag, gesture tag, refusal flag, citation IDs | | ← service | `caption` (JSON) | text chunks with timing | | ← service | audio frames (binary) | TTS audio, 24 kHz mono PCM16 (agree the rate) | | ← service | `answer_end` (JSON) | — | | ← service | `error` (JSON) | code; the client falls back (section 6) | **If an authored answer matches,** the service sends `answer_meta` with `authored: true` and a sequence ID. Unreal then plays that pre-animated Level Sequence instead of streaming audio ([[Tutorial — Sequencer for Monologues]]). ## 3. Create the plugin 1. **Edit → Plugins → + Add → Blank** (a C++ plugin). Name it `ATConversation`. It goes in `AT_Studio/Plugins/`. 2. In `ATConversation.uplugin`: module type **Runtime**, loading phase **Default**. Also enable the engine plugin **AudioCapture**. 3. In `ATConversation.Build.cs`, add the dependencies: ```csharp PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "WebSockets", "HTTP", "Json", "JsonUtilities", "AudioCapture", "AudioCaptureCore", "AudioMixer" }); ``` ## 4. The component (skeleton) One `UATConversationComponent`, added to `BP_AT_<Figure>` (the child Blueprint of the MetaHuman; [[Tutorial — MetaHumans in Unreal]]). ```cpp // ATConversationComponent.h #pragma once #include "Components/ActorComponent.h" #include "IWebSocket.h" #include "ATConversationComponent.generated.h" class USoundWaveProcedural; class UAudioComponent; DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnATText, const FString&, Text); DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnATAnswerMeta, const FString&, GestureTag, bool, bAuthored); DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnATSignal); UCLASS(ClassGroup=(AT), meta=(BlueprintSpawnableComponent)) class ATCONVERSATION_API UATConversationComponent : public UActorComponent { GENERATED_BODY() public: UPROPERTY(EditAnywhere, Category="AT") FString ServiceUrl; // from kiosk config, not content UPROPERTY(EditAnywhere, Category="AT") FString FigureId; UPROPERTY(EditAnywhere, Category="AT") int32 TtsSampleRate = 24000; UPROPERTY(BlueprintAssignable) FOnATText OnTranscript; UPROPERTY(BlueprintAssignable) FOnATText OnCaption; UPROPERTY(BlueprintAssignable) FOnATAnswerMeta OnAnswerMeta; UPROPERTY(BlueprintAssignable) FOnATSignal OnAnswerEnd; UPROPERTY(BlueprintAssignable) FOnATSignal OnServiceError; UFUNCTION(BlueprintCallable, Category="AT") void BeginQuestion(); // push-to-talk pressed UFUNCTION(BlueprintCallable, Category="AT") void EndQuestion(); // push-to-talk released UPROPERTY(BlueprintReadOnly) UAudioComponent* VoiceAudio = nullptr; // lip-sync source listens to this protected: virtual void BeginPlay() override; virtual void EndPlay(const EEndPlayReason::Type Reason) override; private: TSharedPtr<IWebSocket> Socket; UPROPERTY() USoundWaveProcedural* VoiceWave = nullptr; void HandleText(const FString& Message); void HandleBinary(const void* Data, SIZE_T Size, SIZE_T BytesRemaining); }; ``` ```cpp // ATConversationComponent.cpp (key parts) #include "ATConversationComponent.h" #include "WebSocketsModule.h" #include "Sound/SoundWaveProcedural.h" #include "Components/AudioComponent.h" #include "Serialization/JsonSerializer.h" #include "Async/Async.h" void UATConversationComponent::BeginPlay() { Super::BeginPlay(); VoiceWave = NewObject<USoundWaveProcedural>(this); VoiceWave->SetSampleRate(TtsSampleRate); VoiceWave->NumChannels = 1; VoiceWave->Duration = INDEFINITELY_LOOPING_DURATION; VoiceWave->bLooping = false; VoiceAudio = NewObject<UAudioComponent>(GetOwner()); VoiceAudio->RegisterComponent(); VoiceAudio->SetSound(VoiceWave); } void UATConversationComponent::BeginQuestion() { Socket = FWebSocketsModule::Get().CreateWebSocket(ServiceUrl); TWeakObjectPtr<UATConversationComponent> Weak(this); Socket->OnConnected().AddLambda([Weak]() { if (Weak.IsValid()) { /* send "start" JSON; start mic capture (see section 5) */ } }); Socket->OnMessage().AddLambda([Weak](const FString& Msg) { // Callbacks may arrive off the game thread: hop back before touching UObjects. AsyncTask(ENamedThreads::GameThread, [Weak, Msg]() { if (Weak.IsValid()) Weak->HandleText(Msg); }); }); Socket->OnRawMessage().AddLambda([Weak](const void* Data, SIZE_T Size, SIZE_T Remaining) { TArray<uint8> Copy((const uint8*)Data, Size); AsyncTask(ENamedThreads::GameThread, [Weak, Copy = MoveTemp(Copy), Remaining]() { if (Weak.IsValid()) Weak->HandleBinary(Copy.GetData(), Copy.Num(), Remaining); }); }); Socket->OnConnectionError().AddLambda([Weak](const FString&) { AsyncTask(ENamedThreads::GameThread, [Weak]() { if (Weak.IsValid()) Weak->OnServiceError.Broadcast(); }); }); Socket->Connect(); } void UATConversationComponent::HandleBinary(const void* Data, SIZE_T Size, SIZE_T) { // TTS PCM16 audio: queue it and start playback on the first chunk. VoiceWave->QueueAudio(static_cast<const uint8*>(Data), Size); if (!VoiceAudio->IsPlaying()) VoiceAudio->Play(); } void UATConversationComponent::HandleText(const FString& Message) { TSharedPtr<FJsonObject> Obj; if (!FJsonSerializer::Deserialize(TJsonReaderFactory<>::Create(Message), Obj) || !Obj.IsValid()) return; const FString Type = Obj->GetStringField(TEXT("type")); if (Type == TEXT("transcript")) OnTranscript.Broadcast(Obj->GetStringField(TEXT("text"))); else if (Type == TEXT("caption")) OnCaption.Broadcast(Obj->GetStringField(TEXT("text"))); else if (Type == TEXT("answer_meta")) OnAnswerMeta.Broadcast(Obj->GetStringField(TEXT("gesture")), Obj->GetBoolField(TEXT("authored"))); else if (Type == TEXT("answer_end")) OnAnswerEnd.Broadcast(); else if (Type == TEXT("error")) OnServiceError.Broadcast(); } ``` `EndQuestion()` stops the microphone and sends `end_of_speech`. `EndPlay` closes the socket and calls `VoiceWave->ResetAudio()`. **Not compiled yet.** This is a skeleton. Check the exact signatures against the 5.8 API pages listed in the Sources and fix them in this page after the first build. ## 5. Microphone capture - **Use the AudioCapture plugin:** `UAudioCaptureComponent` for a simple route, or `Audio::FAudioCapture` / `IAudioCaptureStream` for raw buffers in C++. - **Convert** the float samples to **16 kHz mono PCM16**. Send 20–40 ms frames as binary WebSocket messages. - **Push-to-talk:** a button (kiosk), controller trigger or pinch (headsets). Background noise in museums makes always-listening unreliable. - **Permissions:** Android headsets need the runtime microphone permission; visionOS/macOS need the `.plist` usage string. Linux may have no capture backend. ## 6. Latency and failure - **Target:** first voice audio under about 1.5 s after the visitor stops speaking ([[AI Mind — Process]]). - **Play the "Thinking" gesture** while waiting. Never use filler speech. - **Underflow:** `USoundWaveProcedural` fires an underflow delegate when the queue runs dry. Log it; it means the network or TTS is too slow. - **Timeouts:** - no `answer_meta` within 4 s → play an authored "Let me think on that another time" line, then return to Waiting; - socket error → set the program to offline mode ([[Tutorial — Authoring a Timed Experience]]). - **Log per turn:** timestamps (question end → first audio), answer ID, refusal flag. **No audio and no faces.** ## 7. Wiring to the figure - `VoiceAudio` is the **audio source for lip sync** (the MetaHuman Audio Live Link Source or Audio2Face; [[Tutorial — Authoring a Q&A Experience with Ambient Idle]]). - `OnAnswerMeta` gesture tag → the State Tree picks the gesture clip. - `OnCaption` / `OnTranscript` → the caption UI ([[Tutorial — Captions UI]]). ## 8. Check before a venue build - [ ] Works in a **Shipping** build on the kiosk PC and one headset - [ ] 50 test questions: median time to first audio under 1.5 s on the venue network - [ ] Offline fallback and timeouts tested (unplug the network mid-answer) - [ ] No keys in content or logs; logs contain no audio - [ ] Microphone permission flows tested on each surface ## Sources - Epic, Plugins in Unreal Engine: https://dev.epicgames.com/documentation/unreal-engine/plugins-in-unreal-engine - Epic, HTTP module API: https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/HTTP - Epic, WebSockets API: https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/WebSockets - Unreal Community Wiki, WebSocket client in C++: https://unrealcommunity.wiki/websocket-client-cpp-5vk7hp9e - Epic, UAudioCaptureComponent: https://dev.epicgames.com/documentation/unreal-engine/API/Plugins/AudioCapture/UAudioCaptureComponent - Epic, IAudioCaptureStream: https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/AudioCaptureCore/IAudioCaptureStream - Epic forums, Audio Capture component for mic capture: https://forums.unrealengine.com/t/knowledge-base-using-audio-capture-component-for-mic-capture/264943 - Epic, USoundWaveProcedural: https://dev.epicgames.com/documentation/unreal-engine/API/Runtime/Engine/USoundWaveProcedural - Full list: [[Sources — 2026-09-20]]