# Tutorial — Speech In *Augmented Traveler local tutorial.* Back to [[AI Mind — Tutorials]] · the runtime loop: [[AI Mind — Process]] · Unreal side: [[Tutorial — Conversation Client Plugin]] · captions: [[Tutorial — Captions UI]] How the visitor's spoken question becomes text. It covers **push-to-talk**, **voice activity detection (VAD)**, the choice of **streaming speech recognition (ASR)**, **custom vocabulary** for Nantucket names, noise, children's voices and testing. A wrong transcript produces a confident wrong answer, so this stage deserves real testing. > **Draft.** Written from the vendor and project docs listed under Sources. The AI conversation engineer owns it; the UX and accessibility designer checks sections 6–7. Correct parameter names here after the first build. ## 1. What arrives, and what must leave - **In:** 16 kHz mono PCM16 frames of 20–40 ms, between `start` and `end_of_speech` ([[Tutorial — Conversation Client Plugin]] section 2). - **Out:** one `transcript` message with the final text, sent before retrieval starts. - **Budget:** about **300 ms** from `end_of_speech` to final transcript ([[AI Mind — Process]]). - **Never stored:** the audio. It lives in memory for the turn only ([[Tutorial — Privacy and Logging]]). ## 2. Push-to-talk first, VAD second Museum rooms are loud: school groups, other exhibits, the figure's own voice. Always-listening is unreliable there. 1. **Push-to-talk is the default** on the kiosk (a large lit button) and on headsets (trigger or pinch). 2. **Release of the button** is `end_of_speech`. No guessing when the visitor has finished. 3. **VAD still helps:** - trims silence before and after the question; - ends the turn if the button is held but nobody speaks for about 2 s; - rejects a press with no speech in it (a child pressing for fun). 4. **Silero VAD** is the usual choice: MIT licence, about 2 MB, supports 8 kHz and 16 kHz, and a 30 ms chunk takes under 1 ms on one CPU thread. 5. **Mute the microphone while the figure speaks,** unless we build barge-in (the visitor presses to interrupt). Decide this with the UX designer. If barge-in is wanted, the mic array must do echo cancellation (section 5). ## 3. Streaming ASR: local or cloud | Option | Type | What the docs say | Our notes | |---|---|---|---| | **faster-whisper** | Local, Python | MIT; Whisper on CTranslate2; fp16 and int8; built-in Silero VAD filter; `transcribe()` takes `hotwords` and `initial_prompt` | Not streaming by itself; batch the utterance at release, or use a streaming wrapper | | **WhisperLive** | Local server | MIT; near-live Whisper server; backends faster_whisper, TensorRT, OpenVINO; takes 16 kHz mono int16 or float32; `use_vad`, `hotwords`, `initial_prompt` | Closest to our protocol; run on the kiosk PC | | **whisper.cpp** | Local, C/C++ | MIT; CUDA build; `whisper-stream` and `whisper-server` examples; Silero VAD via `--vad`; needs 16 kHz input; model sizes about 273 MB (tiny) to 3.9 GB (large) | Good if we want no Python on the kiosk | | **Deepgram** (Nova-3) | Cloud streaming | Keyterm prompting on Nova-3 (multi-word phrases); multilingual keyterms up to 500 tokens (about 100 words) per request | Audio leaves the building; see [[Tutorial — Privacy and Logging]] | | **AssemblyAI** Universal-Streaming | Cloud streaming | Keyterms prompting: up to 100 keyterms per session, 5–50 characters each; 16 kHz mono; applies to final transcripts only; free in beta at time of reading | Same privacy question | **Suggested default:** local Whisper (WhisperLive or faster-whisper) on the kiosk GPU for every venue. Audio then never leaves the site, and offline venues work the same way. Try one cloud service as a quality comparison in the test in section 8. Record the model name and size in the venue config. **Speed check (from the faster-whisper README):** large-v2 transcribed 13 minutes of audio in about 1 minute on an RTX 3070 Ti (fp16, beam 5). A 6-second question should therefore finish well inside 300 ms on a 5090. Measure it; don't assume it. ## 4. Custom vocabulary for Nantucket names Whisper and the cloud services all mishear local names. Keep one **vocabulary list per figure**, versioned with the corpus ([[Tutorial — Building a Figure's Corpus]]). | Term | Why it matters | |---|---| | Nantucket, Siasconset ('Sconset), Madaket, Polpis, Quidnet, Coatue | Place names visitors will say | | Wampanoag | Must be recognised correctly; never guessed | | Maria Mitchell, Lucretia Mott, Absalom Boston, Frederick Douglass | Figure names | | Atheneum, Pacific Bank, Main Street, New Guinea, African Meeting House | Places in the answers | | Essex, Veeder, Brock | Whaling names | | comet, telescope, transit, sextant, meridian | Mitchell's science words | | Quaker, Friends, meeting house, abolition | Period and faith words | How to apply it: 1. **Local Whisper:** pass the list through `hotwords` (faster-whisper, WhisperLive), or a short `initial_prompt` such as "Questions for Maria Mitchell about Nantucket, Siasconset, the Atheneum and the comet." 2. **Deepgram:** send the terms as keyterms (Nova-3). 3. **AssemblyAI:** send up to 100 keyterms; drop any term under 5 characters or it is rejected. 4. **After ASR:** a small correction table for known mishearings ("sconce it" → "Siasconset"). Log each correction so the list improves. 5. **Pronunciations** for the voice are a separate list ([[Tutorial — Speech Out]]). ## 5. Noise and microphones - **Directional mic array** with beamforming, noise suppression and echo cancellation. Kiosk microphone vendors describe exactly this set for public kiosks (see Sources); pick a product with the Kiosk and AV integrator ([[Position — Kiosk and AV Integrator]]). - **Mount it** at mouth height between an adult and a child, aimed at the standing mark on the floor. - **Keep the figure's speaker** away from the mic's pickup direction. - **No always-on processing** in the cloud: the mic only streams while the button is held. ## 6. Children's voices Children are a large share of museum visitors, and ASR is much worse for them. - The **Kid-Whisper** study measured Whisper word error rates on children's speech at about **3–10 times** the adult rate (about 13–21% against about 3% on adult read speech). - **Design for it:** - always show the transcript (section 7) so a child sees a mishearing; - offer the touch alternative on every screen; - include children's voices in the test set (with consent, section 8). - **Fine-tuning** on children's speech helped in that study. Treat it as a later option, not a launch task. ## 7. Showing the transcript, and a touch alternative 1. When `transcript` arrives, show "You asked: …" briefly ([[Tutorial — Captions UI]] section 4). 2. If the transcript is empty or low-confidence, the figure does **not** guess. It plays an authored line ("I did not quite catch that. Ask me again?"; final wording from the historian) and shows question buttons. 3. **Touch alternative:** 6–8 suggested questions on screen, drawn from the authored answers ([[Tutorial — Authored Answers for Top Questions]]). Visitors who cannot or will not speak get the same experience. ## 8. Testing with recorded museum noise 1. **Record 30 minutes of room noise** at each venue, with no recognisable speech kept (walk-through before opening, or crowd noise with no one close to the mic). Delete anything with intelligible conversation. 2. **Record 200 test questions** from the golden set ([[Tutorial — Evaluating Answers]]): staff, volunteers and, with written parental consent, children. Store under the test folder only; never in production logs. 3. **Mix** each question with venue noise at three levels (quiet, normal, school group). The MUSAN corpus (CC BY 4.0) adds extra noise types. 4. **Run every ASR option** and compute word error rate and **name accuracy** (did "Siasconset" survive?). ```python # skeleton — test before use import jiwer def score(pairs): """pairs: list of (reference_text, asr_text).""" refs = [r.lower() for r, _ in pairs] hyps = [h.lower() for _, h in pairs] names = ["nantucket", "siasconset", "wampanoag", "atheneum", "mitchell"] hits = sum(1 for r, h in zip(refs, hyps) for n in names if n in r and n in h) total = sum(1 for r in refs for n in names if n in r) return {"wer": jiwer.wer(refs, hyps), "name_recall": hits / max(total, 1)} ``` **Targets to agree with the CTO:** word error under 10% for adults at normal noise; name recall over 95%; final transcript under 300 ms after release. ## 9. Practice exercise Record yourself asking "Did you see the comet from the Pacific Bank roof?" five times: close, at 1 m, with a noise track, with the mic muted halfway, and as fast as you can. Run faster-whisper with and without `hotwords`. Write down the errors and add any new mishearing to the correction table. ## 10. Check before a venue build - [ ] Push-to-talk works on kiosk and headsets; mic muted while the figure speaks (or barge-in tested) - [ ] VAD rejects empty presses and trims silence - [ ] Vocabulary list for the figure loaded and versioned - [ ] Noise test run with that venue's recording; results filed - [ ] Children's voices included in the test (with consent) - [ ] Transcript shown; "didn't catch that" path and touch questions work - [ ] No audio written to disk anywhere (check temp folders) ## Sources - SYSTRAN, faster-whisper: https://github.com/SYSTRAN/faster-whisper - SYSTRAN, faster-whisper `transcribe.py` (hotwords, initial_prompt, VAD options): https://raw.githubusercontent.com/SYSTRAN/faster-whisper/master/faster_whisper/transcribe.py - Collabora, WhisperLive: https://github.com/collabora/WhisperLive - ggml-org, whisper.cpp: https://github.com/ggml-org/whisper.cpp - Silero, Silero VAD: https://github.com/snakers4/silero-vad - Deepgram, Nova-3 multilingual keyterm prompting: https://deepgram.com/learn/deepgram-expands-nova-3-with-10-new-languages-and-multilingual-keyterm-prompting - Vapi, Deepgram keywords and keyterm prompting: https://docs.vapi.ai/customization/custom-keywords - AssemblyAI, Keyterms prompting (streaming): https://www.assemblyai.com/docs/speech-to-text/universal-streaming/keyterms-prompting - arXiv, Kid-Whisper: Towards Bridging the Performance Gap in ASR for Children vs Adults (2309.07927): https://arxiv.org/html/2309.07927 - Andrea Electronics, Information kiosk microphones: https://andreaelectronics.com/information-kiosk-microphones/ - OpenSLR, MUSAN corpus: https://www.openslr.org/17/ - jiwer, usage: https://jitsi.github.io/jiwer/usage/ - Full list: [[Sources — 2026-09-20]]