# AI Mind — Tutorials **Process note:** [[AI Mind — Process]] · Unreal side: [[Tutorial — Conversation Client Plugin]] · [[Tutorial — Authoring a Q&A Experience with Ambient Idle]] Collected 20 September 2026. Items marked ⚠️ are recent or come from a single source. Check them before relying on them. ## 1. Pipeline building blocks | Part | Resource | Link | Licence / notes | |---|---|---|---| | Retrieval (RAG) framework | LlamaIndex | https://docs.llamaindex.ai | Open source | | | LangChain RAG tutorial | https://python.langchain.com/docs/tutorials/rag/ | Open source | | | Haystack | https://docs.haystack.deepset.ai | Open source | | Vector store | pgvector (Postgres) | https://github.com/pgvector/pgvector | Open source | | Evaluation | Ragas | https://docs.ragas.io | Open source | | | DeepEval | https://deepeval.com/docs | Open source | | | promptfoo | https://www.promptfoo.dev/blog/promptfoo-joining-openai/ | ⚠️ Acquired by OpenAI (March 2026); watch its open-source status | | Guardrails / policy | NVIDIA NeMo Guardrails (Colang) | https://github.com/NVIDIA-NeMo/Guardrails · https://docs.nvidia.com/nemo/guardrails/configure-guardrails/colang | Open source | | | Llama Guard 4 (safety classifier) | https://huggingface.co/meta-llama/Llama-Guard-4-12B | Llama licence | | Speech recognition | faster-whisper | https://github.com/SYSTRAN/faster-whisper | MIT; local | | | whisper.cpp | https://github.com/ggml-org/whisper.cpp | MIT; local | | | Deepgram live streaming | https://developers.deepgram.com/docs/live-streaming-audio | Cloud | | Text-to-speech | ElevenLabs | https://elevenlabs.io/docs | Cloud; the contracted actor voice | | | Kokoro-82M | https://huggingface.co/hexgrad/Kokoro-82M | Apache 2.0; local | | | Piper | https://github.com/OHF-Voice/piper1-gpl | GPL-3.0; per-voice licences vary | | | XTTS-v2 | https://github.com/coqui-ai/TTS/issues/3490 | **Avoid:** non-commercial model licence, and no one left to sell a commercial one | | Face from audio | NVIDIA Audio2Face-3D | https://github.com/NVIDIA/Audio2Face-3D · https://docs.nvidia.com/ace/ace-unreal-plugin/2.5/ace-unreal-plugin-audio2face.html | SDK and plugins MIT; models under NVIDIA Open Model licence | ## 2. Local inference (open-weight models on the kiosk PC) **Why local:** offline venues and schools, no per-question cost, visitor audio stays on site, predictable latency. **Trade-off:** smaller models are weaker. Our design already constrains answers to retrieved sources, which helps a smaller model a lot. | Runtime | Link | Notes | |---|---|---| | llama.cpp | https://github.com/ggml-org/llama.cpp | MIT; GGUF quantised models; the base of many tools | | Ollama | https://ollama.com | MIT; easiest to run; OpenAI-compatible API | | LM Studio | https://lmstudio.ai | Free, proprietary; good for trying models on a desktop | | vLLM | https://docs.vllm.ai | Apache 2.0; best when many requests run at once (streaming server) | | TensorRT-LLM · NVIDIA NIM | https://github.com/NVIDIA/TensorRT-LLM · https://developer.nvidia.com/nim | Fastest on NVIDIA; more set-up | **Current open-weight families (September 2026; check each model card):** | Family | Sizes that fit one RTX 5090 (32 GB) | Licence | |---|---|---| | **Qwen 3.6 / 3.8** ⚠️ | 27B dense; 35B-A3B MoE (3B active) | Apache 2.0 | | **Gemma 4** | 12B, 26B-A4B MoE, 31B dense | Apache 2.0 | | **gpt-oss** | 20B (about 16 GB) | Apache 2.0 | | **Mistral Small 4** | 119B MoE; only fits at about 3-bit | Apache 2.0 | | **Meta Muse Glimmer** ⚠️ | 30B | Apache 2.0 (reported August 2026) | | Llama 4 | — | Llama Community Licence (conditions apply) | **Rules of thumb:** - **VRAM for weights** is about parameters × bits ÷ 8. Q4_K_M uses about 0.6 GB per billion parameters and Q8 about 1.05 GB. Add 2–6 GB for context. - **Leave room** on the same GPU for speech recognition, TTS, embeddings, lip sync and Unreal itself. That argues for a **20–35B model at Q4–Q5**, or a **separate inference PC** in the kiosk cabinet. - **MoE models** with about 3B active parameters give the fastest first token. - ⚠️ **Published speeds** on a 5090 are 40–90 tokens/s depending on the model. They come from one blog; benchmark on our hardware. ## 3. SillyTavern and similar tools | Tool | What it is | Licence | Our use | |---|---|---|---| | **SillyTavern** | Locally installed front end for character and role-play chat. Character cards, **World Info / lorebooks** (keyword-triggered context), connects to KoboldCpp, Ollama, OpenAI-compatible and cloud APIs | **AGPL-3.0** | **Prototyping and persona authoring sandbox only** | | KoboldCpp | llama.cpp-based local server with its own UI | AGPL-3.0 | Local back end for tests | | text-generation-webui | Local model UI | AGPL-3.0 | Tests | | Open WebUI | ChatGPT-style web UI for local models | BSD-3 + branding clause | Internal staff tool at most | **Why not SillyTavern in the kiosk:** 1. **Security:** its own docs warn never to expose it on the open internet without proper security. Basic auth has no rate limiting. 2. **Accuracy:** World Info injects text when keywords match. That is good for **voice and persona**, but it doesn't give **historian-controlled retrieval with citation IDs, refusal rules or an evaluation harness**, which our Ground Rules require ([[AI Mind — Process]]). 3. **Licence:** under AGPL, running a modified copy as a network service means publishing our changes. **Where it helps:** the conversation designer and historian can quickly try a persona card, tone and period vocabulary with a local model, and collect examples for the golden question set. ## Local tutorials (Augmented Traveler) | Tutorial | Covers | Owner | |---|---|---| | [[Tutorial — How the Mind Works]] | Architecture for new staff: authored-first, retrieval fence, policy in code, logging, owners | Everyone on the mind | | [[Tutorial — Building a Figure's Corpus]] | Sources, rights, transcription, shelf marks, A/B/C stores and D file, metadata, versioning, the monograph test | Corpus editor, historian | | [[Tutorial — Retrieval with Citations]] | Postgres + pgvector schema, chunking by dated entry, hybrid search with RRF, cutoff filter, citation IDs | AI engineer | | [[Tutorial — Writing a Persona for a Historical Figure]] | Voice dossier, persona prompt structure, A vs B first-person rules, plain speech, examples | Conversation designer, historian | | [[Tutorial — Policy Layer with Guardrails]] | Cutoff in code, refusals, trauma protocol, companionship ban, children, prompt injection; NeMo Guardrails / Llama Guard | AI engineer, historian | | [[Tutorial — Evaluating Answers]] | Golden question set, metrics (Ragas/DeepEval), regression, blind historian review, red-teaming, ship gates | AI engineer, historians, QA | | [[Tutorial — Authored Answers for Top Questions]] | Choosing top questions, recording and animating, matching rules, answer IDs, updating from logs | Conversation designer, Unreal engineer | | [[Tutorial — Speech In]] | Push-to-talk, VAD, streaming ASR local vs cloud, Nantucket vocabulary, noise, children's voices, testing | AI engineer, accessibility designer | | [[Tutorial — Speech Out]] | Contracted actor voice and consent, streaming TTS, pronunciation lexicon, latency, QA | AI engineer, voice director | | [[Tutorial — Choosing Cloud or Local Models]] | Decision note: criteria, options, data-retention terms, minors, suggested start, questions for the founders | CEO, CTO | | [[Tutorial — Local Inference for Offline Venues]] | Ollama / llama.cpp on the kiosk, model and quantisation choice, VRAM budget, structured output, benchmarks | AI engineer, DevOps | | [[Tutorial — Persona Sandbox with SillyTavern]] | Sandbox only: local install, character card and World Info from Corpus A, exporting to the golden set; never public | Conversation designer, historian | | [[Tutorial — Staff Dashboard]] | Logs, flags, pinning canned answers, re-evaluation, corpus version, access control, review cadence | DevOps, historian | | [[Tutorial — Privacy and Logging]] | What we log and never log, retention, notices, children (COPPA), schools (FERPA), 201 CMR 17.00, vendors; not legal advice | CTO, counsel | ## 4. Every AI-mind tutorial the project needs Status: ✅ written · 🔗 an external tutorial is enough · ✏️ local tutorial to write · ❓ decision needed first. | Need | Status | Where / plan | |---|---|---| | **Architecture overview** for new staff (retrieval first, policy layer, authored answers) | ✅ | [[Tutorial — How the Mind Works]] | | **Corpus ingestion**: transcription, shelf marks, dated chunks, A/B/C separation | ✅ | [[Tutorial — Building a Figure's Corpus]] | | **Retrieval (RAG) with citations**: hybrid search, pgvector, chunking | ✅ | [[Tutorial — Retrieval with Citations]] | | **Persona prompt and voice dossier** | ✅ | [[Tutorial — Writing a Persona for a Historical Figure]] | | **Policy layer**: cutoff date, refusals, trauma protocol, companionship ban, in code not prompt | ✅ | [[Tutorial — Policy Layer with Guardrails]] | | **Golden question set and evaluation harness** (Ragas/DeepEval; blind historian review) | ✅ | [[Tutorial — Evaluating Answers]] | | **Authored answers** for top questions (pre-recorded, animated) | ✅ | [[Tutorial — Authored Answers for Top Questions]] | | **Streaming speech recognition** in a noisy museum (push-to-talk, mic arrays) | ✅ | [[Tutorial — Speech In]] | | **Streaming TTS with the contracted actor voice** (consent, version lock) | ✅ | [[Tutorial — Speech Out]] | | **Lip sync choice** (MetaHuman audio source vs Audio2Face-3D) | ✅ | [[Tutorial — Authoring a Q&A Experience with Ambient Idle]] section 4 | | **Unreal client** | ✅ | [[Tutorial — Conversation Client Plugin]] | | **Local inference on the kiosk** (Ollama/llama.cpp/vLLM, model choice, quantisation, VRAM sharing) | ✅ (❓ model choice) | [[Tutorial — Local Inference for Offline Venues]] | | **Cloud vs local model decision** (quality, cost, privacy, offline) | ✅ (❓ founders decide) | [[Tutorial — Choosing Cloud or Local Models]] | | **Persona prototyping in SillyTavern** (sandbox only) | ✅ (❓ adopt or not) | [[Tutorial — Persona Sandbox with SillyTavern]] | | **Staff dashboard**: pin canned answers, review logs | ✅ | [[Tutorial — Staff Dashboard]] | | **Privacy and logging** (no audio, no faces; retention) | ✅ (counsel review) | [[Tutorial — Privacy and Logging]] | | Red-teaming (attempts to break character or policy) | ✅ | Section in [[Tutorial — Evaluating Answers]] | ## Skills this covers See [[Technical Skills Matrix]].