# Tutorial — Staff Dashboard *Augmented Traveler local tutorial.* Back to [[AI Mind — Tutorials]] · what is logged: [[Tutorial — Privacy and Logging]] · evaluation: [[Tutorial — Evaluating Answers]] · owner: [[Position — DevOps Engineer]] The **staff dashboard** is where the historian, conversation designer and venue staff see what visitors asked and fix problems **without retraining anything**. The key action is to **pin a canned answer overnight** ([[AI Mind — Process]]). This page lists the features, access control, build options and the review cadence from the [[Ground Rules — Historical Virtual Humans]]. > **Draft.** Written from the project process notes and the tool docs listed under Sources. The DevOps engineer builds it; the historian and conversation designer validate it in the first two weeks after opening. Screen and button names are placeholders. ## 1. What it shows, and what it never shows | Shows | Never shows | |---|---| | Question text (after redaction) | Audio | | Answer text, answer ID, authored-or-generated flag | Faces, camera images | | Corpus hits (citation IDs) | Names or contact details volunteered by visitors (redacted before storage) | | Refusal flags and which rule fired | Anything that identifies a person | | Confidence / retrieval score | Device IDs beyond the kiosk's own | | Timings (ASR, retrieval, first token, first audio) | | | Venue, figure, anonymous session ID, corpus version | | The fields come from [[Tutorial — Privacy and Logging]]. If a field isn't logged, the dashboard can't show it; that is the point. ## 2. Features 1. **Question log.** Filter by venue, figure, date, answer type. Full text search over questions. 2. **Refusal flags.** Every refused or redirected question, with the rule that fired. Staff can mark "correct refusal" or "should have answered". 3. **Low-confidence answers.** Generated answers with weak retrieval scores or no citations. Sorted worst first. 4. **Pin a canned answer.** - Pick a question (or a cluster of similar ones). - Choose an existing authored answer, or write a short reviewed reply. - It needs **historian approval** before it goes live. - It takes effect at the next nightly sync (or immediately, for a refusal). - Unpin the same way. Every pin and unpin is logged with who and why. 5. **Trigger re-evaluation.** Run the golden set against the current corpus and model; show the score beside the last run ([[Tutorial — Evaluating Answers]]). 6. **Corpus version.** Which corpus version each venue is running; when it changed; the correction note ([[Tutorial — Building a Figure's Corpus]]). 7. **Per-venue settings.** Model route (cloud or local; [[Tutorial — Choosing Cloud or Local Models]]), offline mode, touch-question list, captions default, open hours. Changes need a second person to approve. 8. **Flag for the golden set.** One click adds a real question (redacted) to the golden-set candidates. 9. **Export.** CSV of questions for a date range, for the monthly review. No other export. ## 3. Access control | Role | Can see | Can change | |---|---|---| | Historian of record | Everything in section 1 | Approve pins; veto answers | | Conversation designer | Everything in section 1 | Draft pins; flag for golden set | | Second historian | Blind review screen only | Scores | | Venue staff | Their venue's log, last 7 days | Offline switch; report a problem | | DevOps / AI engineer | Everything, plus timings and errors | Settings (with a second approver), re-evaluation | | Anyone else | Nothing | Nothing | - **Single sign-on** or named accounts with multi-factor sign-in. No shared logins. - **Least access:** venue staff see only their venue. - **Audit log** of every sign-in, pin, unpin and setting change. Keep it longer than the question log. - **Not on the kiosk.** The dashboard runs on a studio server or cloud host, never on the public screen. ## 4. Build approach options | Option | What it is | Licence | For | Against | |---|---|---|---|---| | **A. Small custom app: FastAPI + a simple web UI** | Python API over the log database (Postgres); a few pages | FastAPI MIT (check); our code | Exactly our features, including pin approval; same language as the AI service | We build and maintain it | | **B. Metabase** for reading logs | Self-hosted business-intelligence tool over Postgres | Open-source edition AGPL | Filters and charts with no code | Read-only views; pins still need option A | | **C. Grafana** for timings | Dashboards over Postgres, Prometheus, Loki | AGPL-3.0 (with Apache-2.0 exceptions) | Good for latency and error monitoring | Not a review tool for text | | **D. Open WebUI or similar** | Chat UI | BSD-3 + branding clause | — | Wrong tool; not for logs | **Suggested:** A for review and pins, plus C for the DevOps engineer's latency and error panels. Use B only if the historian wants self-serve charts. Running Metabase or Grafana unmodified inside the studio avoids AGPL network-service questions; confirm with counsel if we modify them. FastAPI supports OAuth2, OpenID Connect, API keys and HTTP Basic in its security module. Use OpenID Connect with the studio's identity provider. ## 5. Data model (sketch) - `turn` — one row per question: IDs, venue, figure, session ID, question text (redacted), answer ID, flags, citation IDs, scores, timings, corpus version, model version, voice version. - `pin` — question pattern, answer ID or text, status (draft, approved, live, retired), approver, dates. - `setting` — venue, key, value, changed by, approved by. - `audit` — who, what, when. Keep the log database separate from the corpus database. ## 6. Review cadence From the [[Ground Rules — Historical Virtual Humans]]: | Period | Who | What | |---|---|---| | **Weeks 1–12 after opening** | Historian + conversation designer | Read **all** logs. Daily in week 1, then at least twice a week | | **After week 12** | Same | Monthly review of all refusals, low-confidence answers and a sample of the rest | | **Press spikes** (article, social media, school holiday) | Same, within 48 hours | Read everything from the spike period | | **Every corpus correction** | Corpus editor + AI engineer | New corpus version; re-evaluation run; note in the dashboard | **Each review ends with:** pins added or retired, golden-set additions, corpus corrections logged, and a short note filed in the dashboard. ## 7. Practice exercise Using ten made-up log rows for Maria Mitchell (write them yourself; no real data), find one refusal that should have been an answer, one low-confidence answer, and one question worth pinning. Draft the pin, send it for historian approval in the test system, and check it appears the next morning. ## 8. Check before opening - [ ] Sign-in with named accounts and multi-factor; roles set - [ ] No audio, faces or unredacted names reachable from any screen - [ ] Pin → approve → live → unpin tested end to end, overnight - [ ] Re-evaluation button runs the golden set - [ ] Corpus version shown per venue - [ ] Review rota for weeks 1–12 in the calendar ## Sources - FastAPI, Security intro: https://fastapi.tiangolo.com/tutorial/security/ - Metabase, GitHub repository: https://github.com/metabase/metabase - Grafana, GitHub repository: https://github.com/grafana/grafana - Full list: [[Sources — 2026-09-20]]