# Human Aligned Generative Interfaces ![[images/Human-Aligned-Generative-Interfaces.webp]] **Creator:** Jeffrey Yoon (channel [@jeffreyyoon3618](https://www.youtube.com/channel/UCdkukRmGNlasP7tK7etHTNg); speaker Janghan Yoon / 윤장한) · **Published:** 2026-11-08 · **Length:** 18:45 · **Language:** Korean speech, English slides · [Watch on YouTube](https://www.youtube.com/watch?v=hx5KKa9L8Qw) > Caption source: `youtube-transcript-api` **Korean auto-generated** (`ko`, the only caption track; 535 segments, complete). `yt-dlp` (android_vr) hit HTTP 429 on two spaced attempts. A box Whisper pass (faster-whisper medium, Korean) was run as a cross-check: it fixed some words (e.g. 윤장환 / "Human-Line", "LLM" for "LRM", "WCAG") but dropped long stretches (roughly 1:13-5:10 and 15:18-16:20), so the YouTube track is the primary transcript. Technical terms garbled by the captions are corrected from the slides and papers (e.g. "젠UI/ZUI" = GenUI, "언라인/얼라인" = A11yn, "CQ UI/SICK UI" = SeekUI, "엑스코어" = Axe-core, "GPT45" = GPT-4o, "클라우드" = Claude, "코엔 2.5 코더" = Qwen2.5-Coder). **The quotes below are my English translations of the Korean speech**, with the Korean caption text alongside; they are not the speaker's own English. Video: yt-dlp format 134 (360p, video only); frames taken from it, video deleted. Official thumbnail as cover. No chapters in the description. Added from the Summarize playlist rotation. > [!note] Framing > This is a short academic presentation (it looks like a lab or seminar talk recorded over a video call; a "Janghan Yoon" webcam tile sits in the corner). Two of the three papers it covers (GenUI and SeekUI) are by other groups; A11yn is the speaker's own first-author paper. The video description lists the three papers: "[ACL 2026 Findings] Generative Interfaces for Language Models", "[COLM 2026] A11yn: Aligning LLMs for Web-Accessibility Aware UI Generation" and "[CHI 2026] SeekUI: Predicting Visual Search Behavior on Graphical User Interfaces with a Reward-Augmented Vision Language Model". The last part is proposed future work, not results. > [!quote] Thesis > "As AI takes on more of the generating and executing, the human role may shift toward quickly reviewing and judging the results. So I think generative interfaces should develop in the direction of reducing cognitive load, so that people can review results accurately and efficiently." — Janghan Yoon ([18:14](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1094s); "AI가 생성과 실행을 더 많이 맡을수록 사람의 역할은 결과를 빠르게 검토하고 판단하는 쪽으로 옮겨갈 수 있습니다…") ## Outline ### Opening: what should an AI's output look like? ([0:00](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=0s) to [1:00](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=60s)) - [0:04](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=4s) Introduces himself as Janghan Yoon, in the third semester of the integrated program in the Department of AI (title slide: "Department of AI, Yonsei University"). Subtitle: "Turning model capability into usable and accessible interaction". - [0:12](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=12s) Today people use ChatGPT-style AI by typing a question, reading the answer and asking follow-ups until satisfied; the model uses **text as its one interface**. - [0:31](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=31s) But what should AI output look like to actually help? An interface that looks good but hides needed information, nudges people toward wrong judgments, or can't be operated by some users is not a good interface. - [0:44](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=44s) Plan: research on generative interfaces that help people understand efficiently and that diverse users can use, then his own work in progress. ![[images/Human-Aligned-Generative-Interfaces-t0005.webp]] *Title slide: "Human-Aligned Generative Interfaces", Department of AI, Yonsei University, 윤장한* ### Why the interface matters ([1:00](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=60s) to [2:30](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=150s)) - [1:00](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=60s) Models are usually compared by benchmark or reasoning scores, but users don't experience ability as an abstract score. They experience it while reading quickly, finding what they need and comparing options. - [1:13](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=73s) Example: Korea's recent sovereign ("독자") AI foundation model evaluation showed that even with strong raw performance, **model UX became a real factor**. The slide shows Korean news headlines from August 2026 (AI Times, a Naver blog, Yonhap) about the evaluation, including one saying the benchmark leader was eliminated amid a "usability" controversy. Slide line: "Model capability reaches people through the interface." - [1:25](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=85s) The interface is the contact point where users meet AI information, state what they want and check results. Flow on the slide: **model output** (information, recommendations) → **interface** (presentation, timing, controls, feedback) → **human process** (perceive, judge, act) → **outcome** (task result and consequences). - [1:49](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=109s) If key information is hard to see or choices are hard to change, users can't use even good model information. Slide: "Interface failures can hide uncertainty, encourage overreliance, or block operation." ![[images/Human-Aligned-Generative-Interfaces-t0065.webp]] *"Why interface experience matters": August 2026 Korean headlines on the sovereign foundation model evaluation* ![[images/Human-Aligned-Generative-Interfaces-t0095.webp]] *"The interface mediates model output and human action"* ### Human-aligned interface generation ([2:00](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=120s) to [2:40](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=160s)) - [2:00](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=120s) Once the model also generates the interface, the scope of alignment widens: the model now decides what to emphasize, in what order, and what can be manipulated directly. - [2:15](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=135s) He frames **generative-interface alignment** as making those decisions fit users' goals and abilities so they can understand, judge appropriately and act. Slide, three questions: **Generate** (what interface should the model create?), **Guide** (how should the interface shape human decisions?), **Include** (who can perceive and operate the result?). ![[images/Human-Aligned-Generative-Interfaces-t0155.webp]] *"Human-aligned interface generation": generate, guide, include* ### Industry is moving there: GPT-6 Intelligent UI and Google Generative UI ([2:38](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=158s) to [3:35](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=215s)) - [2:38](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=158s) Companies are combining visual elements and interaction in model responses. **OpenAI's Intelligent UI** shows this directly: aware of the limits of chat UIs, OpenAI trained GPT-6 to combine text, visuals and interactive elements to fit the question. Slide quote: "Compose responses using text, visuals and interactive elements, choosing how they fit together based on your question." In the bicycle example the user selects parts on an interactive diagram instead of only reading text. Slide takeaway: "The model now chooses the form of the interaction, not only the content of the answer." - [3:06](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=186s) **Google Research's Generative UI** generates pages and tool simulations to fit prompts, introduced through **Gemini Dynamic View** and Search's **AI Mode**. Slide: model output = "content + layout + interaction"; "Generative UI is becoming an industry-level interface paradigm." - [3:22](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=202s) Effective interface generation reduces people's cognitive load and lifts the model's UX performance, which is why companies are actively using it. ![[images/Human-Aligned-Generative-Interfaces-t0185.webp]] *"GPT-6 makes interface composition part of model behavior" (OpenAI, GPT-6 and Intelligent UI for everyone, 2026)* ![[images/Human-Aligned-Generative-Interfaces-t0215.webp]] *"Google Research treats the whole experience as model output" (Google Research, Generative UI, 2025)* ### The output contract now includes interaction ([3:35](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=215s) to [4:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=268s)) - [3:35](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=215s) Example request: "I want to understand neural networks." A text answer can explain structure and meaning, but a generative UI lets the user learn by interacting instead of reading line by line. - [3:57](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=237s) With text, the core of the output was *what to say*. With interaction the model must also decide what appears first, what state follows a changed input, and how errors are undone. A nicely placed button that doesn't update the result when pressed makes the task impossible. - [4:17](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=257s) So interface generation covers the accuracy of language content, **program behavior** and **the human usage process**. Slide: previous contract "Prompt → text response"; new contract "Intent → information structure → interaction → action"; the model determines what appears, how state changes, what the user can do and how errors are corrected. ![[images/Human-Aligned-Generative-Interfaces-t0245.webp]] *"The output contract now includes interaction"* ### Part 1: GenUI, generative interfaces for language models ([4:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=268s) to [9:10](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=550s)) - [4:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=268s) The pipeline paper is **"Generative Interfaces for Language Models"**, presented at ACL this year (slide: Chen et al., Findings of ACL 2026). GenUI **proactively** generates an interface that helps solve the question even if the user never asks for a website: a learning tool for "I want to learn X", an exploration screen for "I want to compare options". Slide definition: "A task-specific UI that the language model proactively creates in response to a user query." Examples on the slide: "Explain neural network", "How can I learn piano effectively?", analysing three years of e-commerce purchase data. - [5:06](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=306s) How it works: between the user query and the code sits a **structured, interface-specific representation**. An LLM organizes the requirements and expresses the interface's behavior in two layers. - [5:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=328s) Layer 1, the **interaction flow**: the overall path through the task. For "I want to understand the principles of quantum physics" the flow is: read explanations on a home screen, run a simulation, look up terms. Movement between screens and sub-goals is the interaction flow. - [5:47](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=347s) Layer 2, **state changes inside each feature**: clicking the concept-explanation menu moves the panel from closed to open; choosing a simulation moves to a "simulation selected" state. Current state, user event and next state are written as **finite state machines**. Slide: "Interaction flows and finite state machines constrain the UI before code generation." The representation is then handed to a code-generation model as the design. - [6:26](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=386s) A spelled-out design doesn't guarantee the code implements it, so the next stage generates code and improves it through evaluation. Four steps (slide "GenUI planning and iterative refinement"): 1. **Specify the goal and required interaction**: features, components and a problem-solving strategy from the query. 2. **Represent behavior before generating code**: interaction flows and component-level finite state machines. 3. **Generate executable HTML, CSS and JavaScript**, using the query, requirements, structured plan and examples; reusable components such as charts and calculators, and related examples searched from the web. 4. **Score and refine candidate interfaces** with query-specific rubrics. - [7:01](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=421s) The first output isn't assumed good enough, so several candidates are made and evaluated. The key part is **evaluation criteria generated for each question**: a quantum-physics learning tool is judged not only on layout and clarity but on whether it explains the concept well. An LLM builds the criteria, sub-criteria and weights, and combines the per-candidate ratings into a **0-100 score**. - [7:30](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=450s) The top-scoring candidate and its feedback feed the next generation. In the paper the loop **stops at 90+ points or after at most five iterations**. - [7:53](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=473s) Validation by human preference study: the researchers built an evaluation set of **100 questions** across fields such as education and data analysis, generated GenUI interfaces with **Claude 3.7**, and compared against three baselines: Claude 3.7 text answers, GPT-4o text answers, and Claude 3.7 directly asked to solve the question with a UI (the "Instructed UI", included to isolate what the GenUI pipeline adds over just asking for a UI). - [8:30](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=510s) Raters saw two results for the same question side by side (pairwise win rate). GenUI was preferred **84%** vs Claude text, **69%** vs GPT-4o text and **75%** vs the instructed UI. Slide: 428 main-study annotators; dimensions "Functional, Interactive, Emotional"; "Strong evidence for response-form preference." - [8:55](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=535s) Takeaway: organizing information to fit the question and letting users manipulate it directly improves the experience. ![[images/Human-Aligned-Generative-Interfaces-t0305.webp]] *"A prompt becomes a task-specific interactive artifact" (Chen et al., 2026)* ![[images/Human-Aligned-Generative-Interfaces-t0335.webp]] *"GenUI formalizes interaction before generating code": user query → structured interface-specific representation → generated code and UIs, with iterative refinement and an adaptive reward function* ![[images/Human-Aligned-Generative-Interfaces-t0425.webp]] *"GenUI planning and iterative refinement": the four steps* ![[images/Human-Aligned-Generative-Interfaces-t0485.webp]] *"Users preferred GenUI across comparison settings": 84 / 69 / 75% win rate vs Claude chat, GPT-4o chat and Instructed UI; 428 annotators* ### Usability is not enough: accessibility ([9:10](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=550s) to [10:35](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=635s)) - [9:10](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=550s) Next question: can such well-liked interfaces be used by diverse users? That is **accessibility**. - [9:30](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=570s) Two requirements (slide "Generated interfaces must satisfy two user-centered requirements"): **usability** ("Can specified users achieve the task effectively and efficiently?": task success, effort, satisfaction, appropriate reliance) and **accessibility** ("Can people with diverse abilities perceive, operate, and understand the interface?": perceivable, operable, understandable, robust). A Venn diagram puts "Universal" at the overlap. - [9:56](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=596s) An interface that looks well made can still be unusable for some people: an **icon-only button without an accessible name** tells a screen-reader user nothing; broken heading or input structure makes the page hard to understand and navigate; **low contrast**, like cream text on white, is hard for low-vision users to read. - [10:20](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=620s) High visual polish or user preference doesn't remove accessibility problems. Slide: "Visually appealing generated UIs can still be inaccessible"; "Visual quality can hide structural barriers in the DOM and interaction semantics." The example shows a "World of Movies" page with 2 serious and 1 moderate issue next to a version with no violations. ![[images/Human-Aligned-Generative-Interfaces-t0575.webp]] *"Generated interfaces must satisfy two user-centered requirements": usability and accessibility* ![[images/Human-Aligned-Generative-Interfaces-t0605.webp]] *"A useful interface can still exclude users": inaccessible vs accessible versions of the same generated page* ### Part 2: A11yn, training accessibility into UI generation ([10:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=628s) to [15:05](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=905s)) - [10:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=628s) **A11yn** (pronounced "align") asks whether accessibility fixes can be built into the LLM's training objective. - [10:39](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=639s) Background: **WCAG** (Web Content Accessibility Guidelines), developed by the **W3C** through international consensus, gives a shared standard that doesn't depend on one model's or rater's taste (slide: WCAG 2.2). **POUR** = **P**erceivable (users can perceive the information), **O**perable (usable with inputs other than a mouse), **U**nderstandable (content and behavior can be understood), **R**obust (assistive tech such as screen readers can interpret the content correctly). WCAG turns these into concrete success criteria. - [11:18](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=678s) Automated checkers such as **AChecker** and **Axe-core** find WCAG violations quickly. Render the generated page in a browser, inspect the DOM structure, element attributes and color contrast, and you get feedback on which element has a problem. Slide "Machine-checkable feedback": render the page, inspect the DOM and accessibility tree, detect WCAG violations and severity, return a measurable training signal. "Accessibility can constrain the generation policy." - [11:42](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=702s) Is telling the model the rules enough? Three existing approaches: put accessibility requirements in the **prompt**, **audit the UI and feed the problems back** for repair, or use a **bigger model**. The chart's inaccessibility rate (lower is better; Axe / AChecker): Base 7B 0.40 / 0.62; A11y Prompt 7B 0.29 / 0.57; FeedA11y 7B 0.32 / 0.54; A11y Prompt 32B 0.18 / 0.49; A11yn 7B 0.05 / 0.26. - [12:02](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=722s) Prompting helps but leaves many violations. Multi-agent-style audit-and-fix loops help, but fixes can introduce new violations, and every request needs extra generation and checking. Scaling the model doesn't automatically buy accessibility. The point: there is a gap between *telling* a model about accessibility and the model *reliably producing* it. - [12:30](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=750s) So A11yn puts accessibility feedback into training and **post-trains** the model to cut violations at first generation. For one UI request it samples several code candidates (base model **Qwen2.5-Coder-7B-Instruct**), renders each in a browser, audits it with **Axe-core**, converts the audit into one reward, and uses **GRPO** to compare rewards within the group: raise the probability of higher-reward candidates, lower the rest. - [13:02](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=782s) The trained model outputs a more accessible interface **in a single pass**, with no separate audit-and-fix loop at inference time. - [13:09](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=789s) Reward design (slide "A11yn reward design"): (1) **WCAG auditing** finds violations per rollout; (2) **raw penalty** sums violations with **severity-aware weights** so issues that are bigger barriers count more (slide weights: critical 0.4, serious 0.3, moderate 0.2, minor 0.1; P_raw = Σ N_v · w_v); a lower penalty means a higher reward. - [13:44](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=824s) **Reward hacking**: trained only to cut the violation count, the model can find unexpected shortcuts, such as **deleting a button instead of fixing its accessibility problem**. The audit shows fewer problems, but the feature the user wanted is gone. - [14:03](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=843s) Fix: (3) **normalize the penalty by UI size**, the number of rendered DOM elements (P_norm = P_raw / D; R = 1 − P_norm). This measures problems relative to the whole page, removes the advantage of simply having fewer elements, and compares candidates of different sizes more fairly. Slide: "Raw Penalty based Reward favors shorter UI." - [14:37](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=877s) Results: A11yn cut accessibility problems sharply versus the base model, and the gains held on checkers not used in training, i.e. fairly robust. Slide "Accessibility gains transferred across auditors": **Axe-core −87.5%**, **AChecker −58.1%**; violation-free UIs **12/300 → 151/300**; Lighthouse score 97.68 (best in the table, which also lists GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.6, Best-of-8 and larger Qwen models). A per-rule heat map shows consistent reductions (e.g. low text contrast, unlandmarked content, links without names) without trading off other issues. ![[images/Human-Aligned-Generative-Interfaces-t0665.webp]] *"Accessibility provides auditable generation constraints": WCAG 2.2 POUR and machine-checkable feedback* ![[images/Human-Aligned-Generative-Interfaces-t0725.webp]] *"Previous approaches to enhance accessibility of generative interfaces": existing approaches remain insufficient* ![[images/Human-Aligned-Generative-Interfaces-t0785.webp]] *"A11yn learns accessibility during generation": prompt → rollouts → web accessibility reward → group advantages → policy update (Yoon et al., A11yn, 2026)* ![[images/Human-Aligned-Generative-Interfaces-t0815.webp]] *"A11yn reward design": WCAG auditing, severity-weighted raw penalty, DOM-normalized penalty* ![[images/Human-Aligned-Generative-Interfaces-t0905.webp]] *"Accessibility gains transferred across auditors": Axe-core −87.5%, AChecker −58.1%, 151/300 violation-free UIs* ### Part 3 (proposed): can predicted gaze guide layout? ([15:05](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=905s) to [18:05](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1085s)) - [15:05](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=905s) Recap: GenUI generates interaction that fits the user's goal; A11yn builds the accessibility requirement into training. Open question: can a model also learn **what to emphasize and where to place it** so tasks go better? OpenAI itself says Intelligent UI's **design judgment still needs improvement** (slide: "OpenAI: the model's design judgment still needs improvement"). - [15:31](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=931s) His focus: emphasis and placement. Example: a screen comparing travel budgets. Even if every cost is present, if the **total cost** is buried among line items, users waste time finding it. So check not just *whether* information is included but *how easily it is found*. - [15:58](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=958s) Finding a needed item is **visual search**, a concrete process for judging whether placement and emphasis help. The clue for observing it is **eye gaze**: where people look, in what order, until they reach the target. - [16:20](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=980s) His research question: can predicting this search behavior give the model feedback to place information where it's easier to find? (Slide: "Can predicted gaze provide feedback for UI generation?" Focus: information discovery; validate actual task performance with people.) - [16:32](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=992s) **SeekUI** (CHI 2026; slide "Guo et al., CHI 2026") predicts the gaze path when a person looks for a specific target on a UI. A vision-language model trained on real human gaze data, it considers both the screen layout and the target. Input: a UI screenshot and the target's text (example: "English (United States)" at the top right). Output: a predicted sequence of gaze locations; dots are fixations, lines the order. The paper's example shows human gaze (left) next to SeekUI's prediction (right). - [17:20](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1040s) Proposed use: generate UI candidates with **the same information and functions but different placement and emphasis**, pick the items that matter for the user's task, and compare how SeekUI predicts the search for them on each candidate. The SeekUI paper itself shows layout comparisons using such predictions. - [17:39](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1059s) Plan: start by using it to **select** among generated candidates, then connect feedback shown to work into **training** the generator. - [17:47](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1067s) The key test is whether predicted improvements help real users: do they find the information faster and still complete the original task accurately? Slide: "Validate with people: faster target discovery and accurate task completion." ![[images/Human-Aligned-Generative-Interfaces-t0935.webp]] *"Research direction: evaluating information discovery"* ![[images/Human-Aligned-Generative-Interfaces-t1025.webp]] *"SeekUI: predicting how users search a UI" (published example, Figure 1; red box = target)* ### Close: interfaces are part of model behavior ([18:05](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1085s) to end) - [18:05](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=1085s) AI now generates not only the answer but the interface people use. As AI does more of the generating and executing, people shift toward reviewing and judging, so generative interfaces should reduce the cognitive load of reviewing results accurately and efficiently. - Closing slide: "Interfaces are part of model behavior. Model capability becomes useful only when the interface helps people perceive, decide, and act." Three goals: generate the right interaction; improve the human outcome; keep the result accessible. ![[images/Human-Aligned-Generative-Interfaces-t1115.webp]] *Closing slide: "Interfaces are part of model behavior"* ## Key ideas - **Capability reaches people through the interface.** Benchmarks don't capture how fast users can read, find, compare and act. - **Generative-interface alignment** has three parts: *generate* the right interface, *guide* decisions well, *include* everyone who needs to use it. - **New output contract**: intent → information structure → interaction → action. The model now owns state changes and error recovery, not just wording. - **Plan before code** (GenUI): interaction flows plus finite state machines, then code, then query-specific rubrics and iterative refinement (stop at 90+ or five rounds). - **Preference ≠ accessibility.** Polished UIs can hide structural barriers (missing accessible names, broken headings, low contrast). - **Train it in, don't bolt it on** (A11yn): turn auditor output into a verifiable RL reward so accessibility comes out in one pass. - **Watch for reward hacking**: a count-based penalty rewards deleting features; normalizing by DOM size fixes that. - **Next frontier, design judgment**: use predicted gaze (SeekUI) to compare layouts by how quickly key information is found, then confirm with real users. ## Things mentioned ### Books None mentioned. ### Papers - **"Generative Interfaces for Language Models"**, Jiaqi Chen, Yanzhe Zhang, Yutong Zhang, Yijia Shao, Diyi Yang (Stanford / Georgia Tech), *Findings of ACL 2026* (arXiv 2508.19227; code [SALT-NLP/GenUI](https://github.com/SALT-NLP/GenUI)), discussed [4:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=268s) - **"A11yn: Aligning LLMs for Web Accessibility-Aware UI Generation"**, Janghan Yoon, Jaegwan Cho, Junhyeok Kim, Jiwan Chung, Jaehyun Jeon, Seungwon Lim, Youngjae Yu (Yonsei / Seoul National University), listed in the description as COLM 2026 (arXiv 2510.13914; code [jeffrobot/A11yn](https://github.com/jeffrobot/A11yn)), discussed [10:28](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=628s) - **"SeekUI: Predicting Visual Search Behavior on Graphical User Interfaces with a Reward-Augmented Vision Language Model"**, Zixin Guo, Yue Jiang, et al., *CHI 2026* (DOI 10.1145/3772318.3791178), discussed [16:32](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=992s) - OpenAI, "GPT-6 and Intelligent UI for everyone" (2026 announcement), [2:38](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=158s) - Google Research, "Generative UI" (2025), [3:06](https://www.youtube.com/watch?v=hx5KKa9L8Qw&t=186s) ### Tools, models, standards - **Products / models**: ChatGPT; GPT-6 with Intelligent UI; Gemini Dynamic View; Google Search AI Mode; Claude 3.7 (Sonnet); GPT-4o; Qwen2.5-Coder-7B-Instruct (A11yn base); on slides also GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.6, Qwen2.5-Coder-14B/32B-Instruct - **Methods**: finite state machines; query-specific rubrics; pairwise human preference; GRPO (Group Relative Policy Optimization); reward hacking; best-of-8 sampling; scanpath (gaze) prediction - **Accessibility**: WCAG 2.2 and the POUR principles (W3C); Axe-core (Deque); AChecker; Lighthouse; screen readers - **Other**: Korea's sovereign AI foundation model ("독파모") evaluation, August 2026 news (AI Times, Yonhap) ### People Janghan Yoon (speaker). Named only through slide citations: Chen et al. (GenUI), Yoon et al. (A11yn), Guo et al. (SeekUI). No other people named aloud. ## Biographies ### Janghan Yoon (윤장한), channel "Jeffrey Yoon" The speaker introduces himself as a third-semester student in the integrated (MS-PhD) program of the Department of AI at **Yonsei University** in Seoul; the title slide and webcam tile give "윤장한" / "Janghan Yoon". He is first author of **A11yn** (with co-authors from Yonsei and Seoul National University, including Youngjae Yu); the arXiv paper lists the email [email protected] and the code is at github.com/jeffrobot, which matches the "Jeffrey" channel name. The channel ([@jeffreyyoon3618](https://www.youtube.com/channel/UCdkukRmGNlasP7tK7etHTNg)) had 8 views on this video at capture. His research interests, as presented: human-aligned generative UIs, accessibility-aware code generation, and gaze-informed layout feedback. Nothing else is stated in the video or verified here. ## Claims check | Claim in video | Check | |---|---| | OpenAI trained GPT-6 to "compose responses using text, visuals and interactive elements, choosing how they fit together based on your question" (Intelligent UI) | **Correct.** Verbatim from OpenAI's "GPT-6 and Intelligent UI for everyone" post (rollout from 7-8 Oct 2026). The bicycle example matches OpenAI's demo as reported by The Verge. | | OpenAI says the model's design judgment still needs improvement | **Correct.** OpenAI: "There's still work ahead to improve the model's design judgment." | | Google Research's Generative UI surfaces in Gemini Dynamic View and Search AI Mode | Consistent with the slide's citation (Google Research, Generative UI, 2025); not re-verified here. | | GenUI was presented at ACL this year | **Correct.** *Findings of ACL 2026*, pp. 1499-1519 (Chen, Zhang, Zhang, Shao, Yang). | | GenUI uses interaction flows and finite state machines, then HTML/CSS/JS, then LLM-built rubrics scored 0-100; stops at 90+ or after five iterations | **Correct** per the paper ("until a candidate reaches an overall score of 90 or higher, or until we have completed five iterations"). | | 100-question evaluation set; GenUI on Claude 3.7; baselines Claude 3.7 text, GPT-4o text, Instructed UI | **Correct.** UIX has 100 queries; ConvUI used GPT-4o or Claude 3.7; IUI = Claude 3.7 prompted "Please help me solve it with UI". (Captions say "GPT45"; the slide and paper say GPT-4o.) | | GenUI preferred 84% vs Claude, 69% vs GPT-4o, 75% vs Instructed UI; 428 annotators | **Correct.** These are the paper's overall win rates (Table 1) and its 428 unique annotators. Ties and losses are not shown on the slide (e.g. vs GPT-4o: 30% loss, 1% tie). | | WCAG is a W3C standard built on the POUR principles; Axe-core and AChecker automate checks | **Correct.** | | A11yn: Qwen2.5-Coder-7B-Instruct, Axe-core audits as reward, GRPO, severity weights, penalty normalized by DOM elements | **Correct** per the arXiv paper (Axe-core → verifiable reward for GRPO). The weights 0.4/0.3/0.2/0.1 are from the slide. | | A11yn cut the inaccessibility rate 87.5% (Axe-core) and 58.1% (AChecker); IR 0.05 / 0.26 | **Correct** per the current arXiv abstract and Table 2. (An earlier v1 abstract said "lowering the Inaccessibility Rate by 60%".) | | A11yn is a COLM 2026 paper | Stated in the video description. An earlier version was an ICLR 2026 submission that was withdrawn (OpenReview); COLM acceptance not independently verified here. | | SeekUI (CHI 2026) predicts gaze scanpaths from a screenshot plus target text using a VLM trained on human gaze data | **Correct.** CHI '26 (Barcelona, 13-17 Apr 2026); reward-augmented VLM (Qwen2.5-VL-7B base) that predicts scanpaths from a screenshot and text cue. The supplement includes layout-optimization examples, as the talk says. | | Korea's sovereign foundation model evaluation showed model UX mattered | The slide shows real-looking August 2026 Korean headlines about the evaluation and a "usability" controversy; details not verified here. | | Using predicted gaze as feedback for UI generation | **Proposed work**, not a result; the speaker says it still has to be validated with people. | No tutorial section (a research talk on generative and accessible UIs, not a walkthrough of an AI tool). ## Related - [[Jeffrey Yoon]] (channel TOC)