![[images/thumbnail.jpg]] **Creator:** Gary Explains · **Published:** 2026-10-09 · **Length:** 11:33 · [Watch on YouTube](https://www.youtube.com/watch?v=qxO1l5iY33E) > *I ignored Unsloth for way too long... and now I regret it. Unsloth allows you to run local AI models on your PC or Mac. It has lots of features: everything from chat to image creation, including using coding harnesses with local models, and LLM fine-tuning.* (from the video description) Gary Explains argues that running AI locally is one of the best outcomes of the current AI era. He had been happily using **Ollama** and **LM Studio**, and that satisfaction made him ignore **Unsloth**—a choice he now calls a mistake. This video is an overview of Unsloth’s dynamic quantization, the growing Unsloth Desktop feature set, and a hands-on macOS demo. He stresses it is **not a sponsored video**. *Summary written from yt-dlp English auto-captions and TubeAlfred video metadata. On-screen UI text (model names, paths) is taken from screenshots where it clarifies the spoken demo.* ## 1) Detailed outline ### [0:00] Local AI is a big deal—and Unsloth was the missed third option - Gary repeats a familiar thesis: the ability to run AI on your own PC or laptop is one of the greatest things to come out of this AI era. - Until now he relied on **Ollama** and **LM Studio**—both actively developed, with new features almost weekly. - Being happy with those two made him ignore a third option: **Unsloth**. After using it more, he regrets getting on the train late. ![[images/gary-intro.jpg]] ### [0:50] What “local AI” and open weights mean - Local AI = software on your machine (usually using the GPU) that runs models released as **open weights**: the finished weights, not the training data or training recipe. - To fit on consumer hardware you need **quantization**: shrinking weight precision (e.g. 16- or 32-bit down to 8-bit / 256 levels, or 4-bit). Smaller models, but less “resolution” / accuracy—like compressing a photo—if the shrink is done well, the model stays very usable. ### [2:04] Unsloth’s dynamic quantization - Gary first noticed Unsloth because of **dynamic quantization**: instead of one shrink setting for the whole network, more important / more-activated layers keep higher precision and less-used layers get more aggressive compression. - Unsloth often cites roughly **~10% greater accuracy** for the same model size. Using Unsloth means access to those specially quantized (UD) models. ### [3:06] Coding harnesses and OpenAI-compatible API - The desktop app has not stood still; Gary says it can do more than Ollama and LM Studio (“Bionic” in the captions is almost certainly a mis-hear of LM Studio / similar tooling). - Like Ollama, you can drive coding harnesses with a **local** model instead of a cloud frontier model: **Claude Code**, **Codex**, **Open Code** (his current favorite), **Open Claw**, and similar—typically with one command. - Also like Ollama/LM Studio, Unsloth can act as an **API endpoint**: the same Python script that calls OpenAI can be pointed at Unsloth instead (100% compatible request shape) and run against a local model—saving token spend. He often adds a flag to choose cloud (e.g. GPT) vs local. ### [4:29] Images, video, and audio inside the desktop app - **Image generation** with local diffusion models, plus AI editing, **inpainting**, and **upscaling**. - **Video generation** with appropriate local models. - **Audio**: text-to-speech, voice cloning, and music generation—again all local. ### [5:18] Fine-tuning and decision models (JEPA) - A relatively unique desktop feature: **fine-tune** a model in-app—inject custom domain knowledge; Unsloth handles the workflow. He offers to make a full separate video if people want it in the comments. - After the release of **JEPA** (spoken roughly as “Jev” in the captions)—a decision-making / predictive model people wanted a piece of—you can also do decision-making training and run those decision models in Unsloth. ### [6:08] Install Unsloth Desktop - Go to [unsloth.ai](https://unsloth.ai) → download. Available for **macOS**, **Linux**, and **Windows**. - On macOS: download the `.dmg`, drag Unsloth to Applications, launch via Spotlight or Applications. ![[images/unsloth-download-page.jpg]] ### [6:35] Model Hub — Qwen 3.8 27B with UD quantization - Open **Model hub** in the left sidebar; he picks a heavily downloaded model he has covered on the channel: **Qwen 3.8 27B**. - Choose quantization level (16-bit, 6-bit, 5-bit, 4-bit, …). Names with **UD** use Unsloth **dynamic quantization**; he picks a UD 4-bit variant. ![[images/model-hub-qwen.jpg]] ### [7:11] Chat without the internet — The Shard - Select the downloaded model and ask a simple question: *What is the tallest building in London?* - No web; pure model knowledge. The UI shows a **thinking** process; answer: **The Shard**. ![[images/chat-tallest-london.jpg]] ### [7:37] Optional web search - Toggle the **Search** control under the chat box; ask *What is the tallest building in France?* - Model thinks, then searches and asks permission to read a Wikipedia page (e.g. The Link); after allow, it merges internal knowledge with live web results. ![[images/web-search-demo.jpg]] ### [8:13] Local code execution - Toggle **Code**; ask it to write a Python script for the first **500 prime numbers**. - It creates the script in a workspace, asks to authorize a run, then reports that the script was written and tested and explains how it works. ### [8:42] Image creation with Flux - Left sidebar → **Images** (Create / Edit / Transform / Inpaint / etc.). - He uses an already-downloaded **Flux 2** / FLUX.2 Klein-class model, prompts a **koi fish pond** (with a pelican on the edge in the on-screen prompt), hits Generate, and gets a strong local diffusion result. ![[images/images-flux-select.jpg]] ![[images/koi-pond-flux-generate.jpg]] ### [9:11] Music with Stable Audio 3 - Under More → **Audio** / music; pick **Stable Audio 3**, set parameters and a prompt, generate. He plays the result. ![[images/music-stable-audio.jpg]] ### [9:36] Coding harness: `unsloth start opencode` - In a Mac terminal: **`unsloth start opencode`**. - Open Code comes up wired to the **local** `unsloth/Qwen3.8-27B-GGUF` (Unsloth Studio at `http://127.0.0.1:8888`), not a cloud frontier model. ![[images/terminal-unsloth-start.jpg]] ### [9:58] Hard task — FizzBuzz in ARM64 assembly - Prompt: write **FizzBuzz** in **ARM64** assembly for Apple Silicon—not C, Python, Rust, or Go. - As a harness, Open Code writes code, runs it, finds bugs/misunderstandings, rewrites, and loops until it builds and runs correctly, with notes on the assembly. - After exit: directory has `fizzbuzz` binary and `fizzbuzz.s`; running it prints FizzBuzz up to 100. ![[images/opencode-fizzbuzz-arm64.jpg]] ![[images/fizzbuzz-result.jpg]] ### [10:49] Closing - He should have adopted Unsloth much earlier; he was focused on the alternatives and “missed the boat” a bit. - Again: **not sponsored**—genuine local-AI experience. Asks for comments; end card / music. ![[images/gary-outro.jpg]] ## 2) Things mentioned ### Software and platforms - **Unsloth** / **Unsloth Desktop** (Beta) — [unsloth.ai](https://unsloth.ai); macOS / Linux / Windows; Model hub, Library, Images, Train, More - **Ollama** and **LM Studio** (his prior daily drivers) - Coding harnesses: **Open Code**, **Claude Code**, **Codex**, **Open Claw** - **OpenAI** API compatibility / local API endpoint pattern - Install: `.dmg` on macOS; optional `curl -fsSL https://unsloth.ai/install.sh | sh` ### Models and techniques - **Open weights** vs training data - **Quantization** (16/8/4-bit) and Unsloth **dynamic quantization** (**UD-** prefixes; ~10% accuracy claim at same size) - **Qwen 3.8 27B** GGUF (UD 4-bit picks); chat/UI also shows other Qwen GGUF variants in places - **Flux 2** / **FLUX.2 klein** (image) - **Stable Audio 3** (music) - **JEPA** decision-making / predictive models (spoken as “Jev” in auto-captions) - In-app **fine-tuning** for domain knowledge ### Demo tasks and facts - Tallest building in London → **The Shard** (~310 m) - Tallest building in France → web search / Wikipedia (**The Link**, completed 2025, ~242 m in the on-screen search) - Python script: first **500 primes** - Prompted image: koi fish pond (pelican on the edge) - **FizzBuzz** in **ARM64** / AArch64 assembly on Apple Silicon (`fizzbuzz` + `fizzbuzz.s`) ### Concepts - Local vs cloud frontier models; token-cost savings - Agent-style tool use: web search with approval, local code run with approval - Harness loop: write → run → fix → rebuild ## 3) Biographies ### Gary Explains (Gary Sims) Tech educator and YouTuber behind the **Gary Explains** channel (@GaryExplains). He covers practical explanations of tech, mobile, and increasingly local AI tooling. In this video he presents from his studio setup and demos Unsloth on a MacBook Pro (`gary@Garys-MacBook-Pro`). Also linked in the description: [twitter.com/garyexplains](https://twitter.com/garyexplains) and [github.com/garyexplains](https://github.com/garyexplains). Tagline energy matches his “let me explain” brand. ### Unsloth (project / company) Open-source project best known for fast, memory-efficient LLM fine-tuning and for releasing **dynamically quantized** GGUF models. **Unsloth Desktop** is their local AI interface for running and training models—chat, multimodal generation, harnesses, and fine-tuning—available from unsloth.ai for macOS, Linux, and Windows. ### Qwen (Alibaba / Qwen team) Family of open-weight large language (and multimodal) models from Alibaba’s Qwen team. Gary demos **Qwen 3.8 27B** via Unsloth’s UD GGUF builds for chat and for Open Code on Apple Silicon. ### Related tools (context, not people) - **Ollama** — popular local LLM runner/API used heavily before Unsloth in Gary’s workflow. - **LM Studio** — GUI-oriented local model runner he also used. - **Open Code** — coding harness he prefers here; launched via `unsloth start opencode` against the local Qwen model. - **Claude Code** / **Codex** — Anthropic- and OpenAI-oriented coding agents he cites as harness options Unsloth can also wire to local models.