https://subscription.packtpub.com/video/data/9781806675555/p1/video1_7/from-prompting-to-ai-agents-in-snowflake # **From Prompting to AI Agents in Snowflake** Explore the frontier of artificial intelligence in Snowflake. In under 45 minutes, learn how to build an AI-powered application using Cortex AI and Streamlit natively in Snowflake with the Power of Snowflake AI Toolkit. This workshop will take you from the first step into building an entire application for an Industry use case. If you are looking to explore Snowflake, or just starting with it, or even an advanced user, this workshop is for you. --- This workshop, **“From Prompting to AI Agents in Snowflake,”** shows how to go from simple LLM prompts to full gen‑AI pipelines and agent‑like apps inside Snowflake using the open‑source Snowflake AI Toolkit (Streamlit + Cortex AI). --- ## 1. Context, goals, and toolkit overview - Presenters Shankar (Microsoft, Snowflake Data Superhero) and Vivek (Verizon, ML engineer/Kaggle expert) introduce a hands‑on session using their **Snowflake AI Toolkit**, built to complement their 2024 book on Snowpark and ML in Snowflake. - The toolkit is a **Streamlit app** that runs: - Locally or in the cloud. - Natively inside Snowflake as a Streamlit app (and optionally as a native app). - Entirely on **Snowflake Cortex AI** and `AI_` functions, without moving data out of Snowflake. - It is **open source** on the official Snowflake Labs GitHub, modular, and designed as both an accelerator and a playground for building use cases (RAG, fine‑tuning, agents) on real Snowflake data. --- ## 2. AI adoption challenges and design principles - The speakers outline common enterprise challenges: - **Unclear AI strategy**: organizations want “AI” but lack concrete, value‑driven use cases. - **Data chaos**: no single source of truth, duplicated data, unclear ownership, weak data strategy. - **Business–IT disconnect**: business focuses on ROI; IT focuses on security/platform, leading to misaligned expectations. - **Pressure for fast value**: leaders feel urgency to show AI wins as peers appear in media with AI success stories. - **AI out of reach for end users**: unlike consumer tools (ChatGPT), enterprise AI must be contextualized, data‑aware, and aligned with business outcomes. - Toolkit design goals: - Move **beyond hype** to practical, ROI‑oriented use cases. - Leverage **trusted data foundations** already in Snowflake. - Provide a **no‑code/low‑code UI** usable by business users and data scientists. - Act as a **POV accelerator**: quickly experiment, fail safely, and iteratively build an AI strategy. - Make AI “for everyone,” not just data scientists. --- ## 3. Playground: LLM and multimodal features ## 3.1 Text playground - The **Playground** section exposes basic LLM functions: - Users select a **Cortex model** (Snowflake native, OpenAI, Claude, Gemini, etc., as supported by Cortex) and issue prompts directly. - Temperature and other parameters can be tuned; responses are visible immediately in the Streamlit UI. - This mode is meant as a “hello world” for Snowflake‑hosted LLMs and a quick prototyping surface for prompts. ## 3.2 Image understanding - Snowflake supports multimodal models like **Claude** and **Picasso‑large** for image tasks. - Workflow demo: - User chooses a model and a Snowflake database/stage that stores images. - Selects an image and issues prompts such as “Describe this image in English.” - The model returns detailed descriptions (e.g., describing a person taking a photo or summarizing slide charts). - This shows how to run bulk image understanding over images already stored in Snowflake. --- ## 4. Build mode: transforming tables and RAG over documents ## 4.1 Table‑level LLM pipelines (summarization, translation, sentiment) - **Build** mode lets users apply LLM operations over entire tables: - User selects a function (e.g., summarize, translate, extract sentiment) and a model. - Picks a database/table with text columns; the toolkit previews sample data. - Writes a prompt like “You are a summarizer. Summarize the description in less than five words.” - Specifies output table and column names (e.g., `SUMMARY_TEXT`). - After running, the toolkit creates a **new Snowflake table** with original text plus the generated summaries, enabling downstream pipelines or further transformations. ## 4.2 RAG over documents (contract example) - The toolkit supports **RAG** using documents stored in Snowflake stages: - Use **Build → RAG** to select documents (e.g., many contract PDFs) from a stage. - Choose an embedding model and create an “embedding table” (vector store) that holds chunked document embeddings. - Once embeddings are built, users switch back to Playground → **Chat** and: - Select the RAG table and model. - Ask questions like “What are the different types of contracts?” or “List all effective dates.” - The system retrieves relevant chunks and composes answers grounded in contract content. --- ## 5. Cortex Search, conversational querying, and agents ## 5.1 Cortex Search on tables - The workshop introduces **Cortex Search** for semantic search over Snowflake tables: - In Build → Cortex Search, user defines a **search service** by selecting a table, specifying which text columns to index, a document ID, and a model. - Snowflake builds an internal embedding index (not directly visible as a user table). - In Playground → Chat, users select **Cortex Search mode**: - Choose the search service, pick display columns, and limit result counts. - Query examples: “Show developer‑related issues” against a large `QA_DATA` table of textual issues; results show a filtered list of relevant issue descriptions. - Follow‑up questions can dive deeper into specific issues. - Notifications view shows underlying queries/errors if something fails, allowing debugging of Cortex‑generated SQL. ## 5.2 Cortex Agents (early agentic capabilities) - The **Cortex Agent** feature combines **Cortex Analyst** (NL → SQL) and **Cortex Search**: - Users create agents by naming them and attaching tools/resources (currently Cortex Search and Analyst) in the toolkit. - For example, a “Sales Intelligence Bot” agent can: - Translate natural language to SQL via Analyst. - Use Search to retrieve semantically similar records. - In Playground, users select **Cortex Agent** and chat with the agent, allowing it to decide whether to run queries or search based on intent. - The presenters mention this as the initial step toward richer agentic frameworks on Snowflake, with more tools and multi‑agent features planned. --- ## 6. Fine‑tuning models in Snowflake ## 6.1 When to fine‑tune - The speakers stress a hierarchy of techniques: - Start with **prompt engineering**. - If insufficient, add **grounding/RAG**. - Only then consider **fine‑tuning**, due to cost and complexity. ## 6.2 Fine‑tuning workflow in the toolkit - In Build → **Fine‑tune**: - User selects a base model (only certain Cortex models support fine‑tuning). - Chooses training and validation tables formatted with `PROMPT` and `RESPONSE` columns (plus optional metadata). - Example: a table where each row pairs a customer question with the desired answer, providing supervised examples of ideal behavior. - Assigns a name to the fine‑tuned model and launches training. - The process runs asynchronously; users monitor progress in **Notifications** via tracking IDs. - Finished models appear in the Playground model list under a **Fine‑tuned** section and can be used like any other model for prompts or pipelines. --- ## 7. Operational details, limitations, and roadmap ## 7.1 Usage, cost, and trial - Individuals can try the toolkit on Snowflake’s **free trial** (e.g., $400 credits for ~1 month), enough to: - Learn Snowflake. - Explore the toolkit. - Prepare for Snowflake certifications. - Run POCs on small datasets. - The presenters caution **not** to use the open‑source toolkit directly on production data; for production, they recommend working with them on a more hardened “pro” version. ## 7.2 Toolkit structure and extensibility - The app consists of: - An **Overview** page (links, setup instructions, debug mode). - **Playground** (LLM, multimodal, chat, RAG, search, agent chat). - **Build** (table transforms, RAG builder, Cortex Search builder, fine‑tuning). - Debug mode allows developers to modify the underlying Python/Streamlit code and re‑deploy customized workflows. ## 7.3 Differences from Snowflake Copilot - Snowflake **Copilot**: focused on assisting with SQL and Snowflake usage; a chat interface integrated into Snowflake UI. - **Toolkit**: - Builds full **pipelines**: RAG indexes, Cortex Search services, fine‑tuned models, and agent definitions. - Serves as a **framework** and accelerator for custom apps, not just a chat helper. - Intended to **complement** Copilot, not compete with it. ## 7.4 Roadmap highlights - Planned features: - Integration with **TruLens** for LLM observability and evaluation metrics (quality monitoring for RAG and fine‑tuning). - Richer **agentic interfaces**: - More Cortex tools. - Integration with external agent frameworks for things like multi‑agent setups and LLM‑as‑judge workflows. - **MCP client** support: - Toolkit as an outbound MCP client connecting to external agents/data sources (e.g., SAP) beyond Snowflake. - **API layer**: exposing toolkit functionality as APIs for custom front‑ends (React, Angular, containerized apps). - **Industry demo data packs**: preloaded datasets for verticals (retail, etc.) so teams can show value before onboarding their own data. - **Voice interface**: allow users to speak queries in the browser and have them processed via Cortex Analyst; responses would be spoken and displayed, enabling a hands‑free conversational experience. --- ## 8. How to use this as a learning and action guide - To replicate the workshop: - Sign up for a Snowflake trial; enable Cortex AI and required features. - Clone the **Snowflake AI Toolkit** from Snowflake Labs GitHub and run it natively in Snowflake or locally via Streamlit. - Start in **Playground** to experiment with prompts and multimodal features. - Move to **Build** to: - Summarize or translate real tables. - Create RAG indexes over contracts or docs. - Configure Cortex Search over issue/FAQ tables. - Define basic Cortex agents. - (Optionally) fine‑tune small models on curated prompt/response tables. - The presenters encourage starring the repo, providing feedback, and even contributing code or ideas; they actively evolve the toolkit based on community input and enterprise needs. 1. [https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/45e928d4-e174-4e48-b897-6b874c1810c7/paste.txt](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/45e928d4-e174-4e48-b897-6b874c1810c7/paste.txt) 2. [https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/2ee025f7-cabe-4041-992e-39e39dd2d69e/paste.txt](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/2ee025f7-cabe-4041-992e-39e39dd2d69e/paste.txt) 3. [https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/c9fd2284-b316-4126-b35b-873091643fe9/paste.txt](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/c9fd2284-b316-4126-b35b-873091643fe9/paste.txt) 4. [https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/92f15510-9262-495d-8c8a-3cf05c571b33/paste.txt](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/92f15510-9262-495d-8c8a-3cf05c571b33/paste.txt) 5. [https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/aff95306-1187-4c03-8550-57188987c3e3/paste.txt](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/aff95306-1187-4c03-8550-57188987c3e3/paste.txt) 6. [https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/bbb4159c-a892-4e4c-b130-7b7111f7f906/paste.txt](https://ppl-ai-file-upload.s3.amazonaws.com/web/direct-files/attachments/139614499/bbb4159c-a892-4e4c-b130-7b7111f7f906/paste.txt) ---