Building LLMs from scratch - Evaluating fine-tuned LLM using Ollama

In this lecture, we explore the process of evaluating instruction fine-tuned LLMs using Ollama. We demonstrate how to collect LLM responses systematically and save model parameters effectively. Ollama Setup: An introduction to Ollama for managing LLMs is provided, including setup instructions. Llama 3 Utilization: The lecture includes practical examples of querying Llama 3 and using it to evaluate the fine-tuned model. This is a very dense lecture explained through detailed whiteboard notes and live coding. The key reference book which this video series very closely follows is Build a Large Language Model from Scratch by Manning Publications. All schematics and their descriptions are borrowed from this incredible book! This book serves as a comprehensive guide to understanding and building large language models, covering key concepts, techniques, and implementations. Affiliate links for purchasing the book will be added soon. Stay tuned for updates! 0:00 Instruction fine-tuning recap 4:37 The need for LLM evaluation 6:40 Extracting and saving LLM responses 11:58 3 methods to evaluate instruction fine-tuned LLM performance 13:07 Evaluation Method 1 - MMLU 15:43 Evaluation Method 2 - Human preference comparison 16:04 Evaluation Method 3 - LLM measures another LLM 18:20 Collecting LLM responses in JSON format 21:35 Saving the LLM parameters 24:08 Evaluating the fine-tuning LLM introduction 24:33 Ollama introduction and setup 31:58 Query Llama 3 and generate response 35:55 Using Llama 3 to evaluate our fine-tuned LLM 45:24 Steps to improve the fine-tuned model 49:51 Recap and summary Code file: https://drive.google.com/file/d/1FQIzAGdBVgSqR_39Qx4jQYoAOYARkkld/view?usp=sharing Instruction data link: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/01_main-chapter-code/instruction-data.json MMLU paper: https://arxiv.org/pdf/2009.03300 Alpaca Eval: https://tatsu-lab.github.io/alpaca_eval/ Llama 3 8B-Instruct: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct Stanford Alpaca link: https://github.com/tatsu-lab/stanford_alpaca ================================================= ✉️ Join our FREE Newsletter: https://vizuara.ai/our-newsletter/ ================================================= Vizuara philosophy: As we learn AI/ML/DL the material, we will share thoughts on what is actually useful in industry and what has become irrelevant. We will also share a lot of information on which subject contains open areas of research. Interested students can also start their research journey there. Students who are confused or stuck in their ML journey, maybe courses and offline videos are not inspiring enough. What might inspire you is if you see someone else learning and implementing machine learning from scratch. No cost. No hidden charges. Pure old school teaching and learning. ================================================= 🌟 Meet Our Team: 🌟 🎓 Dr. Raj Dandekar (MIT PhD, IIT Madras department topper) 🔗 LinkedIn: https://www.linkedin.com/in/raj-abhijit-dandekar-67a33118a/ 🎓 Dr. Rajat Dandekar (Purdue PhD, IIT Madras department gold medalist) 🔗 LinkedIn: https://www.linkedin.com/in/rajat-dandekar-901324b1/ 🎓 Dr. Sreedath Panat (MIT PhD, IIT Madras department gold medalist) 🔗 LinkedIn: https://www.linkedin.com/in/sreedath-panat-8a03b69a/ 🎓 Sahil Pocker (Machine Learning Engineer at Vizuara) 🔗 LinkedIn: https://www.linkedin.com/in/sahil-p-a7a30a8b/ 🎓 Abhijeet Singh (Software Developer at Vizuara, GSOC 24, SOB 23) 🔗 LinkedIn: https://www.linkedin.com/in/abhijeet-singh-9a1881192/ 🎓 Sourav Jana (Software Developer at Vizuara) 🔗 LinkedIn: https://www.linkedin.com/in/souravjana131/