Author(s): Selina Li Originally published on Towards AI. What they are, how they are related and how to select one for your use case Author: Selina Li, Tianyi Li · Introduction· Recap on how LLM works· What are the strategies, and how are they related to each other? ∘ 1. Pure Prompt ∘ 2. Agent + Function Calling ∘ 3. RAG (Retrieval Augmented Generation) ∘ 4. Fine Tuning· When to use which strategy?· Enjoyed This Story? Introduction In the ever-evolving landscape of Generative AI, certain buzzwords have become commonplace: “Prompt Engineering,” “Functional Calling,” “RAG,” and “Fine-Tuning.” Perhaps you have encountered them in discussions, blog posts, or conference presentations. Like you, I’ve heard about these concepts repeatedly over the past year, but until recently, they remained as separate dots in my mind. Recently these separate dots got connected after I attended a virtual training provided by Prof. Zhigang Sun, the creator of the open-source project ChatAll and founder of AGIClass.ai. I cannot wait to share this newfound clarity with you. This article will describe what each LLM strategy does and how they are related to each other, also simply covering how we can choose among them for our best use case. Recap on how LLM works Before we go further, let’s briefly recap on how LLM works to generate results. In layman’s terms, it is like finishing a cloze test: Given a sequence of words, LLM refers to all words in the dictionary, and evaluates the likelihood of seeing each of them coming as the next The word with the highest likelihood will be chosen and filled into the space The above steps repeat until there is no more space to fill in How LLM generate results. GIF from Prof. Zhigang Sun AGIClass.ai contents Going deeper, in order to work on these “cloze test”, LLMs have gone through a training process to read through all human knowledge and record the likelihood of seeing each token* as the next. (Note: 1 token represents one or several words) The process in which LLMs generate results for the “cloze test” is inference. Training and Inference are the two core processes that enable LLM to work as expected. This explains why sometimes LLM confidently speaks wrong — the human knowledge that it was trained on might not include sufficient information about the topics that we want it to talk about. Therefore, to enable LLM to function as desired, we will need to supply LLM with more context useful information about the topics. All the strategies including Function Calling, RAG and Fine tuning are around this. They are different approaches to provide more context useful information to LLMs. What are the strategies, and how are they related to each other? 1. Pure Prompt Image from Prof. Zhigang Sun AGIClass.ai contents, translated into English by the author The first strategy is pure prompting. By name it explains how this strategy works: You issue a query to the LLM The LLM provides a response This is simply “chatting”. You chat with LLM like chatting with another human. We use this strategy every day when we chat with LLM-powered ChatBots such as ChatGPT, Gemini, and Copilot. 2. Agent + Function Calling Image from Prof. Zhigang Sun AGIClass.ai contents, translated into English by the author The second strategy is Agent plus Function Calling. This is how this strategy works: You issue a query to the LLM Instead of answering you in a straight way, the LLM will refer to a given toolkits, and select a tool from it to perform some tasks first Why would the LLM do so? Could be because it needs some extra information, or it was educated to fulfil some pre-requisites before answering your questions. The toolkits here are a list of pre-written functions or API calls The LLM will decide on the tool that it wants to use, and return the name of the tool (i.e. function name) Upon receiving the tool name (function name), you use the tool (i.e. call the function) to perform the task as desired by the LLM, and get some useful information You pass the useful information back to the LLM, and the LLM uses it to generate a response to your original query If this still sounds abstract, let’s take an example: You issue a query to the LLM as a travel agent e.g. “Plan me a trip for the incoming Christmas holiday in Bali island” The LLM decides that in order to plan a good trip for you, it requires your budget information first. It refers to a given toolkit that includes multiple tools including get_budget(), get_destination_info(), get_weather() and so on, and decides it will use the tool named get_budget() Upon receiving the suggested tool name, you (as the application) call the function get_budget(). Let's say it returns you a budget of AUD1000. You pass the AUD1000 budget information to the LLM, and the LLM generates a list of travel plan for you based on your budget Note: function calling is definitely not limited to a single function. In this example, the LLM may decide it also requires the weather information and the destination information, thus it may select the other tools get_destination_info() and get_weather() as well. What functions and how many functions will be called depending on different factors, including: What functions are given in the toolkit Context including system prompt, user prompt and historical user information etc. As you might have noticed, this process involves LLM (which provides any possible answers) as well as functions/API calls (which comes with preset logics). Different from the traditional approach to use hard-coded logics (e.g. if else) to decide when to call which function/API, this process utilises the power of LLM to dynamically decide when to call which, depending on the context. And different from pure prompting, this process enables LLM to integrate with external systems through function/API calling. 3. RAG (Retrieval Augmented Generation) Image from Prof. Zhigang Sun AGIClass.ai contents, translated into English by the author The third […]
↧