Author(s): Martbakler Originally published on Towards AI. A programmer Tanuki — created with DALL·E 3 TLDR: This workflow responds to customer feedback messages and parses them into prioritized support tickets using GPT4 + Tanuki (open-source). Who is this useful for? Anyone interested in creating applications integrating GPT4 capabilities to complex workflows with little to no effort. How advanced is this post? Basic knowledge of Python is required, but that’s about it. Tanuki takes care of the rest. What this post doesn’t do. We don’t get Tweets (see their API) or post to your customer support service (depends heavily on the tech stack). Introduction GPT4 has taken the world by storm. Potential use cases range from chatbots to classifiers, to data analytics, and now to vision. One remaining challenge is creating applications that require structured outputs, as language models return outputs in natural text. In addition, LLMs in zero-shot fashion are known to be misaligned (language models may understand the task differently) and produce hallucinations. To counter this, we have created Tanuki, a simple way to create aligned LLM-powered applications and ensure structured and typed outputs. Here, I will demonstrate how to use Tanuki in 20 minutes to create: A (Customer Support) chatbot A (Customer support) classifier, and Reliable structured objects for logging into databases. Use-case in a nutshell In this example use case, we will create a workflow that both acts as a customer support chatbot and creates potential issues in an internal system. In particular, we assume a tweet to a company’s support account as input. The workflow will create an empathetic response to the tweet and classify it to determine whether it requires further action from the company’s support team. If the tweet does require further action, then a support ticket is created that you can save to your internal database for further action. Tanuki will be used in this example to create the response, classify the tweet, and create the support ticket if needed. We were able to build this Python app in 20 minutes. Now, we want to show you how to do the same. If you’re impatient, you can find the repo and use-case in the following links: Tanuki repo Github Code for this particular use-case here or in a Google Colab notebook What is Tanuki, and why is it useful for this use case? Tanuki is an open-source library for easy and seamless building of LLM-powered apps with little to no required LLM or prompting knowledge. The useful features of Tanuki, especially for this use case are: Type-Awareness: The outputs will always follow the type hinting structure that the user specifies (a nightmare to deal with using LLMs, as the outputs are free-form text). The outputs can be specified as Python base types (ints, lists, dictionaries) or more complex types such as Pydantic classes or Literals Language model behavior is easily alignable through demonstrative mock examples (follows the idea of In-Context Learning). Adding these align statements is not required but usually significantly improves model performance. Adding mock examples is as easy as writing them down in a notepad. Tanuki carries out automatic model distillation, i.e., the teacher model (GPT-4) is automatically distilled down to a smaller fine-tuned model giving up to 15x reductions in both cost and latency without sacrificing performance. These features are relevant for this use-case as: The outputs of the LLM will be used downstream by other systems. This required the outputs to always be typed to ensure nothing breaks and there will be no data-induced runtime bugs. This is crucial, as the workflow creates structured ticket objects that need to be logged to databases that expect data in a given schema. What should (and shouldn’t) be acted upon is subjective and not obvious. Moreover, how the language model responds to clients' feedback is of huge importance (the tone and message need to be correct to not anger customers who are already having issues). Thus, alignment is crucial, as you wish to ensure that the LLMs understand what is an appropriate response and that any further action (you don’t want to miss a customer request) is aligned with how an employee would address these problems. Aligning the language model to these issues is the only way to ensure performance suitable for production use. The amount of potential support tickets and feedback is colossal in production settings. Thus, reducing costs by a factor of 15 can represent a huge saving and motivation for long-term use, especially as the performance will remain constant, and the workflow will not be affected by potential future version changes to GPT4. To scope out the project, we’ll use the following requirements First, we assume the following general workflow: A user feedback message regarding a product or service is sent to the Twitter account. The LLM analyzes the feedback and responds empathetically (or as best as it can), i.e., the chatbot aspect. Given the customer feedback, the LLM will classify the feedback as “requires action” or not, i.e., classifier aspect. If it does, then the chatbot will create a customer ticket object to be used later in downstream applications. We will use OpenAI’s GPT4 for this use-case. To start, we’ll need to set up some environmental variables with your OpenAI API key. For this, we should create a .env file and add it to the directory. Later the .env file will be read and the environment variables will be correctly parsed. OPENAI_API_KEY=sk-XXX And this is all you need to configure to get started with Tanuki! Then, let's next see how to build the use case. Building the workflow As mentioned before, you could use GPT4 with prompts if cost was not a factor, although getting typed outputs would require additional work. If you had a few weeks, you could even fine-tune an open-source LLM to handle this task. Instead, we’ll use Tanuki to do this in 20 minutes. First line of work — we need to install Tanuki pip install tanuki.py Then let’s lay out some groundwork. […]
↧