| Task | Prompt/Context | Golden Answer/Criteria |
| Information extraction | A 60-year-old male patient, Mr. Johnson, presented with symptoms of increased thirst, frequent urination, fatigue, and unexplained weight loss. Upon evaluation, he was diagnosed with diabetes, confirmed by elevated blood sugar levels. Mr. Johnson's weight is 210 lbs. He has been prescribed Metformin to be taken twice daily with meals. It was noted during the consultation that the patient is a current smoker. | {"age": 60, "gender": "male", "diagnosis": "diabetes", "weight": 210, "smoking": "yes"} |
| Python code generation | Write a python function to check whether the given number is odd or not using bitwise operator. | assert is_odd(5) == True |
| Summarization | Summarize the following into a short paragraph for 3rd graders: "Water is a chemical compound composed of two hydrogen atoms bonded to one oxygen atom, with the chemical formula H2O. The hydrogen atoms are bonded to the oxygen atom via polar covalent bonds, which means that the electrons in the bond are not shared equally between the atoms, resulting in a slight negative charge on the oxygen atom and a slight positive charge on the hydrogen atoms. This gives water molecules a bent, or V-shaped, structure, and allows them to form hydrogen bonds with other water molecules, which are responsible for many of water's unique properties, such as its high boiling point and surface tension." | Water is a special substance that is made up of two tiny parts called atoms, specifically, two hydrogen atoms and one oxygen atom. These atoms are stuck together in a way that creates a bent shape, which makes water have some amazing properties. For example, water can stick to itself, which helps it stay together as a liquid and not just turn into a gas. This also makes water have a high boiling point, which means it needs a lot of heat to turn into a gas. Water's unique properties make it an important part of our lives and our planet. |
| Multilingual | Translate this college homework assignment into advanced Spanish for an advanced writing course: "Write a personal essay of 500 words about a crucial moment in your family, describing the event, its impact on family relationships, and the emotions and thoughts that surrounded it. Your essay should have a clear structure, with an introduction, body, and conclusion, and use expressive and precise language. It will be evaluated based on clarity, coherence, depth, and originality." | Escriba un ensayo personal de 500 palabras sobre un momento crucial en su familia, describiendo el evento, su impacto en las relaciones familiares, y las emociones y pensamientos que lo rodearon. Su ensayo debe tener una estructura clara, con una introducción, un cuerpo y una conclusión, y utilizar un lenguaje expresivo y preciso. Será evaluado sobre la base de claridad, coherencia, profundidad y originalidad. |
| Chat | You are an 8th grade tutor for a Latin class, your job is to help students memorize their weekly word list test. You don’t give the answers to your students but instead help them get to the answer themselves. Begin your tutoring session by asking your student what word list do they need help memorizing. Remember to be encouraging and ask students to "show their work" i.e. articulate how they came up with that answer. | Hello there! I hope you're having a great day. As your Latin tutor, I'm here to help you with your weekly word list test. So, let's get started! Could you please tell me which word list you need help memorizing this week? Remember, the goal is not just to remember the words, but also to understand their meanings and how to use them in context. Once you've chosen a word, I'd like you to try to recall its meaning or use it in a sentence. Then, I'd love for you to "show your work" by explaining how you came up with that answer. This will help reinforce your understanding and make it easier to remember. Does that sound good? Let's dive in! |
Context information is below.
---------------------
{context\_str}
---------------------
Given the context information and not prior knowledge. Generate {num\_questions\_per\_chunk} questions based on the context. The questions should be diverse in nature across the document. Restrict the questions to the context information provided.
| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | * Based on the context and the generated question from the previous prompt template, generate answers according to the following prompt template: |Context information is below
---------------------
{context\_str}
---------------------
Given the context information and not prior knowledge,
answer the query.
Query: {generated\_query\_str}
Answer:
| | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | The data generated by LLM typically needs to be further refined and filtered. Real-world sources, specifically those from experts, are preferable for generating questions. LLM-generated data can, however, be extremely useful when real data is hard to come by or scarce. ### How to evaluate[](https://docs.mistral.ai/guides/evaluation/#how-to-evaluate) #### Metrics-based evaluation[](https://docs.mistral.ai/guides/evaluation/#metrics-based-evaluation) Many public benchmark evaluations follow a metrics-based approach. Analyzing and comparing model-generated answers with golden answers can be done through a variety of metrics. There are several popular metrics, including exact match accuracy rate, recall, precision, and F1 score. * Exact Match Accuracy Rate: Model-generated answers that are perfect matches to golden answers are measured by this metric. By doing so, the model's capability to generate precise responses is evaluated in a rigorous manner. * Recall: This calculation determines the percentage of the relevant information from the golden answers that the model-generated answers are able to retrieve. It can be concluded that a model with a higher recall is capable of capturing more relevant information. * Precision: Precision measures the extent to which model-generated answers contain relevant information. Responses generated by a model with a high precision are more accurate and relevant. * F1 Score: It provides a balanced evaluation of a model's performance in terms of recall and precision based on the harmonic mean of both metrics. Additionally, BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) are other popular NLP evaluation metrics. ROUGE is commonly used for text summarization, whereas BLEU is typically used for machine translation tasks. In spite of this, we do not recommend using these instruments because of concerns about their reliability and the potential for misleading results. #### Example 1: evaluate information extraction with accuracy rate[](https://docs.mistral.ai/guides/evaluation/#example-1-evaluate-information-extraction-with-accuracy-rate) **Evaluation data**[****](https://docs.mistral.ai/guides/evaluation/#evaluation-data) An example of how patient information may be extracted from medical notes is shown in the first example. In the evaluation data, both the medical notes and the accurate/expected answer (referred to as the "golden answer") are required. Listed below are two examples of test cases: ```json prompts = { "Johnson": { "medical_notes": "A 60-year-old male patient, Mr. Johnson, presented with symptoms of increased thirst, frequent urination, fatigue, and unexplained weight loss. Upon evaluation, he was diagnosed with diabetes, confirmed by elevated blood sugar levels. Mr. Johnson's weight is 210 lbs. He has been prescribed Metformin to be taken twice daily with meals. It was noted during the consultation that the patient is a current smoker. ", "golden_answer": { "age": 60, "gender": "male", "diagnosis": "diabetes", "weight": 210, "smoking": "yes", }, }, "Smith": { "medical_notes": "Mr. Smith, a 55-year-old male patient, presented with severe joint pain and stiffness in his knees and hands, along with swelling and limited range of motion. After a thorough examination and diagnostic tests, he was diagnosed with arthritis. It is important for Mr. Smith to maintain a healthy weight (currently at 150 lbs) and quit smoking, as these factors can exacerbate symptoms of arthritis and contribute to joint damage.", "golden_answer": { "age": 55, "gender": "male", "diagnosis": "arthritis", "weight": 150, "smoking": "yes", }, }, } ``` **How to evaluate?**[****](https://docs.mistral.ai/guides/evaluation/#how-to-evaluate-1) * Step 1: Define prompt template Using the medical notes as context, we have designed a prompt. A predefined JSON schema should also be followed by the model in order to ensure that the model produces the expected output. In order to ensure consistent JSON output, we specified \["type": "json\_object"] when calling our models. ```python import os from Ultrasafeai import Ultrasafe def run_Ultrasafe(user_message, model="Ultrasafe-large-latest"): client = Ultrasafe(api_key=os.getenv("Ultrasafe_API_KEY")) messages = [{"role": "user", "content": user_message}] chat_response = client.chat.complete( model=model, messages=messages, response_format={"type": "json_object"}, ) return chat_response.choices[0].message.content # define prompt template prompt_template = """ Extract information from the following medical notes: {medical_notes} Return json format with the following JSON schema: {{ "age": {{ "type": "integer" }}, "gender": {{ "type": "string", "enum": ["male", "female", "other"] }}, "diagnosis": {{ "type": "string", "enum": ["migraine", "diabetes", "arthritis", "acne", "common cold"] }}, "weight": {{ "type": "integer" }}, "smoking": {{ "type": "string", "enum": ["yes", "no"] }}, }} ``` * Step 2: Define how we compare the model response with the golden answer Using step 2, we wrote a function for comparing two JSON objects, one representing the model response and the other representing the golden answer. This example evaluates the accuracy of the JSON output by calculating the percentage of matching values across the keys. ```python import json def compare_json_objects(obj1, obj2): total_fields = 0 identical_fields = 0 common_keys = set(obj1.keys()) & set(obj2.keys()) for key in common_keys: identical_fields += obj1[key] == obj2[key] percentage_identical = (identical_fields / max(len(obj1.keys()), 1)) * 100 return percentage_identical ``` * Step 3:Test cases can now be examined individually so we can calculate accuracy rates across each one. With the prompt template as a starting point, we can create user messages for each case. A comparison is then made between the LLM responses and the correct answers. In order to calculate the overall accuracy rate across all test cases, we must calculate the accuracy rate for each case separately. ```python accuracy_rates = [] # for each test case for name in prompts: # define user message user_message = prompt_template.format(medical_notes=prompts[name]["medical_notes"]) # run LLM response = json.loads(run_Ultrasafe(user_message)) # calculate accuracy rate for this test case accuracy_rates.append( compare_json_objects(response, prompts[name]["golden_answer"]) ) # calculate accuracy rate across test cases sum(accuracy_rates) / len(accuracy_rates) ``` #### Example 2: evaluate code generation[](https://docs.mistral.ai/guides/evaluation/#example-2-evaluate-code-generation) **Evaluation data**[****](https://docs.mistral.ai/guides/evaluation/#evaluation-data-1) Our second example involves generating Python code and assessing the generated code. To conduct the evaluation, both the Python instructions and the corresponding unit tests are required for the evaluation data. Here are two examples of such evaluation data sets: ```python python_prompts = { "sort_string": { "prompt": "Write a python function to sort the given string.", "test": "assert sort_string(\"data\") == \"aadt\"" }, "is_odd": { "prompt": "Write a python function to check whether the given number is odd or not using bitwise operator.", "test": "assert is_odd(5) == True" } } ``` **How to evaluate?**[****](https://docs.mistral.ai/guides/evaluation/#how-to-evaluate-2) * Step 1: Define prompt template We have designed a prompt that generates Python code snippets based on descriptions of specific tasks. ````python def run_Ultrasafe(user_message, model="Ultrasafe-large-latest"): client = Ultrasafe(api_key=os.getenv("Ultrasafe_API_KEY")) messages = [{"role":"user", "content": user_message}] chat_response = client.chat.complete( model=model, messages=messages, response_format={"type": "json_object"}, ) return chat_response.choices[0].message.content # define prompt template prompt_template = """Write a Python function to execute the following task: {task} Return only valid Python code. Do not give any explanation. Never start with ```python. Always start with def {name}(. """ ```` * Step 2: Decide how we evaluate the code generation Next, we can use code\_eval.compute to evaluate whether the generated code passes the pre-defined unit tests. predictions is a list of generated code outputs from the LLM and references is a list of unit tests for each prediction. {% hint style="warning" %} This code is designed to run code that’s been generated by a model, which may not be entirely reliable. While it's unlikely that the model-generated code will intentionally cause harm, it's strongly recommended to run this evaluation suite in a sandbox environment, isolating it from your main system and network. {% endhint %} ```python from evaluate import load os.environ["HF_ALLOW_CODE_EVAL"] = "1" code_eval = load("code_eval") # example using code_eval: pass_at_1, results = code_eval.compute( references=['assert is_odd(5) == True'], predictions=[['def is_odd(n):\n return n & 1 != 0']], k=[1]) # example output: # ({'pass@1': 1.0}, # defaultdict(list, # {0: [(0, # {'task_id': 0, # 'passed': True, # 'result': 'passed', # 'completion_id': 0})]})) ``` * Step 3: Calculate accuracy rate across test cases Now, we can go through all test cases, create a user message based on the prompt template, use the LLM to produce Python code, and evaluate the generated code for each test case. ```python refs = [] preds = [] for name in python_prompts: # define user message user_message = prompt_template.format( task=python_prompts[name]["prompt"], name=name ) # run LLM response = run_Ultrasafe(user_message) refs.append(python_prompts[name]["test"]) preds.append([response]) # evaluate code generation pass_at_1, results = code_eval.compute(references=refs, predictions=preds) pass_at_1 # example output # {'pass@1': 1.0} ``` ### LLM-based Evaluation[](https://docs.mistral.ai/guides/evaluation/#llm-based-evaluation) It is common to use Large Language Models (LLMs) to evaluate or judge the results of another LLM, particularly when labeled data and golden answers cannot be obtained. LLMs as judges were examined by the MT Bench paper, which demonstrated that strong LLMs can perform in a similar manner to humans. As LLMs are capable of processing and evaluating large amounts of data, they are highly scalable and efficient when compared to human evaluations, which are often time-consuming and resource intensive. LLMs may be used as judges in a number of ways, including single-point grading, reference-based grading, and pairwise grading. * Single-point grading: A generated output is given a single score based on the quality or accuracy of its content. There are usually specific grading instructions associated with this score. In the case of multi-faceted outcomes, single-point grading is not always sufficient to capture the nuances. * Reference-based grading: An LLM evaluates the similarity of a generated output with one or more reference outputs and assigns a score as a result. Machine translation tasks often require this approach, since there may be several valid translations for the same input. As a result, reference-based grading often requires a golden answer that is not always available. * Pairwise grading: By comparing two generated outputs, LLM calculates a score based on their relative quality. In tasks such as dialogue generation, where multiple valid answers to a particular query may exist, this method is frequently used. LLM assigns scores based on comparisons of two responses that determine which is more relevant or coherent. Identifying the potential limitations and challenges is also essential. It is possible for LLMs to display inherent biases, for example. It is typically the case that LLMs developed by one company tend to favor answers generated by models of the same company. In the case of evaluations, it can be difficult to ensure that they are accurate and fair. The favouritism shown by Ultrasafe AI is relatively low in our experience. #### Example 3: evaluate summary generation with LLM[](https://docs.mistral.ai/guides/evaluation/#example-3-evaluate-summary-generation-with-llm) **Evaluation data**[****](https://docs.mistral.ai/guides/evaluation/#evaluation-data-2) In this example, we generate news summaries and use LLM single-point grading to evaluate the quality of the summary. To carry out the evaluation, let's use a sample news article that we plan to summarize.