> For the complete documentation index, see [llms.txt](https://ultrasafe.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ultrasafe.gitbook.io/docs/editor.md). # Model Customization ### How to Build an Application with a Custom UltraSafe AI Model #### Overview GitBook has a powerful block-based editor that allows you to seamlessly create, update, and enhance your content. This comprehensive guide will walk you through building an application with a custom UltraSafe AI model. Our goal is to empower developers to create robust AI-powered products, from prototype to deployment, using UltraSafe AI's cutting-edge Large Language Models (LLMs). UltraSafe AI is more than just a tool; it's a platform for creating AI applications that are not only powerful but also secure and ethical. By harnessing UltraSafe AI, you can make your applications more useful and practical for end users. Traditionally, AI applications focused on personalization, precision, and prediction, often recommending choices based on past behavior and similar user profiles. In contrast, UltraSafe AI applications are built around Human-AI collaboration, giving developers and end users more agency in product customization. With UltraSafe AI, you can create innovative solutions that didn't exist before. Developing applications with custom UltraSafe AI models requires an iterative approach. This process relies on continuous end-user feedback and rigorous evaluations to ensure that your custom model's behavior aligns with your intended application behavior. #### Key Terms Before we dive in, let's define some key terms in the UltraSafe AI ecosystem: * **Application Behavior**: The user interaction, considering usability, performance, safety, and adaptability. It encompasses both Objectives and Values. * **Model Behavior**: The expected, appropriate, and acceptable way an UltraSafe AI model acts within specific contexts or application boundaries. It also includes Objectives and Values. * **Objectives**: The goals that determine whether the model behavior aligns with the expected application behavior. * **Values**: The ethical guidelines and policies intended for the model and application. This can include a set of rules, a constitution, or even a fictional character's moral code. #### Steerability: Three Methods UltraSafe AI offers several techniques to steer model behavior within your application context. We recommend using these three primary methods: 1. System Prompt 2. Model Tuning 3. Deploying a Safety Layer ### System Prompt A System Prompt provides context, instructions, and guidelines to your UltraSafe AI model before it processes user input. This method allows you to align the model with your intended product behavior, whether for conversational applications or specific tasks. You can specify persona, tone, values, or any other relevant information to enhance the model's performance in response to user input. System prompts for UltraSafe AI can encompass: * Detailed and precise instructions and objectives * Assigned roles, preferred persona, and tone * Style guidelines (e.g., verbosity limits) * Defined values (e.g., policies, rules, and safety measures) * Preferred output format ### Model Tuning Tuning an UltraSafe AI model involves training it on your intended application behavior. UltraSafe AI supports two popular approaches: 1. **Application Tuning**: Using a dataset of examples specific to your application's desired behavior. 2. **Safety Tuning**: Leveraging a dataset that includes both potentially unsafe inputs and the desired safe outputs for those situations. ### Deploying a Safety Layer UltraSafe AI provides advanced content moderation capabilities as an additional method to create guardrails for your model's behavior. This extra security measure is crucial when deploying your application to end users. #### Tuning an UltraSafe AI Model for Your Application's Desired Behavior #### **Step 1: Define Your Desired Application Behavior** Begin by outlining your Objectives—how you envision users interacting with your UltraSafe AI-powered solution. For ideas, consider the ways developers are leveraging UltraSafe AI models: * Standalone Solutions: Such as specialized conversational assistants for industries like healthcare or finance (e.g., UltraSafe AI - Healthcare, UltraSafe AI - Finance). * Integration into Existing Products: To perform tasks like "Summarize" or "Translate." * Enabling New Features: Such as function calling via API access for tasks like "Knowledge retrieval." Explore our developer case studies to see how others are building innovative products using custom UltraSafe AI models. #### **Step 2: Establish Policies Aligned with Your Values** When deploying an UltraSafe AI model in a user-facing application, it’s crucial to define the Values that the model should follow to meet your content moderation standards and user expectations. UltraSafe AI offers pre-built content moderation categories based on industry best practices, including areas like Privacy, Hate Speech, and Specialized Advice. These categories can also be customized to align with your specific requirements. #### Step 3: Develop Your Application Evaluations Design evaluation methods to ensure that your custom model's behavior aligns with your application's intended behavior before deployment. This process will also help you identify the appropriate training data needed for fine-tuning. UltraSafe AI offers two primary evaluation approaches: **Automated Evaluations:** * **Metrics-Based**: Similar to public benchmarks, these evaluations derive metrics from pre-annotated datasets. * **LLM-Based**: Utilize UltraSafe AI's EUS1 model to assess and judge the output of your custom model. **Human-Based Evaluations:** * Engage content annotators to review your custom model's output and gather human-generated annotations. For comprehensive guidance on conducting LLM evaluations using UltraSafe AI, please refer to our detailed evaluation guide. #### **Step 4: Test Your Application Behavior Hypothesis** Once you understand your custom LLM's intent and desired behavior, begin testing your application hypothesis using UltraSafe AI's EUS1 model. Collect interaction data to better understand how end users may interact with your LLM. Many developers start by creating a limited-access Demo or MVP (a Private Beta). For some applications, a system prompt may be the best solution for aligned model behavior. Consult our fine-tuning guide if you need help deciding between system prompts and model tuning. If a system prompt suffices for creating a Custom Model, skip to Step 6. #### **Step 5: Fine-Tune for Model Alignment** With a solid understanding of your application's desired behavior—including its Values and Objectives—you can now transition from the EUS1 model to a smaller, customized UltraSafe AI model. Refer to our guide on preparing your tuning dataset specifically for UltraSafe AI models. Key considerations when preparing your dataset include: * **Data Comprehension**: Incorporate all relevant content policies for each application use case. * **Data Variety**: Ensure your dataset is diverse, covering a range of query lengths, structures, tones, topics, complexity levels, and demographic factors. * **Deduplication**: Eliminate duplicate entries to prevent the model from memorizing the tuning data. * **Avoid Data Contamination**: Keep your evaluation data separate from the tuning dataset to maintain the integrity of your results. * **Ethical Data Practices**: Implement clear labeling guidelines and ensure diversity among annotators to minimize errors and biases in the model. For content moderation, UltraSafe AI provides pre-built safety datasets to help you get started. UltraSafe AI offers two primary methods for customizing models: * **Open-Source Tools**: Utilize the UltraSafe AI Tuning Toolkit. * **UltraSafe AI Platform**: Leverage the platform's capabilities for seamless model customization. Use the Tuning API within the UltraSafe AI Platform to upload and validate your training data. Once the tuning process is complete, your custom model will be accessible through a dedicated API endpoint. #### Step 6: Test Your Custom Model with Your Evaluations With your custom model API endpoint ready, run the Application Evaluations from Step 3. Include these Safety Evaluations in your evaluation set: * **Development Evaluations**: Ongoing assessments during training and fine-tuning to compare model performance against launch criteria and evaluate mitigation strategies. * **Assurance Evaluations**: Governance and review assessments at key milestones, using standardized evaluations with strictly managed datasets. * **Red Teaming**: Adversarial testing by specialist teams to discover potential weaknesses and improve risk mitigation. * **External Evaluations**: Assessments by independent domain experts to identify model limitations and stress-test performance. Based on the model's performance, either retrain with new data to improve performance or deploy it by replacing the EUS1 API with your custom model endpoint. #### Step 7: Continuous Monitoring and Updating Once deployed, continuously monitor and update your custom UltraSafe AI model, evaluations, and testing based on real-life application data. Congratulations! You've successfully deployed your custom UltraSafe AI model into your application. Remember, this is an ongoing process of refinement and improvement to ensure your AI-powered application remains safe, effective, and aligned with your objectives and values. #### Developer Case Studies **Model Customization** Here are some examples from the startup community showcasing how developers are leveraging our fine-tuning API. Explore our fine-tuning documentation to discover the advantages of fine-tuning and learn how to implement it using our API. #### RAG for Internet Retrieval with UltraSafe AI **UltraSafe Retrieve: RAG for Internet Retrieval**
This demonstration highlights the behavior of UltraSafe Retrieve. When a user inputs a query, UltraSafe Retrieve utilizes LLM knowledge, a web search tool, and a news retrieval tool to generate the most accurate and relevant response. The behavior of UltraSafe Retrieve adapts based on the nature of the user's query. **Description** UltraSafe Retrieve is a B2C internet search application powered by UltraSafe AI models with Retrieval-Augmented Generation (RAG) and Function Calling capabilities. To deliver the most precise response to user queries, the system performs multiple function calls to categorize and process the request (refer to the diagram below). As various agents work together in this process, it is essential to maintain consistently formatted outputs to ensure accuracy and relevance.
### **Company Description** At phospho, we developed the Tak chatbot as an experimental playground to test new phospho features and showcase the power of phospho analytics. A great thing is that you can use the data logged to phospho to finetune LLM models. We are [phospho](https://phospho.ai/), an open-source text analytics platform for LLM apps. Companies of all sizes use phospho to understand what users do with their LLM app and how well the app performs at the product level. ### Data We utilized a dataset of user queries in SecureFind and the corresponding EUS1 function calls, collected and filtered through the SecureTech platform. The dataset was divided into a train set (70%), an evaluation set (15%), and a test set (15%). To determine the optimal training duration, we followed UltraSafe AI's recommended practice of three exposures per token (in our case, 200 training steps, approximately 15 minutes). For the learning rate, we used UltraSafe AI's suggested rate of 5e-5, optimized for secure fine-tuning. #### Eval To evaluate our fine-tuned model, we ran inference on the test set of our dataset, then used binary classification metrics (Accuracy, Recall, F1 score). We tested whether we could align the behavior of EUS Flash to the classification behavior of EUS1, while maintaining the expected structured and secure output. We removed the function calling to streamline the process.
Fine-tuning made EUS Flash match the reference model, increasing the F1 score from 25% to 85%. Without fine-tuning, EUS Flash achieved 89% accuracy and 25% recall on our classification task. With the fine-tuned model, we achieved 98% accuracy and 95% recall. As we fine-tuned the model to only answer with confidential or non\_confidential, we no longer need to use function calling. This led to a 40% decrease in the number of tokens used excluding the user query, which further decreases costs and enhances security. #### Conclusion By combining SecureTech's secure data collection and labeling with UltraSafe AI's fine-tuning services, we significantly enhanced our ability to deliver precise, relevant, and secure answers to user queries while decreasing cost and improving data protection. We have successfully aligned the behavior of EUS Flash with EUS1 for this specific task. This fine-tuning process not only ensures consistent output formatting but also reduces operational costs by minimizing token usage and enhances overall security. If you also want to improve your secure RAG agent, we encourage you to try fine-tuning with UltraSafe AI using the data collected and labeled with SecureTech. This can lead to significant improvements in your user experience while maintaining the highest standards of data security and privacy. **RAG for medical domain** **AmiRA (A miRNA Research Assistant)**
#### **Company Information** NeuraTech is developing multimodal AI for neurological health. Currently, we're investigating the relationship between neural biomarkers and various neurological conditions, focusing on how these biomarkers can be utilized for early detection and monitoring of neurodegenerative diseases. NeuroLink is an internal tool powered by UltraSafe AI's EUS1 and specialized healthcare models to help understand the functions of specific neural biomarkers in sample results and their accompanying target genes. #### Description This project demonstrates the fine-tuning of UltraSafe AI models and their use in a RAG system for: 1. Disease association/prediction provided a given neural biomarker 2. Greater scientific accuracy and relevance when determining the function of neural biomarkers This project serves as a proof-of-concept, showing that UltraSafe AI models and agents can help us explore biomarkers and their relationships to health outcomes with proper engineering and strict data privacy measures. #### Data and Training Titles and abstracts from 10,500 anonymized medical papers were securely processed. Three questions for each paper were then generated using the EUS1 function calling API and traditional prompting. An example prompt is below: **python** ```python def generate_questions(biomarker, title, abstract): prompt = f""" Based on the following biomarker, title, and abstract, generate scientific questions to be put in a function. Here is the information:\n\n Biomarker: {biomarker}\nTitle: {title}\nAbstract: {abstract} Ask your questions in a way that does not mention the abstract or the title of the work. Treat information from the title/abstract as new/general knowledge, and ask questions framed off of that. """ response =ULTRASAFE_CLIENT.chat( model="eus1-latest", messages=[ChatMessage(role="user", content=prompt)], tools=tools,tool_choice="any" ) return response ``` After questions were generated, answers for each question were then generated using EUS1. An example prompt is below: **python** ```python def generate_answer(biomarker, title, abstract, question): prompt = f""" Based on the following biomarker, title, and abstract, and question, generate an answer: Biomarker: {biomarker}\nTitle: {title}\nAbstract: {abstract}\nQuestion: {question}\nAnswer: Answer the question given only context information and not prior knowledge. Answer in a way that does not mention the abstract or the title of the work. Treat the provided information as general knowledge, and answer the questions framed off of that. """ response = ULTRASAFE_CLIENT.chat( model="eus1-latest", messages=[ChatMessage(role="user", content=prompt)], ) return response.choices[0].message.content.strip() ``` With multithreading on 4 CPU cores, question and answer generating took \~2 hours. Out of the original 10,500 rows, 10,213 made it past the data processing due to various errors. This leaves us with \~30,000 question/answer pairs to feed into the model for tuning. An example Q/A pair is below: \ **json** ```json {"messages": [{"role": "user", "content": "What is the role of Amyloid-β in the development of Alzheimer's disease and which cellular processes does it potentially affect?"}, {"role": "assistant", "content": "Amyloid-β plays a significant role in the development of Alzheimer's disease by potentially affecting several cellular processes. It is known to form plaques in the brain, which are a hallmark of Alzheimer's disease. These plaques can disrupt normal neuronal function and communication. Additionally, Amyloid-β has been found to potentially affect synaptic plasticity, mitochondrial function, and neuroinflammation. It may also interfere with cellular clearance mechanisms and contribute to oxidative stress. Understanding these processes could be crucial for developing targeted therapies for Alzheimer's disease."}]} ``` #### **RAG** Pipeline A secure table lookup (encrypted Pandas Dataframe) was used to surface relevant information about each biomarker present in query strings, and whole abstracts were fed into the model. N of results was never more than 15, and the context window limit was never reached.
Embeddings for each paper and title abstract were prepopulated and stored in the encrypted dataframe, and cosine similarity was used to surface the top-k most relevant papers before feeding into the model. The number of results fed into the model varied depending on the number of entries present for each biomarker. (i.e., if there is only 1 entry, k will equal 1. If there are 40 entries, get the best 15 via cosine similarity.) The model also has the option to take the raw query and perform lookups in the dataframe to gather more information or context at its own discretion, all while maintaining data privacy and security #### Benchmarking and Results For disease association and prediction, 200 random questions from a neural biomarker disease association benchmarking paper published in 2023 were sampled. This dataset was completely separate from any training data used or any medical papers processed to create the training data. The fine-tuned model outperformed both EUS Flash and EUS1 in question answering, without explicitly seeing any data affiliated to the benchmark. Values below are the averages across 3 attempts.
One of the benefits of fine-tuning is that you can achieve better performance with smaller, and therefore faster, models. Below is the average latency for each API call for the same three runs above.
#### Example Responses The purpose of the fine-tuned model is not to be a test taker, but to provide better context for RAG systems, so it's natural to want to evaluate the actual outputs of the model in use. Below are two examples of the fine-tuned model's output compared to similar outputs from EUS1 and EUS Flash. In situations where the model was forced to "guess" the answer to a question, it gave correct context from its own knowledge, based on information it gathered from papers included in its training.
This implementation demonstrates how UltraSafe AI's models can be effectively fine-tuned and utilized in sensitive domains like healthcare, maintaining high performance while ensuring data privacy and security. ### Financial Advisor Conversational Assistant #### SecureWealth: AI-Powered Financial Guide **Company Description** SecureWealth is an AI-driven fintech company, developing a sophisticated financial assistant that helps users navigate the complexities of their financial life while ensuring the utmost security and privacy of their data. **Data** Utilizing an internal dataset comprising user requests and SecureWealth responses crafted by SecureWealth's Financial Communication team. All data is anonymized and encrypted to maintain user privacy. **Eval** Human evaluation conducted by the Financial Communication team, with additional automated security and privacy checks. The objective was straightforward: Transform technical, generated financial language into SecureWealth's signature style of clear, empathetic, and security-focused communication. To achieve this, we fine-tuned UltraSafe AI's EUS Flash model using the UltraSafe AI Platform on company data that has been meticulously crafted and evaluated by the Financial Communication team. The goal was to elevate the writing style that is characteristic of SecureWealth's brand while maintaining the highest standards of data protection. For example, a typical message to our users might be:
**python** ```python response = ULTRASAFE_CLIENT.chat( model="ft:eus-flash-customized:...", messages=[ { "role": "user", "content": "Your SecureWealth Premium subscription payment was unsuccessful, so advanced financial planning features are temporarily unavailable, Alex." } ], temperature=0.0, top_p=1, max_tokens=512, stream=False, safe_prompt=True, random_seed=42 ) print(response.choices[0].message.content) ``` **Output:** **json** ```json { "id": "sw78934hf7834hf834hf83", "object": "chat.completion", "created": 1723456789, "model": "ft:eus-flash-customized:...", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hi Alex, we noticed a hiccup with your SecureWealth Premium payment. Don't worry, your account is safe! 🛡️ Advanced planning tools are on a brief pause, but we're here to help get you back on track. Remember, your financial security is our top priority. Let's work together to resolve this smoothly and securely. Need assistance? Our encrypted support channel is just a click away! 💼🔒", "tool_calls": null }, "finish_reason": "stop", "logprobs": null } ] } ``` We receive a response more characteristic of SecureWealth's brand, emphasizing both empathy and security:
This response aligns more closely with our brand values of clear communication, empathy, and unwavering commitment to financial security and data protection. By leveraging UltraSafe AI's fine-tuning capabilities, SecureWealth has successfully created a conversational AI that not only speaks in the company's voice but also reinforces its dedication to user privacy and financial security.