Acceptance Test-Driven LLM Development

First of all: Sorry for the poor audio quality, unfortunately we only realized this afterwards. I hope the content will make up for it :-) The development of Large Language Models (LLMs) and the role of Acceptance Test Driven Development (ATDD) are key topics in AI development. David, an expert in the development and quality assurance of AI-based telephone bots for medical practices, shares his experiences and insights into this process. The challenges and approaches to training and testing LLMs, including the use of prompt engineering and fine tuning, will be highlighted. Of particular note is the approach of applying ATDD methods to LLM developments to improve the quality and effectiveness of the models. Another focus is on the CPMAI process, which represents a modern approach to the development and implementation of AI projects.

“This is relatively demanding. At the end of the day, we have a few components. We first do speech-to-text and then we use a language model on a pure text basis.” - David Faragó

David is a deep learning engineer at Mediform, specializing in fine-tuning large language models, prompt engineering and microservices. He also runs QPR Technologies, a consultancy for innovative quality assurance, and is a member of the steering committee of the GI specialist group Test, Analysis and Verification.

Highlights of this episode:

• David and his team are using Large Language Models (LLMs) to develop a phone bot for medical practices
• They use Acceptance Test Driven Development to ensure the quality of their LLMs
• An important part of their work is fine-tuning the LLM to adapt it to specific tasks
• They developed a solid process and testing framework to ensure high quality
• A special tool, the Language Model Evaluation Harness from Eloifa, is used to verify the LLM
• David sees Acceptance-Test-Driven LLM development as a new approach in the development of AI systems
• They also use other types of testing, such as stress testing and metamorphic testing, to further improve their LLM

Further links:

• EleutherAIs LM Evaluation Harness
• Cognilyticas CPMAI
• zugehöriger CPMAI Kurs mit Zertifikat – Affiliate-Code für 10% Rabatt: “dfarago-10”
• Large Language Models (LLMs): A Technology Gifted by Aliens Without a Manual
• Cross Industry Standard Process for Data Mining 1.0 (CRISP-DM)
• The Seven Patterns of AI

Wie Acceptance Test Driven Development LLMs neu definiert

When developing AI-based telephone bots for medical practices, there are a lot of new quality assurance challenges to overcome. One solution approach in the testing process is Acceptance Test Driven LLM Development

The new fields of application for AI development

David Faragó, a renowned expert in the field of AI and specifically in LLM topics, shares his extensive knowledge on the use of Large Language Models. From prompt engineering to fine-tuning foundational models, David covers all aspects. His current project at Mediform aims to develop a telephone bot for medical practices that can communicate with patients in natural language thanks to modern AI. This innovative application represents a significant step forward and demonstrates the potential of LLM in practice.

Development and testing

At the heart of David’s work is a solidly developed testing process that focuses on acceptance testing. Through careful analysis of real dialogs and iterative improvements, a high-quality model is created. David talks about the challenges and approaches to developing LLMs, including dealing with non-determinism and the black box nature of this technology. By using specialized tools such as Eloifa’s Language Model Evaluation Harness, the team is able to effectively measure and verify the quality of their models.

Acceptance Test Driven LLM Development

Acceptance Test Driven LLM Development is not just a method, but a philosophy. It interweaves agile methods with machine learning, enabling rapid iteration cycles with direct end-user involvement. David explains the process in detail and shows how this approach has made it possible to develop a robust and effective system for medical practices. This approach ensures that the end product meets the exact requirements while remaining flexible for future customization.

The role of CPMAI in the development cycle

Another key element of David’s strategy is the use of CPMAI (Cognitive Process Management for AI), a modern process framework that combines agility with machine learning. This method supports the team at every stage of the development cycle - from understanding the business need to deploying the model. This structured approach allows problems to be identified and resolved quickly, enabling continuous improvement of the system.

The future of AI development

The discussion not only highlights the complexity behind the development of Large Language Models, but also the enormous potential of this technology. Innovative approaches such as Acceptance Test Driven LLM Development are opening the door to a new era of AI development - an era in which quality assurance and agile methods go hand in hand. This episode sheds light on the exciting future of AI technology and its many possible applications.