Quality from and with Prompt Engineering

As a deep learning enigneer, David explores the possibilities of using AI. It’s about his approach to AI-generated test cases, the limitations and the possibilities. We also talked about prompt patterns, what the smallest changes in the prompt do and how they can be designed to increase the quality of answers. Finally, David assesses the development of AI and has interesting tips for testers who want to start integrating AI into their workflow.

“However, I didn’t improve this one prompt in the traditional way, but then moved on to a conversation. It doesn’t have to stay with a single prompt” - David Faragó

David is a deep learning engineer at mediform, a company that develops communication solutions such as telephone bots for doctors’ surgeries. His work there includes fine-tuning large language models, prompt engineering and data science, as well as software engineering, e.g. microservices, Kubernetes and quality assurance. He also has his own company, QPR Technologies, which offers consulting and development services for innovative quality assurance, e.g. using static code analysis or AI. He founded this company with colleagues after completing his doctorate in symbolic AI, formal methods and model-based testing. He is a member of the steering committee of the GI specialist group Test, Analysis and Verification.

Highlights of this episode:

• David talks about the use of artificial intelligence for quality assurance
• Prompt engineering can help overcome writer’s block and develop test cases
• David emphasizes that one should not completely rely on the results of artificial intelligence, especially in terms of correctness
• Experimenting with different prompt patterns can improve the quality and robustness of test cases
• David shares his experience with the React pattern, which can improve the robustness and correctness of results
• Testers are well suited to prompt engineering as they already have an understanding of quality and requirements
• David is currently exploring the possibilities of fine-tuning models to further improve results

Left:

• GI Fachgruppe Test, Analyse und Verifikation von Software
• Mein Vortrag
• Commit-Message-Qualität-Klassifikator
• Chris Beams Commit Message Guideline
• Paper
• Anthropics Constitutional AI
• Poe App mit vielen LLMs
• Paper
• Firma mediform
• Auto-GPT
• LangChain-Bibliothek
• Hugging Face
• Metas LLaMA-Modell
• QLoRA-Finetuning
• Google-Leak

How Prompt Engineering is transforming software test quality

Today we enter the world of prompt engineering and how it is revolutionizing the quality of software testing. David shares his experiences and insights on how artificial intelligence and prompt engineering can be used to create and optimize test cases.

The potential of AI in test design

Artificial intelligence has the potential to eliminate the infamous writer’s block when creating test cases. However, despite the enormous possibilities, he makes it clear that complete automation is not realistic. His main message was clear - AI can do a lot, but it is not a panacea.

The challenge of correctness

One critical issue is the question of the correctness of generated test cases. David emphasized that AI-generated tests must be viewed particularly critically. Despite the impressive ability of AI to generate useful content, errors and inaccuracies cannot be ruled out. This is particularly relevant in the context of test generation, where accuracy is essential.

AI-supported commit messages

David shared a story about his experiment with commit messages using AI for quality assessment. This experience revealed both the potential and the limitations of AI - a single small change could completely reverse the AI’s judgment. This experience taught him to always critically question the results.

The evolution of prompt engineering

We then talked about David’s journey to becoming an expert in the field of prompt engineering. By experimenting with different patterns, he found ways to improve test quality. But despite significant progress in test coverage and structuring, challenges in correctness remained.

Future prospects

David is optimistic that further developments in the field of AI-supported testing are imminent. Nevertheless, a certain skepticism remains regarding the reliability of these methods. As always, it is important to strike a balance between innovation and critical observation.