Testing of and with AI

What an amazing time we are living in. I think it’s super exciting and I’m thrilled about the possibilities that are opening up for us, especially in software development. I’ve just seen Gartner’s hype cycle, where GenAI is just at its peak - i.e. before it falls into the valley of tears. But what comes next will be exciting. Where will AI be used in a sustainable and productive way - apart from SEO optimization and text generators?

We would do well not to fall into blind actionism - but on the other hand not to bury our heads in the sand when it comes to AI. It’s a balancing act! I have also noticed that my personal opinion on AI is constantly changing. Nevertheless - or perhaps because of this - I would like to share my current state of ignorance here 😉

Software testing with AI

…is a big topic right now, as it is in all industries. There is of course a lot of potential here: test case creation, test data generation, execution, error analysis - a playing field for AI tools. It feels like every tool now has the addition AI in its name - even if not much has changed. So take a close look at what’s really behind it.

And we should always ask ourselves: what problem do we actually want to solve with AI? It’s tempting to be dazzled by the latest technology, but the key to success lies in using it in a targeted way to overcome specific challenges in our projects. And when I look at my customers, it’s less about even more efficient test execution and more about things like test data management and the classic: how do I overcome the media discontinuity between the requirements of the requester and the technology?

But how do you get started with AI in testing? Quite simply: start. Experiment. Try it out. This is pioneering work, much is not yet finished, not yet developed and not yet conceived. The first websites were not created with a CMS - but with notepad.exe.

Testing AI

When we talk about the testing of AI itself, we are thrown back to the primal question that we tend to avoid in software development: What does quality actually mean to us?
When I come into new projects and companies, this is always my first question: What does quality mean to you? Oh. Ambiguity. Um. Reference to ISO 25010, specifically? Silence.

Traditional quality criteria such as functionality must be rethought for AI systems. Other criteria such as accuracy, learning ability, adaptability, data quality and statistical criteria are becoming much more important here.

Rethinking necessary

As testers, test managers and quality engineers, we need to rethink both fields. Both in the use of AI and in the testing of AI. We have to let go of what we love - and learn new things. This is not always comfortable, but it is necessary.

And there is help available: The popular protagonists are a good place to start using AI: ChatGPT, Midjourney, Gemini, Copilot, etc. Just give them a try 🙂 Or you can use AI as part of a training course, for example the A4Q Practical Tester And for testers? The ISTQB Certified Tester AI Testing training and certification is a good option here. Incidentally, this certification has been around since 2021 and its predecessors even longer. After all, AI hasn’t just been around since ChatGPT. And somehow it’s just software 😉