Properties Based Testing - Richard Seidl

Properties based testing is a method that checks constant output properties and complements traditional test strategies. This technique is particularly suitable for complex and microservices-based systems, as it allows a large number of inputs to be checked. Nikhil uses several real-life examples to explain how PBT works, what the advantages are and where the method reaches its limits.

“The fundamental proposition of properties based testing is when the system gets to complicated, it is not possible for human to write every test cases.” – Nikhil Barthwal

Nikhil Barthwal is passionate about developing decentralized systems. He has several years of professional experience in large corporations as well as in smaller start-ups and acts as a mentor for various start-ups. He is also a guest speaker at various international conferences and gives talks on topics related to decentralized systems and software quality.

Highlights of this episode:

• Nikhil explained that properties-based testing is used when systems are too complex for humans to write every test case
• The idea is to automatically generate test cases based on the behavior of the service
• Nikhil mentioned that properties based testing includes three components: a modeling language, a generator and a shrinker
• The generator writes test cases from specifications, and the shrinker simplifies long sequences of defects for human understanding
• Nikhil provided an example with an e-commerce system to illustrate how properties based testing works
• He advised starting small with properties based testing and expanding the scope organically
• Nikhil also noted that properties based testing can be costly if testing resources are expensive
• He emphasized that properties based testing supports testers rather than replacing them

How properties-based testing increases quality assurance

In this episode, I spoke with Nikhil Barthwal about the applications of Properties Based Testing. As software systems become increasingly complex, traditional testing methods are reaching their limits, making way for more advanced techniques such as properties-based testing.

The challenge of testing large-scale systems

Testing thousands of microservices is a huge challenge. Nikhil explains how properties-based testing addresses this by automatically generating test cases based on system behavior, reducing the need for manual test case creation.

Core components of properties-based testing

There are three main components: Modeling Language, Generator, and Shrinker. Nikhil uses an example of an arithmetic service to illustrate how these components interact within property-based testing frameworks to efficiently identify potential errors.

Practical applications and frameworks

Nikhil talks about various technologies that support property-based testing. He also highlights QuickCheck in Haskell as the origin of this concept, which has since been translated into several languages.

Achieve reliability beyond human imagination

Properties Based Testing aims to outperform human capabilities by handling an immense number of permutations in system behavior. This ensures reliability and uncovers errors that might only have shown up under certain conditions or over time.

Implementation of properties-based testing in organizations

Nikhil suggests starting small with properties-based testing and gradually expanding its use within an organization. He emphasizes the importance of cultural change and convincing teams to implement new methods for improved results.

Limitations and precautions with properties-based testing

Properties-based testing has limitations. For example, it is not suitable where tests incur significant costs or are destructive. It is crucial to weigh up these factors before implementation.

A new ally for testers

Properties Based Testing does not replace the tester, but acts as an assistant that increases productivity by narrowing down potential problems for human analysis.