Exploratory Behavior With Generate-and-Test

One powerful mechanism of robustness is exploratory behavior, for which the desired outcome is produced by a generate-and-test mechanism.

This organization allows the generator to work and be developed independently of the tester that accepts or rejects a particular result. One can make an analogy to biological evolution, where the generator is random mutation and the tester is natural selection.

We may try a similar organization in our programs. Property-based testing tools can generate proposals from specifications, and design-by-contract tools can test contract clauses for implementations.

Here is an example in Python, which I’ve adapted from this post by Hillel Wayne:

The icontract library is our design-by-contract tool. We use the require decorator to require a precondition of invoking the function, and we use the ensure decorator to ensure a postcondition after the function has returned.

The hypothesis library is our property-based testing tool. We import its strategy mechanisms for generating lists and integers for our tester.

Our generator of test cases is independent of the contract guarding the function under test, which in turn is independent of the implementation of the function.

Running pytest, we uncover a contract violation via a test case generated by hypothesis.

We can filter/qualify our hypotheses with assumptions.

Our fuzzer has unearthed another violation. This time, we choose to change the implementation rather than add an assumption to the generator.

Now, we get nudged to acknowledge the impact of our stated assumptions.

And we decide to explicitly opt out.

Given that research software often supports the real-world generation of hypotheses and testing of models – where models must implement observation-determined “contracts”, I hope this quick example hints at how tools for property-based testing and design-by-contract can complement each other and help you lessen the semantic gap between physical research activities and the software artifacts that support them.