Why Publishing Failures is Important for Science

Not documenting failures causes them to be repeated. This repetition is waste because no new information is generated. Only new failures generate information.

There is an optimum failure rate.¹ We can avoid oversimplifications like “embrace failures” or “avoid failures”. We maximize information gain by making outcomes equally likely. Thus, for a test with two outcomes, seek a 50% failure rate. This is information-theoretic support for the binary search algorithm and for determining LD50 in pharmaceutical toxicity trials.

Not all evaluation is designed to maximize information gain. Validation tests strive for 100% success, and rightly so. Exploratory testing, though – seeking novelty – is a sine qua non of scientific research.

Not all uncertainty is valuable to resolve. We pay for tests with time and money. “Minimizing the economic impact of variability is a profoundly different goal than minimizing variability.”¹