Easy Things Are Hard

In general, we’re least aware of what our minds do best. It’s mainly when systems start to fail that we engage the special agencies involved with what we call “consciousness.”

Accordingly, we’re more aware of simple processes that don’t work well than of complex ones that work flawlessly.

This phenomenon helps to explain the poor performance of many so-called expert systems in the 1980s. There were attempts to fully rationalize human expertise as calculative rules. The effect was often to regress an expert’s knowing how to a novice practitioner’s knowing that.

Skill acquisition in unstructured domains moves not towards abstract rules, but rather from abstract rules to particular cases. And “the distinction between education, a process aimed at drawing out the abilities of the student, and training, in which the student is learning to negotiate a structured domain, is crucial.”

This may help shed light on much of the recent mixed success of “unexplainable” neural-network-based decision systems.