Experimentation and Understanding

Tyler Cowen’s blurb on Jason Crawford’s post Draining the swamp points out that even before vaccination and antibiotics, tuberculosis, influenza/pneumonia, and gastroenteric diseases were on the decline. This was due to making efforts on sanitation, which was “based on data collection and analysis, long before a full scientific theory of infection had been worked out.”

I think this is a very important point. Many people, if only implicitly, take the view that most progress comes from breakthroughs. That is, that we may toil for a while, making tiny amounts of progress, which maybe we can’t even hold on to because we just don’t know what we’re doing and have no real understanding of the underlying problem, etc, etc, and then one day we have a breakthrough, a major advance in understanding and knowledge, and finally we’re able to really put that problem behind us - from that point it’s just a matter of implementation and convincing the skeptics, and of doing away with the old and backwards ways of thinking.

But it’s not that at all. So many of the advances in medicine and elsewhere have been made without a full or deep understanding of the ‘physics-based’ phenomena that were at work - we just made empirical observations, experimented, and pursued what seemed to work. A very pragmatic, empirical approach. It can work because many problems worth solving have numerous contributing causes, so there are many avenues by which to affect outcomes, and thus lots of opportunities to experiment, observe, and iterate.

One of my colleagues has a great aphorism about the approach CMOS fabs take to solve problems - you don’t waste time understanding the problem, you just fix it. And by fix, he meant to create DOEs, observe the results, and iterate until you find something that makes a difference one way or another.

A physics-based understanding can certainly help, and in some cases certainly it has been a game-changer. But it isn’t strictly necessary for progress.

The corollary is that just because you do not understand the root cause of a problem, does not mean that you can’t make a lot of progress toward solving it. And the more complex the problem, or the more interacting parts there are, the more likely you will be able to find some subset that make an outsize contribution to the results. Don’t be dissuaded just because a problem seems opaque - experiment!

In fact, the experimentation that’s required for making empirical progress, is often also a pre-requisite for a fuller, physics-based understanding of the underlying problem. So if you think you need a physics-based, root-cause understanding of a problem in order to make meaningful progress, you’re going to need to do that experimentation anyway.

Looking at the infectious disease data that Jason Crawford cites, a good deal of progress came before the germ theory of disease was properly known, and a clear preponderance of it came before the development of antibiotics & vaccines. The experimentation was more valuable than the understanding.