On levels of understanding of systems:
Technology very broadly is becoming more and more complicated … actually so complex that no one, whether you’re an expert or otherwise, fully understands these things … They have enormous number of parts that are all interacting in highly nonlinear ways that are subject to emerging phenomena.
We’re going to have bugs and glitches and failures. And if we think we understand these things well and we don’t, there’s going to be tons of gap between how we think we understand the system and how it actually does behave.
On modelling reality with a system and then creating a model of that system:
… the world is messy and complex. Therefore, often, in order to capture all that messiness and complexity, you need a system that effectively is often of equal level of messiness and complexity … whether or not it’s explicitly including all the rules and exceptions and kind of the edge cases, or a system that learns these kinds of things in some sort of probabilistic, counterintuitive manner.
It might be hard to understand all the logic in [a] machine learning system, but it still captures a lot of that messiness. I think you can see the situation where in machine learning, the learning algorithm might be fairly understandable. But then the end result … You might be able to say, theoretically, I can step through the mathematical logic in each individual piece of the resulting system, but effectively there’s no way to really understand what’s going on.
On “physics” and “biological” thinking:
[Physics:] A simple set of equations explains a whole host of phenomena. So you write some equations to explain gravity, and it can explain everything from the orbits, the planets, the nature of the tides … It has this incredibly explanatory power. It might not explain every detail, but it maybe it could explain the vast majority of what’s going on within a system. That’s the physics. The physics thinking approach, abstracting away details, deals with some very powerful insights.
[Biology:] Which is the recognition that oftentimes … the details not only are fun and enjoyable to focus on, but they’re also extremely important. They might even actually make up the majority of the kinds of behavior that the system can exhibit. Therefore, if you sweep away the details and you try to create this abstracted notion of the system, you’re actually missing the majority of what is going on.
Oftentimes I think people in their haste to understand technology … because technologies are engineered things … think of them as perhaps being more the physics thinking side of the spectrum.
There’s this idea within complexity science … “robust yet fragile,” and the idea behind this is that a lot of these very complex systems are highly robust. They’ve been tested thoroughly. They had a lot of edge cases and exceptions built in and baked into the system. They’re robust to an enormously large set of things but oftentimes, they’re only the set of things that have been anticipated by the engineers. However, they’re actually quite fragile to the unanticipated situations.