Where we start by observing that both our senses and our interpretation machinery are not designed to capture reality as it is, that instead we are designed to make “useful” mistakes, and end up realising that scientists are stupid (using a strict Cipolla definition of stupidity).
My starting point is the least expected place: a public monograph available on the CIA website (!?). The full citation is: Heuer Jr, R. J. (1999). Psychology of intelligence analysis. Lulu. com.
Of all possible sources, I’m using this because it has a nice, plain-English description and discussion of “Cognitive Bias”: it explains intuitively why cognitive bias happens, and why it is a very big problem when one wants to understand reality as it truly is (as CIA analysts should). Here is the crucial conclusion:
Cognitive biases are similar to optical illusions in that the error remains compelling even when one is fully aware of its nature. Awareness of the bias, by itself, does not produce a more accurate perception. Cognitive biases, therefore, are, exceedingly difficult to overcome.
This further extends the concern explained in my previous post: I concluded that pure thought can easily allow to systematically make undetected mistakes, and decided that I ought to try to minimise this danger. The CIA monograph tells me that cognitive bias is “exceedingly difficult to overcome”, even when you are fully aware of it. Ouch, this is not going to be easy. I will look at the “CIA approved” catalogue of cognitive biases in a future post, but for now I’m more interested in exploring why they exist. As a shorthand, I’ll provisionally conclude that the evidence tell us that we don’t see things as they are, but apply useful heuristics instead.
This isn’t surprising: at some level we all know it, and it is easily established empirically. However, for reasons that I’ll start discussing below, it is also somewhat ignored by classic thinking both in the evolutionary and economics fields. Artem Kaznatcheev and colleagues over at EGG have been exploring this sort “blindness” for quite a while, I could link to many posts, but will start with the one that was able to get my own thoughts clarified, it’s entitled “Interface theory of perception can overcome the rationality fetish“. If you find my own explorations somewhat amusing, I would strongly encourage you to explore EGG in depth (especially if maths doesn’t scare you).
Anyway, I’m indebted with Artem for many reasons (I also wish to note how all the EGG team splendidly exemplifies the power of open science), and I especially like the post linked above because it uses the “rationality fetish” expression, you’ll see why as I proceed. More to the point, Artem’s essay shows how error (intended here as a faulty/misleading/incorrect representation of reality) creeps into two domains: straightforward perception and cognitive representations.
The first bit actually refers to Hoffman’s Interface Theory of Perception [see Hoffman, D.D. (2009). The interface theory of perception. In: Dickinson, S., Tarr, M., Leonardis, A., & Schiele, B. (Eds.), Object categorization: Computer and human vision perspectives. Cambridge University Press, Cambridge.]: the way I understand it, the basic idea is that organisms perceive the world in a way that maximises fitness, and that usually implies hiding the complex aspects of reality and build representations that are useful, not truthful. Hoffman uses the term “interface”, in the sense of “user interface” of computer software. In the case of software (and I know a thing or two about this!) the things you see on the screen are designed to allow you to do your thing in the easiest possible way, and it is fair to say that the function of the interface is to hide the complexity under the hood. This is almost always true for software, but I’m not entirely convinced that it applies to the evolution of perception; to know why, please refer to my comment on Artem’s post here. Anyway, for the purpose of this post, I don’t need to discuss my perplexities in detail, it is enough to note that it’s quite reasonable to say that “the function of perception is to maximise fitness, not to model reality faithfully, so that systematic mistakes and approximations are the norm, not the exception“.
Interestingly, the same applies to indirect interpretations of reality: by this I mean the understanding of complex interactions, or our internal representation of phenomena that can be modelled successfully using derived concepts that are not direct representations of the physical reality (the typical example is how we “understand” people by interpreting their intentions). Once again, Artem comes to my rescue, with another post “Evolving useful delusions to promote cooperation“. The title says it all, right? There is plenty of maths in Artem’s article, so I’ll try to summarise it in plain English (corrections are welcome!).
Classic game theory revolves around a basic principle: given a certain set of mathematical rules that define the interaction between two or more players, one can usually find out what is the optimal strategy for each player. In other words, by modelling somewhat simplified versions of real interactions (in Artem’s case, using a model that describes the possible variations of the classic Prisoners’ dilemma game), maths can show us what are the supposedly “rational” behaviours for the different players. It’s a powerful approach, and a frequently misused one as well.
What is important of Artem’s effort (with Marcel Montrey and Thomas Shultz) is that they produced a game model that allows actors to maintain and modify their own, idiosyncratic representation of reality (in this case, each player acts based on their own evaluation of the actual rules that govern the game, and their experience at each round shapes and updates their own evaluation), and has shown how and when the players in an arena will develop representations that do not match the real rules at all. Crucially, what may happen in certain circumstances is that all players will develop towards a representation of the game that favours collaboration, even when the “best” mathematically defined strategy would be to always “cheat”.
In my interpretation, both domains (perception and representation, or Interface Theory and Artem’s game-theory simulation) provide are examples of how natural selection produces systems that maximise fitness: to do so, they represent the features of reality that happen to be useful, obscuring what is real, but irrelevant, but also what happens to be real and counter-productive. The representation is also a classic case of self-sustaining cognitive attraction: when agents drift towards cooperation, the game changes so that the best behaviour is to keep the magic alive and keep cooperating. The players find themselves in a world that really is different from what the original rules would suggest to a naive, “a priori” observer. Artem’s model is, in other words, a mathematical representation of the mechanism that allows self-sustaining cognitive attractors to shape our experiences.
Perhaps unsurprisingly, both theoretical efforts require us to re-evaluate what we consider to be an error. In the case of a prisoner dilemma arena, where most players drift toward cooperation even if the payoff rules indicate that the rational strategy is to defect, agents are actually maximising their fitness (or profit) and they do so because they evolved a misrepresentation of the actual rules. Is this an error? If your aim is to find out what the rules are, then yes, it is. But the players are designed to maximise their fitness, their representations of what happens are instrumental, not the final goal. Under these circumstances, taking the “rational” (note the scare-quotes, rational here means the fetishistic mathematically derived “best strategy”) stance, and always defect, actually minimises fitness. It means that avoiding the “rational” choice, given the actual purpose, is not an error at all.
This is a cute thought, and something most of us will find it easy to recognise at some deep, intuitive level: the artificial arena settings devised in Artem’s experiment provide a distilled case where acting “rationally” is the stupid option. Once again, note the scare-quotes around “rationally”: they are there to mean the “mathematically calculated optimal strategy”, or what looks like it when one doesn’t account for the full complexity of the arena. Also, “stupid” here follows Cipolla’s definition in a literal way: acting “rationally” is stupid because it does not produce the desired effect.
The corollary observation is that scientist that insist with this interpretation of “rationality” and keep considering the wrong strategy “rational” are actually being stupid (in Cipolla’s sense). They get fooled by maths, and fail to see that the lovely equations they use to calculate the optimal strategy are just plainly wrong: they are wrong because they don’t show the optimal strategy, and they don’t do so because they are not capturing the full complexity of the arena. Specifically, they fail to represent the evolutionary potential of it. In this specific case, the internal representations that players develop are simply less wrong than the standard mathematical solution, but in a deeper way: they represent the situation on the field, not the actual rules of the game, they faithfully model how the game actually does develop.
For me, the implications are overwhelming, and will have to spend quite some time thinking/writing about them. To finish off this post, I’ll list a few implications, and hopefully discuss them in the future:
- Mathematical models are tricky, applying objectively true mathematical theory to reality is a risky business. If one doesn’t capture/represent the relevant aspects of reality (in the game-theory case, the standard approach does not represent the ability of players to learn from experience [Update, following the kind correction in the comments] consider the fact that all players have their own pre-existing bias, nor it accounts for indirect effects across multiple individuals and across generations[/Update]), the whole effort will look as rigorous as it gets, but would still be wrong and/or meaningless. This is a dangerous kind of error, and I suspect it’s quite common in the science business.
- The “frame problem” (will explain what it is in a dedicated post) may be a meaningless issue. We don’t have a mysterious ability to understand how to interpret reality correctly: we are in the business of extracting information that allows us to function, and that doesn’t require to build faithful representations at all.
- I can’t trust my own perceptions, and neither my own most profound “understandings”. They may well be useful in my everyday life, but that doesn’t mean they are faithful representations of what is really out there.
- My own assumption: that “understanding reality as faithfully as possible is a good thing to do” may be wrong, after all. All of the above seem to indicate that it is unnecessary and sometimes counter-productive.
Once more, the outlook is pretty bleak! But hey, no one said it would be easy…
Writing my own user manual 
[…] also note that I do not think I will be able to proceed in a strictly systematic way: I will start with sources of error that I’ve already identified, and then add to the catalogue as and when […]
Thank you for the kind words! I am glad that I was able to ignite some thoughts. I think you provided a very good summary of my posts, the only point I would pick on is:
If you are referring to the useful delusions post then this isn’t quite right. The players don’t actually learn much from their experience. In particular, their conception of the payoffs is genetic, not experiential. The only thing they update is their belief that others will cooperate with them, and this has a much less drastic effect on the outcome than the ‘erroneous’ genetic representations. In fact, you can get similar results without learning as long as you pick some reasonable rule for deciding in cases where both (or neither) pure strategy is subjectively rational.
What I believe to be the key difference between the standard and evolutionary approach in this case is the problem of externalities. The ‘standard’ analysis is overly reductionist and looks only at the pairwise interaction. The ‘evolutionary’ analysis (or what their genetic conceptions converge to), however, seems to take care of indirect fitness effects (i.e. how their actions affect close relatives and descendants) that the objective treatment ignores. I am pretty confident of this conclusion, but I can’t endorse it 100% since I haven’t done a formal analysis of games on random graphs from an inclusive fitness perspective.
Finally, I wanted to push your comment here further:
You can also get this from bounded rationality, not in the ad-hoc econ sense of the word but in the theoretical computer science sense (i.e. see the references to Livnat & Pippenger in that post), and I think this also builds some support for interface theory of perception (or in this case, maybe cognition?). Unfortunately, I was not aware of interface theory at the time of writing that post, and I don’t think Livnat & Pippenger were familiar with it when they were writing their papers either (although I will have to reread the papers and email Livnat to double check). I will have to revisit these connections more in future posts in this ongoing discussion.
Artem,
the kind words are 100% earned. I do my own writing/thinking in my private time, and it quite simply would not be possible if it wasn’t for people like you, the whole EGG and many more.
I’ve changed the bit in your first quote, but I’m still on the fence. First, the new version is way less intelligible, I’m still not sure I’ve got it right, and in all cases, I’m also unsure on whether I want to refer specifically to your own simulations or to the breath of possible simulations (that would account for many combinations of factors, besides pairwise interactions). Overall, I don’t think either way has the potential to change the take home message, so I’ll leave it like it is.
It’s funny how we are suspicious of different concepts: you run away as soon as you sniff an idea of evolutionary ladder (even when I don’t see it), while whenever I hear something about “inclusive fitness” I always think “do we really need the concept?”. I’ll try to explain (will clarify my point on the evolutionary ladder on your blog): my standard understanding is that a given variant will change its frequency in a population depending on the relative fitness. If a variant is facilitated by the presence of more copies of it, there will be a threshold where this facilitation is stronger than the original effect. If I look at the whole population, inclusive fitness is meaningless to me, yes, that particular variant may not propagate through agent X, and still be passed on by her relative Y, but the value I’m looking for is the relative fitnesses of each different variant across the whole population and in that domain “inclusive fitness” looks like an embellishment that explains very little. But again, I may well be wrong.
See also how I’ve changed my original text, but what I’m after is simple: even without knowing the detail of how your model works (say: do you allow more than one round for each genetic generation? Do players eventually die of old age? Is there sexual reproduction or just parthenogenesis?) the whole thing that classic game theory is ignoring is the possible establishment of positive feedback loops. In the real world these can happen horizontally (within members of a generation), vertically (across generations), or both, and favour collaborative alleles in all cases; and again, in the real world relatives have usually more chances to interact with one another, making the horizontal interaction important for understanding evolution in the field. I don’t need mathematical proof to know that under some conditions this sort of inter-individual positive feedback will happen, the maths may tell me where in the whole space of possibility it will happen, but since the space is an artificial simulation, the actual numbers mean little to me. That’s to say that I’m more committed than you: I am confident even without the final formal demonstration.
Bounded rationality indeed has something to do with systematic mistakes, and yes, I see it as exactly the same thing in the case of interface theory. For what I’ve written here, the domain is cognition, interface theory is about both cognition and perception (because it’s about how to collect and classify perceptions so that they can inform cognition in a way that maximises fitness), but the mechanism seems the same to my eye. It’s somewhat amusing to notice who it’s the same also for the “error” of classic game theory: the simpler model makes a systematic mistake because it’s simple, but the model is still popular because the minds that use and evaluate it are bounded, etc., etc… Apart from smart-ass remarks, bounded rationality is part of the picture, but I suspect a crucial part is what you mention in Phenotypic plasticity, learning, and evolution: to what extent does step 2 of the Simpson-Baldwin effect count? Or, in other words, under what circumstances does selection favour brute heuristic generalisations over accuracy? Interesting, and will discuss it further over on EGG.
[…] and try to find out how to detect and possibly neutralise them. The first attempt focused on the generic sources of bias that are the direct consequence of biology, it sparked the discussion with Artem and reached fairly […]
I will continue the discussion of your comment on my comment here.
I actually think inclusive fitness is relatively useful concept, although one that is currently under some dispute with regard to if it is the best way to understand cooperation. However, as long as this dispute is kept mathematical and precise, I think it is fruitful. I find that this goes for all discussions. I use the terminology of inclusive fitness because I want to be able to communicate with biologists. It doesn’t make sense for me to speculate from my armchair and only be understandable to myself.
I believe that the general truth underlying these sort of models (not just the particular of my model) is that the underlying representation will converge towards faithfully representing the inclusive fitness effects instead of the individual fitness effects. This has a precise meaning to biologists, and my hope is to show this correspondence rigorously in the future. The best part about inclusive fitness, is that it also has some measure of empirical meaning to biologists which is not typical. Of course, it isn’t perfect, and there are plenty of things inclusive fitness is bad for modeling.
This is exactly where we differ, I think. The above comment to me was completely contentless and rather meaningless. When you say things as vaguely as “positive feedback loops” then to me you are not saying anything at all, just words. Classical game theory and EGT both incorporate all kinds of positive and negative feedback loops. They include some and they exclude others. It is obvious that any model will miss some effects, but it is not enough to point this out. It is easy to be a critic, but to be meaningful you have to be precise and best if you can offer an alternative.
When you are working with heuristic models, as we are in this case, you should never take their exact numbers as meaningful. You should look at the qualitative effects and compare it to other models with different assumptions. After enough experience you will develop a feel for what ‘sort’ of assumptions lead to what ‘sort’ of results and this can become useful when you are exploring your mental (or other) models that are closer to experimental data.
However, it makes no sense to me to be confident in something without models. All you are saying there is that you are confident in your mental model but not confident enough to try to formalize your mental model into something other people can play with. Sure, sometimes (most times?) it is impossible to perfectly capture your mental model, but that doesn’t mean you shouldn’t try. If you don’t try to understand the models you have in your head, you will just end up lying to yourself.
Sorry for the delayed responses, I get caught up in all kinds of things on the never-ending internet to-do list.
Artem, thanks for taking the time to keep this dialogue going. As you can see, delays are not a problem: I struggle to find the time to catch up as well. I also need time to think: you’re doing an excellent work in challenging my own stance, and since I’m now concentrating on my own errors, I tend to slow down my writing and juggle with different perspectives until I’m confident that the ideas are stable enough.
I think a crucial difference between our approaches is the direct consequence of different competences. I studied and practised biology, and I’m discovering how strong is my drive towards philosophy. I can (barely) follow the maths in your posts, but before reading your answer I’ve never even considered the idea of creating my own EGT arena to see what comes out of it. I suppose I could hack something together in an afternoon of coding (or thereabout, provided that I already had a clear plan to follow), but I’m sure I won’t have the knowledge required to rigorously interpret the output. Who knows, maybe our discussions will give me the needed motivation, life never stops to surprise me.
I appreciate your efforts to make your work accessible to a wider audience, if it wasn’t for that, I would probably find your posts too arduous (as opposed to just difficult).
Anyway, our discussion so far had three effects on me:
1) It contributed to shake my confidence, and that has to be a good result in itself. I’m currently questioning many aspects of my intellectual project, many more that I would have without your contribution.
2) I am slowly clarifying my ideas on what sort of investigation I would like to see done with the kind of tools that you use. With some luck, I hope I’ll be able to write some less incoherent thoughts in the not so distant future. The basic idea is that the study of evolution does indeed need formal mathematical models, but to the best of my (very limited) knowledge, what is currently accepted in the main stream suffers from too many (and sometimes questionable) assumptions. I hope I’ll write something, but it will not happen overnight and I can’t even be sure that I will publish it. The probability that I’ll consider it worthless and amateurish is quite high.
3) I’m also busy clarifying my thoughts about learning, ability to accommodate change, fitness-peaks, evolutionary equilibrium (or lack thereof) and evolutionary ladders. This is likely to be pure thought (no numbers), but is a prerequisite for starting to think about 2). It is also tightly connected to the only serious project of mine: as I’ve hinted here and there, I’ve submitted a paper on consciousness and I’m nervously awaiting the peer review comments (probability that it will be accepted is around 5%, so I should probably not even mention it).
I really would like to think that our discussion was in some way helpful to you as well, but I don’t see why it may be so!
I’m sure we’ll be in touch in the future: will let you know if I publish something that I hope may be of interest to you. In the mean time, I’ll be following you as usual, do send stuff/thoughts in my direction as and when you may see fit. Your input is always welcome.
Send me an email if you decide to go this route. I will be happy to offer what advice I can. If you are interested in the userful delusions work in particular, then I have code for that on github, although it is a private repository for now.
Good luck! Have you blogged about it? You should!
I look forward to continuing our blogging interaction.
Artem,
I’ve been thinking a little about evolutionary models and I’m pretty sure you won’t like what I have in mind. Way too complex, I fear. Also means it would take much more than an afternoon to hack something together. With some luck I’ll find a way to organise it all and write a post as a starting point. Thanks for your offer: you should be afraid, I may accept it one day!
I’m not going to blog explicitly about consciousness before receiving the PR comments. Giving my inclination for silly mistakes, it seems the only wise way to proceed: I do want to be taken seriously on this subject. On the other hand, when the time will come, I’ll write a lot!
PS Just wrote a post on Antifragility where I explain some of my grandiose claims, this time on Natural Selection as a source of Antifragility.
[…] Interface theory of perception can overcome the rationality fetish. That last post even prompted a post-length reflection from Sergio Graziosi. In other words, if I want more discussion then I should replace the technical posts by […]
[…] Theory of Perception sparked a long conversation, and eventually informed another post of mine: Sources of error: we are all biased, but why? This post is the direct result of the fortuitous encounter between Artem and me, something that […]
[…] underline current Western societies. What a surprise: offence does work in this way, it exposes our biases, assumptions and blind-spots. The real problem is that it worked too well: because the show was […]
[…] Having said this, an important disclaimer: I’m Italian, an EU citizen, I live and work in the UK and have no wish to leave. This makes my opinion naturally influenced by a relatively significant conflict of interest: I do wish my personal and professional life not to be threatened by a victory of the leave side. Furthermore, my world view and inclinations clearly point in the same direction: I think the remain option is very obviously the best one, furthermore, I refuse to believe that Britain as a whole will fail to recognise what’s best. Thus, be warned: what follows is an opinion, coloured by the sort of biases that grip all of us. […]