I. Pure Bayesianism
On A Transformative Journey
The theory of probabilities is basically just common sense reduced to calculus; it makes one appreciate with exactness that which accurate minds feel with a sort of instinct, often without being able to account for it. If we consider the analytical methods that the theory gave rise to, the truth of the principles it relies on, the subtle logic that demands its application to solving problems, the public utility goods that is built upon it, and the extensions it has received and can still receive, given its application to the most important questions of natural philosophy and political economics; if we then observe that even in things that cannot be reduced to computation, probability theory allows the most reliable insights to guide us in our judgment, and that it teaches us to steer away from the illusions that often mislead us; we shall see that there is no science more worthy of our meditations, and whose results are more useful.
Pierre-Simon Laplace (1749-1827)
STUMPED BY A STUDENT
At the end of a lecture in probability and statistics I was giving at the Ecole Polytechnique of Montreal, a trolling student came to test me with a simple-looking puzzle. A man has two kids. At least one is a boy. What is the probability that the other is a boy too?
After a few seconds of thoughts, I successfully gave the right answer, which, as we shall see, is not l/2. The student acquiesced, and moved on to the next puzzle. Suppose you now learn that at least one of the kids is a boy born on a Tuesday. What is the probability that the other kid is a boy too?
This time, though, my answer was wrong. The student had stumped
The usual reflex is certainly to regard these two puzzles as mere mathematical games. Sure, there is a right answer. But that answer is only valid in a rigid and restricted mathematical setting. Solving these puzzles is useful in exercises or exams at school. But it's only mathematics.
Yet, the puzzle of the troll student is just an ultra-simplified version of many questions that we face in our daily lives. Should I believe a medical diagnosis? Is the presumption of innocence justified? Do judges racially discriminate? Is terrorism worrying? Can one generalize from one example? From a thousand? A million? Is the argument of authority worth anything? Are financial markets trustworthy? Are GMOs harmful? Why would science be more right than pseudosciences? Are robots about to conquer the world? Is capitalism wrong? Does God exist? What’s good and what’s bad?
For most, such questions have absolutely nothing to do with mathematics. And indeed, math alone is insufficient to address such questions. World hunger will not be solved by only proving theorems. Nevertheless, math likely has a lot to offer. It can help better structure our thinking, identify key challenges, and provide unexpected solutions. This is why many endeavours are more and more mathematized - including human- itary aid.
Despite the flourishing of mathematical models, it seems that most of us still want to distinguish the “real world” from academic courses that schools force us to take. In particular, the real world, it is often said, far transcends the framework of mathematics. As a result, mathematical theorems never seem to really apply to reality. How stupid must one be to think that mathematics has anything to say about the equality of rights?
Sadly, rejecting the usefulness of mathematics is not merely a bad- student reflex. Even years after failing the troll student puzzle, I had not yet realized that my mathematical mishap revealed my inability to correctly reason about the real world. I had not understood that a better understanding of the puzzle would be key to better analyze my traveling friends’ advice to plan my next trip - we’ll get there.
MY PATH TOWARDS BAYESIAN ISM
Granted, I did solve the troll student puzzle later that day, after some obscure and mysterious computations. But it was only three years later, in early 2016, when I investigated the frequentist-Bayesian debate, that I really took the time to meditate about the puzzle. Most importantly, at last, I finally took it out of its confined mathematical setting.
In particular, for the three years that followed, nearly once a day, I kept thinking about the magical equation that solves this puzzle. To my greatest pleasure, this mysterious equation started to reveal its secrets to me. Slowly but surely, this brilliant equation was seducing me. I began to see it everywhere. Months after months, my mind got flooded with the sublime elegance of this untameable equation. It was too much. I had to write about it. And I had to do this well. This is how, towards the end of 2016, I began the writing of the book you have just started.
The untameable equation I am talking about is what I like to pompously call the equation of knowledge. But mathematicians, statisticians, and computer scientists better know it as Bayes’ rule.
Bayes’ rule is a mathematical theorem of remarkable simplicity. It’s a compact equation, which is often taught in high school. It has a one- life proof, and only relies on multiplication, division, and the notion of probability. In particular, it seems vastly easier to learn than many other concepts in mathematics that high school and university students are asked to master.
And yet I’d claim that even the best mathematicians do not understand Bayes’ rule - and there is even some mathematics that explains our inability to grasp this equation! More modestly, there is absolutely no doubt that I still do not understand Bayes’ rule. Indeed, if I did, I would have immediately seen how the fact that at least one kid is a boy born on a Tuesday affects the likely gender of his sibling. I would have instantly given some relevant answer to the troll student. He would not have stumped me.
Over the last two years, I have been torturing my mind so as to never fail like this again. I want to know, understand, and feel Bayes’ rule. I have already learned a lot, and I am still learning so much! I meditate on Bayes’ rule almost every day, as if it were some sort of God I had to devote parts of my days to. And what a pleasure this is! Far from being a repetitive strain, these meditations have continuously fed my curiosity, as they have been discreetly whispering many of the unexpected implications of Bayes’ rule. One after the other.
After long months of thinking, I ended up concluding that few ideas were as deep as Bayes' rule. I fell in love with Bayes' rule to the point where I now gladly claim that “rationality” essentially boils down to applying Bayes’ rule - in which case no one is rational! This is the foundation of what might be call Bayesian philosophy, or Bayesianism.
A UNIFIED PHILOSOPHY OF KNOWLEDGE
Since I have not yet had the time to present Bayes’ rule, for now, I will be intentionally vague about what Bayesianism is. But basically, if I had to sum it up in three clumsy phrases, I would give the following definition. Bayesianism supposes that any model, theory, or conception of “reality” is mere belief, fiction, or poetry; in particular, “all models are wrong”. Empirical data must then force us to adjust the importance, or credence, that we assign to the different models. Crucially, the way credences are adjusted must obey Bayes’ rule as rigorously as possible.
I have long rejected the relevancy of this philosophy of knowledge. It seems to discredit any concept of reality or truth, that many scientists cherish. Yet, it seems to perfectly fit what physics Nobel laureate Richard Feynman once said': “I can live with doubt and uncertainty, and without knowing. I think it’s much more interesting to live not knowing than to have answers that may be wrong. I have approximate answers, I have possible beliefs and different degrees of certainty about different things. But I am not absolutely sure of anything. And there are many things I don't know anything about. But I don’t have to know an answer. I don’t feel frightened by not knowing things.”
You might fancy this viewpoint. Or jam might want to reject altogether this approach to knowledge. Yet, before rejecting or adhering to Bayesianism, I can only encourage you to first take the time to meditate Bayes’ rule and its consequences.
In this book, sadly, the main guide that I’ll be has a very incomplete understanding of Bayes’ rule. To help us in our thoughts, I will invoke a (female) fictitious character, the pure Bayesian, and we will try to imagine how this pure Bayesian behaves in different contexts. More than myself, it’s this pure Bayesian that we shall put to the test. This is what we shall do again and again in this book. We shall repeat thought experiments which will be challenges that the pure Bayesian will have to face. And we shall carefully scrutinize, judge, and criticize the behaviour of the pure Bayesian - although these criticisms will often quickly turn into that of our intuition and of our relentless overconfidence.
Now, the first Bayesian in history worthy of this name, the great Pierre-Simon Laplace, only had a partial description of the pure Bayesian. But over half a century ago, all computations, thoughts, and predictions of the pure Bayesian were rigorously described by the brilliant Ray Solomonoff. Unfortunately, as we shall discuss it in length, the pure Bayesian that Solomonoff described seems to necessarily violate the laws of physics (in particular the Church-Turing thesis).
This forces us to restrict ourselves to some approximate Bayesian- ism, which I shall call pragmatic. Pragmatic Bayesianism, which differs from pure Bayesianism by its need of (fast) computability, will be incarnated by another fictitious (male) character, which I shall call the pragmatic Bayesian. Unfortunately (or not!), my description of the pragmatic Bayesian will be very incomplete, as pragmatic Bayesianism is still a huge and very open field of research - and it’s not clear whether it can one day be fully closed.
As you are probably starting to guess, understanding the pure Bayesian and the pragmatic Bayesian is no easy task. To do so, we will have to discuss numerous fundamental concepts of mathematics, logic, statistics, computer science, artificial intelligence, and even notions of physics, biology, neuroscience, psychology, and economy. We will have to explain logarithms, contraposition, p-values, Solomonoff complexity, and neural networks, as well as entropy, Darwinian evolution, false memory, cognitive biases, and financial bubbles. What’s more, we shall also invoke several cases from the history of science to test our two fictitious heroes.
I know. This is a lot to take in to understand Bayes’ rule.
The good news is that I love explaining modern science - I have my own (French) YouTube channel called ScienceTAll! Thus, rather than reading this book as a treatise in philosophy, I invite you to (also) read it as a science and mathematics popularisation book. In fact, on our way to Bayesianism, I will not hesitate to take some detours through the world of science, with the secret goal to tease you and make you want to find out more about scientific theories!
But let’s get back to philosophy for now. As you can guess, I have surrendered to the appeals of Bayesianism. After long months of meditation, Bayesianism seduced me to the point where I felt the need to write about it. I kept being marvelled by the intelligence of the pure Bayesian. And I now aspire to resemble her more and more. Even long after the beginning of the writing of this book, I have kept discovering, again and again, the uncountable breathtaking wonders of what has since become my favorite mathematical equations of all.
When I started this book, I was an enthusiastic Bayesian. By now, I have become a convinced Bayesian. I would even call myself an extremist Bayesian, especially compared to others that call themselves Bayesians as well. But more importantly, I would like to become a competent Bayesian some day. I dream about the day I'll be able to apply Bayes’ rule, as I have become convinced that this is the only way to finally be a rational being!
Ironically, the emotional momentum that Bayes’ rule has given me sounds like irrational delirium. I cannot deny it. You may justifiably frown at me. You should be frowning at me. Indeed, I’m even pretty sure I am suffering from a huge cognitive bias caused by a sacralization of Bayes’ rule. After all, it’s impossible for me to be indifferent to the many secrets of Bayes’ rule that I have managed to uncover myself - even though many others uncovered these secrets half a century before me.
Having said that, conscious of this bias, I promise I have fought - and I still do - against the pure Bayesian. I have kept trying to prove her wrong; I have kept trying to win a debate against her. In vain.
-  A Set-Partitioning Formulation for Community Healthcare Network Design inUnderserved Areas. M Cherkesly, ME Rancourt & К Smilowitz (2017).
-  Measuring unfairness feeling in allocation problems. Omega. LN Hoang, FSoumis & G Zaccour (2016).
-  :i The Feynman Series - Beauty. Reid Gower (2011).
-  The Universal Turing Machine. ZettaBytes, EPFL. R. Guerraoui (2016).