The best book I read in 2014 was ‘Probability Theory – The Logic of Science’ (2003) by Edwin T. Jaynes. It was recommended to me by an ex-colleague from Toronto and it turned out to be one of the most exciting scientific books I’ve read so far.

E.T. Jaynes was a physicist and one of the main figures in the development of probability theory in the 20th century. He was associated with the school of scientists that, in the tradition of Laplace, viewed probability as a system of logic – what’s commonly known as the Bayesian school of probability. This book was his attempt to summarize his view on probability theory. Unfortunately, he was not able to complete it before his death. As a result, the published version is missing some chapters from the originally planned outline and some of the later chapters in the book are not as polished as earlier ones. Still, its 700 pages make an extraordinary read.

The sense of unease which many of us have towards the subject of statistics is largely a reflection of the inadequacies of the ‘cook book’ approach to data analysis that we are taught as undergraduates. Rather than being offered a few clear principles, we are usually presented with a maze of tests and procedures. […] This apparent lack of a coherent rationale leads to considerable apprehension because we have little feeling for which test to use or, more importantly, why.  A more unified and logical approach is provided by the probability formulations of Bayes and Laplace. […] In more recent times they have been expounded by Jaynes and others.

Devinderjit Sivia, John Skilling – Authors of ‘Data Analysis: A Bayesian Tutorial’

E.T. Jaynes presents Probability Theory as a generalization of Aristotelian logic – as a system of logic in which propositions are associated with degrees of plausibility rather than binary truth values (true or false). Indeed, early in the book, Jaynes dedicates one chapter to the question of how we should define such a system. Why do we define probabilities as real numbers between 0 and 1? How do we end up with the sum and product rules of probabilities?

What follows that early chapter is a sweeping treatise on probability theory. Part of it covers what would make textbook material on probability theory and its applications – how to use the rules of probability to perform parameter estimation and model selection. Another part of it discusses what can be considered as extensions to the theory: principles to assign probabilities in the absence of data (‘prior’ probabilities), how to use probabilities when making decisions (decision theory). A third part covers deeper technical and philosophical issues (normality assumptions, paradoxes, randomness and random experiments) that were the subject of intense controversy with orthodox statisticians in the 20th century.

Most of the technical material in the book had appeared before, both in the author’s work and others’. What makes it exceptional, though, is how Jaynes discusses issues of enormous scope from the vantage point of basic principles. That’s in sharp contrast with most discussions on data analysis I had previously been exposed to, that, in the words of D. Sivia and J. Skilling, came with ‘a maze of tests and procedures’ but little coherent rationale. On the contrary, reading this book helped me clarify my understanding of probabilities and left me with a new way of thinking. I now wish I had read it earlier and that the Bayesian viewpoint were more prominent in academic curriculums.

If you’re not sure if you should read ‘Probability Theory’, you can take a look at its list of contents and first 3 chapters, that are available online in pdf.  If you’re considering reading it, then know that it is technical, philosophical, visionary, and very-very personal – a polemic. It is also a masterpiece that we are lucky exists, even unfinished.