Bayesian probability

From Wikipedia, the free encyclopedia

Bayesian probability is one of the most popular interpretations of the concept of probability. The Bayesian interpretation of probability can be seen as an extension of logic that enables reasoning with uncertain statements. To evaluate the probability of a hypothesis, the Bayesian probabilist specifies some prior probability, which is then updated in the light of new relevant data. The Bayesian interpretation provides a standard set of procedures and formulas to perform this calculation.

Bayesian probability interprets the concept of probability as 'a measure of a state of knowledge', ^[1] in contrast to interpreting it as a frequency or a physical property of a system. Its name is derived from the 18th century statistician Thomas Bayes, who pioneered some of the concepts. Broadly speaking, there are two views on Bayesian probability that interpret the 'state of knowledge' concept in different ways. According to the objectivist view, the rules of Bayesian statistics can be justified by requirements of rationality and consistency and interpreted as an extension of logic.^[1][2] According to the subjectivist view, the state of knowledge measures a 'personal belief'.^[3] Many modern machine learning methods are based on objectivist Bayesian principles.^[4] One of the crucial features of the Bayesian view is that a probability is assigned to a hypothesis, whereas under the frequentist view, a hypothesis is typically rejected or not rejected without directly assigning a probability.

Contents 1 Bayesian probability calculus 2 History 3 Justification of the Bayesian view 4 Personal probabilities and objective methods for constructing priors 5 Scientific method 6 References 7 See also 8 Footnotes 9 External links

[edit] Bayesian probability calculus

The probability of a hypothesis given the data (the posterior) is proportional to the product of the likelihood times the prior probability (often just called the prior). The likelihood brings in the effect of the data, while the prior specifies the belief in the hypothesis before the data was observed.

More formally, Bayesian inference uses Bayes' formula for conditional probability:

where

• H is a hypothesis, and D is the data.
• P(H) is the prior probability of H: the probability that H is correct before the data D was seen.
• P(D | H) is the conditional probability of seeing the data D given that the hypothesis H is true. P(D | H) is called the likelihood.
• P(D) is the marginal probability of D.
• P(H | D) is the posterior probability: the probability that the hypothesis is true, given the data and the previous state of belief about the hypothesis.

P(D) is the prior probability of witnessing the data D under all possible hypotheses. Given any exhaustive set of mutually exclusive hypotheses H_i, we have:

We can consider i here to index alternative worlds, of which there is exactly one which we inhabit, and H_i is the hypothesis that we are in the world i. P(D,H_i) is then the probability that we are in the world i and witness the data. Since the set of alternative worlds was assumed to be mutually exclusive and exhaustive, the above formula is a case of the law of alternatives.

P(D) is the normalizing constant, which in many cases need not be evaluated. As a result, Bayes' formula is often simplified to:

where denotes proportionality.

In general, Bayesian methods are characterized by the following concepts and procedures:

• The use of hierarchical models, and the marginalization over the values of nuisance parameters. In most cases, the computation is intractable, but good approximations can be obtained using Markov chain Monte Carlo methods.
• The sequential use of the Bayes' formula: when more data becomes available after calculating a posterior distribution, the posterior becomes the next prior.
• In frequentist statistics, a hypothesis is a proposition (which must be either true or false), so that the (frequentist) probability of a frequentist hypothesis is either one or zero. In Bayesian statistics, a probability can be assigned to a hypothesis.

Broadly speaking, there are two views on Bayesian probability that interpret the 'state of knowledge' concept in different ways. For objectivists, the rules of Bayesian statistics can be justified by requirements of rationality and consistency^[1][2]. Such requirements of rationality and consistency are also important for subjectivists, for which the state of knowledge corresponds to a 'personal belief' (rather than the objective state of knowledge in the world)^[5]. For subjectivists however, rationality and consistency constrain the probabilities a subject may have, but allow for substantial variation within those constraints. The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.

[edit] History

The term Bayesian refers to Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem. However, it was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence ^[6]. When insufficient knowledge was available to specify an informed prior, Laplace used uniform priors, according to his "principle of insufficient reason".^[7] Laplace also introduced primitive versions of conjugate priors and the theorem of von Mises and Berstein, according to which the posteriors corresponding to initially differing priors ultimately agree, as the number of observations increases.^[8] This early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes ^[9]).

After the 1920s, inverse probability was largely supplanted by a collection of methods that were developed by Ronald A. Fisher, Jerzy Neyman and Egon Pearson. Their methods came to be called frequentist statistics.^[9] Fisher rejected the Bayesian view, writing that "the theory of inverse probability is founded upon an error, and must be wholly rejected" .^[10] At the end of his life, however, Fisher expressed greater respect for the essay of Bayes, which Fisher believed to have anticipated his own, fiducial approach to probability; Fisher still maintained that Laplace's views on probability were "fallacious rubbish".^[11] However, Fisher did use Bayesian methods with informed priors sometimes, for example in mice genetics.^[12] Neyman started out as a "quasi-Bayesian", but subsequently developed confidence intervals (a key method in frequentist statistics) because "the whole theory would look nicer if it were built from the start without reference to Bayesianism and priors". ^[13] The word Bayesian appeared in the 1950s, and by the 1960s it became the term preferred by those dissatisfied with the limitations of frequentist statistics^[9][14].

In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed ^[15]. No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.

In the further development of Laplace's ideas, subjective ideas predate objectivist positions. The idea that 'probability' should be interpreted as 'subjective degree of belief in a proposition' was proposed, for example, by John Maynard Keynes in the early 1920s. This idea was taken further by Bruno de Finetti in Italy (Fondamenti Logici del Ragionamento Probabilistico, 1930) and Frank Ramsey in Cambridge (The Foundations of Mathematics, 1931).^[16] The approach was devised to solve problems with the frequentist definition of probability but also with the earlier, objectivist approach of Laplace ^[15]. The subjective Bayesian methods were further developed and popularized in the 1950's by L.J. Savage.

The strong revival of objective Bayesian inference was mainly due to Harold Jeffreys, whose seminal book "Theory of probability" first appeared in 1939. In 1957, Edwin Jaynes promoted the concept of maximum entropy for statistical inference, which is an important principle in the formulation of objective methods, mainly for discrete problems. In 1979, José-Miguel Bernardo introduced reference analysis^[15], which offers a general applicable framework for objective analysis.

Other well-known proponents of Bayesian probability theory include I.J. Good, B.O. Koopman, Dennis Lindley, Howard Raiffa, Robert Schlaifer and Alan Turing.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to dramatic improvements in hardware and software, and an increasing interest in nonstandard, complex applications ^[17]. Despite growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics^[18]. Nonetheless, Bayesian methods are widely accepted and used, such as for example in the field of machine learning ^[19].

[edit] Justification of the Bayesian view

The Bayesian probability calculus has been supported by several arguments, such as the Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.

Richard T. Cox proved^[2] that Bayesian updating follows from several axioms, including two functional equations and the controversial^[20] hypothesis that probability be a continuous function.

The Dutch book argument was proposed by de Finetti, and is based on betting. A Dutch book is made when a clever gambler places a set of bets that guarantee a profit, no matter what the outcome is of the bets. If a bookmaker follows the rules of the Bayesian calculus in the construction of his odds, a Dutch book cannot be made.

However, Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books.^[21] In fact, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics" following the publication of Richard C. Jeffrey's rule). The additional hypotheses sufficient to specify (uniquely) Bayesian updating are substantial, complicated, and unsatisfactory, according to Bas van Fraassen's book Laws and Symmetries.

A decision-theoretic justification of Bayesian methods was given by Abraham Wald, who proved that every Bayesian procedure is admissible (and admissibility is related to Pareto efficiency). Conversely, every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures. Wald's result also established the Bayesian formalism as a fundamental technique in such areas of frequentist statistics as point estimation, hypothesis testing, and confidence intervals.^[22]

[edit] Personal probabilities and objective methods for constructing priors

Following the work on expected utility theory of Ramsey and von Neumann, decison-theorists have accounted for rational behavior using a probability distribution for the agent.^[23] Ramsey and Savage noted that the individual agent's probability distribution could be objectively studied in experiments.^[24] Since individuals act according to different probability judgements, these agents' probabilites are "personal" (but amenable to objective study).

Of course, personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.

Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probability-distribution for "regular" statistical problems; c.f. well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes: These theorists and their successors have suggested several methods for constructing "objective" priors:

• Maximum entropy
• Transformation group analysis
• Reference analysis

Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challenging statistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice.^[25] Alas, each of these methods gives implausible priors for some problems, and so the quest for "the universal method for constructing priors" continues to attract statistical theorists.

Thus, the Bayesian statistican needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.

[edit] Scientific method

The scientific method can be interpreted as an application of Bayesian inference. In this view, Bayes' rule guides (or should guide) the updating of probabilities about hypotheses conditional on new observations or experiments.^[26]

Some experiments on belief revision have suggested that humans change their beliefs faster when using Bayesian methods than when using informal judgement.^[27] Bayesian methods have been used for hundreds of years, so there are many examples of Bayesian inference to scrutinize. Of the tens of thousands of papers published using Bayesian methods, few criticisms have been made of implausible priors in concrete applications. Such criticisms are themselves welcomed by Bayesian statisticians, as part of the inevitable revisions of science.^[28] Nonetheless, worries about the possible problems of Bayesian methods continue to appear.^[29] Such worries would warrant more attention were they to be accompanied by experimental evidence that would challenge the perception that Bayesian methods facilitate belief revision or were they to discuss published examples of implausible priors that led to practical problems.

[edit] References

• Hald, Anders (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley. ISBN 0471179124. 
• Stigler, Stephen M. (1990). The History of Statistics: The Measurement of Uncertainty before 1900. Belknap Press/Harvard University Press. ISBN 0-674-40341-X. 
• Stephen Stigler Statistics on the Table: The History of Statistical Concepts and Methods. ISBN 0-674-83601-4
• Charles Sanders Peirce and Joseph Jastrow (1885). "On Small Differences in Sensation". Memoirs of the National Academy of Sciences 3: pp. 73–83.  http://psychclassics.yorku.ca/Peirce/small-diffs.htm
• Ramsey, Frank Plumpton; “Truth and Probability” (PDF), Chapter VII in The Foundations of Mathematics and other Logical Essays (1931).
• de Finetti, Bruno. "Probabilism: A Critical Essay on the Theory of Probability and on the Value of Science," (translation of 1931 article) in Erkenntnis, volume 31, September 1989.
• de Finetti, Bruno. 1937, “La Prévision: ses lois logiques, ses sources subjectives,” Annales de l'Institut Henri Poincaré,

de Finetti, Bruno. "Foresight: its Logical Laws, Its Subjective Sources," (translation of the 1937 article in French) in H. E. Kyburg and H. E. Smokler (eds), Studies in Subjective Probability, New York: Wiley, 1964.

• de Finetti, Bruno. Theory of Probability, (translation by AFM Smith of 1970 book) 2 volumes, New York: Wiley, 1974–5.
• Donald Davidson, Patrick Suppes and Sidney Siegel (1957). Decision-Making: An Experimental Approach. Stanford University Press. 
• Pfanzagl, J (1967). "Subjective Probability Derived from the Morgenstern-von Neumann Utility Theory". in Martin Shubik. Essays in Mathematical Economics In Honor of Oskar Morgenstern. Princeton University Press. pp. 237–251. 
• Pfanzagl, J. in cooperation with V. Baumann and H. Huber (1968). "Events, Utility and Subjective Probability". Theory of Measurement. Wiley. pp. 195–220. 
• Morgenstern, Oskar (1976). "Some Reflections on Utility". in Andrew Schotter. Selected Economic Writings of Oskar Morgenstern. New York University Press. pp. 65–70. 
• Edwards, Ward (1968). "Conservatism in Human Information Processing". in Kleinmuntz, B. Formal Representation of Human Judgment. Wiley. 
• Edwards, Ward (1982). "Conservatism in Human Information Processing (excerpted)". in Daniel Kahneman, Paul Slovic and Amos Tversky. Judgment under uncertainty: Heuristics and biases. Cambridge University Press. 
• Edwards, Ward (October 2008). Jie W. Weiss and David J. Weiss. ed. A Science of Decision Making:The Legacy of Ward Edwards. Oxford University Press. pp. 536. ISBN 9780195322989. 
• DeGroot, Morris, Optimal Statistical Decisions. Wiley Classics Library. 2004. (Originally published 1970.) ISBN 0-471-68029-X.
• Berger, James O (1985). Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics (Second ed.). Springer-Verlag. ISBN 0-387-96098-8. 
• Bernardo, José M. and Smith, Adrian F. M. (1994). Bayesian Theory. Wiley. 
• Robert, Christian P (1994). The Bayesian Choice (first ed.). Springer. 
• ET Jaynes' Probability Theory: The Logic of Science (2003)
• Colin Howson and Peter Urbach (2005). Scientific Reasoning: the Bayesian Approach (3rd ed.). Open Court Publishing Company. ISBN 978-0812695786. 
• Ian Hacking (1967) Slightly more realistic personal probability. Philosophy of Science 34: 311–325. (Reprinted in Gärdenfors, Peter and Sahlin, Nils-Eric. Decision, Probability, and Utility: Selected Readings. 1988. Cambridge University Press.)

[edit] See also

• Bayesian brain: the application of Bayesian theory to the functioning of the brain
• Bayesian experimental design
• Bayesian inference: Statistical inference and methods using Bayesian probability
• Bayesian network: Bayesian reasoning for multiple variables in the presence of conditional independencies
• Bertrand's paradox: a paradox in classical probability, solved by Bayesian methods
• Expected utility
• De Finetti's game: a procedure for evaluating someone's subjective probability
• Empirical Bayes method
• Fiducial inference: Fisher's attempt to produce probability-distributions on the parameter space without using a prior.
• Frequency probability: the main alternative to the Bayesian view
• Inference
• Likelihood function
• Maximum entropy thermodynamics: a Bayesian view of thermodynamics due to Edwin T. Jaynes
• Predictive inference
• Probability interpretations
• Uncertainty

[edit] Footnotes

[edit] External links

• A tutorial on Bayesian probabilities
• A. Hajek and S. Hartmann: Bayesian Epistemology (review article)
• On-line textbook: Information Theory, Inference, and Learning Algorithms, by David MacKay, has many chapters on Bayesian methods, including introductory examples; arguments in favour of Bayesian methods (in the style of Edwin Jaynes); state-of-the-art Monte Carlo methods, message-passing methods, and variational methods; and examples illustrating the intimate connections between Bayesian inference and data compression.
• An Intuitive Explanation of Bayesian Reasoning A very gentle introduction by Eliezer Yudkowsky
• An on-line introductory tutorial to Bayesian probability from Queen Mary University of London
• Jaynes, E.T. (1998) Probability Theory : The Logic of Science.
• Bretthorst, G. Larry, 1988, Bayesian Spectrum Analysis and Parameter Estimation in Lecture Notes in Statistics, 48, Springer-Verlag, New York, New York;
• Jeff Miller "Earliest Known Uses of Some of the Words of Mathematics (B)"
• James Franklin The Science of Conjecture: Evidence and Probability Before Pascal, history from a Bayesian point of view.
• Is the portrait of Thomas Bayes authentic? Who Is this gentleman? When and where was he born? The IMS Bulletin, Vol. 17 (1988), No. 3, pp. 276–278