If one has a theory whose predictions are insufficiently accurate to be acceptable, then it is necessary to change the theory. 数据挖掘实验室
For human beings it is much easier to elaborate the theory, or otherwise tinker with it, than to undertake a more radical shift (for example, by scrapping the theory and starting again). This elaboration may take many forms, including: adding extra variables or parameters; adding special cases; putting in terms to represent random noise; complicating the model with extra equations or rules; adding meta-rules or models; or using more complicated functions. In Machine Learning terms this might be characterized as a preference for depth-first search over breadth-first search. 数据挖掘研究院
Regardless of the reasons for elaboration, we are well aware of this tendency in our fellows and make use of this knowledge. In particular we know to distrust a theory (or a story) that shows signs of elaboration - for such elaboration is evidence that the theory might have needed such elaboration because it had a poor record with respect to the evidence.
This knowledge, along with an understandable preference for theories that are easily constructible, comprehensible, testable, and communicable provide strong reasons for choosing the simplest adequate theory presented to us. 数据挖掘研究院
In addition to this preference for choosing simpler theories, we also have a bias towards simpler theories in their construction, in that we tend to start our search with something fairly simple and work 'outwards' from this point. 数据挖掘研究院
In the language of economics we are satisfiers rather than optimizers. This means that it is almost certain that we will be satisfied with a theory that is simpler than the best theory (if one such exists, alternatively a better theory). This tendency to, on average and in the long term, work from the simpler to the less simple is a straightforward consequence of the fact that there is a lower bound on the simplicity of our constructions. This lower bound might be represented by single constants in algebra; the empty set in set theory; or a basic non-compound proposition expressed in natural language.
Biological evolution started from relatively simple organisms and evolved from there. 数据挖掘研究院
Thus the effective lower bound on complexity means that there is a passive drift towards greater complexity (as opposed to an active drive towards complexity, a distinction made clear by McShea, 1996).
There have been a number of a priori the probabilities are established. arguments aimed at justifying a bias towards simplicity - (Kemeny 1953) and (Li, M. and Vitányi, 1992) are two such. The former makes an argument on the presumption that there is an expanding sequence of hypotheses sets of increasing complexity and a completely correct hypothesis - so that once one has reached the set of hypotheses that contains the correct one it is not necessary to search for more complex hypotheses. However this does not show that this is likely to be a better or more efficient search method than starting with complex hypotheses and working from there. The later shows that it is possible to code hypotheses so that the shorter codes correspond to the more probable ones, but in this case there is no necessary relation between the complexity of the hypotheses and the length of the codes that is evident before
Theoretical results in Machine learning (Schaffer 1994, Wolpert 1996) show that, in general, no learning or search algorithm is better than another. In particular that if a bias towards simplicity is sometimes effective, there must be other domains in which it is counter-productive.
It may be that nature is biased towards producing data that is more amenable and, in particular, simple than these extreme cases. Since the tendency towards elaboration is a characteristic of human theory construction, we look to situations where theory construction is not biased towards elaboration to see if simplicity is truth-indicative there. Recently there have been such studies in the field of Machine Learning - where a computer program (rather than a human) attempts the induction. This gives one a test bed, for one can design the induction algorithm to use a simplicity bias or otherwise and compare the results. In one of these studies (Murphy and Pazzani 1994) a comprehensive evaluation of all as to their effectiveness at fitting some initial portion of the data (the in-sample part of the series), secondly as to their success predicting the continuation of this data (the out-of-sample part), and finally, as to the theory's complexity (measured in this case by the size or depth of the formal expression representing the theory). The theories with best success at fitting the in-sample data were selected. Within this set of 'best' theories it was examined whether the simpler theories predicted the out-of-sample data better than the more complex theories. In some cases the simpler hypotheses were not the best predictors of the out-of-sample data. This is evidence that on real world data series and formal models simplicity is not necessarily truth-indicative.possible theories in a given formal language (to a given depth) were analyzed against some real-world data series as follows: firstly 数据挖掘研究院
In a following study on artificial data generated by an ideal fixed 'answer', (Murphy 1995), it was found that a simplicity bias was useful, but only when the 'answer' was also simple. If the answer was complex a bias towards complexity aided the search. Webb (1996) exhibited an algorithm which systematically extended decision trees so that they gave the same error rate on the in-sample data, and, on average, gave smaller error rates on the out of- sample data for several real-life time series. This method was based upon a principle of similarity, which was used to restrict the set of considered hypotheses. A useful survey of results in Machine Learning, that can be seen as a parallel paper to this one is (Domingos 2000).
If someone asks you to guess the next number is the sequence: 2 ,4, 8, 16 you will correctly guess 32, because the nis the simplest pattern that describes these five numbers, and you can rely on the fact that the human will have chosen a simple (albeit possibly obscure) rule for their construction. It would not be sensible to guess the number 31, despite the fact that there is a rule that would make this the correct answer (the number of areas that n straight lines, each crossing the perimeter of a circle twice and such that no three lines intersect in a single point, cut that circle into).th power of two 数据挖掘研究院
The simplicity of these kinds of phenomena is only a hallmark of deliberate, conscious human construction. Products of our unconscious brain or social constructs such as language may be extremely complex for these were not the product of an intentional design process. Thus artists may construct extremely complex artifacts because they do not design every detail of their work but work intuitively a lot of the time with parts and media that are already rich in complexity and meaning.
In order to justify the selection of theories on the basis of simplicity, philosophers have produced many accounts of what simplicity isinformativeness - so that instead of asking whether simplicity is informative he seeks to show that simplicity (as informativeness w.r.t. a specified question) is, in fact, simple.. These have included almost every possible non-evidential advantage a theory might have, including: number of parameters (Draper 1981), extensional plurality (Goodman 1966, Kemeny 1953), falsifiability (Popper 1968), likelihood (Rosenkranz, 1976 Quine 1968), stability (Turney, P 1990), logical expressive power (Osherton and Weinstein 1990) and content (Good 1969). In some cases this has almost come full circle. Sober (1975) characterizes simplicity as
If, as I have argued, simplicity is not advantages that are not directly linked to its success at explaining or predicting the evidence, restoring the correct labels for these advantages will help (rather than hinder) their elucidation.truth-indicative, this whole enterprise can be abandoned and the misleading label of 'simplicity' removed from these other properties. This mislabeling, far from producing insight has produced a fog of differing 'simplicities' and 'complexities' which do much to hinder our understanding of the modeling process. Theories can posses a lot of different
It should be clear from the above that, if I am right, model selection 'for the sake of simplicity' is either: simply laziness; is really due to pragmatic reasons such as cost or the limitations of the modeler; or is really a relabeling of more sound reasons due to special circumstances or limited data. Thus appeals to it should be recognized as either spurious, dishonest or unclear and hence be abandoned.
However, there is a form of Occam's Razor which represents sound advice as well as perhaps being closer to its Occam's original formulation (usually rendered as "entities should not be multiplied beyond necessity"works.), namely: that the elaboration of theory in order to fit a known set of data should be resisted, i.e. that the lack of success of a theory should lead to a more thorough and deeper analysis than we are usually inclined to perform. It is notable that this is a hallmark of genius and perhaps the reason for the success of genius - be strict about theory selection and don't stop looking until it really 数据挖掘研究院
Abridged from: 数据挖掘研究院
Simplicity is not Truth-Indicative 数据挖掘研究院
Bruce Edmonds 数据挖掘研究院
Centre for Policy Modeling 数据挖掘研究院
Manchester Metropolitan University 数据挖掘研究院
By: Moe Hey. 数据挖掘研究院
Image is a painting by Mark Rothko. 数据挖掘研究院
