Critical Thinking and Statistics Statistics is an approach to research in which case data collected from a sample is used to draw conclusions about the population. To this end, statistics provides a crucial basis for decision-making, in addition to understanding population complexities. However, in the light of all their usefulness, statistics have one major flaw that cannot be overlooked -- since their creation is done by human beings and the counting and analyses therein based solely on human definitions, statistics can neither be regarded as magical, nor can they be taken as being always true. In this regard, statistics can mislead a reader as much as they can educate, and can lie just as much as they can depict the truth. The key to discerning good and bad statistics lies in the evaluation approach chosen; naivety and cynicism make people lose a great deal, but a critical mind-set leads a person to effectively evaluate any statistics presented to them and draw comprehensive conclusions.

One may ask; what then constitutes a critical mind-set? This text seeks to provide answers to this particular question, but before embarking on the main discussion, it would be prudent to briefly discuss the two other approaches to statistical thinking; naivety and cynicism (Best 15).

The Naive Mindset: this makes the presumption "that statistics are generally accurate, that they mean what they seem to mean" (Best 15). This mindset fully trusts any statistics presented to it, and neither questions why the figures are the way they are, nor wonders how the producers' interests could have come to play (Best 16).

The Cynical Mindset: this views statistics as nothing but manipulative efforts. In this regard, this mindset does not trust numbers, and neither does it, the people who produce them (Best 16). Cynical mindsets are always "suspicious of statistics; they are convinced that numbers are probably flawed, and that those flaws are probably intentional," and hence statistics can simply not be used to prove anything (Best 16).

The Critical Mindset: this is not hostile or negative, because it understands that even the best of statistics cannot be perfect (Best 17). All a critical mindset does is "evaluate numbers to distinguish between good statistics and bad statistics" (Best 17). This mind-set is, unlike the naive and cynic ones, statistically literate (Schield 1).

A critical mindset evaluates statistics presented to it on the basis of;

i) The study type used

ii) The sample used for the study

iii) The measurements derived

iv) The graphs generated from the gathered data

v) The claims and probability statements made from the data

vi) The study limitations

vii) The quantity of information provided to the end-consumer

(Source: Ganio 6)

Having recognized that no single statistic is perfect, a critical thinker is often more concerned about "whether a particular statistic's flaws are severe enough to damage its usefulness" (Best 17). Some of the questions asked would include; how broad is the definition; does it encompass an overflow of false positives, or leave out significant false negatives? How significant would the statistical change be, if the definition was changed? How does the sample or measurement choice influence the statistic? How would the statistic change if a different sample was used? Has the statistic been correctly interpreted, or is it more of a mutant statistic? How appropriate are the comparisons?

The Process of Critical Thinking

i) Fundamental Distinctions

Critical thinking begins with making three fundamental distinctions about presented statistics; causation vs. association, population vs. sample, and a test's quality vs. its predictive power (Schield 2).

Association vs. Causation: distinguishing causative statements from associative statements is a crucial step in critical thinking because it gives a feel of how strongly the evidence given supports the statistic or claim (Schield 2). A recent magazine article, for instance, claimed that violent TV programs have contributed to the rising rates of antisocial behavior and violent crime. The article expresses that;

The more violent TV entertainment a teenager is exposed to, the higher...

Critical thinking would, however, lead one to realize that the difference between the two is also the difference between causation and association, and that therefore, in as much as claim 2 provides evidence for 3's truth, its truth "is not sufficient to prove the truth of #3" (Schield 3).
Population vs. Sample: distinguishing between a population parameter and a sample parameter is an important component of critical thinking. Failure to make this distinction gives rise to the presumption that "a statistic obtained from a sample is actually a property of the entire population" (Schield 2). Taking, for instance, the claim that 70% of New York adults support the reduction of violent TV programs; unless the results were obtained from a census, this statistic is a result of a sample survey, and hence the terms 'New York adults' do not represent the adult population in New York, in its entirety (Schield 2).

The question that arises here has to do with the sample's degree of representativeness; was the sample collected from a neighborhood mostly inhabited by young adults? Was the sample reflective of New York's population in terms of size and composition?

A Test's Quality vs. Its Predictive Power: another fundamental component of critical thinking involves distinguishing between a test's quality, and its predictive power (Schield 3). Whereas quality "is measured on subjects whose disease status is known prior to the test; the predictive power of a test is measured on subjects whose disease status is unknown prior to the test" (Schield 3). Lack of statistical literacy leads readers to presume that a test's quality automatically reflects its predictive power, which may not always be the case.

ii) Statistic Interpretation

Interpretation has a lot to do with questioning. Having made the three distinctions above, and using the same as a guide, a critical thinker embarks on questioning the truth and representativeness of the statistics.

Is the statistic true?

In some cases the source of doubt is a simple error, but in others, statistics are intentionally "presented in a misleading function" so as to pass a judgment that satisfies the producer's interest (Schield 3). A critical thinker would be quick to notice any such deviations, given that statistics for consecutive years would, in most cases, not display very significant deviations (Schield 3). Establishing the truth entails making comparisons, and establishing the degree of correlation between the evidence given and the conclusions reached. One significant example is the case of the 1994 Statistical Abstract, which erroneously reported a birth rate of 189.5/1000, rather than 89.5/1000 among unmarried Blacks in 1991; more than double the rates reported in 1990 and 1992 (Schield 3). Without criticality, such an error would never have been spotted and corrected.

Is the Statistic Representative?

Sometimes "a true statistic has been selected just because it supports a particular claim -- not because it is representative" (Schield 3). Taking, for instance, a situation where a researcher seeks to reinforce the claim that the number of road accidents and deaths would rise if highway speed limits were raised; if they gathered data only from those states that display this trend, and ignored those that did not, they would end up with a statistic that is true but not representative (Schield 3). In such a case, the sample statistic, though true, would be nowhere near the population parameter (Schield 3).

Is the Statistic Factual, or is it Inferential

Inferential statistics include explanations, generalizations, and predictions, and hence, their truth-values are highly disputable (Schield 3). On the other hand, the truth-value of factual statistics is not often disputable. This is expected given that a statistic derived from the study of a sample would be factual to the elements of the sample, but inferential for the population (Schield 3).

These…