Statistics research report and methodology

Statistical Significance in Published Scientific Works

The study I chose to examine is from the field of psychology: Todd, Hanko, Galinsky, and Mussweiler (2011), "When Focusing on Differences Leads to Similar Perspectives. This study was recently published in Psychological Science, a high-impact journal in the field. The research has to do with perspective-taking in conversation; the authors hypothesize that people are better at taking another person's perspective if they are in a "difference mindset" -- i.e. are more aware of interpersonal differences than usual. The authors propose to induce this "difference mindset" by a variety of means, across five experiments. Below, I will discuss the use of statistics in their study and how well they conveyed the size and significance of the results.

Experimental Design

All five experiments used a similar design: participants were randomly assigned to a similarity-mindset, difference-mindset, or control condition. Experiment 1 tested perceptual perspective differences, Experiment 2 tested the impact of the difference mindset on the use of privileged knowledge in interpreting communication, Experiment 3 tested whether the difference mindset would improve reasoning about false beliefs, Experiment 4 tested the impact of the difference mindset on reasoning about racial prejudice-based false beliefs, and Experiment 5 tested whether the difference mindset would improve or hinder interpersonal coordination in a collaborative task.

Statistical Methods

For all five experiments, the authors reported t-tests of the key comparisons followed by One-way ANOVA results. They also reported values for partial Eta squared, a measure of effect size that is analogous to r2 (correlation). For some experiments, they also report d, or probabilistic hit rate, for items in which subjects' accuracy was measured. Values for t, F, and d from Experiments 1-5 can be seen in the table below.

Similarity

Difference

Control

X1: t (78), from Diff.

2.60 (p = 0.01)

Comparison group

2.10 (p = 0.04)

X1: F (2, 78)

Main effect: 3.80 (p = 0.03)

X1: d

0.59

Comparison group

0.48

X2: t (97)

3.30 (p = 0.001)

Comparison group

2.15 (p = 0.3)

X2: F (2, 97)

Main effect: 5.55 (p < 0.01)

X2: d

0.67

Comparison group

0.44

X3: t (39)

3.05 (p < 0.01)

Comparison group

2.27 (p = 0.03)

X3: F (1, 39)

Interaction (Mind-set x Knowledge base): 7.28 (p = 0.01)

X3: d

0.98

Comparison group

0.74

X4: t (56)

2.59 (p = 0.01)

Comparison group

2.20 (p = 0.03)

X4: F (1,56)

Interaction (Protagonist group x Knowledge base): 5.92 (p = 0.02)

X4: d

0.71

Comparison group

0.59

X5: M (SD)

70.86 (25.21)

55.73 (16.59)

N/A

X5: F (1, 22)

4.61 (p = 0.04)

All reported comparisons with the Difference Mindset groups turned out to be significant. This is interesting because of the diversity of tasks used to measure the contrast between difference-awareness and similarity-awareness. However, the article draws heavily on previous research and is well-grounded theoretically, so the fact that all five experiments showed the effect the authors were aiming for is unsurprising.

Effect Sizes

The authors' choice to report effect sizes strikes me as very honest, although this statistic is more frequently being required by reviewers in the social sciences. The partial Eta2 values for the key comparisons in each experiment varied within a range of 0.04-0.17, with higher values generally obtained for main effects than for interactions. In the social sciences, values for partial Eta2 are generally considered to be minimal at 0.20, medium-sized at 0.50, and large effects at 0.80. It is therefore uncertain whether the effects Todd et al. found can be thought of as statistically valid based on the below-minimal value of all experimental effects.

Study Conclusions