Instructor's Guide: Teaching Statistical Normality

Question 1: Type I Error Rate with Normal Data

Expected result: ~5% (should be close to nominal α = .05)

Good student answer: "The Type I error rate is approximately 5%, which matches our alpha level. This shows the test is working correctly when assumptions are met."

Question 2: Type I Error Rate with Skewed Data (n=20)

Expected result: 8-12% (inflated above 5%)

Good student answer: "With small sample size and skewed data, the Type I error rate increased to about 10%, which is double the intended rate. This means we're rejecting true null hypotheses too often."

Question 3: Type I Error Rate with Skewed Data (n=100)

Expected result: ~5-7% (much closer to nominal, showing robustness)

Good student answer: "With a larger sample size, even though the data is still skewed, the Type I error rate dropped back down to around 5-6%. This demonstrates that larger samples make the test more robust to violations of normality."

Question 4: Why does sample size matter?

Strong answer should mention:

• Central Limit Theorem (sampling distribution becomes normal)
• With small n, violation of normality affects the test directly
• With large n, the sampling distribution of the mean is approximately normal even if the data isn't

Example: "The Central Limit Theorem tells us that as sample size increases, the sampling distribution of the mean approaches normality regardless of the population distribution. With n=100, even our skewed data produces a nearly normal sampling distribution, so the t-test performs well. With n=20, we don't have enough data for the CLT to fully protect us."

Strong Field Guide Characteristics:

• Sketches: Simple drawings showing characteristic shapes
• Key words: Descriptive terms (symmetric, tail, peak, clusters)
• Q-Q patterns: Notes on whether points curve up, down, or show S-shapes
• Action items: What to do next (transform, investigate, proceed)

Example Strong Entry (Right-skewed):

Right-Skewed Distribution
Histogram: Peak on left, long tail stretching right →
Most values cluster low, few extreme high values
Q-Q plot: Points curve UPWARD on right side
Action: Try log transformation or sqrt
Common in: Reaction times, income data, counts
            

Questions 1-2: Recording Results

Expected pattern:

• Small sample (n=25): p-value likely > 0.05 (fails to detect deviation)
• Large sample (n=200): p-value likely < 0.05 (detects same deviation)

Key insight: Same distribution type, different conclusions!

Questions 2-3: Comparing Visuals

Question 2 (Histograms): Should look very similar (both slightly right-skewed)

Question 3 (Q-Q plots): Both should show similar patterns (slight deviation from line)

The paradox: They LOOK the same but get different p-values!

Question 4: Pattern as n Increases

Correct observation: p-value tends to decrease as sample size increases

Strong explanation: "Larger samples give the test more 'power' to detect even tiny deviations from perfect normality. With small samples, the test might miss problems. With large samples, it detects everything—even deviations too small to matter practically."

Advanced insight: Some students might note that this creates a dilemma: when you have enough data to reliably detect violations, you also have enough data that violations don't matter much!

Question 5: The Big Question - What Should You Do?

Ideal answer components:

• With large n: Visual inspection matters more than p-value
• If both visual and statistical tests agree: clear decision
• If they conflict: Depends on sample size and severity
• Remember: t-tests are robust with large samples
• When in doubt: Report both findings

Example strong answer:

"When I have a large sample (n > 50) and Shapiro-Wilk says p < .05 but the Q-Q plot looks only mildly skewed, I would proceed with the t-test because: (1) the t-test is robust to moderate non-normality with large samples due to the Central Limit Theorem, and (2) the Shapiro-Wilk test is overly sensitive with large samples, detecting trivial deviations that don't affect the validity of the test. I would note the mild skewness in my write-up but wouldn't transform unless the visual inspection showed severe problems."

Question 6: Robustness Demonstration

Expected result: Even with "failed" Shapiro-Wilk (p < .05), 95% CIs should still achieve ~95% coverage with n=100

Good answer: "Even though the Shapiro-Wilk test rejected normality, the confidence intervals still had approximately 95% coverage. This demonstrates that with large samples, the t-test is robust - it works correctly even when the formal assumption test says normality is violated."

Question 1: Why Transformations Work

Strong answer includes:

• Log transformation compresses large values more than small values
• This "pulls in" long right tails
• Makes multiplicative relationships additive
• Changes the scale while preserving order

Example: "A log transformation works on right-skewed data because it compresses large values proportionally more than small values. For instance, log(100) - log(10) = 1, but 100 - 10 = 90. This 'pulls in' the extreme right tail, making the distribution more symmetric."

Practice Dataset Results

Right-skewed dataset:

• Before transformation: Shapiro-Wilk p < .05, Q-Q shows upward curve
• After log transformation: Shapiro-Wilk p > .05, Q-Q points fall on line
• Conclusion: Log transformation successfully normalized the data

Left-skewed dataset:

• Before transformation: Shapiro-Wilk p < .05, Q-Q shows downward curve
• After reflection + sqrt transformation: Shapiro-Wilk p > .05, improved Q-Q
• Note: Some students may try squaring or exponential - these will make it WORSE (wrong direction)

Heavy-tailed dataset:

• Challenge: May not fully normalize with standard transformations
• Best approach: Investigate outliers first, might need robust methods
• Teaching point: Not all problems are solvable with transformations!

Question 2: When NOT to Transform

Key scenarios students should identify:

1. Bimodal distributions - transformation won't fix underlying two-group structure
2. Large samples with mild skew - robustness makes transformation unnecessary
3. Data with meaningful zeros - log(0) is undefined
4. When interpretability matters more than normality - original scale may be more meaningful

Example answer:

"You should NOT transform if: (1) you have a bimodal distribution because this suggests two distinct groups that should be analyzed separately, not squashed together, (2) you have a large sample (n > 100) with only mild skewness because the Central Limit Theorem makes the t-test robust anyway, or (3) your data has meaningful zeros (like reaction times or counts) and you'd lose interpretability with a log transformation."

Question 3: Interpretation Challenge

Scenario: Original data in milliseconds, mean = 450ms. After log transformation, mean = 6.1.

What students need to understand:

• 6.1 is the mean of LOG(reaction time), not reaction time itself
• To get back to original scale: exp(6.1) ≈ 445ms
• However, this is now the GEOMETRIC mean, not arithmetic mean
• Differences on log scale = ratios on original scale

Strong interpretation:

"The mean on the log scale is 6.1, which corresponds to a geometric mean of approximately 445ms on the original scale (exp(6.1) = 445). This is slightly lower than the arithmetic mean of 450ms because geometric means are pulled down by skewness. When we report results, we should either back-transform our estimates or clearly state we're working on the log scale."

Prompt: Provide students with 3 datasets (small n with clear skew, large n with mild skew, bimodal). Ask them to:

1. Create and interpret visual diagnostics
2. Run and interpret Shapiro-Wilk tests
3. Make and justify decisions about transformation
4. If transforming, show before/after comparison
5. Explain their reasoning using concepts from all 4 modules

Grading rubric elements:

• Visual diagnostics are correct and well-labeled (20%)
• Statistical test interpretation is accurate (20%)
• Decision-making shows nuanced understanding of sample size/severity (30%)
• If transformation applied, it's appropriate and verified (20%)
• Written explanation demonstrates conceptual understanding (10%)

Format: 50-minute exam, students analyze 2 datasets using RStudio

Dataset 1 (20 points): Small sample, clear violation

• Students must identify problem visually
• Run appropriate tests
• Choose and apply transformation
• Verify improvement

Dataset 2 (20 points): Large sample, mild violation

• Students must recognize robustness applies
• Justify NOT transforming
• Demonstrate understanding of large sample paradox

Short answer (10 points):

• "When does Shapiro-Wilk p < .05 NOT mean you should transform?"
• "Why are bimodal distributions different from skewed distributions?"

Format: Partners create a 5-minute "mini-lesson" teaching ONE concept to the class

Topics to assign:

• Why normality matters (using Module 1 simulation)
• How to read Q-Q plots (with examples)
• The large sample paradox
• When to use log transformations
• Why not to transform bimodal data

Assessment criteria:

• Accuracy of content
• Clear explanations with examples
• Effective use of visuals
• Answers peer questions correctly

Benefit: Best way to cement understanding is teaching others!

A student who truly "gets it" will:

• ✓ Look at visual diagnostics FIRST before running statistical tests
• ✓ Consider sample size when interpreting Shapiro-Wilk results
• ✓ Know when violations matter (small n, severe skew) vs. don't matter (large n, mild skew)
• ✓ Choose transformations based on distribution shape, not trial-and-error
• ✓ Recognize that bimodality signals investigation, not transformation
• ✓ Report findings honestly (including limitations)
• ✓ Make thoughtful decisions rather than blindly following rules

Common Instructor Concerns

"Won't this complexity confuse students who just want clear rules?"

Short-term: Maybe. Long-term: No. Students who learn nuanced thinking become better researchers. Those who learn rigid rules become frustrated when real data doesn't fit the rules (which it never does).

"This takes a lot of class time for one assumption..."

True. But normality is violated more often than any other assumption, and mishandling it invalidates many analyses. Time invested here prevents major errors later. Plus, the thinking skills transfer to other assumptions.

"What if students get different answers from what I expect?"

Good! If they can justify their decision using concepts from the modules, that's more valuable than getting the "right" answer. Statistical analysis isn't always black and white.