Understanding Study Design: Case Study on Intro Stats Students' Conceptual Change, Study notes of Statistics

Download Understanding Study Design: Case Study on Intro Stats Students' Conceptual Change and more Study notes Statistics in PDF only on Docsity!

Random is Random: Helping

Students Distinguish Between

Random Sampling and

Random Assignment

Robert delMas

Elizabeth Brondos Fry

Department of Educational Psychology

Background and Motivation

According to statistics education recommendations (e.g.,

GAISE, 2016), students should understand the following

about the role of randomness in study design:

• Random sampling tends to produce representative

samples, allowing for generalization to a

population.

• Random assignment tends to balance out

confounding variables between groups, helping to

enable cause-and-effect conclusions.

Research Question

A study design unit was created and

implemented to answer the research question:

How does introductory statistics students’ conceptual

understanding of study design and conclusions change

after participating in a learning unit designed to

promote conceptual change in these areas?

Course and Audience

• Undergraduate, 3-credit introductory statistics course that fulfills

general education mathematical thinking requirement at a large,

Research 1 university (using CATALST curriculum; Garfield et al.,

2012; Zieffler et al., 2015)

• Four sections: Three in-class and one online (30-45 students each)

• Taught by advanced graduate students in statistics education

• Engaged students in active learning and discovery, minimal lecture

• Study design unit lasted 2½ weeks, during second half of spring 2016

semester

Day 1: Sampling Methods and Unbiased Estimation

Activity: “Sampling Countries”

Students contrasted central tendency of average life expectancy from

convenience samples ( n = 20) with simple random samples ( n = 10) of

countries

Convenience samples, n = 20 Random samples, n = 10

Population mean

life expectancy

Day 2: Assignment to Experimental Groups and

Establishing Causation

Activity: “Strength Shoe” (modified from Zieffler et al., 2015)

Students simulated random assignment to two groups and

observed the distribution of group mean differences on several

potential confounding variables

“Known” confounder: Height

“Unknown” confounder: Gene

Survey Incentives: Part 1

• Students compare distribution of random samples to population
• Observe that when many samples are taken, sample means are

centered at population mean

Age distribution for one sim ple

random sam ple

Sam ple m ean age for 200 sim ple random

sam ples

Survey Incentives: Part 2

• Students compare groups within a random assignment
• Observe that a single random assignment produces similar groups

(but not identical) and group mean differences tend to balance out

across many random assignments.

Differences in m ean incom e for m any

random assignm ents

Single random assignm ent

Results: Overview

• IDEA changes in total score

• Items with high performance (pretest & posttest)

• Items with significant improvement

– Comparisons to prior studies

• Item distractors related to misunderstandings

IDEA Changes in Score

Mean Diff. SD t p Cohen’s d

Difference in total score

(22 items)

Difference in sampling

subscore (9 items)

Difference in assignment

subscore (13 items)

Results from paired t- tests of differences in IDEA score (posttest-

pretest) for n = 125 students

Items with Most Improvement

Item

Measured learning outcome:

Ability to understand…

Percent correct

( n = 125)

McNemar’s

test p Pretest Posttest

16 Correlation does not imply causation

18 Purpose of random assignment in an

experiment

3 What it means to make an appropriate

generalization to a population, using

sample data

6 Small random sample is preferable to a

larger sample gathered with a biased

sampling method.

After adjusting for multiple comparisons, αc =.

Comparisons to Prior Studies

• The two IDEA items that showed the most improvement were

slightly modified from items on the Comprehensive Assessment

of Outcomes in Statistics (CAOS; delMas et al., 2007).

• Performance on IDEA was compared with performance in

similar CAOS items from:

• A national sample of 13,432 undergraduate introductory

statistics students enrolled in U.S. universities (2005-2017)

• Two samples of introductory statistics students at a small

college (Tintle et al., 2012):

• Randomization-based curriculum ( n = 76)

• Consensus curriculum ( n = 78)

Measured learning outcome: Understanding the

purpose of random assignment in an experiment

Sample

Pretest %

correct

Posttest %

correct

National sample n = 13,

Tintle et al. (2012; n = 76)

Randomization-based curriculum

Tintle et al. (2012; n = 78)

Consensus curriculum

IDEA

Comparisons to Prior Studies

Other Items with Statistically

Significant Improvement

Item

Measured learning outcome

Ability to.…

Percent correct

( n = 125)

McNemar’s

test p Pretest

Post-

test

1 Identify population to which inferences can

be made, based on a sample

5 Understand when sample estimates may be

biased due to lack of a representative sample

21 Understand that assigning subjects to

treatments as they walk into a room does not

help balance out confounding variables

22 Recognize when a randomized experiment

should be used for a particular research

question