Evaluating statistical claims: Observational studies and experiments
Explanation
Evaluating statistical claims requires understanding how data was collected. The method of data collection dictates what type of conclusion or claim can be validly drawn about a population or relationship.
Observational Studies vs. Experiments
Observational Study: Researchers observe and record data without assigning treatments or controlling conditions.
Purpose: To find correlations or associations between variables.
Limitation: Cannot prove cause-and-effect because underlying hidden variables (confounding variables) may influence results.
Example: Surveying people on their coffee intake and tracking their heart health can show an association between coffee and heart health, but it cannot prove coffee causes better health.
Controlled Experiment: Researchers randomly assign subjects to different treatment groups to test the effect of a specific intervention.
Purpose: To establish cause-and-effect (causation).
Key Element: Random assignment ensures that the treatment and control groups are similar in every way except for the treatment being tested.
Example: Randomly assigning 100 participants to either take a new medication or a placebo (sugar pill) to determine if the medication directly causes a reduction in blood pressure.
Random Sampling vs. Random Assignment
The validity of a statistical claim depends on two distinct types of randomness:
Random Sampling (Selection): Choosing participants at random from a population.
Result: Allows you to generalize findings to the broader population.
Random Assignment: Randomly placing chosen participants into treatment vs. control groups.
Result: Allows you to claim causation (cause-and-effect).
Example 1
A sample of 45 sixth-grade students was selected at random from a certain middle school. The 45 students completed a survey about the school's recess schedule, and 36 said they were satisfied with it. Which of the following is the largest population to which the results of the survey can be applied?
A
The 45 students who were surveyed
B
All sixth-grade students at the school
C
All students at the school
D
All sixth-grade students in the district in which the school is located
Choice B is correct. Selecting a sample at random allows the results to be applied to the population from which the sample was drawn, but not beyond it. Here, the sample was drawn from all sixth-grade students at the school, so the results can be applied to all sixth-grade students at that school. Choice A is incorrect; the results can be applied to the 45 surveyed students, but this is not the largest group to which they apply. Choice C is incorrect; the sample was drawn only from sixth-graders, not from students in other grades at the school, so the results cannot be extended to all students at the school. Choice D is incorrect; the sample was drawn only from one school, so the results cannot be extended to sixth-graders elsewhere in the district.
Example 2
A talk-radio host invited listeners to call in and respond to an on-air poll asking, "Do you support the proposed city parking tax?" By the end of the show, the host reported that 64% of callers said "No." Which of the following best explains why these results are unlikely to represent the sentiments of all city residents?
A
The percentages reported may not add up to exactly 100%
B
The listeners who called in were not a random sample of all city residents
C
There were more "No" responses than "Yes" responses
D
The show did not run long enough for every resident to call in
Choice B is correct. For a sample's results to represent a population, the sample must be representative of that population, which random selection is far more likely to achieve than a voluntary-response method. Here, only listeners who chose to tune in and chose to call the show were included, so the respondents are self-selected rather than randomly chosen, and are unlikely to represent all city residents. Choice A is incorrect; this describes a rounding or reporting issue unrelated to whether the sample represents the population. Choice C is incorrect; the relative sizes of the "Yes" and "No" groups do not by themselves indicate bias. Choice D is incorrect; the issue is not the amount of time available to respond, but that only self-selected callers were included at all.
Tips for solving
Match the study design to the correct type of claim (association vs. causation).
Example: A researcher tracks 500 adults for 5 years and notes that those who exercise daily report lower stress levels.
Study Type
Valid Conclusion
Invalid Conclusion
Observational Study (No treatment assigned)
Daily exercise is associated with lower stress levels.
Daily exercise causes lower stress levels.
Determine generalizability and causation based on sampling and assignment methods.
Example: Evaluating conclusions based on how participants are chosen and grouped:
Random Sampling?
Random Assignment?
Valid Claims
Yes
Yes
Can claim causation AND generalize to the entire population.
No
Yes
Can claim causation, but ONLY for the specific group studied.
Yes
No
Can claim association/correlation generalized to the entire population.
No
No
Can claim association/correlation, but ONLY for the specific group studied.
Identify the placebo effect and control group roles in experiment design.
Example: Testing a new memory supplement with 200 volunteers.
Group
Treatment Received
Purpose of Group
Treatment Group
New memory pill
Tests the actual effect of the active pill.
Control Group
Placebo (inactive pill)
Acts as a baseline to account for psychological/external factors.
Practice questions
00:00
Question 1 · easy
A sample of 60 dog owners was selected at random from the client list of a single veterinary clinic. The owners were surveyed about their satisfaction with the clinic's new online appointment system, and 48 said they were satisfied. Which of the following is the largest population to which the results of the survey can be applied?
Question 2 · easy
A sample of 80 tenth-grade students was selected at random from a single high school. The students completed a survey about the cafeteria menu, and 52 said they wanted more vegetarian options. Which of the following is the largest population to which the results of the survey can be applied?
Question 3 · easy
A running club organizer selected 50 runners at random from all participants registered for a single city marathon. The runners were surveyed about their training habits, and 70% reported running at least 20 miles per week. Which of the following is the largest population to which the results of the survey can be applied?
Question 4 · easy
A company with offices in several cities randomly selected 120 employees from its downtown branch office to survey about commute times. Which of the following is the largest population to which the survey results can be applied?
Question 5 · easy
An ornithologist captured and studied 35 birds selected at random from a single wetland reserve. The birds were weighed, and the mean weight was found to be 42 grams. Which of the following is the largest population to which this result can be applied?
Question 6 · easy
A librarian selected 50 patrons at random from everyone who checked out a book at a single library branch last month, and 40 of them said they would use a new mobile app for renewals. Which of the following is the largest population to which the survey results can be applied?
Question 7 · easy
Just before a talent show, the host asked audience members to text in a vote for their favorite act using their own phones if they wished to participate. Which of the following best describes why this voting method is not a random sample of the audience?
Question 8 · easy
A random sample of 90 visitors was selected from everyone who entered a botanical garden on a single Saturday, and 78% said they would recommend the garden to a friend. Which of the following is the largest population to which this result can be applied?
Question 9 · easy
After a checkout, a grocery store printed a receipt inviting customers to go online and complete an optional satisfaction survey if they chose to. Which of the following best explains why the results of this survey are unlikely to represent all customers of the store?
Question 10 · easy
A random sample of 65 members was selected from the roster of a single community gym, and members were asked how many days per week they exercise. Which of the following is the largest population to which the results can be applied?
