Evaluating experiments check-list

• 1. What is the hypothesis?

Score for clarity of hypothesis (0-2):

• 2. Is the hypothesis directional?

Score (no: 0, yes: 1):

• 3. Is the study employing Strong Inference?

Score (no: 0, yes: 1):

• 4. What are the two conditions?

Score for clarity (0-2):

• 5. What is intended to vary between the conditions?

Score for clarity of the intended independent variable (0-2):

• 6. What else is varying between conditions?

• 7. Which if any of these are potentially a confounding variable, given the hypothesis?

Score for level of control of confounding variables (0-2):

• 8. How many people per condition?

• 9. What is measured?

• 10. How is this scored?

• 11. Have they controlled for pre-existing differences? (Detection, matching, large-group randomization)

Score for level of control of pre-existing differences (0-2):

• 12. Is there a substantial issue with attrition?

Score for attrition (0 = serious issues, 2 = no issues, 0-2) : 2

• 13. Have they controlled for participant effects? (e.g. by blinding)

Control for participant effects (0-2):

• 14. Have they controlled for experimenter effects? (e.g. blind scoring, pre-registration)

Score for experimenter effects (0-2):

TOTAL SCORE: (out of 18).