Are there replications of Ludmer et al. (2011)?

As of 10th October 2023, this paper had 156 citations on Google Scholar.

On these articles, at least 18 were not in English (example), at least 10 were not published scientific articles (beware preprint servers e.g. arxiv, bioarxiv, researchgate…), at least 3 were with a species other than humans, and at least one was a duplicate entry. In addition, at least 13 were reviews (example) or theory articles rather than original data. This still leaves a lot of articles.

Not on fMRI of the aha! effect

At least 53 of the remaining articles did not report fMRI data so could not be a replication, and at least 3 had fMRI data but were not on the aha! effect, so we’re down to a smaller number of articles. Below are some examples of articles that cite Ludmer et al. (2011), but aren’t fMRI studies of the aha! effect:

• Castelluccia et al. (2017, Network and Distributed System Security Symposium)

• Chang et al. (2012, Perceptual and Motor Skills)

• Chang et al. (2016, Neuroscience of Consciousness)

• Cui et al. (2021, Psychological Research)

• Danek et al. (2013, Psychological research)

• Danek & Wiley (2020, Cognition)

• Du et al. (2017, Journal of Creative Behaviour)

• Ishikawa & Mogi (2011, Cognitive Neurodynamics)

• Ishikawa et al. (2019, Frontiers in Psychology)

• Kizilirmak et al. (2015, Psychological Research)

• Korovkin et al. (2020, Journal of Cognitive Psychology)

• Lin et al. (2018, Frontiers in human neuroscience)

• Lu & Singer (2023, Cerebral Cortex)

• Luo et al. (2014, Human Brain Mapping)

• Oh et al. (2020, NeuroImage)

• Shen et al. (2013, Science China Life Sciences)

• Shen et al. (2016, British Journal of Psychology)

• van de Cruys et al. (2021, Cognition)

• Wu et al. (2019, NeuroImage)

• Xia et al. (2017, Neuroscience Letters)

• Yi et al. (2021, Psychophysiology)

Not a replication attempt

Just because a study shows amygdala activation, or studies the aha! phenomenon, that doesn’t necessarily mean it’s a replication of Ludmer et al. (2011); it also needs to observe the substantially same result as Ludmer, i.e. that the activation of the amygdala correlates with later memory of the stimulus in an aha! procedure.

Here are some examples of neuroscience studies citing Ludmer et al. (2011) that are not attempts to replicate Ludmer et al. (2011):

• Becker et al. (2019, NeuroImage) - Does not assess subsequent memory nor investigate amygdala activatons.

• Becker et al. (2020, Journal of Cognitive Psychology) - Does not assess memory, nor investigate association of amygdala activation to memory.

• Flounders et al. (2019, eLife) - Uses Mooney images, but does not test memory for them after a delay.

• Gonzalez-Garcia et al. (2018, eLife) - Uses Mooney images, but isn’t investigating the same hypotheses as Ludmer et al (correlation of amygdala activation to memory)

• Gonzalez-Garcia & He (2021, Journal of Neuroscience) - Uses Mooney images, but does not measure retention after a delay.

• Huang et al. (2015, NeuroImage) - Does not assess relation between memory and amygdala activity.

• Haung et al. (2018, NeuroImage) - Does not measure retention after a delay.

• Kozunov et al. (2020, NeuroImage) - Does not report activity of amygdala, does not assess correlation of memory with brain activity.

• Shen et al. (2018, Appl. Psychophysiol. Biofeedback) - Does not report amygdala activity, does not assess memory.

• Yu et al. (2019, Cognitive Affective & Behavioral Neuroscience) - Did not test memory

• Yu et al. (2021, Psychophysiology) - Did not test memory after a delay.

• Wu et al. (2019, NeuroImage) - does not test memory.

Not a direct replication

There are a couple of studies that come pretty close to replicating Ludmer et al.:

• Kizilrmak et al. (2016, Frontiers in Psychology) - Table 4 shows something much like the Ludmer et al. (2011) result. It’s not a direct replication, though, because it uses a different insight task (remote associates task) and a different retention interval (24 hours instead of one week).

• Kizilirmak et al. (2019, Consciousness & Cognition) - For the current purposes, this is largely the same as Kizilirmak et al., (2016). There is some evidence for a similar effect to Kizilirmak et al. (2016), but the evidence is somewhat weaker this time (see p.121).

• Chen et al. (2021, Acta Psychologica) - A bit like Ludmer, but it’s a different task (Chinese riddles), and a different delay (30 minutes). Also, although amygdala is mentioned, they seem to place more emphasis on the hippocampus.

Conclusion

Overall, 143 other articles cite Ludmer et al. (2011), but none of them are direct replications of that study. This supports previous analysis, which shows that published attempts to directly replicate a study are rare in psychology Makel et al., 2014. A direct replication is where you do the same thing and get the same result. The closest the field seems to have come in this case are a couple of studies that show a conceptually similar effect with a different insight task and a different (much shorter) retention interval. There is also the issue that, in the more recent study, the evidence for the result is somewhat weak.

So, the answer to the question is not straight forward, as is often the case! We can say that there is not direct replication, hence the phrasing of the poll in class. We might, however, argue that there is at least one conceptual replication i.e. a result that supports the same general conclusion, but over a shorter time period and with different materials. The latter is arguably a good thing, i.e. it’s good that we can show this effect has some generality. The former is more troublesome - for example, in the context of study skills advice, a retention interval of 24 hours is rather short.