Decoding Academia #2: False Positive Psychology *Video*
Added 2021-12-27 10:53:28 +0000 UTCNot had enough content with the Christmas Special?
Well, here is the second instalment of the Decoding Academia series. This episode is on a classic 2011 paper on 'False Positive Psychology'. This was one of the harbingers of the replication crisis and shows how important the concept of 'Researchers Degrees of Freedom' are for critically evaluating research.
The discussion is a little bit technical at times but I hope some of you find it useful.
Interested in feedback as always!
Merry Christmas & Happy New Year again!
- Copy of the article discussed attached to this post
Comments
No need to apologise for commenting too much! We love the feedback. And yeah we kinda skipped over the advice to reviewers section but it was good. In summary it encouraged reviewers to be accepting of nulls, messy results, and still appropriately critical.
Christopher Kavanagh
2021-12-30 11:11:04 +0000 UTCYeah they wanted the age reversal I think. The other measures were just covariates to give multiple rolls of the dice. But they could have rewritten if they had hit with one of the other outcomes.
Christopher Kavanagh
2021-12-30 11:09:36 +0000 UTCThe most recent XKCD is relevant. 🙂 https://xkcd.com/2560/
Jesse Hodges
2021-12-29 03:03:09 +0000 UTCAt the risk of commenting too much myself, I think it would have been more interesting if listening to music had caused the fathers' ages to decrease. And I wonder if a single different answer to the square root of 100 would have been statistically significant? It would have been really, really interesting if listening to music caused a change in the square root of 100. But, I guess if you're doing studies like these, you have to take what pops out.
Kat
2021-12-28 23:21:17 +0000 UTCAt the risk of commenting too much let me say this paper and discussion helped me clarify some issues we deal with at the academic journal I work for. The field covered by my publication is a fairly new one, and it's been my experience that editors in it have not thus far accrued the power they have in other, especially more scientific, fields. The result being that when we critique a submission, the scholars are often shocked that we are assessing their work rigorously, as though publishing the best possible content on their topic were not in our shared interest! Speaking of which, I hope you will talk a bit more in future about responses to academic papers; I assume you mean for peer reviewers, but such a discussion will surely be useful for any number of people. Thanks!
Sian Gibby
2021-12-28 18:06:05 +0000 UTCIf you have a big enough sample and do random allocation you should get a relatively even average age distribution. Of course, they don't have a big enough sample and they are tilting the scales with the analyses until the test provides a 'significant' result. Their example is intentionally impossible but the same kind of analysis is applied all the time. But you are correct you could reverse the causality as you describe but presuming they didn't take age until after allocation that could be taken as evidence of psychic ability too!
Christopher Kavanagh
2021-12-28 02:32:35 +0000 UTCGood, there's evidence of psychic ability in the scientific literature. Don't tell the gurus.
Kat
2021-12-28 02:28:23 +0000 UTCI don't quite get it. If you want to find out whether listening to certain music affects how old people actually are, don't you need to find out their age before as well as after they listen to the music? It's unclear that this was done. Perhaps there's a correlation, but there's absolutely no evidence of the causation, as implied by the abstract. You might just as well posit people's age caused which music the investigators chose for them to listen to. Edit: Oh, I get it. They used the father's age. That makes it OK then, I guess. But they could have just used the participant's age, it's not like they wouldn't have had one before listening to the music. Or maybe they could have just had them listen to the Beatles for a long, long time? And I think the mother's age would have been better anyway.
Kat
2021-12-28 01:59:56 +0000 UTCGlad you liked it Lucy!
Christopher Kavanagh
2021-12-27 16:32:38 +0000 UTCWow, what a super show! I learned so much, but I need to listen several times because there is so much in here. Thanks!
Lucy
2021-12-27 15:14:09 +0000 UTCI am so excited about this series; thank you for doing it. Also thank you for making the paper available. Have you read Michael Lewis's book about David Kahneman and Amos Tversky's work, The Undoing Project? If you have a chance, I hope you will check it out, if not. Here is the description: "Forty years ago, Israeli psychologists Daniel Kahneman and Amos Tversky wrote a series of breathtakingly original papers that invented the field of behavioral economics. One of the greatest partnerships in the history of science, Kahneman and Tversky’s extraordinary friendship incited a revolution in Big Data studies, advanced evidence-based medicine, led to a new approach to government regulation, and made much of Michael Lewis’s own work possible. In The Undoing Project, Lewis shows how their Nobel Prize–winning theory of the mind altered our perception of reality."
Sian Gibby
2021-12-27 14:06:32 +0000 UTC
