There are many major issues that contribute:
- Pressure to publish in academia, regardless of quality. Perverse motives.
- Peer review is a poor filter for fraud and may not have the effect on credibility it’s meant to.
- No motivation to reproduce some else’s research. Hard to be convinced on that usage of grant money.
- Journals not as likely to publish negative findings (null results), as they’re perceived to have less value. They also don’t typically care about reproduction of previous findings, which contributes to a lack of motivation for researchers to replicate them.
- Prestige and ego cause people to make poor choices.
- Self-admitted questionable research practices: https://twitter.com/cremieuxrecueil/status/1652469608442482689
- Journals could be considered scientific gatekeeping…normalizing open access could help?
- Honestly obtained incorrect results could lead to publication of outliers as they are surprising.
- Poor experimental design…suggestive wording in surveys, unreliable sources for sampling, overfitting, confounding issues, etc. Unfortunately, many studies require deep domain knowledge to understand why it’s badly designed.
- p-hacking or hypothesis changing after the fact.
- View that research is a means of influencing policy and public opinion rather than seeking truth.
- Can’t always publish the data used for outside verification (healthcare).
Some fields like particle physics and astronomy actually do focus on reproducibility.
All boils down to a question of incentives?
How do we spot issues proactively?
Further Reading
- https://examine.com/guides/how-to-read-a-study/
- Eleven strategies for making reproducible research and open science training the norm at research institutions (via https://news.ycombinator.com/item?id=38404261)
- Beware the Man of Many Studies (via https://news.ycombinator.com/item?id=36215222)
- https://scienceintegritydigest.com/about/ by Elisabeth Bik the science integrity consultant looking at biomedical research.