2 hours ago
8
A new set of studies out this month suggests that as many as half of all results published in reputable journals in the social sciences can’t be replicated by independent analysis. This is part of a long-running problem across many research fields – most visibly in the social sciences and psychology, though concerns have also been raised in areas of biomedical research.
The latest work is a seven-year project called Systematizing Confidence in Open Research and Evidence (Score), which has now published three studies looking at 3,900 social science papers. It found that newer papers, and those published in journals requiring extensive sharing of underlying data, were more likely to be reproduced. Separately, medical research faces its own constraints: differing patient caseloads and limited sample sizes mean that, in practice, it can resemble the social sciences more than laboratory physics. Clearly, policymakers should be cautious of any claims that don’t have a wide and robust base of evidence.
Language is key: reproducibility looks at whether results can be recreated from the same data and methods. Replication tests whether the finding holds for new data in different contexts. Science rarely produces exactly identical outcomes, and figuring out why is part of how knowledge accumulates. But increasingly, politicians have looked to turn uncertainty into denial and recast normal scientific uncertainty as evidence of failure. That is why a White House executive order in May 2025 emphasised the “reproducibility crisis” in science, essentially a Trumpian call for doubt and inaction.
Unfortunately, large-scale verification projects, such as those undertaken by Score, are few and far between. Most academic researchers would rather spend time on work that is more likely to enhance their careers. Score reanalysed existing data and, in separate work, replicated studies from scratch across more than 100 papers. Around 49% still failed to replicate the original result. This points to a deeper problem. Reanalysing data is relatively straightforward; carrying out an identical experiment is not. It is tricky to recreate experiments in social and medical research, where outcomes depend on complex human systems. AI may help in deciding what to test, but it can’t reduce the costs and time involved in duplicating a piece of research.
Not every failed replication signals a crisis. Some findings don’t matter much; replication studies can themselves be flawed. Results that don’t consistently replicate should be weighed against a wider evidence base when guiding policy. Treating non-replication as disqualification confuses uncertainty with ignorance. This risks paralysing decision-making where judgment matters most. Greater transparency makes outright fraud more difficult and allows errors to be identified. Major funders such as the UK Economic and Social Research Council already require this, and the approach should be universal.
Some are sanguine, arguing that research “ultimately autocorrects”. The long-term solution – shifting incentives so existing results are tested – would increase confidence. But this requires restructuring of research culture and funding. For now, it remains largely notional. These studies should strengthen the case for change and serve as a warning. Social science is a powerful tool for understanding the world – and that trust will be built by acknowledging uncertainty, not repudiating it.
