When a result becomes knowledge

A new study can open a field, correct a theory, or mislead. How that checking produces a claim one can rely on is easier to see when the first report and the later account diverge.

On 23 September 2011 the OPERA collaboration presented a result that, had it held, would have required a fundamental rethinking of part of modern physics. Neutrinos traveling from CERN to the Gran Sasso laboratory appeared to arrive sooner than the speed of light in a vacuum would allow. The collaboration opened the anomaly to public scrutiny. On 8 June 2012, CERN reported that four experiments at Gran Sasso, Borexino, ICARUS, LVD, and OPERA itself, had measured flight times consistent with the speed of light. CERN's update attributed the original anomaly to a faulty element of the experiment's fiber-optic timing system.

A media story could show, first, the overthrow of Einstein and, then, the embarrassment of the scientists. The research record shows something more interesting: the difference between a recorded result and a warranted claim about the world.

The instrument did register something. The question was whether that registration described the neutrinos or described the instrument.

The same difference accompanies less spectacular reports: a more effective way of teaching, a new material, a link between behavior and health, an algorithm that beats its rivals. Before asking whether a discovery will change the world, it is worth establishing what was actually discovered.

What the study actually showed

Imagine a fictional experiment on learning vocabulary. Participants who use a new application score better on a test than a comparison group. That is an interesting observation. The sentence "the application improves memory" already covers more than the measurement itself.

Were participants assigned to the groups at random? Did they spend the same amount of time studying? Was memory checked a month later, or was the test only recognition of words immediately after practice? Did the test require the skill the application actually trained? Did the effect appear in people of a different age as well?

Each answer sets a boundary on the conclusion. A better score on one task need not mean a general improvement in memory. Even a properly run experiment can support a narrower claim than its title or its press release suggests.

It helps to keep three layers apart. The first is the data: the recorded answers, signals, or observations. The second is the result of an analysis, obtained under stated assumptions. The third is the interpretation: an account of what that result says about the phenomenon under study. The arithmetic can be right, and the reading of it can still be too wide.

Knowledge matures in this way as well, when a conclusion becomes less spectacular and more precise.

What peer review contributes

Publication in a peer-reviewed journal is often treated as the crossing from conjecture to fact. Review is an important check. Other researchers can point out errors of reasoning, missing comparisons, unclear methods, and conclusions that go beyond the data. How far the check reaches depends on the journal, the field, and the materials the reviewers have.

A reviewer usually does not repeat the experiment. A reviewer need not reanalyze all of the raw data, or inspect every line of the code. Acceptance admits the paper to scientific discussion after a defined check. It is not a guarantee that every claim in the paper is true.

A preprint, a paper posted before a journal's formal review, is not wrong by definition. It does report the stage the text has reached. Its quality has to be judged from its contents, including the checking it has already had.

A reader therefore has two separate questions: where the study was published, and how strong the evidence in it is. The prestige of the venue does not answer the second.

A p-value does not measure whether a discovery is true

In many studies a p-value below 0.05 works as a symbolic pass. It is easy to read that figure as a less than 5 percent chance that the discovery is a mistake. That is not what it means.

Informally, as the American Statistical Association put it in its 2016 statement, a p-value is the probability, under a specified statistical model, that a statistical summary of the data would be equal to or more extreme than the value observed. Ronald L. Wasserstein and Nicole A. Lazar published that statement in The American Statistician (doi: 10.1080/00031305.2016.1154108). The statement says that a p-value does not measure the probability that the hypothesis under study is true, or the probability that chance alone produced the data. By itself it also does not say whether an effect is large or practically important.

In the fictional application, a very large number of participants can make a small difference detectable. That difference may still be too small to justify a change in teaching. A small experiment, by contrast, can fail to produce a decisive result even when a useful effect is real.

Significance therefore has to sit beside the size of the effect, the uncertainty around it, and a judgment of the study's design. The same statement warns that a conclusion does not become true on one side of a threshold and false on the other. A result with a p-value of 0.049 and a result with a p-value of 0.051 do not automatically belong to two different worlds, one of discovered truth and one of a demonstrated absence.

Whether others can get a similar result

Checking a study can mean different jobs. In the definitions adopted by the U.S. National Academies of Sciences, Engineering, and Medicine in their 2019 report, Reproducibility and Replicability in Science (doi: 10.17226/25303), reproducibility means obtaining consistent results from the same input data, computational steps, methods, code, and conditions of analysis. The report uses that term interchangeably with computational reproducibility. Replicability means obtaining consistent results across studies that answer the same scientific question, each of which has collected its own data. Fields do not all use the words this way. The report adopts this distinction because everyday usage treats the two as interchangeable.

The first check asks whether the analysis can be run again. The second asks whether the phenomenon appears again. Generalization is a further question: whether the result holds in other populations or other conditions. The report separates that question from the first two.

The distinction matters in practice. A program can correctly recompute, every time, the data from a miscalibrated sensor. The fictional application can help students in a laboratory and bring no similar benefit to children studying at home. Repeating a computation does not, by itself, show that the computation is about the right thing.

A failed replication also has to be interpreted. It may point to a weakness in the original result, an error in the new study, chance variation, or a real difference in the conditions. The National Academies report stresses that one successful replication does not guarantee that the original result was correct, and that a single failed replication does not establish that the original claim was false.

That is not a license to keep excusing a hypothesis. If a discrepancy is explained by the conditions, the explanation should say which conditions matter and how the proposal can be tested. Otherwise any outcome at all could be treated as consistent with a favored theory.

What the psychology replication project showed

In 2015 the Open Science Collaboration published replication attempts for 100 experimental and correlational studies from three psychology journals, in Science (doi: 10.1126/science.aac4716). In the original studies, 97 percent of the results were statistically significant, at a threshold of p below 0.05. In the replications, 36 percent were. The authors report that the mean effect size in the replications was half the magnitude of the mean effect size in the original studies.

Those figures matter, and they are easy to give a meaning they do not have. They do not show that "64 percent of psychology is false." The project covered a defined set of studies, not the discipline. The studies were drawn from 2008 volumes of Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition, and the replications did not include every paper those journals published that year. The authors judged replication with several indicators and stated that no single indicator describes it sufficiently.

What the project did show is that many of the findings it checked received weaker support than the original publications would have led a reader to expect. That is a reason to revise the weight given to that evidence, and a reason to examine how research is done and how it is published.

The more interesting lesson is about how knowledge of science itself is obtained. The reliability of research can be studied: a sample can be defined, criteria set, procedures shared, and the assessment exposed to criticism. An assurance that the community keeps high standards does not supply that information.

Why many publications can mean less than they seem

A hundred articles need not be a hundred independent tests. One study can be described in several papers. The Cochrane Handbook therefore requires a review to bring together the reports of the same study, rather than treat each report as a separate experience. That requirement is set out in Chapter 5, section 5.2.1, of the handbook, in version 6.5 (2024): the study, not the report, is the unit of interest.

Dependence can be harder to see. Imagine ten teams evaluating algorithms on the same database. If the labels in the database are faulty, a diversity of authors does not remove the shared source of error. Agreement among analyses that use one method of measurement is not, by itself, a check that the method measures what it is supposed to measure. The ten teams are a hypothetical case, of the same kind as the vocabulary application.

Independence is not a matter of different names on the first page. It concerns the data, the instruments, the assumptions, and the ways of drawing conclusions. Checks that could fail for reasons other than the reasons the earlier studies could fail are especially valuable.

There is also the problem of work a reader cannot see. If striking results, or results favorable to the intervention under study, are more likely to be published, the available literature can present a skewed picture. An entire study may be missing, or only part of its results. Judging that risk means going, among other places, to registers, protocols, and sources other than the published articles. Matthew J. Page, Julian P. T. Higgins, and Jonathan A. C. Sterne set out that problem in Chapter 13 of the Cochrane Handbook, version 6.5 (2024).

In a hypothetical version of the application, one can imagine ten trials of which readers were shown only the two most promising. Even if no single measurement had been falsified, the story of effectiveness would be misleading.

What a synthesis of evidence can do

A systematic review should state its question, the way studies were sought, and the criteria for including them. A meta-analysis is a statistical method for combining results. The two are not synonyms. A careful review can end by concluding that the available results should not be reduced to one shared number.

Return to the application. Can a test of word recognition after ten minutes be pooled, without reservation, with a test of conversation skill after six months? Both concern language learning. They answer different questions. A shared topic does not make the measurements comparable.

Combining many papers does not automatically remove errors they share. More data can shrink the uncertainty that comes from chance and leave a false assumption sitting under the whole analysis. The result is a more precise answer to the wrong question.

For reviews of interventions, Cochrane uses the GRADE system. Holger J. Schünemann and coauthors describe it in Chapter 14 of the Cochrane Handbook, version 6.5 (2024). GRADE distinguishes four levels of certainty in the evidence for a particular outcome: high, moderate, low, and very low. The judgment takes account of, among other things, the risk of bias, inconsistency, indirectness, imprecision, and publication bias.

The phrase "for a particular outcome" matters. The evidence for a short-lived improvement on a task can be strong, and the evidence for a lasting change in ability can be weak. Giving an entire field one label, "confirmed," does not capture that difference.

When different paths lead to the same explanation

Not every important question can be settled by repeating the same experiment. The history of the Earth cannot be rerun, and a controlled experiment cannot be performed on a second universe. The consequences of an explanation can still be checked against different kinds of observation.

One example is the study of the causes of observed warming. Chapter 3 of the IPCC Working Group I report of 2021 brings together evidence from the atmosphere, the oceans, the land, and the cryosphere. The authors say that considering those lines of evidence together increases confidence in attributing the observed changes to human influence and reduces uncertainty relative to assessments based on single variables. That conclusion concerns the causes of changes already observed. The chapter is the report's synthesis for that question.

The scope of that conclusion matters. It does not by itself settle which future emissions scenario will materialize, what the economic costs of the changes will be, or whether particular technologies for cutting emissions or removing CO2 will prove feasible at the assumed scale. Each of those questions needs its own evidence and assumptions. A strong case for one claim cannot be carried over to every other claim in a wide-ranging report. An objection to a particular scenario is likewise not, by itself, a refutation of the findings about the causes of warming to date.

The value of convergent evidence depends on the quality of the measurements, the soundness of the methods, and how independent the possible errors are. An explanation gains credibility when it can cope with different observations, and when researchers also check discrepancies and the shared limits of the analyses.

A similar logic helps with research on the past. Agreement among a source text, the dating of material, and archaeological finds can be more persuasive than further write-ups that repeat the news in the same document. The value of the evidence also depends on which alternative explanations it can rule out.

Consensus and its limits

Scientific consensus, as the term is used here, is a broad agreement of the relevant research community on a defined matter. It can concern the existence of a phenomenon, its cause, or the best explanation available. It does not require unanimity, and it need not cover every side question.

Agreement of opinion has to be distinguished from the grounds it rests on. A vote does not change the properties of nature. For someone outside the specialty, the convergence of competent researchers is still important information, especially when transparent syntheses, independent checks, and an open account of the limitations go with it.

The strength of a consensus is better judged through questions: what exactly it concerns, what evidence sustains it, whether alternative explanations were really tested, and where important disagreements remain.

The presence of a dissenter does not void a well-documented position. Being in the minority does not make an argument wrong either. What decides the value of the argument is its ability to explain the data and to pass checks, not the appeal of the lone rebel.

The same holds for institutions. A position taken by a scientific organization deserves attention, and it should lead the reader to the justification. The most useful summary of knowledge lets a reader see both the findings that are strong and the disputes that have not been resolved.

Self-correction takes work

Science does not correct itself merely because time passes. Someone has to collect the data again, rerun the analysis, check the device, or find the results that were left out. That takes money, time, and credit for work that does not always produce a striking headline.

One way to limit the selection of results is to state the questions and the plan of analysis beforehand. Registered Reports go further. The Center for Open Science describes the format as one in which peer review of the research question and the methods takes place before the results are known. A journal may then accept the paper in principle. Publication depends on meeting the agreed requirements of quality, not on obtaining a desired result. The description of that process is the Center's own.

The format does not remove every error. It does change the incentives. A well-designed experiment that did not confirm its prediction can still be informative. Exploratory research has a place as well, provided a search for new relationships is not presented as a test of predictions that were stated before the data were seen.

Access to data and code makes checking easier. Posting the files does not mean that anyone has already checked them. Transparency creates the possibility of verification. Credibility rises when that possibility is used.

When a reader can rely on a result

There is no universal number of publications after which a result acquires the status of knowledge. What matters more is how well the claim has been specified, which tests it has passed, and how many credible alternative explanations remain.

The grounds are generally stronger when a result holds up under a change of team, of data, and of reasonable methods of analysis; when different kinds of observation lead to a consistent picture; when the scope of the conclusion is known; and when discrepancies can be explained without patching on exceptions after the fact. Those conditions are not always met in the same order. A single, exceptionally well-designed study can change a great deal, while dozens of dependent analyses can add little.

Empirical knowledge can still be corrected. That does not make every claim equally uncertain. A first promising observation and a finding sustained by a wide set of agreeing checks are different kinds of support.

For a reader of science news, the useful change of question is a small one. Rather than stop at "has a study been published?", one can ask what is known after this study that was not known before, and how strong the grounds for that knowledge are.

In the OPERA case, the answer was not the replacement of one authority by another. The discrepancy directed attention to the measurement, and the later checks found an explanation. The work between a sensational result and a warranted conclusion is often the most important part of the discovery.

The vocabulary application, the ten teams using one database, and the ten trials of which only two are shown are hypothetical. They are here to explain the methods. The other projects and documents named above are the ones the links point to.