Not junk: Negative results in research present an opportunity to learn

In science, positive findings conforming with established hypotheses are celebrated via publication—the coin of the realm in academia—whereas nonconforming or negative results are often frowned upon and discarded by the researcher. This is surely also true for optics and photonics.

Many scientists do not proceed further with negative findings because the related value in the scientific community tends to be much lower than for so-called positive data. However, null outcomes sometimes need to be demonstrated and, in some cases, scrutinizing negative results from an alternative perspective can help to understand a larger problem.

In the early 1900s, Lord Rayleigh derived the famous λ^–4 formula to predict the spectral radiance of electromagnetic radiation from a blackbody—a derivation that led to the Rayleigh-Jeans law. This derivation relied on the classical physics theories and on empirical observations for low frequencies. However, that derivation implied that the emitted energy was infinite at high frequencies. This idea was at odds with the then-known fact that total emitted energy is finite, and the prediction significantly diverged from observations above 100 THz.

From its first derivation in 1900, it took approximately five years for the idea of the “ultraviolet catastrophe” to catch on, mostly thanks to the contributions of Lord Rayleigh himself and Albert Einstein, who managed to convince the scientific community that classical electromagnetism had to be rejected in favor of the quanta of light theory derived in 1900 by Max Planck (who, ironically, wasn’t initially convinced of the reality of the theory, and thought of it just as a mathematical trick). This negative result was one of several that led to the birth of quantum mechanics.

So, disregarding negative results not only represents an obstacle for the development of science, but also encourages researchers to dismiss results contradicting existing literature, regardless of potential for leading to significant breakthroughs. Validating negative results by further experimentation or analyses, ensuring reproducibility and statistical significance, is sometimes necessary.

Researchers, editors, publishers, and funding institutions  should be aware of the significance of negative findings and support their dissemination. We need a change of mindset to transfer in-depth knowledge gleaned from negative results to next-generation researchers. In other words, it’s time to be positive about negative results.

Negative findings can be defined as results that contradict research hypotheses, established scientific knowledge, and previous evidence or predictions. They are typically characterized by a different, opposite, or absent correlation between observed phenomena. Of course, these results must still rely on sound theories and carefully performed experiments. Erroneous results cannot be considered negative results per se.

Negative results are usually rejected or even regarded as failures by researchers and their peers. As a consequence, positive results in published articles sometimes lead to conclusions different from those arising from unpublished negative results. When this is systemic, we say there is a publication bias.

Lord Rayleigh (right) and Albert Einstein. Photo credits: The Nobel Foundation archive

Published negative results also tend to be less cited than positive results, or only cited by a small group of researchers, leading to exclusion from meta-analyses or literature reviews. This so-called citation bias also has indirect consequences: In academia, well-cited articles often lead to a higher probability of continued research funding.

So, it is crucial for the advancement of knowledge to publish both positive and negative findings. The publication of well-documented, well-designed, and well-executed “failures” could add important perspectives and records to the scientific literature.

For example, results considered negative are often statistically more reliable since they are reproduced at multiple occurrences. They are valuable elements in the scientific literature that can help to evaluate data, reveal undiscovered relationships, or point out wrong assumptions and flaws in theories. They can also be used to steer research strategies, and provide inspiration for innovative development of theories, simulations, or experiments. What’s more, their dissemination avoids duplication of effort by other groups.

Instead of rejecting negative results, we might allow greater room for nature’s complexity by placing greater trust in our data, rather than outright rejecting that which does not immediately fit our preconceptions. This practice may mean acquiring more data and discussing it with peers in our own research groups and institutions, as well as with colleagues from other institutions.

When negative results are encountered, the path forward for the research team can vary. If obtained in an early stage of a project, decide whether to investigate further depending upon, for example, significance of the data, the project timeframe, project objective, and researchers’ instincts. If secured in a late stage of investigation, negative results should be disseminated.

This practice is true for both early-career and established researchers. The former should take advantage of such publications to practice technical writing, which is a valuable skill. The latter typically have the time to address time-consuming questions that arise from negative results and enough credibility to question widespread assumptions in their fields.

Fortunately, an increasing number of peer-reviewed scientific journals aiming to reduce publication bias have emerged in recent years. All reinforce the idea that, in science, failure can be as important as success. In addition, open-access broad-topic journals such as PeerJ, PLOS ONE, Scientific Reports, and F1000Research also allow the publication of negative results. Still, the number of articles reporting negative findings remains a small percentage of the overall literature.

Ideally, results should be published regardless of negative or positive conclusions. Even if journals favoring novel, impactful, and positive results cannot be avoided, publishers could give negative findings more room. Conference proceedings, like SPIE’s, are one such venue for publications of this type.

Publishing scientific articles is not an end unto itself. As researchers, we should not be disappointed or frustrated by negative results but strive to reach unbiased conclusions driven solely by the data. Such research has value.

Sébastien R. Mouchet, Priyanka Dey, Michele Gintoli, Jayakrupakar Nallala, and Bob-Dan Lechner are researchers in the Department of Physics and Astronomy, University of Exeter, UK. Contact the authors at s.mouchet@exeter.ac.uk; or B.Lechner@exeter.ac.uk