Fail fast, fail often is one of Silicon Valley’s best known mantras. Is there something to this belief that if you’re not failing fast and often you may be stuck in a stagnating endeavor? After all, social scientists have their own Matthew Effect, - i.e., the rich get richer and the famous get more famous, - as evidenced in our age of rising economic inequality and superstar dynamics.
In early March, - just before Covid-19 brought normal life to a screeching halt, - I attended a very interesting seminar by Northwestern professor Dashun Wang. Over the past several years, professor Wang, along with various collaborators, has been investigating the impact of early-career setbacks on the future careers of scientists.
“Setbacks are an integral part of a scientific career, yet little is known about their long-term effects,” notes a recent article he co-authored in Nature Communications. “Despite the ubiquitous nature of failures, it remains unclear if a setback in an early career may augment or hamper an individual’s future career impact.” The Matthew Effect would suggest that early-career success is an important determinant of a successful scientific career. On the other hand, early setbacks could well be a marker for future achievement, teaching young scientists, entrepreneurs, artists, etc, valuable lessons that they wouldn’t have otherwise learned, i.e., what doesn’t kill you makes you stronger. Which is the most likely scenario?
To find out, Wang and his collaborators examined the relationship between early-career failure and success for young scientists. They did so by analyzing a data set containing all grant applications submitted by individual investigators to the National Institutes of Health (NIH), - the largest public founder of biomedical research in the world, - from 1985 to 2015. The data set has information on over 750,000 applications submitted by almost 140,000 investigators, including the evaluation score for each application and the funding decision.
To asses the impact of early-career successes and setbacks, the research team focused its analysis on junior scientists who early in their careers applied for NIH grants between 1990 and 2005. In particular, the analysis focused on two groups of individuals: the near misses, - who narrowly missed funding because their evaluation score was just below the threshold; and the narrow wins, - who narrowly succeeded in getting funded because their evaluations scores were just above the threshold.
The research team found 561 narrow wins and 623 near misses. They looked at the characteristics of these two groups of investigators and found them to be statistically indistinguishable along all examined dimensions. The only difference between the two groups is that one narrowly succeeded in getting an NIH grant of $1.3 million on average for five years, and the other just missed getting the grant.
Next, they examined the scientific output of each group, - the narrow wins and the near misses, - over the next 10 years and found that the two groups published a similar number of papers per person. They then looked at the number of hit papers found in each group, - defined as the top 5% of papers based on the number of citations they received in their field as indicated in the Web of Science database.
Both groups had a substantially higher percentage of hit papers than the 5% rate for average scientists in their field. Over the next five years, 13.3% of the papers published by the narrow-win group turned out to be hit papers. But, much to the researchers surprise, the near-miss group had a 16.1% hit paper rate, a 21% difference. This performance advantage persisted over the subsequent five years. To ensure that the surprising results weren’t limited to hit papers, the research team quantified the performance of the two groups using other measures, such as average citations received within five years of publication. Overall, the near-miss group outperformed the narrow-win group over the 10 years after the initial NIH grant.
“This finding itself has a striking implication. Indeed, take two researchers who are seeking to continue their careers in science. While both near-miss and narrow-win applicants published high-impact papers at a higher rate than their contemporary peers, comparing between the two groups, it is the ones who failed that are more likely to write a high-impact paper in the future.”
These results are particularly surprising given that, by definition, the narrow-win group started out with an initial NIH funding advantage. In fact, the near-miss group received significantly less NIH funding in the five years after the initial awards, about $0.29 million less per person. The funding difference between the two groups disappeared over the subsequent five years. Similar results were found when investigating other sources of funding, such as the National Science Foundation (NSF). Overall, the narrow-win group received more funding from all sources in the first 5-year windows, but the difference disappeared over the next five years.
“Together, these results demonstrate that over the course of ten years, near misses had fewer initial grants from the NIH and NSF,” notes the Nature paper. “Yet they ultimately published as many papers and, most surprisingly, produced work that garnered substantially higher impacts than their narrow-win counterparts. Is the uncovered difference in outcomes causally attributable to the early-career setback? Or, could it be explained by other alternative forces?” To shed light on these questions, the researchers considered two distinct hypotheses.
Survival of the fittest. In the years following their initial grant application, the near-miss group had a higher attrition rate of investigators that continued to apply for NIH grants, - around 11% per year, - than the narrow-win group. Perhaps failing to get a grant weeded out the weakest investigators, leaving only the most determined, higher performers to continue doing research and writing papers. Further analysis of various versions of this hypothesis concluded that it was insufficient to account for the observed difference between near misses and narrow wins.
Learning from failure. After exploring a number of other possible explanations, the research team concluded that narrowly failing to get a grant caused the rejected scientists to try even harder to succeed. For those with the grit to persevere, early failure isn’t taken as a negative signal, but as a way to learn valuable lessons going forward.
“Overall, these results document that an early-career setback has powerful, opposing effects, hurting some careers, but also, quite surprisingly, strengthening outcomes for others. As such, these findings show that prior setback can indeed be a mark of future success.” In other words, what doesn’t kill you does indeed make you stronger.
Irving, I wonder if some of this is due to the way that we think about risk. According to Kahnemann and Tversky, a team becomes more risk-seeking when it has less to lose and the most risk-seeking when they are "underwater". For a team that's gotten the funding, they will be more risk-averse because they have the money and the research and don't want to lose it. For the team that didn't get the funding, they need to do something really interesting! From this, you'd expect the really interesting thinking to come from those with nothing to lose!
Posted by: RobertSchlaff | July 26, 2020 at 05:26 PM
Irving, Many thanks for a provocative article. One variable not noted in the research could be significant, however. I have seen enormous variability in the presentation quality of proposals; i.e. some are far more coherently presented and argued than others, irrespective of the underlying scientific or mathematical bases. Perhaps the linguistic and organizational communications facility is a factor. (This would be very difficult to analyze, of course.) Thanks for the thoughtful, and well-presented, contribution.
Posted by: David Matthew | July 28, 2020 at 08:28 PM