UPDATE ON LAURANCE ET AL.'S “PREDICTING PUBLICATION SUCCESS” IS FUNDAMENTALLY FLAWED (MAYBE).

Diego (above) still has not published any papers, so he is still doomed professionally.

Last year Bill Laurance and colleagues published an article in Bioscience whose take home message was that you could predict the longish-term publication success of scientists by simply looking at their pre-PhD publication record.  Not surprising, perhaps, but they then went on to argue that this would be a useful tool for identifying “rising stars” in Biology.  As might be expected, this study got LOTS of attention [e.g., Nature, Science Daily, tweets a plenty], especially given it’s potential implications for the way we hire researchers. As I posted soon after the study came out, I agree that we should be encouraging students to publish, but I had some nagging concerns about the analysis that I felt could really alter their conclusions. Though I tried writing them directly and posting on one of the co-authors' blog  to get answers to my questions, they never wrote me back.  So, I had to go old school and submit a response directly to the journal and see what they had to say in response.  My letter and their reply are now out.  The summary is below.

I had three main problems with their analysis:

1. They failed to control—statistically or in their sampling—for the type of institution where their focal researchers were based. Given differences in obligations and resources, scientists are likely to have very different relationships between pre- and postdoctoral productivity if they are based at large research universities, smaller colleges focused on undergraduates, or government research institutes.

2. They neglected to correct for the fact that not all researchers, even those at the same institution, devote the same proportion of their time to research. I think they should have scaled productivity by Full-time Equivalent (FTE), since FTE defines how much time you devote to research, and hence your productivity.

3. They pooled researchers from different countries in their analyses without including national identity as a factor in their model. Without explicitly considering the influence of national identity it is difficult to determine whether their results are widely applicable (BTW, they didn’t give a complete list of countries from which researchers were selected or the sample sizes of authors from each country).

Their responses?

1. Correcting for the type of institution where researchers work: It doesn’t make sense to include the type of institution where you work as a factor because this is actually a consequence of your pre-PhD publication record, not a driver of your post-PhD success. In their words, “productive scientists will clearly be better than unproductive ones at securing positions at research-intensive institutions and at devoting more time to research.”. Translation: the reason you didn’t get a job at an R1 is because you weren’t productive enough in grad school to get one. Suffice it to say that the job market must operate way differently in Australia than other places.

2. Productivity should be scaled by FTE:  They didn’t respond to this criticism.

3. One should consider the effects of the country in which a research is based: Their response was “If one wanted to include country as a random effect, would one use the country (or countries) where a researcher was born and raised, the country where he or she received his or her PhD, or the country (or countries) where he or she was subsequently employed?” The answer to their question is “yes”, but for starters Employed, since this context in which someone’s productivity is evaluated and which in part motivates it.  That they didn’t have the replication to do so suggests their sampling was inadequate to draw they general conclusions they put forward. By the way, the issue of author origin vs. author location of employment is something people doing scientometric work struggle with all the time, and they have to justify their choices in the papers or at least explore the implications of these different options (e.g., here).

Not particularly satisfying you say? Don’t worry - their concluding argument for why their work is sound is that it was recommended on Faculty of 1000 and a “popular synopsis” (i.e., blog post) they wrote about it has had 15,000 reads.  QED!

Scientometrics is challenging and requires careful design, sampling, and analysis. The data collection and statistical issues Laurance et al. avoided in favor of “simplicity” are struggled with all the time in the pages of Scientometrics or the Journal of Information Science.  Given that their conclusions could be used to make decisions that affect the future of students with which we work, I wish they had struggled with them a little bit more.