Behind the scenes with Nature

I just published a letter in Nature.

Well, a letter only in the colloquial sense. More properly, it’s correspondence. It can be seen here. It reads:

Lifetime word limits would unleash woe

If science’s current predicament has taught us one thing, it is that we should beware of perverse incentives (see M. Edwards and S. Roy Environ. Eng. Sci. 34, 51–61; 2017). So let us imagine the cascade of woe that could follow from Brian Martinson’s thought experiment of allocating scientists a lifetime word limit (Nature 550, 303; 2017).

Papers could become shorter and more obtuse, with content moved to appendices. ‘Pre-prints’ might never be published and instead would be squirrelled away on personal websites — dodging peer review. A new type of predatory journal that falsified word limits could stoke demand and become pervasive. Research collaborations would decline.

Supervisors would leave their names off papers, relying on the force of association to boost their reputation. And, graduate students could have their lifetime word limits exploited, particularly if they do not continue with an academic career (see Nature 550, 429; 2017).

As my colleague and mentor Sarah Boon said about her own ‘non-traditional’ piece in Nature, “it’s a relatively short piece that looks deceptively easy to write.” I suspect I had an easier time of it than Sarah, since I had no need for interviews, and the editor’s role in this section is closer to gatekeeping than overhauling.  Nonetheless, there’s a lot that went on behind the scenes, some lessons to learn, and some questions raised. This piece owes a lot to Sarah’s — not only the obvious similarities, but the well-timed publication of her piece is part of what inspired me to submit my letter in the first place.


This letter started life as an angry rant. First on facebook, then slightly tidied up and copied onto my blog. Here’s the first lesson: angry rants don’t have to stay on facebook. If you have something to say about a scientific publication, write to the publisher! This is particularly true if you’re a junior researcher, and if you can easily substantiate your claims.

To change my angry blog post into a reasonably argued letter took me a couple of hours. Much of that was spent looking up the requirements – finding DOIs and formatting citations, and wishing I could publish in the same ‘World View’ column as the original. To be honest, I’m not sure what’s up with that column. Yes, it’s important to have a venue for opinion pieces, but it appears to be averse to citations even where they would be appropriate, and the editorial process is very opaque.

In any case, my original letter rang in at close to 300 words, instead of 150 for the final version. I won’t reproduce the entire text, since the content is fairly close to the blog post, though the tone is not. In particular, it included a paragraph about the utter infeasibility of implementing a word limits system, and a concluding statement encouraging both creative thinking and a healthy fear of unintended consequences.

Here, we come to the second lesson: let it go. When the tentative approval came back, the whole section about feasibility had been cut. And it was better without it. It was shorter and less aggressive, and in any case the feasibility of a thought experiment doesn’t matter as much as its unintended consequences.

I was somewhat amused by our email chain back and forth. The editor addressed me as “Dr.”, even though I am not. I addressed her as “Dr.” – because, even though she did not include it in her email signature, a quick google search confirmed that she was. I wondered how many people didn’t, and I was conscious of the usefulness of “Dr.” as a form of address that is both genderless and unrelated to marital status. (Lesson #0: treat people with respect.)

There were a couple of other points in our editorial back-and-forth. The phrase “cascade of woe”, and the title, are hers. It’s not a word I would have thought to use, but I quite like it. I had pushed for “would” in a couple of places that now read “could”, and I had wanted to include a closing sentence about the need for more creative solutions. Most importantly, I had wanted to use the phrase “disposable grad student husks”, which so many people liked in my earlier drafts. Sadly, this was not to be.


My biggest unresolved question: how serious was the original suggestion? The first response I received said “This shortening has been achieved by making the tone more satirical, in the spirit of the original article.” So…was it a satire all along, and did I miss the joke? My response is “if it was a joke and I missed it, then others must have missed it too.” But this is somewhat self-serving, since I wanted to publish.

My best answer is that the suggestion of lifetime word limits was a serious idea somewhere between a thought experiment and a fantasy, which was written in a satirical tone. I say that it had a component of fantasy because I think that if Martinson had treated it as a serious thought experiment, he would have come up with some of the same issues as I did. He has a fairly substantial career in research integrity; this leads me to think that either he knows something I don’t, he’s frustrated enough to propose his perfect-world solution without thinking of undesirable consequences, or he just wanted to throw it out for fun.

The semi-serious nature of this topic is illustrated in another response, advocating cap-and-trade for word limits. This author suggests auctioning researchers’ love of verbiage such as “it is easily shown that …” as a fundraiser.

I will admit that this correspondence was partially published with the goal of getting my own name out there. That seems important, if I want to situate myself in the field of studying and improving the scientific process. But this publication is also worthwhile to the broader community. I saw a decent amount of uncritical coverage of the original piece, including on Retraction Watch. I think that record is worth correcting.

This brief letter also helped me develop my ideas. In particular, I think that as a bare minimum, creative ideas to improve the scientific process should be feasible, and shouldn’t create more problems than they solve. This sounds simple, but is quite a tough requirement. Feasibility means among other things that it should be scalable and not require a system-wide overhaul. Attempting to assess the consequences of theoretical ideas is often quite tough, but should consider existing power structures & axes of marginalization, subtle systemic pressures, and malicious compliance.

I look forward to further developing these ideas in future posts or publications. In the meantime, I have a piece in Nature, and only spent 150 words.

Bad incentives in science apply to your “solutions” too

This is really just a hasty screed copied over from facebook, which I thought I might keep in a slightly more permanent place.

It’s based on this opinion piece in Nature: Give researchers a lifetime word limit.

It starts with a good observation (though at this point it’s a fairly widely held belief):

The purpose of authorship has shifted. Once, its primary role was to share knowledge. Now it is to get a publication — ‘pubcoin’ if you will. Authorship has become a valuable commodity. And as with all valuable commodities, it is bought, sold, traded and stolen.

The proposed “solution” — a lifetime word limit per author, would, according to the author, have the following miraculous effects:

“Lifetime limits would create a natural incentive to do research that matters.”
“It would also lead to increased value being placed on concision and clarity, improving readability and efficiency.”
“Predatory publishers would vanish.”
“The task of evaluating candidates for jobs, advancement and prizes would become less scattershot.”

No. Have we learned nothing from communism? If you think that it’s a problem that something has been commodified, putting a hard cap on it is *not* a solution. I’ll get to the tremendous implementation difficulties later. For now, let’s see some ways your great idea would backfire due to the terrible incentives it creates.

Papers would become shorter and more obtuse, with more of the content shoved into appendices, “pre-prints”, personal websites, and anywhere else that doesn’t count.
Honourary authorship would become less attractive — but a shadow “authorship” would emerge, of (e.g.) grad students leaving off their supervisors, but still making it known as “their” work.
Many of these grad students, who aren’t going to get academic jobs anyway, would just be disposable husks used up for their word count.
Researchers would face new anxiety about whether their current topic was “important enough”.

I think the deeply unethical disposable-grad-student-husks should alone be enough to stop it. But perhaps that’s naive; it certainly doesn’t seem to have stopped our current system. The big point is this would not help science.

The implementation side is also, shall we say, nontrivial.

It’s technically easy (ish) to require all authors to have ORCIDs, and tally everyone’s word count. But that’s the only easy part.

You’d have to get all journals to agree to this system. Getting all journals to agree on anything is more or less impossible. Even if you did in your fantasy world, you’d have a new type of predatory publisher: those that faked compliance with this system.

And what happens with a submission that’s over someone’s lifetime word count? Is it auto-rejected? Or does that author just get dropped?

What happens when a faculty candidate is near their lifetime word limit? Are they automatically rejected?

What happens when a recognized authority is no longer allowed to publish — both for their own career, and for the field?

How is it phased in? Does it only apply to publications from this day forward, and hence disproportionately hit younger researchers? Or does it force a large cohort of senior academics into immediate retirement?

Who sets these word limits anyway? The author acknowledges “Some subjects and pursuits might be inherently wordier than others.”

But then he undermines that with “Exceptions might have to be made for experts such as statisticians and bioinformaticians whose skills are required on many papers — but perhaps this would boost the quality of collaborations. Perhaps researchers could apply for word bonuses for careful reproductions, cautious interpretations and meticulously described methods.” Noooooo! This just commodifies the bureaucratic filing of appeals.

Lifetime word limits do not help science, and it’s bad enough that I can’t see a way to make it work. It’s lucky that this can’t get off the ground. But for next time: please, please think about perverse incentives before proposing brilliant new solutions.

 

Is the US education system producing a society of “smart fools”?

The great thing about headlines as questions is that responses to the article can use the same headline. Though I’d be tempted to reframe it: is America a  minmaxing RPG character? (Jargon time: in roleplaying games, minmaxing is the character-building strategy of maximizing a specific desirable ability, skill, or other power of a character and minimizing everything else, seen as undesirable. The result is a character who is excessively powerful in one particular way, but exceedingly weak in others. As an example, it’s pretty common to trade off wisdom for intelligence.)

 

0. The article is here.  A summary:

This is an interview with psychologist Robert Sternberg, about the risks of standardized testing. In particular, he argues that they’re overly narrow, and develop intelligence at the expense of creativity, common sense, and wisdom. He cites the 30-point rise in average IQ scores over the 20th century (the Flynn effect) as evidence in favour of this narrow focus. He notes a lack of focus on wisdom – which he defines as working with other people’s interest to a common good – and noting the lack of famous wise people or of practice in ethical reasoning. His career has included some initiatives to train these skills, and has found that they are much more predictive of college performance than standardized tests.

 

  1. My issues with what it says: There are so many of them. The challenge is coherence.

I’ll start with the statement that “it’s really hard to think of wise people”. Regardless of what definition of wisdom you’re using, I don’t think that it involves seeking fame and recognition. I am fortunate to know grandparents, former teachers, mentors, and other figures I’d call wise; I think it’s the same for many of my friends.

For that matter, I’m not entirely on board with the definition of wisdom. I think that wisdom includes such other traits as foresight, judgement, experience, and emotional control. It’s not that these aren’t important in working together on a common good, but I do think the definition in the article is a bit narrow. Furthermore, I think this definition is tilted liberal. Even if it may not always be demonstrated, I would hope that wisdom is a nonpartisan trait.

I very much support at least the broad idea, captured in the reference to McGuffey readers, of teaching content at the same time as teaching reading. I would say that one of the major failings of modern education is an overemphasis on reading for the sake of reading scores, which is really boring and devoid of content that could be built into it, such as history or, as Sternberg argues, wisdom.

However, I would be extremely cautious of this kind of moral curriculum. I have not read the McGuffey readers, but it is deeply ironic to rail against tribalism and “people who view the world as being about people like themselves” while praising 19th-century moral education. From Wikipedia: “[McGuffey] attempted to give schools a curriculum that would instill Presbyterian Calvinist beliefs and manners in their students. These goals were considered suitable for the relatively homogeneous America of the early- to mid-19th century, though they were less so for the increasingly pluralistic society that developed in the late 19th century and early 20th century.”

Finally, so much of the article’s claims of lack of creativity and of wisdom are incredibly anecdotal. Though the article doesn’t single out young people, the tone still reads as “kids these days/millennials are ruining everything”. Like so many such articles, there’s a glaring opportunity for a lack of historical perspective. Consider the examples of Greg Gianforte’s election after according-to-the-courts-I-must-say-allegedly-assaulting a reporter, or violence at campaign rallies. In terms of political violence, 1921 alone was orders of magnitude worse than the 2010’s, with both the Tulsa race riots and the Battle of Blair Mountain. I am not a historian, but I would say these low points of American wisdom were induced and not prevented by 19th-centuy moral education.

 

Next up, I want to write about what I wish the article had said instead. Because I do agree with the premise, at least enough to think it’s worth talking about. America may be becoming less wise, and there’s a lot to tease out about incentives in education and society, which rests on a much firmer foundation than anecdotes. But that’s a project for another day.

 

Physics simulator error

This experience happened over a year ago, but oh well. Our current snowscapes reminded me of it.
I was walking through a parking lot, in the winter, when there were little clumps of partly-frozen snow on the ground. I kicked one accidentally. Out of the corner of my eye, I saw it bouncing along on the ground.
Something seemed strange. I looked again. It was kind of floating along higher than it should be. I mean snow is light, and there was a bit of wind, but not that much. I hadn’t kicked it hard at all – just a nudge. It skidded along the ground a couple of times. No little bits of it fell off. That was strange.
Then it clicked. This wasn’t snow – it was a piece of Styrofoam.

Now for everyone’s favourite part – the moral of the story. This was a cool experience of refactored perception – the lower level PHYSICS SIMUALTOR module was doing some calculations, which resulted in an error message when run on the input “snow” from the higher-level OBJECT IDENTIFICATION module. Fortunately, changing the contents of this module resolved the error without system failure. The whole episode was a visceral & fun way to be wrong.

NASA and the political climate

The Trump administration has revealed plans to axe NASA’s Earth Sciences programs. This news was revealed in this article in the Guardian, which is short and worth reading for 3 reasons:

  1. The actual statement of Trump advisor Bob Walker’s views;
  2. The rhetoric and actions associated with those views;
  3. The completely ineffective response by senior scientists quoted in the article.

Here are some reasons – both glaring and subtle – why these cuts are a bad idea, and some proposals for what to do about it.


First, before engaging in any leftist hysteria, let’s look at what was actually said.

“‘We see Nasa in an exploration role, in deep space research,’ Walker told the Guardian. ‘Earth-centric science is better placed at other agencies where it is their prime mission.’

‘My guess is that it would be difficult to stop all ongoing Nasa programs but future programs should definitely be placed with other agencies. I believe that climate research is necessary but it has been heavily politicized, which has undermined a lot of the work that researchers have been doing. Mr Trump’s decisions will be based upon solid science, not politicized science.’”

Reorganizing programs so that the space agency deals with space and earth agencies deal with Earth seems like a pretty reasonable position. And deep space research, going back to the glory days of crewed missions to the moon and beyond, would be a crowning jewel in Making America Great Again. It’s certainly better than sending multi-billion-dollar satellites up just so they can look back down on liberal climate navel-gazing. That’s all that Obama was doing, while he paid lip service to the idea of going to Mars and simultaneously cutting funding from deep space missions. Why are those whiny leftists so upset about common-sense reorganization like this?


Well, this warrants some explanation. I hope to provide arguments that lay out the substance of my criticism, rather than skipping to indignation. So here are my concerns:

  1. This is a misunderstanding of how earth research works: out of 12 or so key climate indicators (temperatures, cloud cover, vegetation cover, ocean algae levels, atmospheric composition, …) 50% of them are either best monitored from space or can only be monitored from space.
  2. Spaced-based Earth monitoring data also leads to cool & interesting research in other fields besides just climate change – mysterious & previously unknown total glacier collapse, or economic trends associated with land use.
  3. There is no sincere plan to move this research to other agencies.
    • The National Oceanic and Atmospheric Administration, which is the most likely contender, has been under Republican attack for years based on similar arguments
    • The Environmental Protection Agency is headed by a climate skeptic
    • In any case, would these “other agencies” have the budget/capacity/mandate to hire satellite engineers & rocket scientists & launch coordinators? Really?
  4. Capacity building for deep space missions will be severely compromised. Getting things to space & communicating with them is very hard, but it’s also kind of all the same. Earth monitoring missions produce lots of opportunities to practice. Both the personnel and the technology are very similar to what’s required for deep space missions.
  5. Pure deep space missions are often boring. At great expense, you can go to some faraway rock and measure how extremely unlikely it is to support life. That’s not to say there aren’t still interesting and worthwhile missions – but life is already happening down here. There’s a whole lot to see, and it directly affects us in a way that Mars and Titan do not.
  6. This threatens to effectively destroy the evidence base on climate change, protecting the Trump administration from future criticism over how much damage their actions do to the climate.
    • This is analogous to the Canadian example of Harper removing the mandatory long-form census, which erased any ability to measure his government’s performance on social or economic issues
    • Corporate interests will stand to benefit very quickly from the reduced scrutiny and accountability
  7. Finally, I want to draw attention to a pattern of Trump and his staff’s engagement with the media (pointed out on Twitter; lost the link), which is to give moderate statements in interviews followed by much more radical ones to their supporters; their actions tend to match the more radical ones, and I suspect that’s the case here. This may be contentious in general, but I think the lack of plan or support to place Earth-based science at other agencies shows it is the case here.

Ok. So those are the problems behind Walker’s seemingly reasonable statements: they betray deep ignorance of the practicalities & role of space research, and his conciliatory words are patently insincere. (There are also factual errors that should stand corrected.)


Now what do we do about it? As I said at the top, I think that indignation and loud yelling about the importance of this work are poor strategies. Clear communication is important both in science and in politics, and I’m disappointed in the responses I’ve seen thus far.

Here’s what we can do instead:

  1. Clearly and patiently explain what Earth science is done at NASA, how it fits into NASA’s expertise, and its benefits, using arguments that are more meaningful and more politically engaging than “monitoring climate change.”
  2. Fight by political means. Little more to say about this here, except: fight hard and fight smart (e.g. by appealing to your audience’s values and not just your own), and nonetheless be prepared for it to fail. This sucks, but it’s normal. Changing political regimes mean changing priorities. And we’re not done yet.
  3. Be prepared to find additional funding sources. Private companies – certainly insurance companies have substantial capital, and are paying very close attention to climate change. Consider crowdsourcing for funding; this could be quite successful given NASA’s excellent public relations. However, there’s a lot of factors to manage here, such as concrete mission objectives and avoiding political smears as “liberals’ pet project.” The Breakthrough Institute, among others, would be natural partners.
  4. Work with private space companies. SpaceX is of course a prime contender, but they’re hardly alone. All of these would be good avenues for ensuring continuity of expertise. As private companies, they could much more freely continue with “politicized” work.
  5. Don’t be afraid to go to other countries. It is highly unfortunate that so much earth monitoring capacity is subject to the whim of presidential elections. But internationally, there’s huge value in the research being done.
  6. Really, go to other countries. If the EU or China has the scientific expertise & output that America just gutted, and the international recognition formerly due to NASA, this will send a strong signal about how misguided this plan is.

As with so many other articles, the moral is: explain yourself clearly, before getting angry. And when you are angry, don’t despair: organize.

 


Addendum: some thoughts on “politicized science”

 

Note that one of Walker’s points I did not try to refute is “this is politicized science”. After a certain point, when huge swaths of the population have bitterly entrenched ideological positions either for or against an issue, it’s counterproductive to loudly yell “this is not a political issue!” as so many American scientists have done. No matter who started it.

 

But then again, all research is informed by values that are to some extent political. The important thing is to make sure your research appeals to various political perspectives. This is not to say that all research must justify itself to everyone. Basic research will always be niche, but have an appeal to curiosity and love of knowledge that can strike across the political spectrum. Applied research, especially large-scale projects dependent on societal support, should have wide-ranging interest. (Note that I didn’t say “benefit”, since that’s often very hard to define and cheapens research; nor did I say “majority support”, because that would be ridiculous and capricious.) For example, the long-form census data is useful for almost any social issue in Canada, whether you’re concerned about the decline in church attendance or about institutional barriers to LGBT people. Data is data, so it’s easy to believe this should not be political. But in earth monitoring, data is still data, only it’s very expensive data and it’s used to support a policy position that (at least in the US!) is predominantly left-wing. It’s not entirely unreasonable for this to be questioned. But I think when it is, we should have answers ready.