Category: retractions

My input to the ASC publication committee

The ASC Publication Committee asked for input on policy and process for publication complaints. The background is of course the now retracted papers in Criminology for reasons detailed by Justin Pickett, and the statements published by the ASC, as well as the video from the forum on scientific integrity. I have previously commented upon it here and here.

I submitted the following to the ASC Publication Committee:

Dear ASC publications committee,
First of all, and I am glad to see the ASC taking steps to improve procedures, and I appreciate you giving everyone the opportunity to give input.

One important issue in the recent debates is access to data and the reproducibility of the results. To re-analyse the original data is clearly crucial when there are allegations of research misconduct. At the more general level, when there are such difficulties, then it also becomes clear that the data used in the publications have not sufficiently well documented in the first place. I think this needs to improve.

There are now strong moves towards what is often referred to as “open science”. Obviously, if data were made publicly available in the first place, it is much easier to check the results by others. However, while making data openly available to all is in many respects desirable, it is also very often not possible with the kinds of sensitive data criminologists typically use. But many of the ethos of “open science” are general principles of science, and some minimum measures should be taken even without committing to any specific “open” framework. At the core is the documentation of research procedures, including data collection and data management. The focus should be on such documentation, and I would like to see some minimum standards of such reporting to be implemented for all studies.

Others have probably provided more thorough suggestions, but I think the following could be a good starting point. My suggestions are simple and should not require much additional effort by anyone (neither authors or editors). I suggest that all published quantitative studies should include the following information:
a)       Regardless of data sources, there should be a note detailing how others can get access to the same data. If special permissions needs to be obtained, information on where to apply must be provided as well as the main conditions for access. If data cannot be made available to others, then the reason for this must be stated. If data will be made available to others at some later point in time, then information on when and how should be included.
b)      When and who collected the data. If a survey company have been hired, there should be some reference to contract or other documentation.
c)       If data have been obtained from an existing study (e.g. AddHealth or NLYS) there should be a reference to when and how the data were handed over, including specifications of sub-samples (when relevant). Thus, others should be able to get access to the exact same data.
d)      If data have been obtained from administrative records, there should be references to who handed over the data, including dates and permissions etc.
e)      Most studies require ethics approvals. Reference to such approvals should always be provided.
f)        Reproducible code should be made available for all studies regardless of data availability. This code should at least cover the estimation procedures, but preferably also the entire workflow from raw data to end results. Whether code is stored as supplementary files at the journal or some repository is of no importance as long as it is specified.

These suggestions are primarily relevant for quantitative studies, but some would apply to qualitative studies as well. One should also create similar guidelines appropriate for qualitative studies.

Please recognize that I expect all researchers to be able to provide this information with minimum effort. It is simply providing basic documentation. Indeed, if researchers cannot do so, then journals such as Criminology should not publish the article at all simply because the study is not well documented. I consider this to be a minimum requirement.

I would also like to see journals to make conditional acceptance of articles based pre-registration, but that would require a bit more work on the principles. I consider also pre-registration as a kind of documentation of ideas and plans. I do not think it should be mandatory, only encouraged.

I think Criminology would benefit from this in at least two major ways: 1) Increase the quality of the work published, just by making the studies more reproducible and well documented. 2) Increase the status of the journal, and gaining international reputation for being at the forefront in this development. 3) Increase trust in the results published in the journal.

I should probably have added that if there are complaints regarding errors in the above documentation (which cannot be fixed within a couple of weeks or so), retraction should be considered based on that alone.

I could have referred to e.g. the Open Science Framework (which is great), and others have probably written more thoroughly on such issues. But I think such documentation is so basic that it is embarrassing it is not already standard requirements.

Taking raised concerns seriously – but do not regret other statements?

Yesterday, the American Society of Criminology posted two statements regarding the Chronicle of Higher Education article of September 24. The first is by the ASC executive committee, stating support for how the editorial team is handling the matter, and ensuring that the process follows the COPE framework. This is very good. Even though Criminology is not a member of COPE, their guidelines are very sensible, and similar to Wiley’s guidelines. COPE has a flow chart that describe the process.

The second statement is from the co-editors of Criminology. This statement explains how the journal handles cases where there is raised concerns about an article. The main approach is a comment-and-reply model, where critics submit their comment to the journal and the original author is offered to reply. They also state that this is not appropriate in all instances, and additional steps may be necessary, including retractions if the evidence is strong. This is all fine, and I agree.

The comment from the co-authors also details the time line from when they got an anonymous email on the May 29, 2019 and up to today. The also emphasize that they did issue a statement July 26 notifying that investigations were being done.

This is all good. I expect nothing less.

However, the statements are not really a comment directly on the article in the Chronicle, although on the same topic. I guess their main message is just to ensure that they are pursuing the case, as is clear from the following statement:

“Social media attention to Dr. Pickett’s online statement led to what we perceive as a rush to judgment against the authors and the journal, including the mischaracterization that we are not taking the issue seriously and are not committed to resolving it. Nothing could be further from the truth. We have taken several steps aimed at obtaining a fair and transparent resolution.”

From my point of view, the editorial statement July 26 was fine, and I trusted the journal to do an appropriate investigation as stated. I did think it now took a bit long time, but I have no problem accepting that there might be good reasons for that.

I was alarmed and disappointed only when I read the article in The Chronicle. There were stories, speculations and rumors that are not the responsibility of the journal, whether true or false. Criminology is not to blame for any of that. However, the chief editor, David McDowall, was quoted in the article saying things that gave the impression that Criminology did not carry out an appropriate investigation. I believe it was precisely his statements that made people doubt whether Criminology took the issue seriously. I think there are three main points:

First, the chief editor was quoted questioning Pickett’s personal motives. It seemed like McDowall actively defended Stewart, and tried to make Pickett look bad. Given that the journal’s investigation is not ready, it is highly inappropriate for the chief editor to make such statements.

Second, the chief editor was quoted on claiming that the journal has published “complete gibberish” before, referring to one specific instance. He even seems to be fine with that as it appeared to be an argument against retracting the article in question. Let’s just hope he was misquoted.

Third, the chief editor was portrayed as “no fan of the move toward more scrutiny in the social sciences, which he sees as overly aggressive”. That was not a direct quote, but there is a direct quote where he refers to such scrutiny having a “blood-sport aspect to it” (which obviously does not sound positive). Scrutiny should be at the heart of social science, and so should reproducibility and accountability. While I do expect journals to handle such instances in a professional manner (no blood-sport), it is hard to accept that the chief editor is not in favor of such scrutiny.

My point here is that the statement from the co-editors do not clarify these three conserns following from the quotes in the Chronicle. It would be good to know if the chief editor was misquoted or cited out of context. Or maybe he was just sloppy and did not really mean those things, or even regretted that it came out that way. Whatever. Does he and the journal stand by these things or not? I would have hoped that the statement from the co-editors would 1) apologize for prematurely questioning Pickett’s motives in public and hopefully also state that it was not the intention at all, 2) ensure that Criminology do not accept publishing “complete gibberish”, but will now look into also the other article mentioned by the chief editor to check if that was actually the case, and 3) ensure that Criminology supports the move to increased scrutiny in the social sciences.

In any case, the co-editors have been very clear that they are taking the issue seriously, and the ASC executive committee ensures the process will follow the COPE guidelines. I trust that is happening.

Clearly, there are ways of improving research integrity and accountability without any aspects of blood-sport. Some improvements might even be easy. I might come back to that in a later post.

UPDATE: The chief editor just sent an email to all ASC members where he clarifies that some of the words he used were regrettable and do not reflect what he really means neither about editorial policy nor about persons involved. That is good! It goes a long way answering my concerns in this blog post.

The former flagship journal Criminology

I’m so incredible disappointed in the journal Criminology. It is meant to be the flagship journal in our field, but it is clearly not up to the task these days.

The journal is published by Wiley, so lets start reviewing the publishing house’s general policy on retractions here: https://authorservices.wiley.com/ethics-guidelines/retractions-and-expressions-of-concern.html. Just take a look at the first point:

“Wiley is committed to playing its part in maintaining the integrity of the scholarly record, therefore on occasion, it is necessary to retract articles. Articles may be retracted if:
– There is major scientific error which would invalidate the conclusions of the article, for example where there is clear evidence that findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error).”

What I know about the story has been in the public for a while. In July, Justin Pickett posed this paper on SocArXiv here: https://osf.io/preprints/socarxiv/9b2k3/ , explaining that an earlier paper has fundamental errors. Surprisingly, a survey of 500, but the article reports n = 1,184. While I can understand errors can lead to duplicates, I do not understand that it can happen without noticing. Pickett details numerous other errors, and asks for the article to be retracted. That seems like a perfectly reasonable request, and I fail to see how it could be declined. But it has.

A story in The Chronicle Review (behind paywall, but is also available here) reveals astonishing statements from the chief editor, David McDowall, who even says he has not read Picketts letter thoroughly. Any editor receiving such a letter should be highly alarmed and should indeed consider all details very carefully. Apparently, the editorial team does little or nothing. Or at least: fail to communicate that they are doing anything.

I find the following quote particularly disturbing:

First, McDowall seems to think a correction of errors has the goal of ruining other people’s career. I have to say that Pickett’s letter seems to me to be sober and to the point. Pickett gave his co-author more than fair chance to make the corrections himself before publishing his note. It seems like a last resort, not a blood sport at all. If the authors had just admitted the errors and agreed to retract, it would have been a regrettable mistake, but now it is a scandal.

Second, a flagship journal should never publish “complete gibberish”! That some (or even many) articles turned out to be wrong, fail to replicate and contains errors is not that surprising (although not desirable, of course), but “complete gibberish” should not occur. If it nevertheless happens, those articles should be retracted.

The unwillingness of the journal’s chief editor to take this matter seriously reveals a serious lack of concern with the truth. That should be unacceptable to Wiley as well as the American Society of Criminology.

I am just so very, very disappointed.

P.S. I do not have any solutions to the systemic problems here, but improvements should be easy. Criminology as a field has to improve in terms of making data available with full documentation and reproducible code. That would make errors detectable sooner.