The ASC Publication Committee asked for input on policy and process for publication complaints. The background is of course the now retracted papers in Criminology for reasons detailed by Justin Pickett, and the statements published by the ASC, as well as the video from the forum on scientific integrity. I have previously commented upon it here and here.
I submitted the following to the ASC Publication Committee:
Dear ASC publications committee,
First of all, and I am glad to see the ASC taking steps to improve procedures, and I appreciate you giving everyone the opportunity to give input.
One important issue in the recent debates is access to data and the reproducibility of the results. To re-analyse the original data is clearly crucial when there are allegations of research misconduct. At the more general level, when there are such difficulties, then it also becomes clear that the data used in the publications have not sufficiently well documented in the first place. I think this needs to improve.
There are now strong moves towards what is often referred to as “open science”. Obviously, if data were made publicly available in the first place, it is much easier to check the results by others. However, while making data openly available to all is in many respects desirable, it is also very often not possible with the kinds of sensitive data criminologists typically use. But many of the ethos of “open science” are general principles of science, and some minimum measures should be taken even without committing to any specific “open” framework. At the core is the documentation of research procedures, including data collection and data management. The focus should be on such documentation, and I would like to see some minimum standards of such reporting to be implemented for all studies.
Others have probably provided more thorough suggestions, but I think the following could be a good starting point. My suggestions are simple and should not require much additional effort by anyone (neither authors or editors). I suggest that all published quantitative studies should include the following information:
a) Regardless of data sources, there should be a note detailing how others can get access to the same data. If special permissions needs to be obtained, information on where to apply must be provided as well as the main conditions for access. If data cannot be made available to others, then the reason for this must be stated. If data will be made available to others at some later point in time, then information on when and how should be included.
b) When and who collected the data. If a survey company have been hired, there should be some reference to contract or other documentation.
c) If data have been obtained from an existing study (e.g. AddHealth or NLYS) there should be a reference to when and how the data were handed over, including specifications of sub-samples (when relevant). Thus, others should be able to get access to the exact same data.
d) If data have been obtained from administrative records, there should be references to who handed over the data, including dates and permissions etc.
e) Most studies require ethics approvals. Reference to such approvals should always be provided.
f) Reproducible code should be made available for all studies regardless of data availability. This code should at least cover the estimation procedures, but preferably also the entire workflow from raw data to end results. Whether code is stored as supplementary files at the journal or some repository is of no importance as long as it is specified.
These suggestions are primarily relevant for quantitative studies, but some would apply to qualitative studies as well. One should also create similar guidelines appropriate for qualitative studies.
Please recognize that I expect all researchers to be able to provide this information with minimum effort. It is simply providing basic documentation. Indeed, if researchers cannot do so, then journals such as Criminology should not publish the article at all simply because the study is not well documented. I consider this to be a minimum requirement.
I would also like to see journals to make conditional acceptance of articles based pre-registration, but that would require a bit more work on the principles. I consider also pre-registration as a kind of documentation of ideas and plans. I do not think it should be mandatory, only encouraged.
I think Criminology would benefit from this in at least two major ways: 1) Increase the quality of the work published, just by making the studies more reproducible and well documented. 2) Increase the status of the journal, and gaining international reputation for being at the forefront in this development. 3) Increase trust in the results published in the journal.
I should probably have added that if there are complaints regarding errors in the above documentation (which cannot be fixed within a couple of weeks or so), retraction should be considered based on that alone.
I could have referred to e.g. the Open Science Framework (which is great), and others have probably written more thoroughly on such issues. But I think such documentation is so basic that it is embarrassing it is not already standard requirements.