Anyone for metascience?
Anyone for metascience?
Reasonable People #43: metascience, aka. metaresearch aka research on research, is taking up a lot of my recent attention.
The research ecosystem is a form of collective reasoning. And one that, just like individual reasoning, it is possible to study. That, and the fact that I’m spending increasing amounts of time doing research on research, is my justification for including some of this work here.
But first, just what is metascience, and is it different from other fields which study research, such as science and technology studies (STS)?
Mark Rubin has a great post summarising some thoughts on this topic from Felipe Romero. His argument, as I understand it, is that metascience is defined by the overlap of three areas:
the open research / research transparency movement
research reform activists
traditional science of science (which I think includes STS)
It is the activism - the desire to not just study but reform the research ecosystem - which makes what is now called metascience unique.
For what it is worth, I prefer metaresearch as a term, since the purview is wider than mere metascience, but maybe that ship of nomenclature has sailed.
If this sounds like it might be of interest, read on for some interesting metaresearch which caught my eye recently… In the future I’ll probably restrict metaresearch updates to the METASERV listserv. Join here: jiscmail.ac.uk/METASERV
That’s right, social media is over, and we’re partying like it is 1994 with a new academic mailing list for sharing and discussion of metaresearch. Load up on email subscriptions and bring your friends!
ITEM: Evidence the UK research system responds to the incentives of national institutional evaluation.
More papers submitted just before the REF deadlines, compared to just after, and those papers submitted just before are less highly cited, less diverse and more likely to be retracted.
(Bonus, a nice example of a regression discontinuity design in science policy)
Groen-Xu, M., Bös, G., Teixeira, P. A., Voigt, T., & Knapp, B. (2023). Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework. Research Policy, 52(6), 104729.
ITEM: Mark Rubin has a newsletter you should read
Link:
ITEM: Research funding could be redesigned to be less conservative
Short, lucid, piece by Stephan & Franzoni
Link: Encouraging high-risk high-reward research at NIH
ITEM: Workshop on The Future of Peer Review
Chaired by me, but I don’t say much. Makes clear how wide the space of possible innovations is in this space
Recording: (link)
Slides provided are available here: https://osf.io/ce3ts/
CONFERENCE: AIMOS, 21 to 23 November 2023 Brisbane
AIMOS 2023: The fifth annual conference of the Association for Interdisciplinary Metaresearch & Open Science (AIMOS)
https://agbarnett.github.io/aimos.conference/
TALKS: Meta-research Methods
Seminar series organized by Tracey Weissgerber and Maximillian Frank, funded by Volkswagen Foundation and offered in cooperation with the QUEST Center for Responsible Research (Berlin Institute of Health at Charité), the German Reproducibility Network and Meta-Rep.
YouTube: https://www.youtube.com/@Metareseasrch_methods
ITEM: Open Editors dataset
I've been playing with the Open Editors data - a web scraped dataset of academic journal editors, and their affiliations
https://openeditors.ooir.org/
Like reviewing, journal editing is often under-recognised. Editors have a key role in shoring up research quality, so it seems important to understand how this labour is distributed across the research ecosystem.
Here's a graph of UK universities, showing their average REF score (a crude indicator of research kudos) vs the proportion of editors at that institution
See the link for caveats and a rollover version so you can identify your favourite University: https://tomstafford.github.io/editors/
The thought was to try and identify outliers - are there institutions which are “under-editor-ed”. Currently scoring highly in terms of research kudos, but not part of the academic community in a potentially important way? I’m still working out what I think of this, comments welcome
Link: https://tomstafford.github.io/editors/
And finally…
Classic from Nick Kim
END