🔈This is the transcript of the Open Update. Find the original audio on Anchor.fm.
[00:00:00] Chris Hartgerink: Hello, and welcome to the Open Update. I'm Chris Hartgerink for Liberate Science, joined by Sarahanne Field, actually in person today.
[00:00:07] Sarahanne Field: In Berlin.
[00:00:09] Chris Hartgerink: We're in a recording studio right now for the first time since we started co-hosting. So that's, that's something else. Maybe you notice the dynamics change a bit from how it is usually, hopefully for the better.
Today we wanted to talk about various things. This might end up being a series or not, depending on how this conversation goes, but we've been talking about. We've been talking about various specific themes over the past few months, but we can talk about them also in a historical context because for me, I don't know about you Sarahanne, but I've been around for, I think around a decade in "open research."
How long has it been for you?
[00:00:48] Sarahanne Field: My first research project that was meta-scientific, I started at the start of 2014, so close to a decade. Yeah.
[00:00:56] Chris Hartgerink: We're gonna combine some topics and take a recent history. We're not that old of course, but a recent history perspective to understand where things have gone over the past decade ish, and also how that might inform us for the next decade because, you know, it's the year of open science in the us There's been a lot of fuss about that, but what does it mean in a larger sense to have this year of open science? I wouldn't have imagined this 10 years ago. It felt like it was very fringe still, and now it's very accepted.
We wanted to start out with talking about anonymous review and double anonymous review because when we talk about open science very often, it ends up very quickly going to just, how do we do peer review. We're gonna be specifically talking about anonymous review because blind review is insensitive to say it that way and anonymous review is just as clear, so it's completely unnecessary to call it anything else.
I'm just gonna look to you Sarahanne, anonymous review, yes or no?
[00:02:01] Sarahanne Field: No,
[00:02:03] Chris Hartgerink: I got a very clear answer.
[00:02:06] Sarahanne Field: sorry, you wanted nuance from me.
[00:02:08] Chris Hartgerink: Yes.
[00:02:11] Sarahanne Field: I think that my perspective is somewhat shaped by the more philosophical approach I take to science. So I'm someone who leans on the side of constructivism. I'm not someone that really thinks there's necessarily, an objective truth with, with a lot of things. I like the negotiation of certain things and the idea that things can be constructed and that they change as the context changes.
In relation to that. I think that when it comes to anonymous review, I'm not of the opinion that we're able to really circumvent our biases. So the idea that we can try to do that using anonymity in the context of peer review, I think that lulls us into a bit of a false sense of security, especially when you have a smaller field.
In research where, you know, a lot of the people in your field, you know, how they write, you know, the kind of research they're working on. The whole idea of anonymity, it isn't really applicable anymore and so then you have this sort of false anonymity, or a pseudo anonymity. And I, I find that problematic.
I would much rather have everything open and then things that need to be declared are, are kind of out there. And I recognize that that comes with the problems that you, that you would imagine that there are some people that get discriminated as a result, but I think I would prefer to have open discussions.
That's how, for me, research should go rather than keeping everything behind closed doors.
[00:03:39] Chris Hartgerink: When you say those problems, you mentioned discrimination. I can imagine for example, just career stage or gender discrimination or race discrimination. Are there any other things that are on top of your mind as you say this?
[00:03:56] Sarahanne Field: I think for me, you know, as someone who's an ECR (Early-Career Researcher) I think largely for me, those problems have to do with the stage of career you're at. When I've done a review for someone who's a much more senior researcher than me, signing my review is, is not always easy because I'm concerned that they won't take what I'm saying seriously because they are more senior.
So that, that's the kind of thing I'm thinking along the lines of. I am privileged to be in the field that I'm in because I think in the field I'm in, which is meta science, ECRs are generally valued. They do a lot of the labor. So I think I'm very lucky, I'm privileged to be able to say this, that I'm happy to sign all of my reviews because my field is like that.
But for certain fields that are very hierarchical, biomedicine for example, is a quite a, a difficult field in that respect. ECRs in that kind of field are gonna have a lot more trouble, and so I'm, I'm definitely sensitive to the problems that they would come across.
[00:04:55] Chris Hartgerink: Also from the perspective of like, how was it 10 years ago? Regardless of my own opinion at this very time, it feels like open peer review was a very radical position in 2013. People would agree with it, but there wouldn't be much practice simply because it's, it was "difficult." Of course, you could always sign, but it was not, as far as I know, not very common.
One of the things that I notice is that it's become much more accepted to sign your name. More journals have set guidelines saying that they explicitly ask whether they can attribute. This aspect of retribution, I find it very interesting because I've had one experience myself, and if you actually start reading some of these open peer reviews that people get very nasty despite it being open review. Sometimes the argument, oh, people will be nicer because, you know, their comments will be made public and with name and a lot of people just don't really register this. But the flip side being, if you have anonymity indeed, this point of can you actually truly be "anonymous", especially if it's such a specific area, or such a specific way of writing.
And I feel like they've been trying to circumnavigate this a bit by providing sort of reviewer forms instead of freeform, free text, reviews.
We've been talking now about "simple anonymity." Simple anonymous review. But then in the clinical field or in the biomedical field, there's double anonymous where even the editor doesn't know who is being reviewed from this idea of double anonymous trials where even the clinician doesn't know the condition.
I mean the design of this idea sounds very sound, sounds very sound, but it feels completely not practical to me. In a clinical trial, this is like very closely monitored to ensure that there's no indication that the, the pills are exactly the same. The bottles are exactly the same, they taste similarly. So people can't figure this out. And I feel like this just, it's a nice idea, but it really isn't possible in research.
[00:07:18] Sarahanne Field: It's the thing, I think how possible is this? So you may be able to create a reviewer form, and that can take away a lot of these issues. However, we have topics of research. So say we're talking about the paper itself being reviewed, right? And these double anonymous setups are also supposed to be protecting the author of the paper.
If that paper is written in such a way, as to indicate the author, I mean, I know what the people in Tilburg or what the people in Utrecht are doing that, the open research groups there, I know what they're doing. And so even if you were to be able to hide the identity of the reviewer, being able to hide the identity of the author of the paper in the first place. Again, the other party that's supposed to be being protected, I think that's where you can really run into some problems.
So how possible is it? and I don't see that, you know, over the time that you and I have been in meta research that that has improved at all. That, that the possibilities have have changed there.
I don't see that.
[00:08:19] Chris Hartgerink: In that time, I have noticed a change in how people talk about this because let's say 2015 it was in my experience it was very much open peer review. Yes, do it. There are no, nothing that really blocks us from doing it. And of course there will be some "collateral damage," but in the grand scheme of things that's worth it.
I'm very glad that that is no longer the sentiment, that there's much more nuance in it and people recognize there are limitations to it and that it's also okay that. There's groups of people or just individuals or journals where they say, "okay, we're gonna try and be anonymous, even if it's a fault. Like if it's not perfect, that's okay, we just wanna make sure we're not gonna share all these reviews with the names on the website." And then there's other journals that say, well, no, we're just gonna do all of that. When people review, that's what they agree to others, it's an opt-in. So I really enjoy that there's this, this diversity of options.
At the same time, I also noticed that even though it's been around for 10 years, all of this infrastructure has popped up, people can register their reviews with different services. Journals are sharing those reviews. They're getting a DOIs, so they can be cited. But I also noticed that I had this recently with the Royal Society for Open Science, where I reviewed several papers and they said, we will add the reviews to the paper. Paper came out, click the link, and it was like, can't find the, the information. So there's also just almost these very basic things by very big organizations like Clarivate who offer this service that are just not really working.
Regardless of the principle debate around should we be doing this? Could we be doing this? How should we do this? Then subsequently the implementation feels sometimes very, amateuristic still. So yes, we've come a long way in the past decade, but at the same time what is on offer to me isn't really it.
[00:10:29] Sarahanne Field: Couple of thoughts. I love seeing the nuance coming in. Part of the things that I do is I study this research reform movement, if you could call it that. And one thing that I see is, is a maturation of the movement. You see a little bit more nuance. So in the beginning, when I first started in meta meta research, it was, yeah, the idea was that everyone should be signing all their reviews. That would fix all the problems, and now it's sort of we're coming into a, a place where we're actually debating, you know, under which cases that is best. I probably have a bit of a binary view on this.
I edit for two journals and I do recommending for PCI RR. And so I think for me, this is a little bit exaggerated because I see so many papers from my field, as a reviewer. I do loads of reviews. I do these editing work, and I also obviously write my own papers.
So I have a pretty good handle of the literature in my field, and I think the average researcher probably has that a little less. I have this really strong sense of who does what and who's submitting what to papers and that kind of thing. And, and maybe. Most people have, you know, less of a sense in their field.
But back to my, my point is just that I see a lot of this, this nuance, this maturation. So I think a lot of it's really quite good. But implementation, I think that's always an issue, right? There's a, there's a mismatch between the theory and practice of something. So I think although we might have matured a little bit and added nuance and complexity to the discussions, what actually washes out in practice is, is a little bit different, and it doesn't always work.
[00:12:06] Chris Hartgerink: This is a very interesting counterfactual in that sense because yes, we've gotten more nuance in the opinions in the various communities, but on the flip side, the implementations have become much more concentrated. Of course, technically if you write a review, you can upload it yourself somewhere, whether that is according to author guidelines or something else.
But just like these submission systems, they're very concentrated. There's like, scholar One or Manuscript Central? I think it is the same product and it might be integrated into that, PeerJ has their own system, PLOS has their system, even though that might also be scholar one.
And it's this question of, you know, how many of these ideas actually get translated into practical options because, we might have all these ideas and not making reviews public is very easy because you don't need any infrastructure. But then the flip side of that is if you want to do a very specific form, you need to implement it.
I know PCI RR is doing a lot of implementation work. We've done that with ResearchEquals a bit as well. But there's this question of is the review being made openly available? Where the review is happening, what additional work does it cost? Does it get registered as a bit of your career credit, so to speak, if that's relevant, of course for you, you should have the option to say, that's not relevant. For example, for me, it's no longer relevant. I just do it for myself.
But there's this, this counterfactual of, okay, the means to realize our world are very concentrated, but the ideas are more dispersed. I've said "ideas come cheap" before, and I think here, this counterfactual justice is very, very present.
[00:13:59] Sarahanne Field: I think we see a lot of that in, in the science or the research reform movement, the open research movement. There's so much discourse, there's so much debate and so many suggestions about what can be better. The conversion into action. For most things, probably the conversion into action, there's a little bit of a, an off ratio there between the talk and the action.
I mean, at the same time, you know, you and I have seen so many huge changes in open research in the last decade, right? So I'm not, I'm not saying there's not action being done, but I do think that we're seeing a little bit of what you're talking about.
[00:14:38] Chris Hartgerink: I think that's a very good point because the flip side of that also is, is those ideas that have been converted, like we've seen in the past decade, a lot of fantastic initiatives get acquired by the established publishers.
Just to name a few. Very recently it was Protocols io, Springer Nature. Publons got bought Springer Nature. the original Altmetrics organization plum Analytics, was bought by Elsevier. We can keep going with this list, but it's very often a small organization being acquired by a already bigger organization.
And so this concentration of, let's say the means to create change have become more concentrated as well. Of course, there's still very good independent projects or you know, welcome trust funds a lot of stuff. eLife is still independent thank goodness, to a certain degree , because they also get funds, of course from stuff, but not from the, from the big publishers. PeerJ is independent, but there's also so many fantastic projects that got bought out.
And so it becomes this question of, you know, I know we started with anonymous review and now we're at this stage of, in the past 10 years, the means to realize change have become more concentrated. It feels, of course, we're speculating here, despite that progress in the past decade, if I say this, it makes me feel like the next decade might actually be stalling.
[00:16:12] Sarahanne Field: When I was interviewing for the faculty position I have now, I was talking to the hiring committee about exactly this issue. I was saying that I'm interested in continuing my research into the, the open research or research reform movement, and they had questions about, "okay, what if this movement dies? Then what do you study?"
I've had people have this concern before, when I say I'm, I'm in the meta research field. They say, well, when everything gets fixed, will that still be a field? And what you are saying, kind of, I think awaken some concerns of mine that are largely in the background for the most part. What if this movement, like other social movements has a death? I mean, there tends to be a lifecycle for social movements and there's a reason for that, but it, it does make me wonder, if we will see a decline of this movement because it, there's a, there's a lot of building up, but if there isn't change being created, then what does that mean for say the next decade?
[00:17:25] Chris Hartgerink: These are some perfect questions that we can segue into in the next episode. But for for now, we're gonna pause the discussion there.
And just to let you know, we still have our Signal group, which you can join. Don't forget to join us if you, you have thoughts on these topics because we really want to have this conversation, not just among ourselves, but with you.
Of course, podcasts , it takes a long time to get an audience, so we really do appreciate every single piece of engagement that we get. It acknowledges that, that we exist in a way and we'll continue this topic in two weeks. Today we've talked a bit about anonymous review, double anonymous review, and how that has changed over the past decade.
And then next time we will continue on this track of how do we actually create change? Has change happened, is it stagnating? And what does it mean for research fields like meta research or social movements? Do we still have those if we "fix everything"?
?unique=f469942)