🔈This is the transcript of the Open Update. Find the original audio on Anchor.fm.
[00:00:00] Chris Hartgerink: Hi and welcome to the Open update. For Liberate Science, I'm Chris Hartgerink and I'm joined by co-host Sarahanne.
[00:00:06] Sarahanne Field: Hello.
[00:00:07] Chris Hartgerink: In this season of the Open Update, we talk about power imbalances in research, and we've been talking about many different topics over the past 15 weeks .
[00:00:17] Chris Hartgerink: We wanted to take a bit of a different approach to not just talk about the power imbalances that exist already, but also to imagine alternatives and to think about, well, how could we even start addressing some of these power imbalances.
[00:00:32] Chris Hartgerink: Today, we would like to chat specifically about how an alternative peer review system might look, or how alternative forms of peer review, might look and what questions come up, what forms might they take? Do we even want to call that peer review and to simply have the space and create the space to explore these alternatives without the pressure of coming up with a final proposal that we can implement after this episode.
[00:01:00] Sarahanne Field: We're so often absorbed in these discussions about what goes wrong and what you know isn't working properly and what we hate about the peer review system, and everyone complains about the peer review system right as it currently is, and yet viable alternatives don't really, no one's really come up with something that most people seem to think, yeah, you know, that might really work.
[00:01:23] Sarahanne Field: So I like the idea of sort of just, Spit balling a little bit and thinking about some alternatives and sort of dreaming up ideals. That's, that's kind of cool.
[00:01:32] Chris Hartgerink: What I really like, if I look back on the past five years or so, that a lot has changed in review, in the review landscape. Before it was always, preprints, non-peer reviewed, and by now we're in a landscape where reviewing preprints is becoming more and more accepted and, hmm, dare I say, standard.
[00:01:56] Chris Hartgerink: So of course we have the PREReview initiative, which has really put this on the map, uh, reviewing of preprints and then also Peer Community In has gotten much bigger where researchers come and they simply say, Hey, please review this Preprint. And it's completely outside of the journal landscape.
[00:02:16] Chris Hartgerink: But this is still the same peer review system and we already know from science and technology studies and meta research that peer review is creating a lot of problems. Like gatekeeping, publication bias, just peer reviewer two, who gives harsh criticism that isn't really enjoyable nor constructive.
[00:02:37] Chris Hartgerink: And so what would we need to think about to create an alternative?
[00:02:41] Sarahanne Field: I've had a pretty good experience or track record with peer reviews. I haven't had too many experiences where I've really seen, in practice, the system falling apart at the seams. One thing that always strikes me is that it's very difficult to take the peer review system itself, outside of the broader system of academic publishing and the academic culture within which it's situated.
[00:03:09] Sarahanne Field: But I wonder if we kept the peer review part and somehow were able to separate it from the negative and toxic culture that it's situated in, if there was a, there would be a way to make that work.
[00:03:20] Sarahanne Field: And so I think if it was all completely open, The peer review system may be a little bit better, but again, the toxic culture of academia itself, that's, that to me is the bigger problem.
[00:03:35] Chris Hartgerink: You know this question of does the peer review system as it is actually work, there's also very little evidence for that, and it's always preventing changes in that sense.
[00:03:44] Chris Hartgerink: Because people say, well, we need evidence for the changes that we're trying to make. One of the things that I really would like to see in an alternative, Is also to reduce the scope of the review, because if you would send me a manuscript, Sarahanne, you wouldn't ask me necessarily to check every single thing.
[00:04:03] Chris Hartgerink: You would maybe ask me to look at the methods or you would maybe ask me to look at the discussion, whether that's going deep enough and I feel like with just inviting one or two people, which is already a very closed process so there's also space to open it up, that there is space to say, well focus on what you can give feedback on the most and go in that direction.
[00:04:26] Chris Hartgerink: And then also that highlights that it's about giving feedback . I know reviewing might sound like feedback for a lot of people, but for me the connotation of the current system is very much trying to criticize and poke holes into a manuscript instead of providing feedback for the authors to do something with.
[00:04:44] Sarahanne Field: We're colleagues, giving colleagues feedback. And I love that idea in principle, but yeah, indeed, it gets completely skewed.
[00:04:51] Sarahanne Field: I'm interested in what your take is on the peer communities in , initiative because I like that as an idea. I like the idea of taking peer review out of the publisher's hands and that being a community sourced activity and then giving it to the publisher.
[00:05:11] Sarahanne Field: What are your thoughts on that?
[00:05:14] Chris Hartgerink: Peer community in, I really like that they take away all the reviewing from the journals because it's pretty much we're doing it anyway ourselves, so why not do it outside of the journal space in any case.
[00:05:26] Chris Hartgerink: I think in that sense, there's two limitations to that model that I personally would still like to address.
[00:05:32] Chris Hartgerink: That's the fact that, one, the reviews happen outside of the journal, but then you can carry them with you to a journal. Again, and then the journal doesn't have to do the review work. I still see this as a subsidy to the journal system. People are volunteering their time. And then who benefits in the end? It's a journal. Of course, there's a bit more nuance in there because not all of these journals are closed access and subscription based. But anyway.
[00:05:57] Chris Hartgerink: Then the other point is that it is literally taking peer review at journals and putting it outside of it, but It doesn't so much ask the question, how could we do this differently or completely. And peer community in is changing there with peer community and registered reports. I think that's already moving in the right direction. They're innovating in terms of like scheduled reviews, which is good. So everything goes a bit faster. Technically, you could submit anything that has a DOI for review, so that could also be a dataset. It doesn't need to be a preprint, or it could be some code if you share that in a repository. I haven't seen it happen, but I think that's my main gripe. It's this idea of what is the feedbackable unit.
[00:06:43] Chris Hartgerink: That sounds like a very weird word. If I say it 10 times, it'll probably sound even weirder. Feedbackable.
[00:06:50] Chris Hartgerink: But in any case, so this question of what is the feedbackable unit?
[00:06:54] Chris Hartgerink: And I think in that sense, that's really where my issue lies because I think of something like a YouTube community, right? There's a whole culture around, commenting on people's videos to help improve their quality. Of course, also a lot of toxicity just as we know from peer review. And it's this question of how could you, how could we create. A community driven exchange of feedback and also appreciation that goes way beyond what we have now.
[00:07:26] Chris Hartgerink: I would love to be in a space where, you could get a comment from somebody and it would actually get you a bit excited instead of, oh, here's a reviewer reports again.
[00:07:41] Chris Hartgerink: If you would sit down and just write out or imagine a. What's a world in 20 years might look like? No holds barred, right? Anything goes. What would that be? And then to start thinking, well, what would we need to do to get there?
[00:07:58] Sarahanne Field: I would love to just get rid of journals entirely.
[00:08:02] Chris Hartgerink: Is peer review inherently linked to journals for you?
[00:08:07] Sarahanne Field: No, I don't think so, but I think that, It makes sense to have a journal overseeing the whole process as a quality control mechanism. I can see that, I can see the idea of someone overseeing that, but I don't think it has to be.
[00:08:24] Sarahanne Field: One thing that, that it, I just wonder is if we completely rely on people from the community just finding reviews or putting the hand up for reviews, would we be able to get enough reviews or would we be in a situation where we're just, there's a whole bunch of self-selection within reviewers and the pool gets small and homogenous because I think with one thing that journals do is look for reviewers and maybe being seeked out as a reviewer, you may not have put your hand up at the outset to do it, but now you're being asked to think, okay, maybe I'll do it. And I wonder if that helps maybe a little bit. Look, there are obviously other issues with journals selecting reviewers.
[00:09:09] Sarahanne Field: I dunno, I'm not convinced that just community sourcing peer reviewers is overall the best idea.
[00:09:17] Chris Hartgerink: We're getting into it now and so this question of, okay, well journals are sourcing reviewers. Why do they need to source it? I know from my own experience that sourcing reviewers is a ton of work. It's very difficult. People getting people to commit it, it's very difficult.
[00:09:33] Chris Hartgerink: Also because it's the scale of the work. But then it becomes this question, how could we indeed make it such that people sharing their feedback in a generous way is something that is feasible and you can create this gift economy of people actually doing it.
[00:09:49] Chris Hartgerink: I understand research isn't like YouTube, but we never hear somebody about sourcing commenters for YouTube, except if it's maybe spammers and trolls. How do we create a gift economy where people have the space and the resources to indeed provide that feedback from their own volition to then subsequently also have a way of giving people recognition for providing good feedback.
[00:10:20] Chris Hartgerink: Very much an issue that we have now is that all of that goes into closed rooms. Even on publons you might see how many reviews somebody has done are on all these other services that are coming up, but the quality is much better.
[00:10:33] Chris Hartgerink: I really like the culture that Stack Overflow has created, not even the mechanisms of the technology, but just the culture around providing answers for people and how to get recognition for it.
[00:10:47] Chris Hartgerink: And I feel like there's opportunity for feedback in research to go in not the stack overflow route, but to create its own gift economy that sustains itself instead of.
[00:10:59] Chris Hartgerink: Somebody running around saying, Hey, please review. Oh, please. Please reminder, uh, your review is due. Oh, you didn't submit. I hope you can still submit your review because otherwise our process will be very much delayed.
[00:11:15] Sarahanne Field: I still think that this idea of the gift economy, I. I see a lot of issues with it. I do see that people will be self-selecting that, a subsection of researchers will be often doing reviews just because they see that as their duty to the academic community.
[00:11:31] Sarahanne Field: And there'll be a whole bunch of people, maybe more senior people, for instance, potentially, who just don't feel like they have time for it. And so they never end up doing reviews. So I'm not sure that, I'm not convinced that's the answer either.
[00:11:44] Chris Hartgerink: I give you God powers and you, of course, it's not gonna work perfectly, but it's about
[00:11:49] Sarahanne Field: Oh,
[00:11:49] Chris Hartgerink: and, and, and so.
[00:11:51] Sarahanne Field: oh, I see.
[00:11:52] Chris Hartgerink: Free to sort of like, of course there's gonna be issues and, and, and these things. Yes. But it's about what would be the culture that we want to build and then, we're gonna fail.
[00:12:02] Sarahanne Field: So if everyone thought, for example, the way I do about this, I think it would work really well because I really like helping and giving feedback, but I can also self-regulate and say no when I'm overloaded and give a recommendation to another person to do a review in my, I really like that idea.
[00:12:27] Sarahanne Field: I think that would be really nice. Now I'm actually thinking about it like if, if we did have magical Christmas land.
[00:12:32] Sarahanne Field: This comes back to what I said at the start about the system and the culture in academia. The fact that so many people just feel as though they don't have any time to do this cuz they're so squeezed in other areas.
[00:12:46] Sarahanne Field: I don't think it's necessarily that people are shit. I think that the system and the culture are shit. the first thing to do would be just to fix the culture. To take some of the pressure off researchers to not have this crazy thing where people are trying to teach and research or get out of teaching and try and do research.
[00:13:04] Sarahanne Field: And there's a lot wrong with that, that I think sort of spills over. And influences other aspects. I mean, in an ideal world, it would be nice if we could actually enjoy doing peer review because we felt like we had the time. Like that would be great. I think that would really change things.
[00:13:23] Chris Hartgerink: I remember from my days when I very legally torrented Linux Distros, that they would have these measurements of, how much do you download and how much do you upload? And then it would sort of be a health measure for you.
[00:13:39] Chris Hartgerink: So if you had a one-to-one ratio, that would be good. You're contributing as much as you're taking out.
[00:13:45] Chris Hartgerink: If you had a very low, so for example, 0.1, then you would be, downloading 10 times as much as you were contributing. And if it was 10, you were contributing 10 times as much as you were downloading.
[00:13:57] Chris Hartgerink: Something like this is a very good, health measure. If somebody's publishing in a journal, I don't know, submitting 10 papers every year and whenever they get a request, they say, no, don't have time. I feel like that, As a part of this, this research place that we're in, we should also be able to make visible, and not just have, these people reap the benefits of publishing when they can and then not provide the reviews.
[00:14:21] Chris Hartgerink: I, I bet there's a lot of very high profile researchers who. Who will say no to so many review requests. Of course, they might also get more, but as a result, they might end up doing fewer. It's a bit like when people, talk about donating money. The people with more money end up spending lower percentages of their disposable income, than people with less money, even though, you would expect people to be more giving when they have more.
[00:14:50] Chris Hartgerink: But anyway. That's my idea world maybe.
[00:14:55] Sarahanne Field: This is a little bit depressing because it's not gonna happen, or am I being really negative and cynical?
[00:15:04] Chris Hartgerink: Well, maybe that's also part of the lesson today, right? We're trying to imagine something, but that muscle hasn't even been trained enough to do that in an easy way. That we feel like we have the permission and this, and the mental tools to actually do it. So, you know, maybe also a call on, like practice it more often and share those ideations.
[00:15:26] Chris Hartgerink: And even if nothing comes of it, maybe even just the tidbit will inspire somebody else.
[00:15:31] Sarahanne Field: You've definitely made this an exercise for yourself. In a sense, liberate science and research equals in particular, are you making a dream into reality. Like to, to just saying, I have the freedom to exercise this, this idea and, and bring it into fruition.
[00:15:50] Sarahanne Field: Like I think that's, if anyone could be telling people to do that, it would be you, because you've, you've actually done that.
[00:15:58] Chris Hartgerink: But will people listen?
[00:16:01] Chris Hartgerink: Because will you allow yourself to imagine, because, I'll be frank, it does sound like you're a bit stuck.
[00:16:13] Sarahanne Field: Maybe I am stuck. Without sounding combative, I think it's easy for you to say that though because you are outside of academia, so you are not having to, I mean, you've got a whole bunch of other things to think about, but you are not thinking about teaching for you publications and not the kind of currency that an academic sees them as.
[00:16:32] Sarahanne Field: In a way I am stuck, but I don't find that problematic because I like the idea of trying to help academia from the inside. I mean, you and I have had discussions about this in the past about, whether you think that's possible or whether it's just the system will eventually end up corrupting everyone inside it. I still think that things can be changed despite how I probably seemed a minute ago. I think there is a lot that can be changed.
[00:16:56] Sarahanne Field: And I think the meta science community, the open science reform movement is they're making changes, So maybe I am stuck, but I don't know. I think I'm still in a learning process and I think that's okay. So I don't think I would frame it as being stuck, but I'm learning a lot, for example, from people like you, of how to rethink academia and what it could be.
[00:17:20] Sarahanne Field: I think that's a process, that's a journey I think you started quite a long time ago, but, that's, for me, a journey that's only just beginning and I think that's okay.
[00:17:31] Chris Hartgerink: I ended up doing this, imagining these things while in academia and felt the need as a result to also leave because it felt like, there was such a big discrepancy, between what was there and where I thought it could go.
[00:17:49] Chris Hartgerink: But that's us for this week's open update. We are not resolving any of these issues. This is only a start. We're trying to, tickle you to think about alternative forms of, let's call it peer feedback, to also indicate it's a different, space.
[00:18:06] Chris Hartgerink: You can really open up your thinking and we would love for you to join us in our Signal group and to just throw in an audio message or a few words around what you thought, what resonated with you this episode, what you found disputable, or maybe just some ideas you yourself have.
[00:18:24] Chris Hartgerink: So join us in that journey and thank you for joining us in the journey of the Open Update. Sarahanne, any final thoughts before we send people out?
[00:18:32] Sarahanne Field: I would say that if the community and scientific community were open to really imagining and just letting their minds run wild. That we could come up with a solution for this that might just work. I think that's possible. .
[00:18:48] Chris Hartgerink: So join us again in a few weeks. If you have topics that you would like us to discuss, drop them in the signal group chat as well. Otherwise, have a great week and you'll hear from us soon.
