This is a transcript of the interview with Brian Nosek (Season 2, Episode 1). Listen to the podcast here. The transcript is slightly edited for readability and available under a CC0 Public Domain Dedication.
Chris Hartgerink: Welcome to the second season of the Open Update. For Liberate Science - I'm your host, Chris Hartgerink.
In this season we interviewed 10 guests over the course of nine weeks, about the UNESCO Recommendation on Open Science. We talk about what it means to them, their work and the future of research, so that we can also better understand what it means to us, our work and our future in research.
What new ways of being lie in store for us. And I encourage you to find agreements and disagreements throughout the series and leave voice messages with your thoughts — like you would share thoughts with a friend. Find how in the show notes, and don't be shy. Don't be a stranger. Your voice matters.
Today we're joined by Brian Nosek, director of the Center for Open Science and professor of psychology at the University of Virginia.
In doing research for the podcast, I found out he was an undergraduate in computer science before switching to psychology. And am I glad that he made the switch. Who knows whether the influential projects Brian co-founded in the past decades would have even existed. Projects like the Open Science Framework, the Society for Improvement of Psychological Science and Project Implicit.
The exception may be the Reproducibility Project Cancer Biology, because it has neither anything to do with computer science or psychological science. Regardless, all of these projects together have influenced thousands of people and researchers around the world. He is here today to share how his vision for a better science has evolved, especially in light of the recent UNESCO recommendation on open science.
Brian welcome to the Open Update.
Brian Nosek: Thank you for having me, Chris.
Chris Hartgerink: One question we'd like to ask everyone throughout this series is what is a low hanging fruit of the UNESCO Recommendation on Open Science?And what do you think is going to be much more difficult to realize?
Brian Nosek: The low hanging fruit, I think, is the normative power that the recommendations have to make a lot of these conversations easier. One of the big challenges of advancing open science is that it is in opposition in a way or another to the system as it currently exists. And so the nature of conversation of change of adopting new practice of shifting policies requires bridging something between the current state and an ideal "state" that necessarily has friction.
Systems tend to persist through inertia, or because people want to rationalize the status quo, or because change is hard and there's just lots of work that needs to be done to realize that change, or because the infrastructures and tools and knowledge doesn't exist and isn't disseminated.
So for me, the emergence of the UNESCO recommendations is the strength of the cumulative efforts of so many in the community over the last dozen plus years to articulate what the value proposition is, what the need is. And then the manifestation of that in a document that has that power of collaborative action that UNESCO represents.
It is very well informed about what the open science potential is and what that means translated into practice. A collective document from such a high credibility source, I think is just going to make a lot of these conversations, dozens hundreds, thousands of people are having in their department, in their institution, in their collaborations, a lot easier to have, and we'll reduce that friction in the aggregate.
Chris Hartgerink: I do want to push a bit more on what a difficult thing to realize is because you say, okay, it's really this culmination of work that's been going on for quite a long time. It's written in a very good language. It covers almost all the bases, and it sets this norm. In realizing those norms, what do you think is going to be really a key thing to having the UNESCO recommendation, not in 10 years, be another recommendation that sits somewhere on a shelf? But that it's actually something where we go "that was the impact it had"?
Brian Nosek: One answer is that the project of advancing open science and shifting the research culture does not live or die based on the UNESCO recommendation.
As much as I think it is an important document and statement from the participating nations and organizations, this movement will occur, regardless. It's just, this is going to be one additional lever of support, of persuasion, that can be used to keep it going. So in that sense, I would say that it doesn't matter.
That's too strongly worded. It does matter! Of course it matters. I guess the more pointed response to the phrasing of the question is that it should not either give people pause of "oh, I guess I don't need to do my part of collective action work anymore because now UNESCO said it" this doesn't end any of that.
If anything it says "here, now you have a new tool in your toolbox to bring to the table when you're doing whatever it is, your actions are." It is a facilitator of the existing efforts. And ideally we'll expand the community of reformers in those efforts by raising awareness, providing common language.
And providing some pretty specific behavioral recommendations of what it should look like if all of this works. That alignment work is really the key contribution, I think. And the rest of it, of translating that into action is what the reform movement is doing and has to just keep doing with this additional resource.
Chris Hartgerink: I hear you say that it's a very important document and it brings together a lot of people and that translation, do I understand correctly that translation is going to be the difficult part because many people will translate it also in their own ways and their own situations?
Brian Nosek: Yeah. And that is what's most impressive to me about the language of the document, that it manages to be quite concrete in "here are things that need to happen" while also representing pluralism on the value proposition. What is open science for? It's for increasing reproducibility and rigor?
No, wait, what's open science for? It's increasing democracy. What's open science for? It's increasing access and equity, right? There's lots of different value propositions. Meaningful, not mutually exclusive, but also different areas of interest in advancing open science. And it leans into that.
It doesn't say this is the one thing and why we're doing it. It says, here are the different reasons that we would care about open science. And here's how some of these behaviors contribute to meeting those objectives. If there was like, here is the answer and this answer works in all contexts for all people and all situations, then the document could say that, but of course that's not how it is.
We know how hard it is to translate this stuff into practice and even something like open data. Is there a consensus, sort of statement of what that means? What data are you talking about? Which level of analysis are you talking about?
What is it FAIR? What does FAIR mean? It's all of this stuff, it's complicated. And so if UNESCO try to impose a standard that is so specific that you wouldn't see diversity in implementation, then it would not succeed. It would fail even faster then a version of it that is totally ambiguous where it's just "openness is a good thing."
From my perspective, it threads the needle really well of providing direction, but not imposing standards that are not yet tested, not yet known for how they are getting implemented.
The key thing that is the biggest barrier is getting started. And once the community starts saying look, just take data sharing as a specific example, once the community starts doing that, all kinds of problems emerge. Oh "I don't understand the data" or they said "the data is not available, but it isn't available or it wasn't preserved."
All of those things will always emerge in new behaviors, new practices, but once you've started, then those conversations can happen. Then that work can get done. That it's more then about refining the behavior rather than getting people to do the behavior at all.
Chris Hartgerink: This is immediately related to that inertia you spoke about
Brian Nosek: Exactly. So it's I've never shared data but now I've shared data. Okay. But then you email me and say, yeah, this data makes no sense. I think the mistake that we make as idealists in reform is that we want it all to be right, right away.
Chris Hartgerink: Has that process influenced how the Center for Open Science is gonna move forward or maybe also in a more general sense, has the UNESCO Recommendation made you reevaluate certain of your own personal objectives?
Brian Nosek: Yeah. I think what the document does well is represent so many different parts of the openness movement in terms of activity like organizationally, we spend very little effort on open access per se, right?
We offer preprint services and obviously we promote Open Access, Green in particular. But the documents spends a lot more engagement with that. And in that sense is reflecting what has been the lion's share of the effort in the academic openness community in promoting open access.
So it certainly reaches out beyond a lot of the things that we have, we are resourced to do specifically. Also, we're just one organization of many, we don't need to try to cover everything that is in that document. And I don't think it's productive for us to try to do that because there's a lot of work to do and it needs to be a lot of actors, organizations, individuals and otherwise that are contributing to that. So for us, it's oh, let's play to our strengths. And a lot of our strengths are represented there. That's really awesome.
Chris Hartgerink: Yeah, it's very easy to sometimes feel the way to some of these big societal issues on your shoulders.
I know that a lot of people have this throughout the years. Maybe they leave academia as a result of it. And I think that what you say is also a great reminder: the UNESCO recommendation is not just a guiding tool into the future. It's also a reflection of all the work that has been done and almost a celebration of that work.
I have one final question on this topic of specifically how reproducibility within the recommendation comes forward, how the strategy has been affirmed or shifted. And I think one of the questions that I want to ask specifically is: there are people who say that the reproducibility movement has also pushed for fairly homogenous forms of doing research, elevating preregistration over a more diverse set of research approaches. How do you feel about these comments, these criticisms, and how can we provide more space to different ways of knowing?
Brian Nosek: Great question. So the idea of diversity in how we establish credibility of findings or reproducibility of findings is very important.
There is no singular silver bullet solution to how to do good research. We're never going to be done on how do we innovate in terms of methods and practices to make our research better. Preregistration is one tool in a toolbox of many things. Many of those things, not yet discovered of ways to do research, better to get to how we know things.
So I expect and hope that not only will be continuing critique of any new practice preregistration or otherwise. With evaluation to see how it can be maximized in terms of its benefit, how its costs can be mitigated and how its appropriate use can be determined. But also that we keep advancing a lot of other interesting approaches and solutions .
One that I think is particularly useful that doesn't get the same degree of attention is blind analysis. Parallel with preregistration where, you have access to the data set, but it's been scrambled in some way. That's not meaningful, but you can at least do all the work to set up the analysis pipeline to even address things like outliers and exclusion rules without being exposed to the hypothesis.
And then when you finalize that, then you peel away the randomization and you get your outcome. So when you're peeling away, that's effectively the preregistration in the abstract concept, but you can apply that model of blinded analysis in ways that, that move outside of the standard workflow of how pre registrations tend to work.
And that's a really effective and interesting way to do research, and is in practice in some areas, quite common in some areas like astronomy. Innovation in, how is it that we get to knowing most effectively is I hope accelerating in excitement. And one of the things that a new practice like preregistration can do when it gets a lot of attention and engagement is to actually facilitate that innovation and alternatives. That's very productive. That's what scholarship is supposed to be about.
Chris Hartgerink: I was just about to say, this is such a scientists' response also to be curious about what does and doesn't work. And I think that indeed also what I get from this, what you're saying is: we're on a continuing journey. Preregistration is not the destination. It's it's not the panacea.
So keep going at it and keep reflecting on where you are.
I want to switch it up a bit: Science always takes place in society. And I wonder, since your scientific utopia that you wrote the papers back in 2012, a lot has happened in society. I wonder how the national or world events in the past decade have affected your perspectives on scientific utopia.
We know that the Center for Open Science is in Charlottesville. A major event happened there. I remember just after we had a conference where we sat down, the Unite the Right rally happened, but there's also bigger movements around MeToo, and around police brutality.
Do these play into the UNESCO recommendation, open science more in general and how you see science improving into the future?
Brian Nosek: It's a very good question. There are interesting strands that are commonalities with some of the challenges that have emerged misinformation, disinformation being another in how do we know what to trust, who to trust, how institutions be trustworthy and not.
Is democracy going to continue in places like the United States? The thing that sticks out in there's many ways that we could talk about, especially on equity and justice lens, but a particular thing that sticks out is the limits of complete openness. A lot of the advocacy in some of the movements toward openness presents an idealized form of just, if everything is completely open, then it'll all be fine.
Because people just do what they want to, certainly extreme on the libertarian perspective of people just do their thing and the market will work it out. And, like the early versions of Reddit where there was all kinds of subreddits that were really, the bottom of the humanity barrel. Then the problems that have emerged in Facebook and Twitter and other social media areas where the initial organizational response from the leadership of of the social media giants has been "it's free expression. We're just giving people a platform to say what they think."
What we've had is an accumulated understanding over the past several years is that may not completely work because the structure of some of those systems may facilitate bad actors in a way that really actually ended up interfering with equity, justice, dealing with accuracy, factual information and otherwise. So there was some evolution in thinking about what the responsibility of those platforms are. So what has changed? So to make that more concrete, what is a challenge from my perspective?
Origins of thinking about COS and particularly infrastructure, Open Science Framework that we provide, is where is it that moderation is needed. And so I spend a lot of time now looking more at what is it that we need to be thinking about? Long-term for OSF as a platform for sharing all resources. To manage that in the same way.
We're not there yet. We have 450,000 users. It's not like it's a trivial number of users. But it isn't a central hub of information globally that we have the bad actors flocking to it in the same way. But that may come. We need to have some kind of framework that is promoting openness, right?
Maintaining those core values, but developing some standards that are not, if Brian doesn't like this it's off the project, or if COS doesn't like this. No one should be in charge of the information in science. So how do we create platforms that support, bring exchange, debate, even sometimes pretty tough debate, and make sure that it is stays within scholary boundaries. And that gets to a lot of the hostility part that has emerged as part of the Reproducibility movement. Openness movement is that sometimes it's not just, "I don't like your research", it's "I don't like you." And a lot of things that come on in terms of hostility towards persons. So this is a wandering answer to your question, but that really, to me is the core where I translate it to.
What are our responsibilities and our opportunities to learn from some of the challenges that are happening societally and to how we try to manifest and advance the values of openness, but all the values of openness transparency. Inclusivity engagement, equity, justice.
Chris Hartgerink: Yeah, there's a lot in there. I hear you say openness is not an end. It's a means. In your 2012 paper, you say "publishing should be trivial." And in essence, here's a sort of like a nuance—maybe we also need a bit of moderation. And then the question, how do we do that? It comes to comes through the play.
So I think meandering thoughts meandering answers are exactly fantastic for these broad questions, because there's so much to cover wherever at a conference together. We can talk more about this.
Brian Nosek: Yeah, it's one of my favorite topics these days is trying to just get some handholds on a framework for thinking about those things.
Chris Hartgerink: And I know that the Open Science Framework has gone through many developments on this. So it'll be very interesting to follow it. And. Just as a note to our listeners also, if in five to 10 years there is more power in your hands. We'll you have the audio clip now that you didn't want it.
To start wrapping up the interview, I want to move to towards a bit of a lighter subject, and I want to play a short audio clip before I ask you another question.
I'm Haven. I am Joni. And we are part of the breakfast bunch. If you don't know what that it's a little sort of mission that we're doing. Us and our dad and our mom. And basically we've gone around and tried over 50 different breakfast places in Charlottesville and we have rated them.
And we have basically just taste tested a bunch of different foods.
Chris Hartgerink: So Brian, your kids seem to really have taken over your YouTube channel and I came across this this video of the breakfast bunch. And I've seen you share a bit about this on social media as well. I think it's just lovely to see you're going out with your kids, measuring things, evaluating systematically, and it made me think of whether you can share a bit more about how you teach your kids about science and what you have learned from your kids to take back into how we improve science as a whole.
Brian Nosek: Yeah. Breakfast bunch is the manifestation of a few different activities that we have as a family. And it's, it reflects my personality in part but also has just turned into a very fun way to engage with the kids on the question you raised earlier, how do we know.
And my Haven and Joni are 15 and 12 now, but for the last four years, we have been going to restaurants on every once, every couple of weeks for breakfast all over Charlottesville. And we set up rating criteria at the start. We don't want to just rate on taste. What are the dimensions on which we decide whether we like a restaurant or not?
So we talked about that and then we said, okay how do we decide. If we like it or not, how should we rate them? And so we set rating scales for these, and then how do we decide which ones are the best or the worst and what if we differ? And how do you think about that? And so we had conversations about reliability and about aggregating across different raters and then what the ratings mean.
We're now into test retest reliability. We're going back to some of these 50 restaurants, rating them again and comparing, "oh, wow. This is so interesting. This was your favorite now it's your least favorite?" So the process of doing the ratings has just excited conversations about about things that I care about a lot, but it was in a way that engages with.
On starting from things that they care about breakfast, and pulling them into things that they found exciting and interesting, like how you rate stuff and how you decide what things are good or bad or otherwise. And this has caught hold so much for them. That my older daughter, Haven for one of her birthdays, organized a blind taste test of ice creams for the kids that came over. And so we always set up the rating scales. We set up the Google spreadsheet and went and bought all the icecream, have a long conversation in the frozen food section or at the grocery store. "Should we buy all the same flavor of ice cream from different brands should we buy within a brand different flavors?"
And then "should we compare by price to see if it's really worth it for the price? " We're just, we're talking about measurement, we're talking about validity of scales. We're talking about how do you translate stuff into what the participant in these cases would be able to report?
How do we visualize what it is we're learning from it. And. It's just fun really. And that's really the goal is we're just having fun together doing something that is interesting. And then we learn stuff.
Chris Hartgerink: And that brings us full circle with where you started on the episode. It's how do we get people started?
Thank you, Brian, for sharing your perspectives here on the Open Update and we'll get back to you in five years, to check in on you when you have more power.
You just listened to our original interview with Brian Nosek here on the second season of the Open Update. What do you think? What insights came to mind and what resonated? But also, what did you disagree with? We would love to hear from you because this is not only a conversation on a podcast. It is a conversation for us all.
So I encourage you to leave us a voice message. It doesn't need to be perfect. You don't need to have deep insights, and no, you also don't need to create an account. We appreciate it, even if you just end up saying hello and thank you for now, have a good rest of your day.
Next week, we'll be back with our interview with Sam Moore, where we'll talk about the desired limits of open.
