RESEARCH STORIES
Dr Stefan Müller in conversation
In the second of our Research Stories, let’s meet Dr Stefan Müller, Assistant Professor in the School of Politics and International Relations at University College Dublin. Stefan established the Text and Policy Research Group and is a core member of the Connected_Politics Lab, as well as co-author of the quanteda R package. Stefan was recently selected as a member of the Young Academy Ireland and is a recipient of a UCD University Teaching Award. At NexSys, Stefan is researching what academic research publications are focusing on when it comes to net zero, environmental policies, and climate policies. He also develops and classifies approaches to measure stances towards energy and environmental policies in political speech.
Can you tell us a bit about your background and research?
I’m a political scientist by training. In most research projects, I use quantitative methods, such as survey data and a lot of textual data, to address substantively meaningful research questions. Not only in politics but in society in general and in our everyday lives, we produce a lot of text. This can include speeches, party manifestos, social media posts, abstracts, full papers of journal articles, or a combination of these. Generally speaking, textual data reveal quite a lot of important insights about politics and society. In the past, it used to be very hard to analyse and structure these texts. Qualitative coding involves having to read every single sentence in every single document and then assign labels. This is extremely important and still is part of every single project we conduct to ensure our findings are valid. If the quantitative scores do not match human judgement, they don’t help us understand politics and society. With thoroughly validated automated methods – such as quantitative text analysis, machine learning, and new large language models – you can analyse texts more efficiently and scale up the analysis.
You recently published a paper on nostalgia in party politics, and wrote about it in The Conversation. Can you tell us about your method and findings?
We wanted to find out which parties communicate nostalgic rhetoric, and whether there are differences over time, across parties, and across countries. In this paper, we compared six different computational methods of measuring nostalgia in 1,648 party manifestos across 24 European democracies from 1946 to 2018. Some of them were better, some worse. Overall, we found that nationalist, radical right, and conservative parties, as we would expect, are more nostalgic. Another finding was that nostalgic appeals are not a recent phenomenon but have been present ever since the end of the Second World War.
You currently lead two research projects, one of which is funded by NexSys. What is it about?
Within NexSys, I’m one of the social scientists, working on a truly multidisciplinary and cross-disciplinary project within the NexSys water strand with Yen-Chieh Liao, a postdoc at NexSys.
We address three research questions. The first is about finding out what researchers actually focus on in publications on energy policy, climate change, and net zero, and also investigating differences over time across disciplines and journals. Databases allow us to extract the abstracts, titles, keywords and locations of authors, based on either the journal name or keywords in abstracts. We have assembled a very large dataset about energy, net zero, and climate change, comprising over 300,000 articles. Our data show that research on these topics has really skyrocketed since around 2018. We want to understand similarities and differences between international agreements and research on these topics.
Our second research question looks at which of these papers actually make policy recommendations. We are training and validating machine learning classifiers to assess which papers make policy recommendations or proposals in the abstract, thereby trying to communicate to policymakers or society more broadly. Although our results are still preliminary, we are finding interesting differences across disciplines, and we see how papers with policy recommendations have increased almost exponentially, from fewer than 50 publications in 2000 to almost 17,000 in 2022.
The proportion of policy mentions is highest in the social sciences and humanities, followed by health sciences. It is considerably lower in multidisciplinary journals and in the life sciences. We see that it has increased in some areas, but not in all of them, and that’s something we want to understand in more detail. It seems that a lot of fields that can or could make a real impact might not speak to policymakers as much as they could.
The third question is about the politicians, looking at parliamentary speeches and manifestos to understand which politicians mention scientific evidence because we expect differences across countries, parties, families, and politicians’ backgrounds. We also want to investigate what kind of evidence politicians provide for their proposals or arguments when they talk about climate, environment, and net zero.
It’s fascinating when we get together as a strand and have meetings. We are from completely different backgrounds, but we still have similar research interests. Looking at our research questions from various angles is something I really enjoy.
Another recent paper on which you are co-author concerns the relationship between the journal impact factor and characteristics of peer review reports. This is part of your second project, funded by the Swiss National Science Foundation. Can you tell us about this research?
The paper recently published in PLOS Biology is the result of a project that started in 2019. We first categorised helpful and thorough aspects of peer review reports. We looked at questions such as: does the sentence give a suggestion? Does it praise the paper? Is it critical? Is it providing examples? We’ve based these categories on prior research, and then we assessed whether the impact factor of the journal makes reviews more helpful or more thorough.
We also wanted to look at grant peer reviews, using reports from the Swiss National Science Foundation. We’re interested in whether a peer review comment on a certain topic actually addresses what it’s supposed to address, how helpful it is, how long it is, and whether it provides rationale or reasons behind the comments or suggestions. We have about 35,000 grant peer review reports from the last few years. This includes anonymized information on the reviewer, the applicant, and the co-applicants. We first relied on manual coding to identify features of reviews that can be coded reliably. Currently, we are using cutting-edge machine learning approaches to scale this up to all reviews.
Staying on this topic, what is your view of research evaluation practices in grant applications?
High-quality academic research publications are extremely important, and definitely one of the core aspects of our daily work, so they should be considered when it comes to hiring scholars. However, researchers do a lot of other things too, such as communicating with the public, writing policy reports, or developing software which is used by many researchers. This might not always be acknowledged. So it’s important to not focus only on academic publications, and in particular impact factors because we know this is highly problematic. We need a broader perception of excellent research. There are initiatives like CoARA, the Coalition for Advancing Research Assessment, which help to promote best practices.
Some of your research is about gender. Can you tell us about that?
We used computer vision to detect emotions, in particular happiness and anger, and applied this to five televised leaders’ debates in Germany. The videos are transformed into images. Each second consists of 25 images, and each of these images is then classified as to whether a candidate is shown, and which emotions are conveyed. We matched this with the vocal pitch of each politician in each second, which is an indicator of emotion. We also used the transcripts to estimate whether they talked positively or negatively and what policy area they discussed. An audience watched each of the debates in real time and indicated their agreement with the candidates throughout the debate. We used these data to test whether emotional expressions influenced or related to the approval of candidates. We applied this to all debates that involved Angela Merkel, and we found that when she expressed happiness, the viewers had a more positive perception of her. For anger, it was the opposite for Merkel – viewers had a more negative perception. But when the male candidates expressed anger, they tended to get rewarded. So there are definitely gender perceptions of acceptable or desirable traits of leaders and also of emotional expressions.
Sharing data and openness is clearly very important to you and your work. Can you say more about this?
I always publish all data and the entire workflow for publications because I think transparency is super important. If the data and the code are not out there, it’s impossible for other researchers to reproduce our findings and spot errors. And if they can detect an issue or want to improve the study, that’s the way science should work.
What advice would you give to early career researchers when it comes to pursuing an academic career?
First of all, there’s always luck involved in some way and that’s important to note. Secondly, unfortunately, there are many more early career researchers than permanent positions. So my main advice would be to think about alternative career paths too. Consider what happens if you don’t want to stay in academia, struggle to find a job, or if you don’t want to move countries frequently. If you become an academic, it’s extremely likely that you will have to work abroad or at least in a different city, which can be difficult if you have a family or caring responsibilities. As researchers, we have a lot of skills that are transferable to other sectors, but sometimes we are not confident enough about these really valuable skills. For example, logical thinking, working on projects for a very long time, completing projects, collaborative work, presenting, teaching, and explaining complex phenomena.
Your PhD was in Trinity. What led you to Ireland?
When I was 16, I did a transition year in Fethard, County Tipperary. I went there to live in a new environment and improve my English language skills, and I had a fantastic time. After that, I often went back to Fethard. I saved money for flights and the bus, and I spent many holidays there with friends. This is why I always wanted to come back to Ireland. I’m also passionate about GAA, hurling and Gaelic football. I follow it very closely and have a publication in a sports science journal on the effect of having a game in a neutral venue. Let’s say Tipperary play Limerick and Cork: we looked at the probability that the favourite team, based on rankings before the match, is going to win. We found a strong moderating effect of neutral venues: if a game is on a neutral venue, the favourite is less likely to win compared to a game at home. So I’m really interested in Irish politics, society, and also Irish sports and culture.
What are some sources of inspiration for you?
There are many. For example, people I’ve worked with in the past, or very good lecturers. I try to incorporate some of their practices into my own teaching. Another source of inspiration is when I receive an email from a student who really liked my module, or from other researchers who say I read your paper and enjoyed it. I always save a PDF version of these emails in a folder. When you’re not feeling well, you can go back to that and see that other people appreciate what you’re doing.
What do you like to do in your spare time?
I enjoy bouldering, playing five-a-side football (not very well, but every week) and watching sports. I live in Stoneybatter, and I’m involved in the community group there. I also enjoy spending time with our cocker spaniel, Baileys.
