What the Figures don’t tell you
Using Qualitative Methods for Analyzing Potential Bias in Research Funding
By Marlene Hock
Posted on October 21st 2020
For years, there has been data-based, empirical evidence on different funding success rates for female and male researchers. Additionally the leaky pipeline remains an issue in academia, especially in view of increasingly balanced numbers among female and male students in STEM. In this respect, the question is: what are the reasons for the persistent gender differences?
Implicit gender bias within research funding processes, career development or even in the research system are possible explanations for remaining gender disparities in various fields of science. Since numbers alone do not explain remaining differences between female and male researchers, it is necessary to use additional approaches. Therefore methods are needed that are able to make underlying processes and practices of research organisations more tangible in order to identify potential sources of implicit gender bias in the stages of researcher`s careers.
The GRANTeD research project investigates the process of decision-making on grant allocation. The starting points of our research are different success rates for female and male applicants, which cannot be entirely explained through differences in merit. After all, it cannot simply be assumed that women are principally less capable and competent researchers than men are. So, if there are gender differences in the outcomes of research funding the question is how could these different success rates be explained?
Funding programs’ formal criteria of evaluation are often clearly communicated by the funding organizations but other aspects of the funding process remain unclear. For example, how members of evaluation panels are selected and what panellists are asked to do is not widely known. Grant peer review is commonly perceived as guaranteeing a more or less fair evaluation process, legitimizing the distribution of resources and affirming the broader system of academic careers. Nevertheless, the principle of meritocratic distribution of resources has received criticism as privileging often the dominant and penalizing the subordinate groups (e.g. Nielsen 2016, Lamont 2009).
The decision on who will be funded reflects a process of social negotiations, that cannot always be anticipated. Human behaviour is never completely predictable, meaning it does not follow the natural (and measurable) law of cause and effect. Social processes, in this case decision-making, are always relationally embedded in power structures and dynamics, relationships, organizational logics, established and routinized practices, political frameworks, historically grown processes, specific value systems and individual conceptions about reality.
A previous qualitative study (Herschberg et. al. 2018) on selecting early-career researchers showed, for example, how macro-discourses of excellence shape formal and applied selection criteria at the meso-organisational and micro-individual levels. The study demonstrated how tensions between the various levels produce inequalities in evaluation, showing that Committee members do not operate in a vacuum and that their actions are inseparably linked to the meso- and macro-context (Lamont 2009; Musselin 2010; Van den Brink and Benschop 2012).
The results of a previous (case) study further suggests that the inconsistency in assessment results across reviewers may be related to differences in the way reviewers conduct their assessments and how they apply institutionalised criteria (Abdoul et. al. 2012b). Consequently, a deeper understanding of how assessment criteria may be applied differently is crucial for illuminating and changing practices (Higgs et. al. 2009).
Decision making in panels is in this sense a social interaction, a practice that needs to be understood and explained in terms of underlying assumptions, conditions, strategies, informal processes and actions. As actors may not always be aware of these structures and conditions, decision-making has to be analysed and considered in the light of these social circumstances.
Since the GRANteD Project aims to explain the outcomes and emergence of the process of grant decision making, we first need to understand how the system of grant allocation functions in practice and how internal procedures work. Therefore, we need to gain insights in the specific logics and references of the social dynamics that decide what research is eligible for realization. Consequently, to identify and explore potential sources and mechanisms of gender bias in grant allocation we have to understand and explain procedures, policies and social practices as well as individual views and assumptions that structure awarding or selection processes.
More concretely, studying practices in GRANteD translates into analysing selection procedures of review panels, investigating the use of (formal) assessment criteria as well as identifying modes and indicators for assessing merit and excellence and find out how they are reflected in the practice of assessing scientists.
On a more general level, the qualitative approach will help reconstructing and therefore understanding the organisation of decision-making processes in different panels and research organisations. This way, it can be determined whether assessments are influenced by (implicit) gender stereotypes or individual gender prejudices.
Consequently applying qualitative methods helps understanding phenomena and processes by considering why and how they occur. In the light of our project, this means, aiming at a better and deeper understanding of the complexity and dynamics of gender inequalities in research and innovation funding. Conducting qualitative interviews with RFO staff and panel members will help gaining insights on practices, assumptions and perceptions, related to the assessment of applicants, research projects and to the decision-making. As many of these practices and routines are unconscious and actors may not always be aware of an underlying bias, methods to capture and reconstruct these unconscious (latent) processes are required. In this context, qualitative research can find out what people believe about their practices (e.g. their assumptions and perceptions towards excellence) and it can explain the gap between the intention and the result of a practice (Higgs et. al. 2009). This is in particular relevant when we analyse how excellence or merit is constructed and understood and how gender is or is not discussed.
Therefore, to evaluate the effectiveness of interventions and instruments in place, qualitative approaches to address researchers, research organization managers and policy makers in the area of science and innovation are needed to understand the ways people perceive, act and manage situations in the selection of grant applications in panels. As already discussed in a previous Blogpost, the relative lack of studies that fully explain existing gender differences in grant allocations calls for further research to increase our understanding of the various factors, which can explain this situation. (Head et. al. 2013, Tricco et. al. 2017)
Higgs, J., Horsfall, D., Grace, S. (Eds.). (2009). Writing Qualitative Research on Practice. Leiden.
Abdoul, H., Perrey, C., Amiel, P., Tubach, F., Gottot, S., Durand-Zaleski, I., & Alberti, C. (2012): Peer review of grant applications: criteria used and qualitative study of reviewer practices. PloS one, 7(9), e46054. Online [19.10.2020]: https://doi.org/10.1371/journal.pone.0046054
Herschberg, C., Benschop, Y. & Van den Brink, M. Selecting early-career researchers (2018): the influence of discourses of internationalisation and excellence on formal and applied selection criteria in academia. Higher Education 76, 807–825. Online [19.10.2020]: https://doi.org/10.1007/s10734-018-0237-2
Head, M., Fitchett, J., Cooke, M., Wurie, F., Atun R., (2013): Differences in research funding for women scientists: a systematic comparison of UK investments in global infectious disease research during 1997–2010, BMJ Open. Online [19.10.2020]:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3865386/
Tricco, A., Thomas, S., Antony, J., Rios, P., Robson, R., Pattani, R., Ghassemi, M., Sullivan, S., Selvaratnam, I. Tannenbaum, C., Straus, S., (2017): Strategies to Prevent or Reduce Gender Bias in Peer Review of Research Grants: A Rapid Scoping Review, PloS one 12(1): e0169718. Online [19.10.2020]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5218731/
Lamont, M. (2009): How professors think: inside the curious world of academic Judgment. Cambridge (Massachusetts).
Musselin, C. (2009): The market for academics. London.
Nielsen, M.W., (2016): Limits to meritocracy? Gender in academic recruitment and promotion processes. Science and Public Policy Vol. 43 Issue 3. Online [19.10.2020]: https://doi.org/10.1093/scipol/scv052.
Van den Brink, M., Benschop, Y., (2011): Gender practices in the construction of academic excellence: sheep with five legs. Organization, Online [19.10.2020]: https://doi.org/10.1177/1350508411414293.