Considerations for developing a rigorous mixed methods diary study

Based on these reflections, we offer several suggestions to researchers considering adopting a mixed methods diary approach in higher education settings:

• • Reflect on the role diaries play in a mixed methods study. As outlined at the start of our chapter, mixed methods designs can vary greatly, depending on the research questions and study aims (Cohen et al., 2018; Creswell & Creswell, 2018; Creswell & Plano Clark, 2017; Greene et al., 1989). When developing a mixed methods study, it is important to critically reflect at the start of the design process about the overarching goals of the study and carefully consider how different research methods might best address the complex experiences of students in higher education (Creswell & Creswell, 2018). The role, purpose and priority of diary research for answering research questions has strong implications for when diary data is collected and how diary instruments are designed to capture the reflections of students or other stakeholders (Filep et al., 2017).
• • Consider methods of integration throughout a study. A frequent critique in mixed methods research is that researchers often only consider the integration of the findings at the conclusion of the study (Bazeley, 2017; Woolley, 2009). However, a stronger form of integration should be developed throughout the research design, data collection, analysis and dissemination of findings. For example, considerations might be made about how the preliminary findings from diary data can inform the instrument design or purposeful sampling of follow-up data collection about experiences in higher education (or vice versa).
• • Consider the strengths and weaknesses of each research method. One important added value of a mixed methods approach in our own study was balancing the strengths and weaknesses of the different methods used to form a more complete picture of doctoral students’ social transitions. By carefully considering those strengths and weaknesses at the start of our study design, we could order and design our data collection in a way that was more logical for addressing the research questions. Therefore, it is recommended to reflect on what is gained by including a diary method into a higher education research design and what gaps or weaknesses remain at the end of each data collection phase so that these can be compensated for in the next method employed in a study (Creswell & Creswell, 2018; Creswell & Plano Clark, 2017; Greene et al., 1989). In particular to diary research in higher education settings, challenges such as self-reflection bias, participation fatigue, or lack of depth in responses should be considered, especially in consideration for distracting periods during the academic year (such as exam times).
• • Consider mechanisms for motivating continued participation. One challenge for mixed methods research is that participation often occurs over a longer period or over multiple phases, which may lead to participation fatigue or participant dropout. This is of particular concern for diary studies in higher education settings, where there is often a longitudinal element that requires sustained effort and contribution (Harvey, 2011), in addition to participants’ competing priorities and deadlines as university students or staff Although there is debate about the merits of financial incentives or gifts for participants (Head, 2009), it is worth noting that mixed methods diary designs often do need some form of incentive or motivation for continued participation. The appropriateness of these choices for participants in higher education are likely circumstantial and it is suggested to reflect upon this at the start of a study design process.
• • Develop a plan for anonymisation and merging of the data. As experienced in our own study, one challenge for mixed methods diary research is that the triangulation of multiple methods might make it easier to identify participants in the analysis and dissemination of findings. This can be exacerbated for higher education researchers working in their own departments, where participants may be known to the researchers or other staff members (Malone, 2003). It can also be difficult to merge data from multiple methods without a designated identifier for individual participants. To avoid ethical or organisational problems, it is suggested to create a strict anonymisation protocol at the start of a study for all data collection methods. When working in a team, assigning one researcher to be in charge of anonymisation and data merging can also be beneficial, particularly if they have limited power over participants’ progression, completion or promotion. Finally, we found it was good practice in our own research to allow student participants the opportunity' to review their own anonymised data and suggest changes to make them less identifiable to each other and departmental staff This approach is especially well-suited for higher education research, where participants are likely' to be highly literate and potentially aware (in the case of our doctoral participants) of research processes and ethical protocols.
• • Create strong ethical protocols to guide the research. As identified previously, the nature of mixed methods diary research design means that large volumes of personal and identifiable information are often collected about participants. For this reason, the research team needs to reflect carefully about potential issues that might arise in the study, including in the data collection, analysis and dissemination (Filep et al., 2017). Researchers in higher education who are developing such studies should consider issues such as power dynamics between researchers (likely university' staff) and participants (students or colleagues), the privacy of non-participants, identification of participants in the dissemination, anonymisation of people or university' place markers discussed in the data, and the increased risk of identification due to triangulation of multiple pieces of information about participants.