Challenges of adopting a diary method
As our research project was longitudinal and sequential, participants were asked to contribute to several data collection phases (an initial survey, a 6-week diary collection, and an interview). As is common in longitudinal and mixed methods designs (Lavrakas, 2008), some participants in our project shared informally with the researchers that they experienced participation fatigue during the research process, leading to a decline in motivation and attention. For example, there were frequent comments in the follow-up interview data that reflected on participants’ experiences using diaries, such as:
I probably should have said this in my diary, but I was a little bit lazy ...
(Interview Participant 2)
I ran out of time so didn’t write much about this ...
(Interview Participant 11)
The consequences for diary collection included data quality deterioration over time, shorter diaries and repetition in the entries. Various factors that contributed to this were informally shared by participants, such as enthusiasm fading when writing the diary became a routine, when there was a lack of new experiences or events to reflect on, or when participants had to balance participation with a busy agenda (such as towards the semester end). As a result, we observed diary quality discrepancies between the earlier and later phases of data collection. Hence, one significant challenge for researchers using a mixed methods diary approach is how to design mechanisms for motivating participants to ensure consistency in the diary data’s quality and volume. This has particular implications for higher education research, as researchers will need to ensure that diary' collection timing does not coincide with busy or stressful periods (such as during exams) or when attention is likely elsewhere (such as during university breaks).
Second, we recognised that there were likely self-selection or self-reflection biases that might have occurred in our data collection in terms of the experiences that participants chose to share in their diaries. Unlike private diary writers, participants in diary research are aware that their diaries are being read by researchers (Wechtler, 2018). In our own study, one consideration was the implications of perceived power dynamics between the doctoral students and the research team, who were university staff members. Participants might share experiences and emotions that they assume would meet the researchers’ expectations, perhaps leading to an unwillingness to be critical of the institution or its staff Participants might additionally select certain parts of their lives to create a particular image, such as highlighting behaviours to portray themselves as ‘good’ students. In our study, for example, we found there were instances when a participant portrayed themselves as active within their doctoral social community in a face-to-face interview, but this was unsubstantiated by evidence in the social network analysis survey or diaries. Therefore, a distinct challenge for mixed methods diary research is disentangling such self-reflection biases in the findings by triangulating multiple pieces of evidence.
As noted previously, integration is at the heart of mixed methods research (Bazeley, 2017; Woolley, 2009). However, many mixed methods theorists argue that integration often remains superficial and focused on research conclusions, rather than substantially embedded across the research design (Bazeley, 2017; Woolley, 2009). However, getting integration ‘right’ can be exceptionally challenging, particularly as it requires careful planning and consideration (Bazeley, 2017; Plano Clark & Ivankova, 2016). In our study, we found that carefi.il considerations were needed at the project’s start to map and plan to ensure that integration was purposeful. This included considering the order of the methods used related to our research questions (first outlining what with the social network analysis survey, then unpacking why and how with the diaries and interviews). We also needed to consider the research instrument development and plan for how preliminary analysis and findings would shape subsequent data collection. For example, preliminary findings from the diary study informed the questions developed for our semi-structured interviews. Finally, we needed to carefully consider from the start of the project our approach to analysing the data and how to integrate findings to develop a more complete picture of the student communities we were investigating. Altogether, in line with Bazeley (2017), we found that data integration was a dynamic (and time-intensive) process which required integrating multiple forms of data at different research stages. At times, this meant unexpected challenges that required collaborative reflection by the research team, such as how to handle missing data or contradictions in findings between different methods. As we conducted the project in different steps with information unfolding over time, it was important to balance both careful planning at the start of the project and flexibility to adjust approaches as findings were developed. Nonetheless, we found that incorporating a mixed methods approach encouraged collaboration and methodological reflection to a degree that might not have been as valuable using one method alone.
