Interviews and Content Analysis
Another frequently adopted data collection approach among the current L2 LOA studies is interviews. “Interviews” in this chapter broadly include two types: task-performing interviews refer to the data collection method where the researchers instruct their participants to perform specific tasks and collect the participants’ written responses or document their spoken responses through video- or audio-recordings (e.g., Bailey & Heritage, 2014; Wolf et al., 2016); and semi-structured interviews refer to the data collection approach where the researchers use a list of pre-determined questions to guide the interview process but at the same time allow the interview responses to digress, within reason, in order to elicit further
Figure 4.1 Common CA transcription key.
information (e.g., Kim & Kim, 2017; Leung, 2020). Interestingly, it appears that the studies that adopted task-performing interviews mostly focused on the learning-oriented features of an assessment (e.g., through learning progressions or scaffolding), and the participants’ responses were used to guide the construction or evaluate the validity of the assessment. On the other hand, the studies that adopted semi-structured interviews usually used the interviews as an opportunity for the participants to reflect on their teaching or learning. For the interviews that focused on the learners’ perspectives, they were usually accompanied by the interviewed students’ graded work (e.g., written assignments with feedback) that served as the focal point for reflection.
Similar to observations, interviews often yield audio or video data for transcription. However, because the topics or themes of the interviews are often determined beforehand, CA is not suitable as an analysis approach given its emphasis on using data that are naturally occurring. Instead, the researchers often perform content analysis to systematically organize the data, categorize the patterns, and identify the themes or topics of interest. It should be noted that in some LOA studies, as seen in Table 4.1, the researchers specified that thematic analysis was used as their data analysis approach. While content analysis and thematic analysis have been defined as similar, but different analysis approaches in other fields (e.g., Braun & Clarke, 2006; Neuendorf, 2019; Vaismoradi et ah, 2013), such a distinction has not been explicitly and clearly made in the field of L2 research. For the purpose of being inclusive and allowing a broader range of discussion, in this chapter thematic analysis and content analysis are used synonymously as a qualitative descriptive approach used to analyze communicative language systematically.
In her overview of how content analysis has served as a useful tool in L2 assessment research, Galaczi (2014) stated that the main purpose of conducting content analysis is to “reduce texts into content categories based on explicit rules of coding” (Galaczi, 2014, para. 1). She further explained that one of the most important processes when conducting content analysis is coding the data using a well established coding scheme. It is reasonable, thus, to see that content analysis has been a popular approach to analyzing transcribed interview data since it allows researchers to logically organize the transcription into categories based on the target themes or research questions.
Among the current LOA studies, content analysis has been used to address a variety of research purposes, such as: to identify the patterns of the learners’ written or spoken responses in relation to learning progressions or scaffolded assistance (e.g., Bailey &t Heritage, 2014; Wolf et al., 2016); to extract the learners' perceptions on the feedback received (e.g., Kim, 2017; Leung, 2020); and to use the participating expert examiners’ comments to thematically develop a “checklist” to inform the teaching and testing of interactional competence (e.g., May et ah, 2020). These studies showcase the potential of using content analysis to capture different aspects of LOA in L2 contexts.
Narrative Inquiries and Analysis of Narratives
Narrative inquiry is a data collection method that connects stories and research either by using stories told by the participants as research data or by using storytelling as a data analysis approach (Barkhuizen et ah, 2013). Considering that narrative inquiry can be approached in different ways, Polkinghorne (1995) made the distinction between “narrative analysis” and “analysis of narratives,” with the former referring to the analysis method “in which storytelling is used as a means of analyzing data and presenting findings” (Barkhuizen et ah, 2013, p. 3), and the latter referring to the analysis method that treats stories as data.
Although narrative inquiry is not new in L2 research, it is not commonly used in L2 assessment research (Baker Germain, 2020). One possible explanation might be that narratives are highly individualized; therefore, it is not easy for narrative inquiry to yield generalizable results, a goal that, conventionally speaking, most L2 assessment research strives to achieve. However, Baker and Germain (2020) advocated for the use of narrative inquiry in CBA research because narratives provide an opportunity for teachers to reflect upon the assessment practices in their instruction, particularly those unplanned.
Currently, only two LOA studies in L2 contexts have been found to employ narrative inquiries as their data collection method, both conducted under close collaboration between the researchers and teachers to better understand the complexity of LOA in language classrooms through analysis of narratives. Specifically, in Baker and Germain (2020), the researcher interviewed the teacher four times over the course of four months, during which the teacher narrated and reflected upon her teaching and assessment experiences since they last communicated. Then, after the completion of the phone interviews, the researcher and the teacher reviewed the transcripts together in person. The teacher also provided photographs to help contextualize the narratives. The collaboration allowed both the researcher and the teacher to probe how spontaneous questioning from the learners could lead to unplanned assessment and further instruction, portraying the process of LOA in a nature-based indigenous language classroom. In Scarino (2020), on the other hand, the narratives consisted of various language teachers’ written rationales and commentaries of their curriculum design and assessment procedures. Scarino explained that, by engaging teachers in the thought processes of their own teaching and assessment practices over an extended period of time, their ability to understand, question, problematize, and theorize or retheorize the “processes, meanings, and meaningfulness of the processes” of CBA can evolve and be strengthened (p. 58). While taking a different approach to collecting the narratives, both of these studies well illustrated the value of narrative inquiries in facilitating a detailed understanding of LOA processes in language classrooms.
In sum, these three pairs of data collection and analysis approaches showcase how LOA research has been, to some degree, systematically conducted in L2 contexts. However, it should be noted that the purpose of highlighting these three pairs of approaches is not to set restrictions on how LOA research should be conducted. Rather, the goal is to offer insights into how data may be collected and analyzed when conducting LOA research, especially considering the complex nature of LOA.