Methods for Examining L2 Learners’ Texts

As discussed, measures and indices of L2 texts can be estimated using human rating (e.g., rating of accuracy), human manual coding, and/or computer analysis. Historically, human rating and manual coding were the main methods for analyzing L2 learners’ texts. However, with the increasing availability of computer programs that can analyze written texts, it is more common nowadays to use such programs to analyze L2 learners’ texts. Each method has its advantages and limitations, however. Human rating and manual coding allow a more in-depth, qualitative analysis of L2 learners’ texts, but are not practical with large samples of learners’ texts. These methods also suffer from low reliability Computer analysis, in contrast, is highly reliable and more efficient and effective, particularly when analyzing large samples of learners’ texts. Computer analysis also allows the identification of patterns, differences, and similarities in the linguistic characteristics of large samples of texts across learners, contexts, and time, that human raters or coders cannot identify consistently.

However, computer analysis has its limitations, particularly when analyzing L2 learners’ written texts. First, computer programs can only count specific writing features, but cannot assess their quality such as their appropriateness and effectiveness for the writing task or audience. Second, one issue when using computer programs to analyze texts written by L2 learners is that these texts often include several linguistic and other inaccuracies (e.g., misspelled words, incorrect punctuation). As a result, computer programs may not accurately identify and estimate the frequency of linguistic and discourse features if an essay includes several and/or severe spelling and/or punctuation mistakes. In order to address this problem, some previous studies corrected spelling and punctuation mistakes before conducting computer analyses of L2 learners’ texts (e.g., Crossley & McNamara, 2014; Riazi & Knox, 2013).Third, automatic analyses may not be 100% accurate. For example, Lavolette et al. (2015) found that Criterion was able to identify only 54% of the errors manually identified in a sample of L2 essays and that the errors that Criterion found were coded correctly only 75% of the time.

The approach adopted to address these challenges in the present study was to combine the three methods of computer analysis, human rating, and manual coding. The combination of these three methods was intended to offset some of the limitations of each method as well as to enable us to capture changes and differences in both micro- and macro-aspects of writing. Specifically, while computer analyses and manual coding can reliably identify and measure micro-features such as number of errors and syntactic and lexical features at the word and sentence level, human rating can detect differences and changes in macro-aspects such as text organization and argument structure. In addition, while computer analyses and manual coding focus on counting specific features (e.g., number of errors, number of content units from source texts included in an essay), human rating focuses on evaluating the quality of specific features such as the effects of errors on essay comprehensibility or the quality of source use. It is expected that such an approach is more likely to detect changes and differences in writing features across proficiency levels, tasks, learners, and time (cf. Barkaoui & Knouzi, 2012; Schiftner, 2013).