Essay Analyses

To analyze the essays, we used the framework in Table 2.2. As explained in Chapter 2, the analytic framework builds and expands upon Connor and Mbaye’s

(2002) Model of Writing Competence and previous research. To analyze the essays in this study, we also matched the criteria used to evaluate responses to TOEFL-iBT⁹ writing tasks to the framework in Table 2.2. Table 4.3 includes the same information as Table 2.2, but adds three new columns. Column 1 lists the five competences. Column 2 lists the criteria from the rating scales used to rate responses to TOEFL-iBT®¹ writing tasks, thus connecting the criteria to the components in the Connor-Mbaye model (Column 1). Columns 3 to 5 list the constructs, measures, and specific indices used in this study, respectively. Column 6 indicates for which task (independent, integrated or both) each index was examined. Column 7 lists the methods of analysis for each index (manual coding, human rating, or computer analysis). As Table 4.3 shows, all the indices we used are directly related to criteria on the rating scales of the TOEFL-iBT⁹ writing section, except for indices related to metadiscourse use and register, which are not explicitly mentioned in the TOEFL-iBT⁹ rating scales since TOEFL-iBT⁹ writing tasks are decontextualized and do not aim to elicit or measure sociolin-guistic competence (Xiaoming Xu, personal communication, December 4, 2019). It was expected that the indices in Table 4.3 would enable us to detect differences in the characteristics of learners’ responses to TOEFL-iBT⁹ writing tasks across proficiency levels, tasks, learners, and time.

Grammatical Aspects

Four constructs were examined in relation to grammatical competence: fluency, linguistic accuracy, syntactic complexity, and lexical complexity.

Fluency: We used one index of fluency: number of words per essay. It should be noted here that the number of sentences, number of clauses, and number of T-units were highly correlated with the number of words in the essays (r > .80) in this study.

Linguistic Accuracy: We used two approaches to measure linguistic accuracy, computer analysis and human rating. First, the computer program Criterion (him & Kahng, 2012) was used to identify and count the number of linguistic mistakes in each essay. Criterion can identify, categorize, and count four types of mistakes: Grammar (e.g., sentence structure errors, ill-formed verbs), usage (e.g., article errors, incorrect word forms), mechanics (e.g., spelling, punctuation), and style (e.g., passive voice). After Criterion identified all errors in each essay, an error ratio (number of errors per 100 words) was computed for each essay.

Second, a four-point holistic rating scale developed by Chan et al. (2015) was used to evaluate impressionistically (from the viewpoint of the reader of a whole essay) the impact of errors involving grammar, punctuation, spelling, and word choice on the comprehensibility of the essay (cf. Cumming et al., 2006; Di Gennaro, 2009; Knoch, 2009; Kuiken & Vedder, 2014). Raters were instructed to consider all types of errors (i.e., lexical [e.g., word choice], morphosyntactic, semantic, and/or mechanics), but to make an overall judgment as to the overall impact of the errors on the comprehensibility of each essay (see Appendix A for a copy of the accuracy rating scale and instructions).

Syntactic Complexity: Based on recommendations in the literature (see Chapter 2), several measures of syntactic complexity were examined in this study using the computer programs Coh-Metrix and L2 Syntactic Complexity Analyser (L2SCA, Lu, 2009, 2010, 2011). L2SCA is a web-based program for identifying specific linguistic structures (e.g., sentences, clauses, T-units) in written texts (Lu, 2009, 2010, 201 l).The following indices of syntactic complexity were examined:

• • Global complexity: Two indices of global complexity (from L2SCA) were examined, mean length ofT-units (MLT) and mean length of sentence (MLS). However, the two indices were highly correlated (r = .77). As a result, only MLT was included in the analyses below.
• • Complexity by coordination: The coordination index, or mean number ofT-units per sentence (T/S) as computed by IASCA. was computed for each essay.
• • Complexity by subordination: Four indices of complexity by subordination were computed by L2SCA for each essay: mean number of clauses perT-unit (C/ T), mean number of dependent clauses per clause (DC/C), complex T-units perT-unit (CT/T), and mean number of dependent clauses perT-unit (DC/ T). However, the four indices were highly correlated (r > .75). Consequently, only one, C/T, was included. Note that C/T was highly correlated with MLT (r = .92) in the present dataset.
• • Clausal complexity: Mean length of clause (MLC), computed by IASCA. was examined.
• • Phrase complexity (sub-clausal complexity): One index, obtained from Coh-Metrix, was examined for each essay: Noun-phrase (NP) density, that is mean number of modifiers (e.g., determines, adjectives) per NP.
• • Structural variety: Syntactic variety was measured using syntactic similarity indices computed by Coh-Metrix. Syntactic similarity measures the uniformity and consistency of the syntactic constructions in a text. Generally, high syntactic similarity indices indicate less complex syntax (McNamara et al., 2014). Coh-Metrix estimates mean sentence syntactic structure similarity for adjacent sentences and for all sentences across paragraphs.The two indices were highly correlated, however (r = .87). Consequently, only one is included in the analyses below: Mean sentence syntactic structure similarity for all sentences across paragraphs.

Lexical Complexity: Various computer programs were used to examine lexical density, lexical variation, lexical sophistication, and lexical bundles as follows.

Lexical density: Ratio of content words was computed using the Tool for the Automatic Analysis of Text Cohesion (TAACO; Crossley et al., 2015).

TABLE 4.3 List of Measures and Indices of Essay Linguistic Features Used in the Study

★Method: Whether the feature is measured by manual coding (MC), human rating (R), or computer analysis (CA).

Lexical variation: For each essay, we computed both D-indices and MTLD using Coh-Metrix. However, the two indices were highly correlated (r > .70). Consequently, only MTLD is reported below.

Lexical Sophistication: Coh-Metrix was used to estimate average word length (AWL) and word frequency for each essay, while the Tool for the Automatic Analysis of Lexical Sophistication (TAALES; Kyle, Crossley, & McNamara, 2015; Kyle & Crossley, 2015, 2016) was used to compute several indices based on the academic word list. These indices are calculated by counting the number of tokens in the essay that occur in the academic list (raw scores) and dividing by the number of words in the essay (normed scores). TAALES computes indices for the entire academic word list as well as for each of ten sub-lists. Two indices, AWL and word frequency, were highly correlated in the present dataset (r > —.75). Consequently, only AWL is reported below. As for the academic word list, only indices for the entire list are reported below.

Lexical Bundles: Using the computer program TAALES, we estimated two indices concerning lexical bundles (cf. Kyle & Crossley, 2015, 2016). TAALES computed a normalized frequency of bigrams and trigrams in each essay based on bigram and trigram frequency lists developed by Crossley et al. (2012) from the 80-million word written and the 10-million word spoken subsets of the British National Corpus (BNC, BNC Consortium, 2007). For each list, normalized counts are calculated using the number of words in the text as the denominator and the number of bigrams/trigrams in the text that are also represented in the frequency list as the numerator. BNC written bigram frequency was computed for each essay. We did not include trigrams because TAALES identified a very small number of trigrams in the essays. In addition, TAALES was used to compute the frequency of multi-word units from the Academic Formulas List (AFL) in each essay. Because the correlations between the three AFL sub-lists (AFL spoken, AFL written, and AFL core) and entire AFL were high (r > .70), only the normed frequency of formulas from the entire AFL for each essay are reported below.