Do Quantitative Ratings and Qualitative Teacher Evaluations Align?

Many authors have sought to associate the quantitative features of writing (lexical richness, syntactic complexity) with more qualitative or holistic evaluations. For example, Daller, Turlik, and Weir (2013) found a latent growth curve using holistic ratings to measure vocabulary development. Shaw and Liu (1998) analyzed data from collegelevel non-native speakers over a relatively short period and thus found more development from spoken to written forms (e.g., a shift away from first person) rather than development in advanced vocabulary and syntactic complexity (p. 246). More recently, Plakans, Gebril, and Bilki (2019) found that complexity was less important than accuracy or fluency in the ratings of essays selected from the Internet-based TOEFL. (The fluency measure was simply a total word count in their analysis

The Development of Writing in an I EP 217 [Plakans et al., 2019, p. 166], which is of some concern. This choice of word count is obviously a very crude one and probably not how an IEP teacher would define ‘fluent’ writing. The use of this measure is of course due to the need to process text computationally.) However, in no case were either the intended meaning of the student interpreted or other features of the text (such as coherence and cohesion) considered. Ortega (2003) provided an overview of complexity and proficiency in college writing. Relevant to our discussion here is that she reiterates the warning that more complex does not necessarily mean ‘better’ and that the development of L2 writing involves more than just CAF features (Ortega, 2003).

In order to look at the overall human ratings of the 48 essays, three instructors from the IEP were asked to rate the essays based on the IEP rating scale (see Chapter 2, Appendix I). All the essays were anonymized (LI and level information was removed, although of course a guess could be made about the LI based on the content in some cases), and the essays were put into a list in random order, mixing authors, levels, and Lis. In other words, the essays were just supposed to be judged on the rubric alone. The raters were asked to simply rate the essays, not to correct them. If they desired, they were invited to make a comment but not required to do so. Each evaluator was permitted to use half points, so the scale ran from ‘1’ = no writing proficiency to ‘6’ = very good writing proficiency. Thus, the scale was really a 12-point scale if 0.5 points counts as points on the scale. (This was done in order to create a possible range of scores, as correlations may not be found when the range is restricted.) The three scores from each rater were averaged and used to correlate

Table 7.10 Correlations Among Average Essay Rating and Measures of Syntactic Complexity (N = 48 in All Cases)

(*) Significant at p = .05, two-tailed. (**); Significant at p = .01, two-tailed.

with two syntactic complexity measures (complex NPs per clause + mean length of clause), word count, and Advanced Guiraud based on the Pitt Service List (AG-PSL3), which excludes some key frequent ESL words (see Chapter 4). Each student provided 2 essays per level, for a total of 48 essays.

These correlations indicate that a reliable relationship exists between MLC and the average essay writing score. Moreover, the number of words in the essay also correlates with the average score assigned to each essay (Bulte & Housen, 2012). However, MLC correlates very strongly with CNC (indicating co-linearity) but not with number of words. This is significant, as it suggests that MLC and length of essay contribute independently to the ratings given by the human raters. Using stepwise regression analysis (which is used when a researcher has ‘no theoretical axe to grind’ [Larson-Hall, 2010]), it was found that MLC and number of words together accounted for approximately 25% of the variance of the average rating of the essays, while CNC and AG-PSL did not enter the regression. (Note that vocD scores are not correlated with human ratings in these data.) This means that in judging the essays, the raters paid significant attention to both the length of clause and number of words, but the MLC of clauses was taken into consideration even in shorter texts. As an example, consider the lowest-rated text [(14) in the Appendix, score = 2.83], which is 142 words long and had a CNC score of 0.45 and MLC of 7.1, versus the highest-rated text (S in the Appendix), which received a unanimous ‘6’ on the scale and was 695 words long and had a CNC score of 1.45 and MLC of 12.63.

Interestingly, the human raters’ scores did not differ by level [means (standard deviations): Level 3 = 4.37 (.67), Level 4, 4.52 (.71), Level 5, 4.87 (.60), F(2,45) = 2.314, p = .11]. From these data, it seems that the human raters are judging the quality of the writing as if it were within the level that they assumed the student was writing, which is a topic that can be followed up on in future research. It is also worth pointing out that quantitative measures are accounting for only 25% of the variance in the average human rating scores. This result suggests that human raters are paying more attention to meaning and organization than length of essay or mean length of clause.

Although there were no statistical differences by level in grades, on reading the essays, it is possible that those essays which were written in the first person were shorter, contained lower lexical sophistication, and had lower MLC. The intuition is that if a student can rely on his or her own experience, the output that they produced may be limited to their existing vocabulary and structural style. In other words, they will not need to reach for new vocabulary and structures to explain a topic that they are less familiar with.¹ In order to explore this intuition statistically, the essays were coded either ‘Impersonal’ (not related to the student’s direct experience and written in the third person) or ‘Personal’

Table 7.11 Impersonal and Personal Texts by Level in the Sample

(directly related to the student’s own experience and written in the first person). The distribution of Personal vs. Impersonal by level is provided in Table 7.11.

Clearly, students write more texts in the first person in Level 3 and more impersonal topics in Level 5. Statistically, it is hard to separate out Impersonal/Personal topics from length of essay (words), as students write much more at Levels 4 and 5 than at Level 3. To reliably examine the effect of first-person vs. third-person texts, a selection at each level that is balanced would be necessary. Yet it is interesting to note that all five of the Personal essays at Level 5 had MLC scores of less 8.42 or less (only one Impersonal essay had a score lower than 8). Only one of those Personal essays had an AG-PSL3 score of over 1.0, while ten of the Impersonal essays had AG-PSL3 scores over 1.0.

This result is only based on a limited number of texts and must of course be treated with some caution before a larger study can be carried out. Certainly, one might want to consider further measures of lexical sophistication and diversity. Overall, it is somewhat surprising that AG scores, increasing as they do over time from level to level (see Chapter 4), did not play a key part (statistically) in influencing how the human raters evaluated the texts, although the comments on the Level 5 essays’ AG-PSL3 scores suggest that they might play a role. One essay, b!7’s 'Happy Chinese New Year’, was the only text to receive the highest possible score of 6 from the raters. This essay was not ‘academic’ in the sense of other essays on social topics, but the vivid description and enthusiasm of the ‘voice’ of the writer come across very strongly. This observation is an important one, and such ‘intangibles’ cannot fully be accounted for by any quantitative measures.