Rubrics for Assessing Development

In each of the skill areas, the IEP uses rubrics for evaluating student performance. Rubrics have become standard tools in ESL and lEPs to assess and encourage development (Becker, 2010). Holistic rubrics tend to be used in summative assessment (e.g., www.ets.org/Media/Tests/ TOEFL/pdf/Writing_Rubrics.pdf), whereas analytical rubrics may be better adapted for formative assessment. The balance of each one in different tasks remains the focus of research (see Knoch, 2011, for a recent research article and overview of types of scale). More recently, Yamani-shi, Ono, and Hijikata (2019) compared rubrics that required holistic assessments (which take less time and are more useful for summative assessment with larger numbers of test-takers) and analytic assessments (which take more time but are more helpful for providing developmental feedback). A version of the ETS rubric is used by the IEP. Yamanishi et al. (2019) carried out both qualitative and quantitative analysis of a hybrid rubric for evaluating summary writing for EFL learners in Japan, focusing in particular on paraphrasing. They found that the scoring on the two aspects was significantly related, produced inter-rater reliability, and allowed teachers to provide both holistic and developmental feedback.

As with most assessment tools, Vercellotti and McCormick (2019) point out that validity, reliability, practicality, and beneficial consequences (positive washback) are important. We can unpack these terms briefly. Validity refers to what the test is measuring - that is, whether the construct, skill, or knowledge that the test seeks to assess is actually being measured by the test. The old paper-based TOEFL test, for example, may not have been a very valid test of ‘readiness to succeed in an English-speaking university degree course’ (=construct) because simple discrete point items are not able to measure the ability to understand and respond to longer spoken and written texts. Reliability refers to the potential for the assessment to produce similar scores based on the performance of the test-taker no matter who is grading and that similar performances by different students receive similar scores. Rubrics may be deficient and unreliable if they are too vague and permit impressionistic and subjective rater opinions to influence scoring (Knoch, 2009, p. 277). Practicality is very important for lEPs in which large numbers of students are enrolled and that have significant numbers of teachers: Can all the teachers apply the rubric well (that is, with good reliability) and with a degree of ease under time pressure? Finally, the impact of the rubric on the classroom instruction and student learning should be beneficial.

Goodrich Andrade (2001) provides useful criteria when considering the quality of instructional rubrics that effectively assist students in their learning. Rubrics should be written in language the students understand, define and describe quality work, refer to common weaknesses, and assess work in progress and guide revision and improvement. The criteria should list what counts and describe strong, average, and weak work. These descriptors should take account of student developmental readiness. In her study of high school students in rhe United States, Goodrich Andrade found that providing students with a rubric enhanced their awareness of the qualities of good writing, but the students had more challenges transferring that knowledge into good writing. The IEP rubrics currently in use provide a stage in which scores are not provided in order to focus students’ attention on the writing rather than the scores.

Final Exams

The final exams in the Pitt IEP are coordinated across sections of the same skill and the same level. Wherever possible, teachers do not evaluate their own sections but rather grade students from other sections. Scores on final exams have a very high impact on whether students can advance to the next level in the program.

The testing formats and exact test questions and prompts cannot be published for the obvious reason that (a) they must remain confidential from students and (b) they change over time. However, some general guidelines can be shared for teachers and administrators to compare with the exam systems in their lEPs.

Reading

Reading exams consist of both comprehension and vocabulary sections. The comprehension passages have to be checked for appropriateness with https://readability-score.com/ by comparing the examination text with a text at the end of term for that level. (Levels 3 and 4 typically have a passage from a textbook, whereas Levels 5 and above have an authentic text that may be slightly edited.) The texts are chosen to be of a length that would allow a student to read a passage twice in 20 minutes. Assuming a reading speed of 60 words per minute for Level 3 students, a Level 3 final exam text might be 1000 words long, for example. (For comparison, Rayner, Slattery, and Bélanger (2010) report that typical expert college-age speakers of English read on average between 207 (slow readers) and 337 (fast readers) wpm.) Higher levels would require longer texts, perhaps up to 200 wpm at the highest level. IEP guidelines state that questions on the texts to be included should be related to topic/main ideas (15%), inferencing (10%), and details (25%), with other skills and comprehension strategies from the course being targeted (50%).

The vocabulary that is assessed as part of reading includes both core IEP academic word list vocabulary (see Chapter 4), as well as vocabulary from the textbooks. The ratio of the AWL and textbook vocabulary increases in favor of AWL words as the levels get higher, from 25% to 50%. In addition to the basic words, collocations are also assessed. Both fill-in-the-blank and matching words with definitions question types are used.

Writing

Vocabulary is also part of the writing assessment. Students are provided with a word bank of 15 words and required to use at least 3 of them in their essays. Students are permitted paper dictionaries but no electronic resources. Students are provided with several topics and required to write as follows: (a) Level 3 is required to write one essay of two paragraphs, (b) Level 4 is given one essay type with three topic options, and students chose one. (c) Level 5 students are given three types of essay with one topic each and required to choose one. The type of essay required at Levels 4 and 5 is a traditional five-paragraph essay, with each paragraph required to be at least five sentences long.

While this approach seems highly structured, it is necessary to provide students with very clear guidelines on how to prepare and to ensure that students write enough to be evaluated.

Listening

In general, the listening finals consist of three sections: (a) 60-second listening segments that are loosely related to subject or topics that have been covered during the course of the semester. Many of them are drawn from National Geographic programs. Students are familiar with the format of these 60-second segments, as they are done throughout the term as a listening exercise, (b) Longer listening segments that are designed to include ‘social listening’, for example, conversations and/or interviews and ‘lecture’ texts, (c) A dictation component of ten sentences that includes items from the vocabulary of each level. Level 3 students are not required to take notes, but all other levels are required to provide notes from their listening. Levels 5 and 6 may be required to write a summary from their notes.

Speaking

In speaking, the evaluation consists of two activities, both of which are recorded: a conversation roleplay and a monologue. The roleplays require pairs of students to engage in a conversation, the topic for which is based on a prompt. Students in Levels 3 and 4 receive two prompts; students in Levels 5 and 6 receive three. Interviews at Levels 3 and 4 usually take about 4-5 minutes, while those in Levels 5-6 would take a little longer time. Their recorded monologue is called a ‘recorded speaking activity’ and is described and analyzed more fully in Chapter 6 (McCormick & Vercellotti, 2013). Students record two versions of their monologue and can choose which one they submit for evaluation.

Grammar

Grammar is assessed in two modes: listening and writing. Both modes for each level probe knowledge of all grammar points (AGP) taught at that level. The written grammar test consists of a variety of item types: traditional multiple choice, fill-in-the-blank, sentence completion, answering questions, and writing a short paragraph. The listening part of the grammar test begins with a comprehension ‘True or False’ section that taps students’ understanding of grammatical points and is followed by a test of cloze items. The listening section is played twice for each subsection.