Placement Assessment

In this section, we consider the following specific questions:

• 1. What theoretical and empirical assessment tools can inform IEP administrators’ decisions?
• 2. What kinds of statistics and qualitative evaluation can be used for assessment?
• 3. Is there a justification for relying more or less on one placement tool?

Description of the Placement Tools in the Intensive English Program

The first decision that a program must make concerns which level to place students in based on their proficiency at entry. This process is an important one for both students and administrators. The placement test is a high-stakes task for the student because this evaluation affects the time that they must spend studying English before they enter an academic program or feel ready to enter the workforce using English. (Students who are sponsored by their government may also have time limits for reaching a particular level. This situation may make them either eager to be in a higher or lower level depending on their specific circumstances. However, placement decisions should be made on the basis of proficiency fit with the curriculum level and not on time and funding available.)

It is also a high-stakes activity for the IEP administration. IEP students are not like international students who are going to take university degree courses, have made a large tuition deposit, and are therefore almost guaranteed to come to campus for their programs. For an IEP, it is possible that only 60-80% of students who have been issued I-20s to get a student visa will actually attend the institution. Thus, the Pitt IEP usually does not even know how many students will arrive to study or what their level will be on the first day because they are not required to present proficiency scores before they arrive. In this context, placement testing, subsequent decisions on splitting a class that is over-enrolled, and the consequent need to hire part-time instructors is an intense process. Student testing, placement, and orientation must be accomplished within a short turn-around time, often no longer than two days. A key factor is that placement cannot result in students clamoring to change levels - either because they think they should be in a higher level or because they want to be in a lower level for some other reason. Some testing researchers suggest consulting learners themselves as part of the diagnostic process (e.g., Harding, Alderson, & Brunfaut, 2015), but student self-assessment is not, and probably could not be, part of the process in the Pitt IEP.

Until 2012, the IEP used the MTELP for placement into three levels. For information purposes and for context for the data in later chapters, Table 3.1 lists the range of scores and cut scores (see Section 3.3.2) that

Table 3.1 Pitt IEP Placement Test Scores - Historical Viewpoint

Level MTELP IEP Writing CEFR Descriptor - Typical Beginning

List. Sample Equiv. Student

Comp.

Test

• • Can communicate only with much difficulty in most situations.
• • Fluency is very limited.
• • Comprehension is very limited.
• • Makes very many errors that lead to misunderstanding.
• • Can produce some discourse of several sentences (speaking and writing).
• • Can communicate with limited success in many situations.
• • Fluency is limited.
• • Comprehension is limited.
• • Makes frequent errors that lead to misunderstanding.
• • Can produce extended discourse beyond several sentences but still limited (speaking and writing).
• • Can communicate with success in many situations.
• • Has several weaknesses in fluency.
• • Has several weaknesses in comprehension.
• • Difficulties with accuracy cause some misunderstandings.
• • Can produce some extended discourse beyond the paragraph level (speaking and writing); writing shows some organizational awareness.

Placement and Assessment in an IEP 65 were used until 2015 during the period of data collection described in Chapters 4-7.

The MTELP was originally designed to test higher-level students’ readiness for academic study and to distinguish among higher-level students; it is a discrete point test, and contained rather dated language that sometimes might appear in a nineteenth-century novel. Thus, the IEP found that this test did not distinguish adequately at the lower levels. In other words, there were ‘floor effects’, which means too many learners scored low and made it difficult to make level-placement decisions distinguishing Level 3 from Level 4, even though large proficiency differences exist between those two levels. This issue is a very important one for programs because it is necessary to sort learners into manageable groups that match the materials prepared for different proficiency levels of students and the associated student learning outcomes that were discussed in Chapter 2.

Therefore, in order to address this problem, the IEP added the use of an IEP in-house writing test in order to ‘finesse’ the placement of students into appropriate levels. This test took the form of two writing prompts. Possible topics were:

• 1. Describe the place you grew up in. What are the advantages and disadvantages of XYZ for children?
• 2. Describe how you use X in your daily life. How has X changed people’s lives?

These writing tasks were designed to assist in determining students’ writing skills that would indicate where in the program a student might best start. Only 30 minutes were allowed for this task, so not a great deal of output was possible. These writing samples were scored on a range of 1-6, based on the following categories:

• 1. Grammar and punctuation, capitalization
• 2. Spelling, appropriate meaning, correct derivational form, variety
• 3. Content: does the sample address the prompt, are ideas fully developed (not formulaic), and is meaning clear?
• 4. Organization: main ideas supported with details and examples (includes reference to topic/support and cohesive devices)

The writing rubric thus targets the components of writing that are the focus of coursework in the IEP. This rubric is provided in Appendix I.

Subsequently, the IEP also developed a listening test that was matched to the tasks that the students would complete in the curriculum as tasks became progressively more sophisticated and more linguistically demanding from Level 3 to 4 to 5. There were four sections in the test:

• 1. General listening: 3 passages and 6 questions
• 2. Short academic lectures: 4 passages and 12 questions

• 3. Conversation, 6 conversations: 12 questions
• 4. Picture interpretation (listening and observing a picture): 2 pictures and 2 questions

All answers were multiple choice, with 32 possible points, converted to a percentage. Thus, the MTELP, the writing sample, and the listening tests scores were then used for placement.

As new tests became available, the IEP adapted its approach to placement assessment beginning in 2013, with cut scores revised in 2015 as additional levels were created. The IEP now uses a newer test, specifically designed for placement: The Michigan English Placement Test (Michigan EPT), https://michiganassessment.org/institutions/products-services/ michigan-ept/. (The test, previously known as CaMLA ept, was originally developed by Cambridge Assessment English and the University of Michigan’s English Language Institute Testing and Certificate Division.) The IEP uses the following parts of this test: the ept Listening (25 points), the ept Grammar (20 points), Reading (15 points), and Vocabulary (20 points). A combined listening and grammar-reading-vocabulary ept score out of 80 is used primarily to place students. The new established cut scores for the program are set out in Table 3.2.

After the adoption of the EPT, the IEP continued to use the in-house writing and listening tests with the EPT for several years. The analysis in Section 3.4 includes both scores from the EPT and these two tests and will show how the IEP made decisions to change its placement practice based on the analysis.