SECTION II. Empirical Research and Practical Considerations on the Development of K–12 ELP Assessments
Collaborating with Educators on the Development of English Language Proficiency Assessments
In the U.S. K-12 educational system, accountability requirements mandate that states administer standards-based summative assessments annually to measure the progress English learner (EL) students have made in attaining English language proficiency (ELP). The development of such assessments is a complex endeavor, involving a range of stakeholders including state Departments of Education (DOE), educational researchers, test publishers, and classroom educators, among others. Classroom educators are, of course, significantly impacted by the implementation of these assessments, as they need to ensure their instruction supports students’ ELP development in a manner that is aligned with standards. While these classroom-level practitioners are often involved in the design and development of summative ELP assessments, relatively little documentation is available as to how their input is incorporated and what the impact of their input is.
The purpose of this chapter is twofold: (1) to offer a comprehensive description of classroom educators’ involvement in the various stages of the development of K-12 English ELP assessments and (2) to consider the impact of educators’ roles both on the validity of test score interpretations for such assessments and on the professional development of the educators themselves.
In discussing the educators’ involvement in the development of ELP assessments, we use contexts in which a state chooses to design and develop its own K-12 ELP assessment program (rather than joining a consortium or adopting an existing assessment), and we focus primarily on the development of an annual summative ELP assessment to be used for accountability purposes. Most of the examples in the chapter are drawn from the K-12 ELP assessment program of the state of California entitled the English Language Proficiency Assessments for California (ELPAC).
The chapter begins with a summary of the roles of key parties involved in the design and development of such assessments: state DOE, test publishers, and classroom educators. An extended example of the role of classroom educators in the development of a representative K—12 ELI’ assessment, the ELPAC, is provided, followed by a discussion of the benefits to the involvement of educators, including both a possible increase in test score validity and benefits to the participating educators themselves. The chapter ends with a conclusion providing recommendations for future practice.
Key Parties Involved in the Design and Development of K-12 ELP Assessments
When a new assessment program for the K-12 students in a given state is to be developed, three parties play key roles. These parties are the state’s DOE; a testing vendor, which may include one or more sub-contractors with special areas of expertise; and classroom educators. Given the focus of this chapter, the roles of the first two parties will be described briefly and the role of classroom educators will be covered in greater depth.
The state DOE is, in a real sense, the “author" of the assessment, the responsible and accountable party with the broadest authority for all aspects of the testing program. The state DOE typically initiates the test development process by releasing a Request for Proposals (RFP), which provides requirements for the major aspects of the testing program. Such RFPs commonly require that classroom educators be included in the development process, although states differ in the manner and degree of educator involvement called for. Once the contract has been awarded to a testing vendor, the state DOE provides oversight for all activities related to the testing program, including assessment design, test development, test administration, scoring, and score reporting.
Testing vendors are contracted by the state DOE and have primary responsibility for executing all requirements specified in the RFP (and in the resulting contract). They may also be called on to provide technical advice and solutions as needed to fill gaps in the RFP or to respond to changing needs and conditions. The requirements for the testing vendor cover a broad range of tasks from the highly conceptual (e.g., designing the assessment to ensure that meaningful, reportable evidence of students’ abilities is gathered) to the very practical (e.g., printing and shipping large numbers of test books to test administration sites and then receiving them back for scoring). The testing vendor is also responsible for implementing the RFP’s requirements for the involvement of classroom educators in various stages of the assessment design and development.
In the development of K—12 state assessments (including but certainly not limited to ELP assessments) asking classroom educators to participate in activities is common practice, although assessment programs vary in the type and number of stages in which educators are included. Classroom educators, by definition, work directly with students every day and provide instruction that constitutes the most practical interpretation and implementation of the state’s educational standards into the students’ education. As a result, they are well positioned to make informed judgments about grade-level appropriateness—a factor that is often referred to in state standards but rarely specified well enough in the standards to be directly operationalized in the assessment. Educators can be effective in representing the perspective of students in the test-development process because their specific knowledge of the classroom context enables them to make informed judgments about what constitutes appropriate evidence that students are (or are not) meeting the expectations described in the standards. This detailed knowledge of students and the classroom, which largely complements that of the DOE and the testing vendor, allows educators to help to ensure both that test items are well aligned with standards and instructional practice and that test items elicit evidence of student abilities at a variety of levels that will result in meaningful scores.
There are also significant benefits to educators in terms of professional development. A useful concept for analyzing the impact of involvement on classroom educators is assessment literacy (Popham, 2009; Stiggins, 1995, 1999) or more specifically language assessment literacy (Fulcher, 2012; Herrera & Macias, 2015; Inbar-Lourie, 2013; Malone, 2013; Taylor, 2013). Both assessment literacy (with a focus on assessment in general) and language assessment literacy (with a focus on language proficiency assessments in particular) define the knowledge, skills, and abilities necessary for classroom educators to successfully accomplish those parts of their roles that relate to assessment.
In this chapter, we will use Stiggins’ (1999) model of assessment literacy (one that is relevant for the subcategory of language assessment literacy as well) to analyze the impact that participating in assessment development activities has on educators’ professional development. Stiggins’ model identifies seven content requirements or competencies that constitute a comprehensive foundation in assessment practices including:
- 1) Connecting assessments to clear purposes
- 2) Clarifying achievement expectations
- 3) Applying proper assessment methods
- 4) Developing quality assessment exercises and scoring criteria and sampling appropriately
- 5) Avoiding bias in assessment
- 6) Communicating effectively about student achievement
- 7) Using assessment as an instructional intervention
During the development of the ELPAC assessments, educators had opportunities to meaningfully develop their competencies in these seven areas, as will be discussed in the next section. This section will also illustrate the major impacts of the involvement of educators—their contributions to the validity of the tests and the professional development they receive—by means of an extended discussion of the role that classroom educators have played in the design and development of the state of California’s ELPAC program.