Inter-Rater Reliability

Methods

An observational study using nine insulin-controlled adult patients and three trained pairs (MPH students) of interviewers (A-B, A-C, B-C) was conducted twice on three consecutive days during a four-week period. It established the objectivity of rater assessments by three trained staff. The methods noted in Table 4.10 can be used to determine the degree of interview/observer agreement. A standardized instrument, discussed in the section on criterion validity, was used, and a series of questions were asked by rater pairs of the nine patients to determine (1) demographic, (2) educational, (3) instructional, (4) cognitive, and (5) self-care behavioral characteristics.

table 4.10 Patient Assessment Methods to Establish Levels of Rater Agreement

rater pairs

patients day

o

1

o

2

A & B

1, 2, 3

1

0

x

0

B & C

4, 5, 6

2

0

x

0

C & A

7, 8, 9

3

0

x

0

O1 = observation; X

= staff training; O2

= O1

+ 4 weeks

All nine patients were also given a standard set of materials and were asked to show how they perform at-home insulin injection. After the baseline level of agreement in Week 1 for the patient-education assessment staff (Rater A-B-C) was established (Oj) for each category, using the standardized instrument presented in Table 4.5, a patient interview/observa- tion training program was conducted in Week 2 and 3. The objective of this study was to reduce rater error in the patient assessment process noted in the first observational study (O1).

The rater-training sessions required approximately three hours. It consisted of systematic question-by-question discussions of the instrument and data of the nine patients among the three assessment staff, a group lecture-discussion of interview/observation techniques, and individual question-by-question discussions with the primary investigator (PI). After completion of the training, a second rater study of nine patients, replicating O1 methods, was performed one month after the first study: O2.

 
Source
< Prev   CONTENTS   Source   Next >