Results

For brevity, we report only models where the primary findings are significant. Responses to the race-related questions are summarized in Table 11.1. Our first research question is whether the kinds of gender- and race-of-interviewer effects observed with human interviewers are reproduced with Vis, and if so whether they are as prevalent with Vis as with human interviewers. We observed no gender-of-W effects in our data (not shown) so

Number of Turns and Eents

We first applied latent class analysis (e.g., Kreuter, Yan, and Tourangeau 2008) to the interviewer observations. Briefly, this multivariable modeling approach takes as input multiple categorical variables (e.g., the post-survey interviewer observations from each survey listed above) that an analyst believes indicate some latent (unmeasured) categorical trait (e.g., overall response quality). The analyst specifies a hypothesized number of categories, or classes, for the latent categorical trait, and the estimation procedure results in predicted probabilities of membership in each class for each case in the data set (e.g., survey respondent). In addition, the estimation procedure generates conditional probabilities of each category on the input items, depending on the predicted class membership; this allows analysts to profile the latent classes in terms of distributions on the observed categorical measures. One can evaluate the fits of competing latent class models with different counts of hypothesized categories for the latent trait and select the model that provides the best fit to the observed data.

Number of Words

TABLE 12.1

Model Variance Components Predicting Log #Words Littered, and the Occurrence of Interviewer and Respondent Filled Pauses and Respondents' Uncertainty Markers

Log #Words Uttered

Interviewer Filled Pause

Respondent Filled Pause

Respondent

Uncertainty

Var

ICC

Var

SD

Var

SD

Var

SD

Null model

Mode

0.13***

0.07

0.63**

0.79

<0.01

<0.01

0.00

0.00

Question

0.63***

0.64

1.82*

1.35

1.29*

1.14

4.96*

2.23

Respondent

0.18***

0.05

1.62*

1.27

0.88

0.94

0.92

0.96

Model fit statistics

Log-likelihood

-3,939.90

-2,160.2

-3,127.5

-1,176.1

AIC

7,889.90

4,328.3

6,263.0

2,360.2

Note: n = 108 respondents, 81 questions, 2 modes, total n = 6,629. * p < 0.05, ** p < 0.01, *** p < 0.001.

(Shapiro W=0.69, p < 0.01, skewness=3.3), log transformation of the variable yielded nearly normally distributed data (Shapiro W = 0.98, p < 0.01, skewness 0.01). The log transformation enabled us to use parametric tests, which allowed for analysis of effects of several independent variables.

Base model. The base model (containing no covariates) predicting the logged number of words is shown in the first column of Table 12.1. Variance terms for the question, respondent, and mode were significant, indicating significant variability due to all three sources. The intraclass correlation coefficients show how much percent of the variance is due to the source; i.e., 6.7 percent was due to the mode of interviewing, 64.2 percent was due to questions, and 5.2 percent was due to respondents.

Question and respondent characteristics. Table 12.2 (first column) shows a model that includes all of the question and respondent characteristics. This model, compared to the base model, explained about 57 percent of the initial variability in number of words uttered due to questions, whereas 87.4 percent of the variability due to mode was explained. Several interactions between mode and question characteristics were significant. Mismatch and show card questions yielded longer interactions (coef = 0.96, p < 0.01, and 0.55, p < 0.01, respectively), but in CAPI these questions yielded shorter interactions (coef=-0.47, p <

0.01, coef=-0.35, p < 0.01, respectively). Similarly, closed nominal questions (as compared to open-ended questions) yielded longer interactions (coef = 0.86, p < 0.05), but shorter for CAPI interviews (coef=-0.96, p < 0.01). In CAPI, closed numeric, closed ordinal, and yes/ no questions also yielded shorter interactions (coef=-1.27, p < 0.01, coef=-0.96, p < 0.01, and coef=-0.54, p < 0.01, respectively), whereas main effects for these question types were not significant. Questions with longer lists of response options yielded shorter interactions (coef=-0.09, p < 0.05), but in CAPI these interactions were longer (coef=0.07, p < 0.01). In addition, sensitive questions yielded more words, regardless of mode (coef=0.38, p < 0.01). Evaluation of respondent characteristics showed that age and gender were not associated with the number of words uttered, but level of education showed that respondents with an educational level of high school or lower uttered more words (coef=0.10, p < 0.01) than respondents with higher levels of education. Inclusion of respondent characteristics reduced variance due to respondents by 4.24 percent.

Log #Words Uttered Coef. (SE)

Interviewer Filled Pause Coef. (SE)

Respondent Filled Pause Coef. (SE)

Respondent Uncertainty Markers Coef. (SE)

Constant/Intercept

2.64 (0.30)***

-4.24 (0.98)***

-0.04 (0.74)

-5.57 (1.77)**

Mode (CAPI)

0.15(0.10)

1.77 (0.76)*

-1.17(0.06)

-0.20(1.63)

Necessary question features

Question length

-0.00 (0.00)

0.02 (0.00)***

-0.2 (0.06)

-0.01 (0.02)

# Response options

-0.09 (0.04)*

-0.10(0.13)

0.01 (0.01)

0.28 (0.22)

Type of question Behavior (ref)

_

_

_

_

Attitude

-0.03 (0.21)

-0.84 (0.55)

-0.42 (3.75)

0.80(1.02)

Demographic

-0.30 (0.17)

0.47 (0.51)

-1.70(0.37)***

1.02(0.94)

Format of response options

Open-ended text (ref)

-

-

-

-

Open-ended numeric

-0.13 (0.45)

-0.34 (1.27)

-1.18(0.26)

-1.46(4.63)

Closed numeric

0.75 (0.45)

-0.34 (1.46)

-1.17(1.07)

-4.26 (2.41)

Closed nominal

0.86 (0.36)*

-0.15(1.31)

-0.91 (0.98)

-2.65 (2.09)

Closed ordinal

0.29 (0.31)

-0.74 (0.95)

-1.73 (0.69)*

-3.35 (1.69)ยป

Yes/no

-0.08 (0.29)

-1.01 (0.87)

-1.87 (0.68)**

-0.31 (1.65)

Respondent task complexity

Question reading level

0.03 (0.02)

0.05 (0.05)

0.03 (0.03)

0.04 (0.08)

Mismatch Q and response options

0.96 (0.47)***

-0.54(1.21)

2.31 (0.78)**

-3.12 (2.36)

Sensitive question

0.38 (0.13)***

0.02 (0.34)

-0.19(0.22)

1.80(0.64)**

Unknown terms

-0.06 (0.18)

-0.04 (0.45)

0.13(0.29)

-0.34 (0.86)

Showcard question

0.55 (0.20)**

1.19 (0.61)

0.69 (0.40)

1.25(1.04)

Interviewer instructions

0.41 (0.13)**

0.44 (0.34)

0.28 (0.21)

0.81 (0.63)

Female = 1

0.02 (0.04)

0.04 (0.00)

2.01 (0.19)

0.17 (2.42)

Age (centered)

0.00 (0.00)

-0.05 (0.03)*

-0.01 (0.02)

-0.05 (0.02)

HS degree or less = 1

0.10 (0.04)**

0.34 (0.29)

0.09 (0.22)

0.02 (0.26)

Choice of mode = 1

0.01 (0.05)

0.59 (0.37)

-0.43 (0.29)

0.18 (0.53)

Mode* question length

0.01 (0.00)***

0.04 (0.00)***

0.02 (0.00)**

0.02 (0.55)

Mode* # response options

0.07 (0.02)***

0.11 (0.11)

0.01 (0.09)

-0.26 (0.19)

Mode* attitude

0.09 (0.05)

0.67 (0.38)

-0.15 (0.30)

-0.18 (0.55)

Mode* demographic

-0.09 (0.04)*

-0.57(0.41)

0.62 (0.40)

-0.26 (0.82)

Mode* open-ended numeric

0.00 (0.37)

-1.62(1.99)

1.28 (2.59)

1.70(4.63)

Mode* closed numeric

-1.27(0.17)***

-1.90(1.21)

-0.04(1.07)

3.37 (1.67)*

Mode* closed nominal

-0.96 (0.17)***

-0.47(1.21)

-0.07(1.03)

1.70(1.69)

Mode* closed ordinal

-0.96 (0.10)***

-1.52 (0.75)*

-0.12(0.69)

1.42(1.24)

Mode* yes/no

-0.54 (0.08)*โ€œ

-0.55 (0.67)

-0.32 (0.62)

-0.08(1.41)

Mode* question reading level

0.01 (0.00)*

0.00 (0.03)

0.01 (0.02)

1.58(0.05)

Mode* mismatch Q and

-0.47 (0.10)***

-0.29 (0.83)

-1.36(0.70)

-0.14 (2.15)

response options

(Coninued)

TABLE 12.2 (CONTINUED)

Model Coefficients and Standard Error (in Parentheses) of Question Characteristics and Respondent Characteristics Predicting Log (#Words Uttered) and the Occurrence of Interviewer and Respondent Filled Pauses and Respondents' Uncertainty Markers

Log #Words Uttered Coef. (SE)

Interviewer Filled Pause Coef. (SE)

Respondent Filled Pause Coef. (SE)

Respondent Uncertainty Markers Coef. (SE)

Mode* sensitive question

0.01 (0.03)

0.11 (0.23)

0.24(0.18)

-0.42 (0.37)

Mode* unknown terms

0.01 (0.04)

0.04 (0.30)

-0.15(0.22)

-0.14 (0.48)

Mode* show card question

-0.35 (0.06)***

-0.83 (0.44)

0.37 (0.34)

0.14 (0.68)

Mode* interviewer instructions

0.02 (0.03)

0.21 (0.23)

0.21 (0.17)

-0.34 (0.35)

Random effects

SD - mode

0.04***

0.01

0.02

0.00

SD - question

0.43***

0.91โ€œ

0.59

1.87

SD - respondent

0.17***

1.22โ€œ

0.93

0.97

SD - residual

0.40***

-

-

-

Log-likelihood

-3,896.2

-2,104.5

-3,067.0

-1,147.5

AIC

7,836.4

4,287.1

6,212.0

2,371.0

Wald chi-square

6.98**

n= 108 respondents, 81 questions, 2 modes, total n = 6,629. * p < 0.05, ** p < 0.01, *** p < 0.001.

Interviewer and Respondent Filled Pauses and Respondent Uncertainty Markers

In addition to the number of words uttered, we explored the occurrence of interviewers' and respondents' filled pauses and whether respondents uttered uncertainty markers in mixed effects logistic regression models.

Base model. The base models (containing no covariates) are shown in Table 12.1 (columns 2, 3, and 4). For interviewers' filled pauses, variance terms for the question, respondent, and mode were significant, indicating significant variability due to all three sources. For respondents' filled pauses and uncertainty markers, only the variance term for the question turned out to be significant.

Question and respondent characteristics. Table 12.2 (columns 2, 3, and 4) shows models for question and respondent characteristics for filled pauses and uncertainty markers. Only for interviewer filled pauses, mode showed a significant effect, indicating that odds of interviewer filled pauses were higher in CAPI than in CATI. A significant interaction effect was found between mode and question length for interviewer filled pauses. As expected, longer questions yielded more interviewer filled pauses (coef=0.02, p < 0.001, odds ratio 1.02) and this effect was stronger for CAPI than for CATI (coef=0.04, p < 0.01, odds ratio = 1.04). In addition, as compared to open-ended text questions, only in CAPI, closed ordinal questions decreased the odds of interviewer filled pauses (coef=-1.52, p < 0.05, odds ratio = 0.22). Respondent filled pauses increased due to question length only in CAPI (coef = 0.02, p < 0.01, odds ratio = 1.02), and demographic questions, as compared to behavior questions, decreased the odds of respondents' filled pauses (coef=-1.70, p < 0.01, odds ratio = 0.18). For respondents' uncertainty markers, a significant effect was found for sensitive questions, indicating that sensitive questions increased the odds of an uncertainty marker (coef=1.80, p < 0.01, odds ratio = 6.05). Evaluation of respondent characteristics (Table 12.2, columns 2, 3, and 4) shows that for both interviewers' and respondents' filled pauses, gender and education of the respondent were not associated with the occurrence of filled pauses, but for age, a significant effect was found; with increasing respondents' age, the odds of an interviewers' filled pause were lower (coef=-0.05, p < 0.05, odds ratio=0.95).

 
Source
< Prev   CONTENTS   Source   Next >