Validity

As we move from reliability to validity, our focus changes from the tools we use in research to the manner in which we use them. Reliability considers the instruments we use to gather information or data, whereas validity looks at the process we use to draw conclusions from our data.

Validity is the approximate truth of suggestions, inferences, or conclusions. W hen our work in establishing theories and conducting experiments can be characterized as valid and rigorous, we have done our job. Validity is a difficult concept for many researchers because it is difficult to quantify’. There is no threshold or yardstick you can use to determine whether you are conducting a valid study or not.

Validity can be best understood by looking at the four different types of validity and considering how they build upon each other in a sequential manner.

Face Validity

Face validity is the starting point of all research. The question is whether the approach you are taking or the way you are measuring something or the results you are getting, make sense on their face. Do they pass the laugh test?

As a practical example, Ewing and co-authors have been operationalizing urban sprawl and its impacts since the early 2000s (Ewing et al., 2002; Ewing & Hamidi, 2017). Previous sprawl metrics had been based on gross density calculations, and by these metrics, Los Angeles was the least sprawling/most compact region of the country, even less sprawling than New York, San Francisco, Portland, Ann Arbor, and Madison. That certainly isn’t the public's perception of Los Angeles, nor the sense you get traveling around Los Angeles’s endless medium density. In part for this reason, Ewing et al. developed compactness measures that capture the distribution of density within metropolitan and urbanized areas, plus three other dimensions of compactness— land use mix, population and employment centering, and street connectivity. All of these contribute to accessibility of places to one another, the ultimate definition of compact development. They all provide alternatives to long-distance automobile travel. By these alternative metrics, New York is the most compact metropolitan area (Hamidi, Ewing, Preuss, &• Dodds, 2015) and San Francisco the most compact urbanized area (Hamidi & Ewing, 2014). Los Angeles is well down on both lists. This result has face validity (Figures 6.5 and 6.6).

In the work of Ewing et al., compactness/sprawl measures have been the main independent variables, related empirically to many dependent variables (Ewing &

Figure 6.5 Endless Medium Density of Los Angeles County

Source: Google

Figure 6.6 Centered Development of Arlington County, Virginia Source: Google

Hamidi, 2014). In one study, compactness was related to housing costs and separately to transportation costs (Hamidi & Ewing, 2015). Compactness, as they measured it, is positively related to housing costs. This seems reasonable. After all, compact places like New York and San Francisco are known for their exorbitantly high housing costs. At the same time, compactness, as they measured it, is negatively related to transportation costs. Again, this seems reasonable. Compact

Validity and Reliability 97 places like New York and San Francisco are known for their walkability and high-quality transit service, providing alternatives to long-distance travel by automobile. So their findings are intuitively plausible or, stated another way, they have face validity.

Construct Validity

Construct validity is a measure of how well we operationalize our constructs. In Chapter 5 on Conceptual Frameworks, we made much of the distinction between constructs, which are big abstract concepts, and the operational variables we use to imperfectly represent them. What we are trying to do in this step is to show that we have actually measured what we set out to measure, in our example, sprawl or compact development. This is an independent variable in much of Ewing et al.'s research. We also want to properly operationalize our dependent variables when we study the costs of sprawl.

Construct validity is subtly different from face validity. But it is different, as the following examples illustrates. Ewing and colleagues operationalized compactness not only at the metropolitan and urbanized area scales, but additionally at the county scale (Ewing, Meakins, Hamidi, & Nelson, 2014). Two measures of county compactness were calculated, an original index from 2002 based on six variables related to development density and street connectivity, and a refined index from 2014 based on 16 variables related to development density, land use mix, population and employment centering, and street connectivity. The refined index, on its face, would seem to be superior because it captures more dimensions of the construct compactness/ sprawl. That is, the refined measure has more face validity. But as it turns out, it also has more construct validity.

The most compact counties are largely the same when measured by the original and refined indices. The top ten include, in both cases, the four boroughs of New York City, San Francisco County, Philadelphia County, the District of Columbia, etc. But the ten most sprawling counties are entirely different for the two indices. Many of the counties rated as most sprawling according to the original index have development patterns that are not what one would expect. They would best be described as small towns surrounded by farmlands. The small towns have moderate densities and gridded streets. The fact they are part of larger census tracts, their units of analysis, depresses their densities and compactness scores. They are not classic examples of suburban or exurban sprawl.

On the other hand, the counties rated as most sprawling according to the new four-factor index have census tracts with very low residential densities, commercial strips, and cul-de-sac street networks. They are classic examples of suburban and exurban sprawl (Ewing & Hamidi, 2017). Put another way, the refined index represents the construct of sprawl more accurately than the original index (Figures 6.7 and 6.8).

Construct validity has also been an issue with dependent variables in costs of sprawl studies. Ewing, Hamidi, Grace, and Wei (2016) wanted to relate compactness/sprawl to upward social and economic mobility in the United States. The best available measure of intergenerational mobility was the likelihood that a child born into the bottom fifth of the national income distribution will reach the top fifth by

Figure 6.7 Most Sprawling County According to the Six-Variable Index (Jackson County, Kansas)

Source: Ewing & Hamidi (2017) age 30. Is this the best metric for representing upward social and economic mobility? Probably not. The obvious problem is this metric’s failure to measure smaller gains in social and economic standing, say, from the bottom half to the top half of the income distribution, or from the first quintile to the second quintile of the income distribution. Still, the metric they used is the best available and has enough construct validity to be credible.

Another dependent variable illustrates the idea of construct validity. Ewing and colleagues have long wanted to relate sprawl to traffic congestion, but until recently, the only available measures of congestion were modeled rather than observed (until 2010) and limited to the largest urbanized areas (until 2015). There are now observed measures of congestion from the Texas Transportation Institute for all urbanized areas, and they have enough construct validity to be credible. A recent study relates sprawl, as they have measured it, to traffic congestion, as measured by annual delay per capita from TTI (Ewing et al., 2017). Both measures have construct validity.

Figure 6.8 Most Sprawling County According to the Four-Factor Index (Oglethorpe County, Georgia)

Source: Ewing & Hamidi (2017)