For decades, the relationship between travel and the built environment has been one of the most studied in urban planning. Built environments that are high on the D-variables—development density, land use diversity, street connectivity, destination accessibility, and distance to transit (which is low in compact developments)—are often described as compact. Those that are low are described as sprawling. A major tenet of the literature both on regional development and neighborhood design is that compact development reduces driving (Ewing & Hamidi, 2015a).

In his article in this issue of JAPA, “Does Compact Development Make People Drive Less?” (Stevens, 2017/this issue), Professor Mark R. Stevens has done exactly what we called for in the last sentence of our 2010 JAPA article, “Travel and the Built Environment: A Meta-Analysis” (Ewing & Cervero, 2010). We said, “As more built environment-travel studies appear in the planning literature, it will be important to update and refine our results” (Ewing & Cervero, 2010, p. 13). There is much to commend in Stevens’s article. It is methodical, novel, and clearly written. We have no doubt that it will be widely cited. But it also raises questions.

Both Stevens and we measure effect sizes in terms of elasticities of vehicles miles traveled (VMT) per capita with respect to the five D-variables. So we are measuring the same thing but getting different results, characterizing them differently, and reaching different conclusions. The questions are why the differences, and who has come closest to capturing the truth about travel and the built environment?

The clear advantage of the Stevens article relative to our earlier article is the expanded sample of studies that he summarizes through meta-analysis. The cut-off date for our meta-analysis was 2009; Stevens’s is 2015, and this literature, perhaps the most extensive quantitative literature in planning, continued to expand over this six-year period. Stevens’s expanded sample is all to the good.

Also, we are inclined to agree that the judicious use of meta-regression is an advance over our conventional meta-analysis, which produced weighted average values of elasticities as our summary measures of effect size. We emphasize the word “judicious.” At the time of our meta-analysis, meta-regression was not as widely accepted as it is today. Tom Stanley, who popularized the use of meta-regression, has produced two major works on the subject since 2010 (Stanley & Doucouliagos, 2012, 2014). We were aware of meta-regression back in 2010 but rejected it due to the small sample size of studies available at the time.

After reviewing Stevens’s study we find ourselves asking four fundamental questions about the methods he uses, the findings he presents, and the conclusions he draws from the findings. We believe that JAPA’s readers might raise the same questions. We organize our commentary around those four questions and our own reasoned responses to them. In some cases we include insights from responses to the questions that we obtained directly from Stevens; in other cases, we imagine that he will give his take on the issues in his written response to our commentary that will appear in a later issue of JAPA.

Does Stevens Draw Appropriate Conclusions From His Findings?

By far our greatest concern is that Stevens has overreached in his conclusions. Stevens’s takeaway message is that “compact development features do not appear to have much influence on driving” and that “planners should not rely on compact development as their only strategy for reducing driving unless their goals for reduced driving are very modest and can be achieved at a low cost” (Stevens, 2017/this issue, p. 7). These conclusions are not consistent with his results, or our earlier results (see below).

His “not much influence” conclusion may be all the casual reader will take away from his article. In this respect, Stevens’s article is reminiscent of an earlier article in JAPA by Echenique and co-authors titled “Growing Cities Sustainably: Does Urban Form Really Matter?” (Echenique, ­Hargreaves, Mitchell, & Namdeo, 2012). Their answer was “No.” Their article generated a firestorm of criticism on the then-active PLANET listserv for academic planners (Ewing, 2012). We would never equate Stevens’s well-documented, well-reasoned, empirical study to Echenique’s poorly documented simulation study, but it may have the potential to do more harm simply because of its relative rigor combined with its overreaching on conclusions.

Does Stevens Appropriately Characterize Our Results and His Results?

Our second concern is related to the first. We are not comfortable with the way results are characterized. Stevens summarizes our own results, from our 2010 meta-analysis, as follows:

What we actually said was:

Saying that relationships are “inelastic” is not the same as saying that relationships are “small.” Inelastic means that elasticities have an absolute magnitude of less than 1.0, which means that a 1% change in an independent variable may produce up to a 1% change in a dependent variable. No one would call that upper limit “small.” Indeed, we don’t think an elasticity of −0.22 is small. A halving of distance to downtown leads to a 22% reduction in VMT. Stevens’s own estimate of the same parameter suggests that halving of distance leads to a 63% reduction in VMT (see Table 1). Imagine the typical U.S. metropolitan area if 63% of the traffic were taken off of the roads. Virtually all phenomena we as planners routinely study are inelastic. It is almost a universal truth. The only elasticities we described as “small” were those of VMT with respect to population density and job density, which had magnitudes of −0.04 and 0.00. And we did say that the combined effect of the Ds, which are multiplicative, could be “quite large.”

The same problem arises when Stevens describes his own results. Stevens describes the elasticity of VMT per capita with respect to population density as “small”:

We have to ask: Small compared with what? From another meta-analysis, the short-run elasticity of VMT with respect to fuel price is only −0.10, while the long-run elasticity is on the order of −0.29 (Goodwin et al., 2004). In our own recent analysis in the journal Urban Studies, only one independent variable, urban area size, has an elasticity of VMT per capita with respect to an independent variable greater in absolute magnitude than 0.30 (Ewing, Hamidi, Gallivan, Nelson, & Grace, 2014). The elasticity of VMT per capita with respect to fuel price is just −0.20. We mention fuel price here because urban planning critics often advocate the use of pricing rather than land use regulation to rein in driving and its social costs. This is easy for critics because planners have no control over the cost of fuel or the generalized cost of driving. But we have some control over the D-variables.

Does Stevens Appropriately Depict the Costs and Benefits Associated With Compact Development?

Our next concern relates to the way costs and benefits are depicted. From Stevens’s article, you might conclude that the sole benefit of compact development is a reduction in driving, while the costs of compact development are substantial. Only in one brief reference does he acknowledge other benefits of compact development. This reflects the kind of reductionist trap that planning academics all too frequently fall into: framing the world from the lens of a single field or sector, whether transportation, housing, or the like.

We cannot begin, in this short commentary, to ­document all the benefits of compact development beyond that of reduced driving. In our own articles in other ­journals, and in the articles of many others, researchers have documented benefits in increased walking and transit use (A. L. Brown, Khattak, & Rodriguez, 2008; Cao, Handy, & Mokhtarian, 2006; Cervero, 2001; De Bourdeaudhuij, Sallis, & Saelens, 2003; Frank & Pivo, 1994; Gallimore, Brown, & Werner, 2011; Hamidi & Ewing, 2014; Hamidi, Ewing, Preuss, & Dodds, 2015; Humpel, Owen, Iverson, Leslie, & Bauman, 2004; Moudon & Lee, 2003), reduced residential energy consumption (Ewing & Rong, 2008; Pitt, 2013), reduced pedestrian and motor vehicle fatalities (Ewing & Hamidi, 2015b; Ewing, Hamidi, & Grace, 2016), increased physical activity and reduced obesity (Atkinson, Sallis, Saelens, Cain, & Black, 2005; Cervero & Duncan, 2003; Doyle & Kelly-Schwartz, 2006; Durand, Andalib, Dunton, Wolch, & Pentz, 2011; Ewing, Meakins, Hamidi, & Nelson, 2014; Forsyth, Hearst, Oakes, & Schmitz, 2008; Frank, Schmid, Sallis, Chapman, & ­Saelens, 2005; Handy, Sallis, Weber, Maibach, & ­Hollander, 2008; MacDonald, Stokes, Cohen, Kofner, & Ridgeway, 2010; McCormack, Giles-Corti, & Bulsara, 2008; Rundle et al., 2007; Saelens, Sallis, Black, & Chen, 2003), reduced ­household transportation costs (Hamidi & Ewing, 2015), decreased crime (Colquhoun, 2004; Hillier, 2004; ­Landman, 2009; Samuels, 2005), increased traffic safety (Dumbaugh & Rae, 2009; Marshall & Garrick, 2010), increased sense of community (Kim, 2007; Kim & Kaplan, 2004; Lund, 2002; Wood, Frank, & Giles-Corti, 2010), increased ­upward social and economic mobility (Ewing, Hamidi, Grace, & Wei, 2016), increased social interaction and neighborliness (B. B. Brown & Cropper, 2001; Lund 2003; Podobnik, 2011; Wilkerson, Carlson, Yen, & ­Michael, 2012), and increased social capital (­Leyden, 2003; Rogers, Halstead, Gardner, & Carlson, 2011; Wood et al., 2008).

Stevens’s article may also cause readers to despair for the high costs of compact development. Stevens focuses exclusively on one D-variable, density, and correctly states that single-family neighborhoods are generally opposed to densification in their backyards.

In Salt Lake City (UT), where one of us lives, there has been considerable opposition from single-family neighborhoods to high-density development nearby, particularly to transit-oriented development. What has not generated opposition is high density at a distance, for example downtown or in commercial centers, or land use diversity and the other D-variables nearby. Our own stated preference study, in Housing Policy Debate, suggests that housing consumers may not want to live in dense development or want dense development nearby, but they do want shopping and transit nearby, complete streets in their neighborhoods, and jobs within reasonable commuting distance (Tian, Ewing, & Greene, 2014). That covers most of the D-variables other than density. National surveys suggest an even more favorable view toward compact development than in the conservative state of Utah (Ewing & Hamidi, 2015a).

Does Stevens Appropriately Characterize His Meta-Regression Analysis as More Rigorous Than Our Meta-Analysis?

Finally, we question the characterization of this particular meta-regression as more rigorous than our meta-analysis using weighted average elasticities. Here is what Stevens says: “Meta-regression analysis, which I use to produce my findings, is the most objective and statistically rigorous approach to systematic reviews of a body of literature ­currently available” (Stevens, 2017/this issue, p. 15). This might be true but for the limitations of his particular meta-regression. Most important, Stevens’s final sample consists of just 37 studies, only 10 of which control for residential self-selection (see below). For individual D-variables, ­samples are as small as 18 studies. Readers will recall from their basic statistics course the rule of thumb that multiple regression requires a minimum sample of 50 cases or 10 cases per independent variable. Stevens’s sample sizes are inadequate for meta-regression. In our original article, we had an issue with sample size too, but simply reported descriptive statistics rather than applying inferential ­statistics via meta-regression analysis.

Stevens says that meta-regression has three advantages relative to conventional meta-analysis. “Meta-regression analysis can correct for the effects of sampling error by placing more weight on elasticities that were measured with more accuracy (or ‘precision’)” (Stevens, 2017/this issue, p. 9). Like Stevens, we weighted study elasticities by sample size to produce weighted averages; that is, we placed more weight on elasticities with smaller sampling errors.

The second presumed advantage is in an ability to control for residential self-selection. In this study, residential self-selection refers to the tendency of people who wish to drive less to locate in neighborhoods where they can drive less. These residents would probably drive less than their neighbors wherever they lived. Studies that fail to account for residential self-selection are believed to overstate the impact of the built environment on ­household travel behavior. Quoting Stevens:

We applaud Stevens’s attempt to control for residential self-selection. Self-selection bias is a genuine threat to the validity of studies that simply report the strength of relationships between D-variables and travel outcomes. However, as Stevens acknowledges, his ability to control for this effect is approximate at best due to his small sample of studies that account for residential self-selection (10 of 37 studies): “I thus make no distinction between the different methods that the 37 studies use to control for residential self-selection: I only distinguish between those that do control and those that do not. This is a limitation that can possibly be addressed in future research as more data become available” (Stevens, 2017/this issue, p. 13).

In our meta-analysis, we identified studies that controlled for self-selection and acknowledged that the phenomenon could lead to either underestimates or overestimates of the effects of the built environment on travel behavior. We did not compute separate average elasticities for studies of the two types for the reason just mentioned (small sample size). In fact, Stevens’s results suggest that residential self-selection actually increases elasticities in most cases relative to our weighted average values. So, as advocates of compact development, we may like his results but question his fix.

Finally, Stevens has used Stanley’s procedures to correct for “selective reporting bias.” This is related to “publication bias,” or the greater likelihood that results that show statistical significance will be published than those that do not. Readers are referred to Stevens’s article in this issue for a more nuanced discussion of the subject. We question whether it is possible to control for reporting bias when all of the studies in the sample have been published. It is possible to control for residential self-selection with a dummy variable because some studies account for self-selection and others do not. But variables by definition have to vary, and there is no variation in his sample with respect to publication.

On this score, we acknowledge we are on shaky ground. We bought Stanley’s book, carefully read the chapter on reporting bias, and still are not convinced. When we first saw this meta-regression study, we asked Bill Greene (of Econometric Analysis fame) about this particular fix, and he was as perplexed as we are. One point we are certain of is that many built environment travel studies report insignificant results for certain variables, but significant results for others. It has become almost universal practice in our field to report all results, not just those for significant variables. So any effect of reporting bias may be mitigated by common academic practice.

Making Sense of Different Results

None of this would be particularly problematic if it did not lead to different results for the two methods of meta-analysis. In Table 1 we report our original weighted ­average elasticities, weighted average elasticities for ­Stevens’s larger sample of studies, Stevens’s estimated elasticities controlling for residential self-selection, and ­Stevens’s estimated elasticities controlling for both ­residential ­self-selection and reporting bias. In some cases, the values in the four columns are very similar. For ­example, elasticities of VMT with respect to distance to transit are virtually identical. In such cases, it probably does not matter which values planners use. In other cases, results are very different. For these, planners have a distinct choice. We assume that the second column of results is superior to the first (though we wonder about the elasticity of VMT with respect to distance to downtown; see below), and think that the third column may be superior to the second. We are unsure about the fourth column. Readers, take your pick.

Our use of weighted averages suggests that the effect of density itself on travel is limited. This seems reasonable. Density has always been viewed as a proxy for other D-variables, and if these other variables are controlled, density should not have a big impact on travel. That is, dense development, without the other Ds, is still auto dependent. As an example, Portland (OR) Metro has relatively high suburban densities, but is auto dependent outside of light rail transit (LRT) corridors. It is hard to believe that VMT, walking, and transit use are much different in Portland’s suburbs than in any other suburbs (Ewing & Hamidi, 2014).

Another difference is in the effect size of land use diversity. Our use of weighted averages suggests that mixed uses reduce VMT per capita, whereas it has the opposite effect after Stevens’s adjustment for publication bias and residential self-selection. Common sense and the preponderance of the literature tell us that land use mix, more than anything else, should enable and promote walking and bicycling. Again, what is going on with meta-­regression that can produce such different results?

A third difference is in the effect size of distance to downtown. Our own use of weighted averages suggests that VMT per capita increases with distance to downtown, as destination accessibility declines moving from urban core to distant suburb. However, when Stevens expands the sample, the effect disappears from weighted average elasticity. Stevens presents the literature as ambiguous: “Naess’s elasticity of +0.31 suggests that driving goes down as households get closer to downtown, whereas Guerra’s elasticity of −0.22 suggests the opposite. Meta-analysis in general (and meta-regression analysis in particular) can help to resolve this type of confusion” (Stevens, 2017/this issue, p. 8). Yet, looking at the studies that comprise ­Stevens’s expanded sample, Guerra’s elasticity looks like an outlier that probably should have been dropped from the sample. For 9 of 11 studies in Stevens’s sample, VMT increases with distance to downtown.

This very comparison raises apples-and-oranges ­concerns. Naess’s experiences are from bike-friendly ­Copenhagen (Denmark), where owning and using a car is extremely pricey, whereas Guerra studied travel in worlds-apart Mexico City. Large concentrations of Mexico City’s low-income workers live in cheap housing in the far-flung periphery, courtesy of the Infonavit housing subsidy program, and their reliance on public transport to get around lowers VMT in the region’s outskirts, as documented by Guerra (2014) and Monkkonen (2012). This contrasts with wealthier residents concentrated in the urban core who, on average, drive far more. Mexico City is likely not the only atypical outlier in Stevens’s sample frame, given it also includes cities from Chile, Malaysia, and Germany. Might including such cases in an already small sample risk skewing results every bit as much as possible self-selection or publication biases? Normative statements about density and driving shaped in part by experiences in such widely different settings is, in our view, problematic.

Indeed, because of these anomalies and caveats, we are now of a mind that our meta-analysis and Stevens’s meta-regression may not be the best approach to deriving measures of effect size for built environmental effects on travel in the U.S. context. For years, we have been accumulating household travel databases and built environmental data from metropolitan regions across the United States, and are currently up to 23 regions as diverse as Boston (MA) and Atlanta (GA), Portland, and Houston (TX). Our combined database now contains 81,056 households and 815,204 people, with precise geocodes for all households and trip ends, and 11 D-variables for buffers of different widths around households. This combined database allows us to model VMT with at least as much precision and external validity as Stevens’s meta-regression. Our preliminary results, published in Urban Studies, suggest that all types of D-variables have statistically significant effects on VMT (Ewing, Tian, et al., 2014). Our regression coefficients can be used in the exact same way as the elasticities from Stevens’s meta-regression or from our original meta-analysis.

Many academic planners, in an effort to avoid the appearance of bias, throw up their hands when 90% of the evidence points in one direction and 10% in the other. We believe that Stevens has fallen into this trap, and that the practitioners who rely on our research deserve better.