![ORCID Icon]({"type":"image","src":"/templates/jsp/_style2/_tandf/images/orcid.svg"} "ORCID Icon"){kind=small}
http://orcid.org/0000-0001-6548-7970
![Sample our Urban Studies journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/110826/TOC-Subject-Sample-2021-Urban-Studies.jpg"})
Problem, research strategy, and findings: The American Community Survey (ACS) is a crucial source of sociodemographic data for planners. Since ACS data are estimates rather than actual counts, they contain a degree of statistical uncertainty—referred to as margin of error (MOE)—that planners must navigate when using these data. The statistical uncertainty is magnified when one is working with data for small areas or subgroups of the population or cross-tabulating demographic characteristics. We interviewed (n = 7) and surveyed (n = 200) planners and find that many do not understand the statistical uncertainty in ACS data, find it difficult to communicate statistical uncertainty to stakeholders, and avoid reporting MOEs altogether. These practices may conflict with planners’ ethical obligations under the AICP Code of Ethics to disclose information in a clear and direct way.
Takeaway for practice: We argue that the planning academy should change its curriculum requirements and that the profession should improve professional development training to ensure planners understand data uncertainty and convey it to users. We suggest planners follow 5 guidelines when using ACS data: Report MOEs, indicate when they are not reporting MOEs, provide context for the level of statistical reliability, consider alternatives for reducing statistical uncertainty, and always conduct statistical tests when comparing ACS estimates.
Acknowledgments
The authors thank the the editor and two anonymous reviewers, as well as Sy Adler, Charles Rynerson, Nick Chun, and Paul Lask for their feedback on earlier versions of this manuscript. We are also grateful to those planners participating in this research, including the Professional Development Officers of the American Planning Association chapters. The opinions, findings, and conclusions expressed in this publication are solely those of the authors and do not necessarily reflect the views of the supporting agency, and all errors and shortcomings are our own.
Supplemental Material
Supplemental data for this article can be found on the publisher’s website.
Notes
1. We provide sampling odds as a way to compare the sampling frame of the decennial long form alongside the ACS. However, to be clear, the ACS sampling frame is based on a fixed number of housing units, not a sampling rate. Furthermore, ACS estimates are pooled over multiple years to produce estimates. For more information, see USCB (2017).
2. The USCB increased the ACS sample size in 2011, and thus sampling error will be lower in post-2011 estimates. For more information on the effective sampling size of the ACS, see USCB (n.d.).
3. Addressing the importance of statistical uncertainty in ACS data, Williamson (2008, p. 46) notes, “ACS data users will have to review all their data and decide if some tabulations should be discarded because of high margin of error and/or imputation rate. Conversely, tabulations may also be strong and cited as more reliable compared to the others.”
4. Testing for statistical significance in ACS data is fully explained in Appendix 4 of the USCB ACS compass handbook (USCB, 2008). In graphical representations of ACS data, it is often tempting to assume that two estimates are statistically different from each other if their confidence intervals do not overlap (or conversely not significant if the confidence intervals overlap). Using the test for statistical significance as outlined in the ACS compass handbook provides data users with a paired t test that recomputes the MOE to assess whether the difference between the estimates is statistically significant. Therefore, we caution users against drawing inferences from graphical illustrations without conducting their own statistical tests.
5. Comparing the city of Portland’s child poverty rate in 2007 (20.4%, ±3.3%) with the 2011 rate (27%, ±4%) yields a statistically significant t of 2.0937. Comparing the 2007 rate for the Portland MSA (14.7%, ±1.3%) with those of 2010 (17.6%, ±1.4%), 2011 (20.3%, ±1.5%), 2012 (18.3%, ±1.3%), 2013 (16.9%, ±1.3%), and 2014 (17.3%, ±1.4%) yields statistically significant ts of 2.4970, 4.6409, 3.2211, 1.9685, and 2.2387, respectively.
6 These percentages are based on a total of 141 census tracts in Multnomah County (out of 171 total tracts in the county) that lie either wholly or partly inside the Portland city limits. Forty-one tracts (29%) have coefficient of variation (CV) values of 75% or higher, 61 tracts (43%) have CV values between 41% and 74%, and 39 (28%) have CV values between 12% and 40%.
7 The CV expresses the error relative to the estimate by dividing the standard error by the statistical estimate and multiplying the result by 100. Given the importance of statistical techniques in making policy decisions, the USCB publishes a series of ACS compass handbooks guiding data users on constructing confidence intervals and conducting tests of statistical significance. For more information, see http://www.census.gov/acs/www/guidance_for_data_users/handbooks/.
8. APA chapters sponsoring sessions focusing on ACS/MOE topics include Federal Highway Administration, Florida, National APA, Nebraska, Nevada, Pennsylvania, Planetizen, and the Southern California Association of Governments.
9 The highest level of education for survey respondents was a bachelor’s degree (17%), a master’s degree (60%), other professional degree (1%), or a doctorate (11%); some respondents did not answer (12%). Respondents who had completed a master’s degree had attained one of the following types of degrees: MA, MSc, MURP, MPH, MEng, MBA, or an equivalent degree.
10. The total number of years (M = 14.5 years, SD = 9.7 years) survey respondents had been working in planning was less than 5 (16%), 5 to 9 (21%), 10 to 14 (16%), 15 to 19 (10%), 20 to 24 (8%), 25 to 29 (7%), and 30 or more (12%); some respondents did not answer (13%).
11. The primary scale/focus of survey respondents’ planning organizations was neighborhood (5%), city (8%), county (19%), regional/metro area (37%), state (15%), nation (6%), global (2%); some respondents did not answer (10%).
12 The planning specialization(s) that respondents reported included long-range planning (48%), transportation planning (40%), health services planning (14%), historic preservation (4%), environmental or natural resource planning (12%), economic development (31%), land use planning (25%), housing planning (22%), and other (respondent specified; 34%); some respondents did not answer (10%).
13. Non-USCB sources of demographic data used by survey respondents included state governments (70%), other federal agencies (57%), municipal or regional governments (46%), school districts (38%), their own organization (41%), and county governments (34%).
14. We asked survey respondents the extent to which they agreed with the following statement: “I feel like the Census Bureau does an adequate job of explaining the ACS and margins of error.” Eighteen percent (n = 37) of respondents agreed with the statement, whereas 9% disagreed (n = 18) and 7% (n = 13) were neutral. A total of 132 respondents did not answer this question.
15 Chi-square tests of cross-tabulated data, specifically years of planning experience compared with correct responses to ACS questions (e.g., “I should be more careful when using American Community Survey data for small geographies [census tracts] than for large geographies [counties]”), revealed no statistically significant relationship between ACS knowledge and planning experience (χ2 = 2.92, p = .81). This suggests that formal planning education and professional continuing education should both serve as arenas for improving knowledge of ACS data.