Purpose: Individual perceptions of one's neighborhood environment influence decisions about physical activity participation. Differences between single-family housing neighborhoods versus multi-family housing neighborhoods may affect perceptions and lead to varying responses on surveys designed to assess perceptions of the neighborhood environment for physical activity. This study tested the factorial invariance for the Physical Activity Neighborhood Environment Survey (PANES) between residents of single-family versus multi-family housing neighborhoods. Method: This study was a secondary data analysis of PANES ratings from African American and Hispanic or Latina women (n = 324) who participated in the Health Is Power study (NCI R01CA109403), a multi-site, community-based trial to investigate the relationship between neighborhood factors and physical activity adoption and maintenance. Factorial invariance was tested using a series of nested confirmatory factor analysis models. Results: The final model was a 2nd-order factor structure with partial invariance of item intercepts. The 2nd-order factor structure and the relationships of the PANES items to the 1st-order factors (amenable, unsafe, and walkable) and of the 1st-order factors to the 2nd-order factor (environment) were invariant between the single-family and multi-family housing neighborhood groups. Conclusion: These findings support the construct validity of PANES, which can be considered valid for measuring neighborhood perceptions among residents of neighborhoods with different housing types.