Mixed-methods evaluation of a New York City resource guide for low-income residents

Abstract The Citywide Guide (“Guide”) is an online and bound publication in New York City that lists low- or no-cost resource and service providers—organized by 21 different service categories—available at no cost to New York City residents and service providers by the Women Creating Change (formerly Women’s City Club of New York). Women Creating Change (WCC) aims to empower underrepresented women with the information and tools they need to become civically engaged change-makers. The primary purpose of the evaluation was to determine the value of the Citywide Guide to New York City residents living in or near poverty, and to determine how web-based information resources can be better marketed and utilized by the less advantaged populations in large urban settings. This mixed methods evaluation of the Citywide Guide includes information collected from end-users of the Guide including results of an online survey with 57 total respondents and of a focus group discussion of seven participants that was held at WCC headquarters. Forty-seven percent and 38% of the survey respondents reported using the online and hardcopy versions of the Guide, respectively, and “Housing and Homelessness” was the category explored most frequently by respondents when using the Guide. We conclude that the Citywide Guide is a valued resource for New York City residents, however recommendations for improvement were provided regarding format, distribution, and content of the Guide.


PUBLIC INTEREST STATEMENT
With nearly 20 percent of the New York City population living at or below the poverty level, it is important to provide access to beneficial resources and services that can aid this population. The Citywide Guide is a free web-based and printed resource guide that provides information about the low-or no-cost services available across New York City's five boro.ughs in categories such as healthcare, homelessness, and education. This paper evaluates the quality and utility of this resource guide that could have a potential beneficial impact on more than 1.7 million indigent people living in New York City. Additionally, the key findings reported here may be useful for other organizations in large urban settings with high poverty rates that wish to create or improve upon a similar resource designed for disadvantaged populations.

Introduction
An evaluation of an on-line guide of social services and other resources for low-income urban residents is presented in this paper. A corresponding bound version of the guide is also described. Despite the several studies evaluating web-based information resources, none have evaluated information resources targeted to low-income residents in large urban areas. Indeed, most of the evaluated resources specifically are devoted to information about health conditions, as opposed to information about a variety of social services that are available in a community. For example, a PubMed search of the literature published in the past five years using the terms "information and referral resource" and "evaluation" and "Internet" yields only 8 publications; in contrast, a search of "consumer health information" and "evaluation" and "Internet" results in 129 publications over the past five years.
Women Creating Change (WCC) is a non-profit, non-partisan organization in New York City (NYC) founded in 1915 to advance women's rights. In its second century of service, WCC is working to empower New York women to become change-makers through increased civic engagement and activism. The Citywide Guide is a list of resource and service providers-organized by service categories-made available at no cost to New York City residents and service providers by Women Creating Change. Since its inception, WCC has encouraged civic engagement, activism, and leadership to decrease barriers to equal opportunities for all citizens. The organization has tackled several public policy issues including labor, living conditions, education, and reproductive rights for women. The implementation of the Citywide Guide by WCC in 2012 was consistent with several goals of their mission, including promotion of safe and secure communities, reduction of income inequality, and improved public education.
The Citywide Guide is available both online and in bound publications, one for each of the five NYC boroughs (Brooklyn, the Bronx, Manhattan, Queens and Staten Island). The online Guide is linked through the Women Creating Change website and is searchable by keyword, zip code, borough, and service category. The online Guide also features a language filter (Bengali, Chinese, French, Russian, and Spanish) so that users may see only providers that speak the selected language. There are 21 different categories of resources and services included in both the online and bound copies of the Citywide Guide, such as child care; health and mental health; adult education; legal services; and housing and homelessness.
Although the Citywide Guide is open to the public and available to all New York City residents, it is targeted towards the nearly 20 percent of NYC residents who live in poverty or near the poverty line (Chatterjee et al., 2018). With a NYC population of 8.6 million people, the Citywide Guide provides applicable and potentially beneficial information to about 1.7 million residents (US Census Bureau, 2017). With such a large possible outreach, it was important to evaluate the Citywide Guide to establish its value to New Yorkers, learn how the Guide is being utilized, and determine what end-users and service providers like best about the Citywide Guide and areas where improvement might be possible.
The vast majority of evaluations of web-based resources seem to be devoted to consumer health information. In recent years, just a few examples of evaluations of web-based information related to consumer health that have been published include such issues as breast cancer, dental caries, swallowing disorders, dengue, medications for individuals with dementia, malaria in pregnancy, advance directives, senior health and palliative care (Blizniuk, Furukawa, Ueno, & Kawaguchi, 2016;Dana, Sobczak, & Tarver, 2019;Hamwela, Ahmed, & Bath, 2018;Horne et al., 2018;Nghiem, Mahmoud, & Som, 2016;O'Connell Ferster & Hu, 2017;Pereira, Paolucci, Daumas, & Souza, 2017;Stuart, 2017;Tanabe et al., 2018). Most of these studies represent not evaluations of specific websites that offer information, but rather more generally of health information that is available from the Internet The primary purpose of the evaluation presented in this paper was to determine the value of the Citywide Guide to New York City residents living in or near poverty, and to determine how this specific web-based source of information on social services and other resources can be better marketed to and utilized. A companion hardcopy version of the Citywide Guide was also included in the evaluation. This paper will describe the methods used to evaluate the Citywide Guide and the key findings of the evaluation. We will also present recommendations for the improvement of the Citywide Guide, as well as the implications of our findings for low-income residents of other large urban populations.

Methods
The evaluation of the Citywide Guide was conducted utilizing a mixed quantitative and qualitative approach that is very commonly used to assess web-based information and resources. The quantitative findings were based on surveys of WCC guide participants, and the qualitative findings were based on a focus group of participants.
A mixed quantitative and qualitative method of analysis, such as reported in this paper, has been adopted in other evaluations and studies of web-based information resources in order to determine the value and usefulness of information resources. The U.S. Department of Health & Human Services' Office of Family Assistance (2010) suggest collecting both qualitative and quantitative data when conducting evaluations. In their review of assessment tools popularly used to analyze the readability, emotional content and comprehensibility of online health information, Beaunoyer, Arsenalt, Lomanowska and Guitton (2017) found that mixed strategies combining quantitative and qualitative evaluations were the most effective.
Focus groups, questionnaires, surveys, and interviews are common metrics utilized in studies involving the creation, evaluation, or usability of online resource tools (Hawton et al., 2012;Kukafka et al., 2007;Madle, Berger, Cognat, Menna, & Kostkova, 2009;Rosenfeld, Shepherd, Agunwamba, & McCray, 2013;Street et al., 2007). An iterative evaluation of a web-based health information resource was conducted using a mixed-methods approach that included: an online survey, interviews, focus groups, and assessment tools. The evaluators found that their resource was valuable, but that formal evaluations must be conducted regularly for information resources targeted for public use (Rosenfeld et al., 2013).
Another study conducted by Tang et al. (2009) sought to evaluate the feasibility of using a webbased information resource made for health services providers of reproductive health services in rural China. This study used baseline and follow-up surveys and in-person workshops to determine the value and feasibility of their information resource. They found that using a web-based information resource was feasible, willingly implemented, and allowed health educators and health services workers to disseminate information to villagers and students in rural China (Tang et al., 2009). Using mixed-method approaches for data collection allows increased flexibility of participants, as well as a broader analysis of the data to better determine the value and usefulness of the information resource tool (Evans et al., 2017).
Women Creating Change (WCC) contracted analysts at the Center for Long-Term Care (CLTC), housed within New York Medical College, to conduct an evaluation of their Citywide Guide solely as a business quality-improvement initiative. WCC gathered the quantitative and qualitative data that was analyzed in completely de-identified form by the CLTC evaluation team. In consultation with WCC and in consideration of their goal to improve their business product (the Citywide Guide), CLTC analysts advised WCC on the design of the survey that was used to gather quantitative data about satisfaction with the Citywide Guide, and on the structure of script used to run the focus group to gather qualitative information. In consideration of the business quality-improvement nature of the project, and of CLTC's strictly evaluative role, the involvement by CLTC analysts was granted an exemption from the Institutional Review Board at New York Medical College.
Quantitative data from the survey results were analyzed both from detailed reports available in SurveyMonkey as well as Stata/SE 14.2 statistical software. Audio recording of the focus group discussion was made via Zoom web conferencing, then transcribed into Microsoft Word.

End-user survey
A survey to gain insight and feedback from users of the Guide was distributed using SurveyMonkey, an online survey production and implementation software. The survey, which was designed by CLTC evaluators on behalf of WCC, who distributed it to users. The questionnaire included 33 multiplechoice questions and six open-ended (free response) questions. After a common set of demographic questions, the survey branched into two sets of questions-one set for users primarily of the online version of the Guide, one set for users of the hardcopy version. While the questions were similar for online and hardcopy users, they were tailored to the version of the Guide that respondents reported using the most. For example, only users who reported using the online version of the Guide were asked about their experience using the website keyword search. A survey question regarding Guide utilization type determined which set of questions the respondent would answer after the initial demographic questions. It was determined that those respondents who selected using both Guide versions equally would answer questions that were specific to only the online Guide.
Survey responses were collected through a link to the survey via the Women's City Club website. Additionally, users of the Guide who are on a Citywide Guide email list or who receive a hard copy of the Guide were individually contacted to participate in the survey. The survey was open for response collection for approximately four months, from 23 February 2018 until 30 June 2018. All respondents were informed at the beginning of the survey questionnaire that by completing the survey they would automatically be entered into a drawing for a $50 gift card. Once the survey was closed, two participants were randomly selected as winners and received the $50 gift card. Additionally, survey feedback about end-users' experiences with the Guide were described via open-ended questions, including how the Guide was accessed, how users heard about it, their frequency of use, the types of services and resources searched most, and their ability to find needed information when they used the Guide.

Focus group
A focus group of end-users of the Citywide Guide was held 6:00 to 8:00 p.m. at the Women Creating Change headquarters in New York City on 6 June 2018. The objective of the focus group session was to hear about the personal experiences of users of the Citywide Guide, to gain insights from actual users on how they use the Guide, and to identify opportunities for improvement. The focus group was conducted as a part of an evaluation of the Guide, the transcript of which was analyzed by analysts of the Center for Long-Term Care (CLTC) at New York Medical College.
Seven focus group participants were recruited from among those who completed the end-user survey and who responded that they would be interested in being contacted about possibly participating in a focus group. A total of 21 people indicated their interest in participating in the focus group. To be eligible for focus group participation, an individual's survey was required to indicate that they had used the Guide at least once within the previous 12 months. Fifteen of the 21 respondents indicating interest were eligible and invited to participate The age, gender, race/ethnicity, employment status, and borough of residence of the seven focus group participants are summarized in Table 1. A total of seven Citywide Guide users participated in the focus group. All focus group participants were women, which is reflective of the fact that 93% of the end-user survey respondents were female. The age of participants ranged from 40 to 71 years, with the median age being 60 years. Additionally, all participants were employed or self-employed and from various racial and ethnic groups, though some race/ethnicities were undisclosed. Focus group participants reported living in Brooklyn, Manhattan, and The Bronx.
The focus group was facilitated by a lead investigator, and a research assistant attended to ensure that the proceedings were recorded properly so that a transcription could later be made for analysis by the CLTC evaluation team. The session lasted approximately 70 minutes and was recorded with a digital voice recorder; signed consent was collected from all participants. Each participant in the focus group received $50 and an MTA MetroCard-such incentives are standard to ensure sufficient attendance (Focus Group Institute, 2018). The consent specified that only first names would be given by participants, and that the discussion would be recorded.
The audio-recording of the focus group session was transcribed by CLTC evaluators. The transcript was separately reviewed by three CLTC analysts to identify primary themes suggested by the focus group discussion. Each CLTC evaluator separately and independently reviewed the transcript for key ideas, recommendations, and criticisms discussed by participants and developed primary themes independent from one another. The CLTC evaluators then shared and discussed their independent transcript analyses to identify common primary themes and recommendations. Only verbal responses were evaluated; non-verbal cues were not, although we doubt given the noncontroversial nature of the focus group content that these cues would be important. Nevertheless, the exclusion from the evaluation of non-verbal cues might be considered a shortcoming.
The aim of the focus group was to achieve a greater level of detail, through anecdotes, about the experiences of users of the Citywide Guide that the survey could not capture. Goals included learning further about whether, and how, people were connected to social services and about how being connected to those services impacted their lives. Another primary aim of the focus group was to hear about occasions when WCC's Citywide Guide provided some assistance, but the individual seeking services was unable to access them, and determine the source of the breakdown so that WCC and Citywide Guide could be improved. No effort to triangulate quantitative and qualitative data from the focus group was attempted; rather, the focus group results were intended primarily to illicit anecdotes, favorable or not, about user experience with both the Citywide Guide and the social service agencies listed in the Guide, with a view not of validating the quantitative results from the survey, per se, but of adding color and context to those results.
A possible shortcoming of the evaluation is that no effort was made to ensure that respondents of the survey and participants of the focus group were representative of those in need of social services and other resources categorized in the Citywide Guide. However, since the primary shortterm goal of WCC was to evaluate how the Guide could be improved for present users (as opposed to would-be users), the findings of the evaluation certainly shed light in that regard. A more ambitious, longer-term evaluation of the Guide might consider the extent to which the findings of this evaluation would be replicated if "non-users" could somehow be reached.

Summary of end-user demographics
Demographic information about end-user respondents to our survey are summarized in Table 2.
A total of 57 end-users of the Guide completed the survey. Each demographic category is compared to the available New York City data to determine if the survey end-users were a representation of the NYC population.
Borough of Residence About one-third of the respondents lived in Manhattan and one-third lived in Brooklyn. The remaining respondents lived in the Bronx and Queens; and none reported living in Staten Island, which is about 5% of the total NYC population.
Gender, Age, Race, Ethnicity, Marital Status A large majority of the respondents were female (only 7% were male). The age of respondents ranged from 18 to 90 years old, with a mean age of 57 years old. Three-fourths of the respondents were not Hispanic or Latino; about 42% were white, 13% were black, 6% Asian, and 26% from multiple ethnic groups. About two-thirds of the respondents were married, divorced, or widowed; while a third have never been married.
Employment Status & Income About 58% of respondents are employed or self-employed and about 40% are retired. About three of every four respondents reported a total household income above $50,000. Figure 1 shows which version/type (online or hardcopy) of the Citywide Guide respondents used the most. Forty-seven percent of respondents reported using only the online Guide and 9% reported using both versions, but mostly the online Guide. Thirty-eight percent of respondents reported using only or mostly the hardcopy Guide. Only about 6% of respondents reported using both versions of the Citywide Guide equally. Figure 2 shows that 77% of the respondents reported learning of the Citywide Guide through the Women's City Club mailing or from "other," a source that was not listed as a survey answer choice. Additionally, only 4% of respondents reported hearing about the Citywide Guide from the public libraries or from a church. Overall, it seems that many users are learning of the Guide from Women's City Club mailing and other sources that are so far unknown (Figure 2). Figure 3 shows that respondents who reported using the hardcopy Guide generally utilized it more frequently than those who used the Online Guide. About a third of respondents who used the online Guide used it more than once in the previous 12 months, while nearly 2/3 of respondents who used the hardcopy Guide used it more than once in the previous 12 months, although there were several less respondents who reported using the hardcopy. A quarter of the respondents for both versions of the Citywide Guide reported not having used the Guide in the previous 12 months.

Guide utilization
Respondents ranked the top three service and resource categories that they have explored in the Citywide Guide. Table 3 shows the top 10 categories that respondents reported exploring the most. The table shows the total number of times that the category was ranked 1 st , 2 nd , or 3 rd by a respondent. "Housing and Homelessness" was the category that was explored by the most respondents, followed by employment; after-school programs; food and hunger; education for adults, immigration, seniors, library, preschool & childcare, and education for youth. Approximately 20% of all respondents reported contacting a resource or service provider from the resource categories that they explored. Several of the subsequent questions pertained to the user's experience when contacting the resource or service providers. Figure 4 shows the results from the survey question that asked respondents about their ability to find the information that they needed when using the Citywide Guide. More than half of the respondents agreed with the statement "When using the Citywide Guide, I was able to find the information I needed,"   Figure 4. "I was able to find the information I needed."n = 43.
while a third of respondents were undecided on whether or not they agreed with the statement. Twelve percent of respondents did not agree that they were able to find the information that they needed in the Citywide Guide.
More than half of the respondents who use the online Guide reported using the keyword search. Of those who reported using the keyword search, more than 80% agreed that the keyword search helped them find exactly what they were looking for.
When asked about services or resources that could not be found in the Guide, respondents listed several. These included: Elder abuse services; free tax preparation services; parks and environmental resources; exercise and wellness programs; senior support groups; Medicaid specialists; and services in Southern Brooklyn.
Respondents also gave several suggestions for ways that they believed the Guide could improve. Some of these suggestions were: having more copies for popular boroughs; making the guide easier to print; adding more language options; and having a pocket-sized version.
Although there were 57 total respondents to the online survey, some respondents skipped certain questions, as shown through the differing sample sizes (n) in the figures and tables.

Focus Group
Three key themes were generated from the analyses of the focus group transcript. These themes involve utilization, availability, and content of the Citywide Guide.
Primary Theme 1-Professional Utilization Several focus group participants said that they used the Citywide Guide in a professional capacity rather than for personal reasons. The Guide was used in a professional capacity in occupations including law, academic administration, social work, and housing assistance. The participants utilized the Guide most in assisting low-income populations. Additionally, it was discovered that residents utilized the Guides that were specific to the borough where their clients lived.
Primary Theme 2-Guide Availability Comments by focus group participants indicated that the Guide is not well-known. In several instances, it was discovered by chance through a neighbor, colleague, or internet search. Many participants reported that they would utilize well-known community organizations, colleagues and professional connections, Google, and other informational resources rather than consult the Citywide Guide first. Several participants also mentioned that the Guide should be targeted to multi-service, community-based organizations. Participants saw merit in both the online and the hard copy versions.
Primary Theme 3-Optimize Content There was strong consensus among participants that the information and content provided in the Guide could be improved. Participants suggested that in order to make the Guide a better resource, it should be more than a directory of organizations, addresses and phone numbers. Several participants said the Guide is comparable to a Google search, but that a Google search is substantially easier to do. Therefore, several participants suggested the need for additional information beyond addresses and phone numbers, including: languages spoken by providers; highlighting the organizations that serve specific cultures; and especially eligibility requirements for services (such as age, US citizenship, veteran, etc.). Although some of this information is available for some providers in both the hard-copy and on-line versions of the Citywide Guide, the information is not systematic and is not included for many providersand when it is included, the information is sometimes not comprehensive.

Discussion
This evaluation sought to determine the value of the Citywide Guide to New York City residents living in or near poverty, and to determine how web-based information resources can be better marketed and utilized by the less advantaged populations in large urban setting. Some of the several findings from the survey of end-users and the focus group were unexpected, which helped inform the Recommendations developed by the evaluation team (as described in Conclusions and Recommendations). As discussed in the Findings section, one of these unexpected outcomesperhaps the biggest surprise to WCC-was that the Guide is used perhaps primarily by social workers and other service professionals, rather than by individual residents. Other unexpected findings included the relatively high income of individual users, the appeal of the hardcopy version of the Guide, and the types of services users of the Guide most commonly searched.
Through the online survey, it was discovered that most Citywide Guide users are women and middle-aged or older-the Guide tool is not generally used by younger people. Additionally, not many Guide users were "unemployed and looking for work," such as might be expected for a list of resources and services designed for low-income individuals; rather, 90% of users were either employed or retired, and three-in-four had an annual household income of greater than $50,000.
Another unexpected finding was the value placed by users on the hardcopy version of the Guide. The survey revealed that both online and hardcopy versions of the Guide are used, although those who utilized the hardcopy Guide use it more frequently during the year. A New York City study on mobile phone access found that at least 66 percent of even the lowest income residents (income of $0 per week) had access to an internet capable smartphone (Menin, 2015). So, this finding of hardcopy users utilizing the Guide more often than online users is inconsistent with the fact that most NYC residents, regardless of income, have access to the Internet via a smartphone, and yet still prefer to use the hard copy version of the Guide. This inconsistency could indicate a lack of easy access to or ease of navigation of the Citywide Guide website.
It was found that Citywide Guide users searched most for Housing and Homelessness services during their search. While perhaps this might not be considered a completely unexpected finding, it might be considered somewhat surprising that more people searched for this category than for "Employment," or "Food and Hunger." On average, more than 63,000 adults and children sleep in a homeless shelter in NYC every night, while several thousand more homeless residents go unsheltered as they occupy subways, streets, and cars for the night (Coalition for the Homeless, 2019; Department of Homeless Services [DHS], 2019).
The survey found that most users of the Citywide Guide were able to find the information that they sought. The focus group found three main themes: 1) getting the Citywide Guide into the hands of individuals who will benefit most or of service organizations who may provide the most benefit to the community, 2) optimizing the content of the Guide so that it is more efficient than other paper and online information resources, and 3) customizing the format (online or hardcopy) of the Citywide Guide to those that use it most frequently so that they may utilize the Guide in the most efficient manner regardless of the format they are using.
Due to the relatively small sample size and to possible selection bias, caution is advised when interpreting the findings of our evaluation. Selection bias might exist if respondents of the survey are systematically different than non-responders. For example, resondents might have had a more -or less-positive experience than non-responders when using the Guide. In addition, it is possible that the findings of this evaluation, based upon a resource Guide available in New York City, may not be generalizable to other metropolitan areas.
Nevertheless, the key themes that emerged seem likely to benefit non-profits and other organizations who provide, or are considering providing, a similar resource Guide in other metropolitan areas, which we describe in the Implications subsection of Conclusions and Recommendations. Although several findings from the online survey of end-users and the focus group were unexpected, these findings allowed WCC and the CLTC evaluation team to better determine the value and utilization patterns of the Citywide Guide and to propose key recommendations for its improvement, as discussed next.

Conclusion and recommendations
Overall, the Citywide Guide is a valuable resource in New York City, although there are opportunities for improvement. This section synthesizes conclusions for this evaluation and provides recommendations for the improvement of the Citywide Guide. The study confirms the value of the Citywide Guide for similar organizations in other cities who may consider investing in a social service guidebook.

Guide format
Several focus group members used the online version of the Citywide Guide exclusively. Other focus group participants valued the hardcopy version. The end-user survey also showed that both versions of the Guide are utilized. Social workers and other service professionals who serve multiple boroughs, regardless of their office location, indicated a preference for a single bound publication including all five boroughs as opposed to five individual publications unique to each borough. In addition, focus group participants and survey respondents recommended distribution of the Guide in places with waiting rooms for people who would likely utilize the services in the Guide.
Key Recommendation 1-WCC should retain both the online and hardcopy formats of the Citywide Guide, as both are valuable for those who utilize them.

Guide distribution
We found that the Guide was being used more frequently by social workers and people acting as social workers, and not as much by individuals for personal reasons. Therefore, the most effective distribution of the Guide would be through social work channels and the channels where populations most eligible for social services such as the clients of social workers congregate. These are primarily public places, such as hospital emergency rooms and urgent care centers, federally qualified health center waiting rooms, the offices of local politicians such as City Councilmen and State Assemblymen, multiservice organizations and social work organizations for use by both social workers and clients of social workers.
Additionally, although the online Guide is downloadable as an Excel file, it is not downloadable as an easily printable PDF. The ability to print the list of providers either by borough or for all boroughs combined might be more strongly highlighted in the WCC website. Lastly, making the list savable and downloadable as a PDF as it is savable and downloadable as an Excel file likely would be useful to providers and end-users alike.
The popularity and ubiquity of social media platforms such as Facebook, Twitter, and Instagram could be used to better advertise the existence of the Citywide Guide, but especially the online website link. The link can be posted to those platforms during days, weeks, or months of awareness for resources that are in the Guide. These could include: mental health, women's health, food and hunger, homelessness, and immigration awareness.
Key Recommendation 2-The hard copy of the Guide should be distributed to public spaces with waiting rooms such as Federally Qualified Health Centers, Emergency Departments, Social Work offices, libraries, and the offices of local politicians, while leveraging a social network presence for distribution of the online version

Guide content
The Guide provides names, addresses, phone numbers and emails for the majority of services providers included. Because a simple internet search can provide a similar set of information, WCC may improve the Guide for members by developing relationships and recommending a small subset of service providers and providing a list of information users will need to have before calling or visiting the organization. Additional information that users would find beneficial is any eligibility requirements necessary to utilize the resource, such as age, income, citizenship status, gender, and residential location. As mentioned previously, some information is available for a relatively small number of providers in the Citywide Guide, but the information when offered is not always comprehensive.
Key Recommendation 3-Focus or highlight a few key partner organizations for each category and provide more information than could be found from a web search.

Implications
In the evaluation presented here, we examined the importance and usefulness of the Citywide Guide to organizations and individuals in New York City. The Citywide Guide is aimed specifically for residents living in or near poverty, and although other studies have evaluated other web-based information resources, ours is the first to examine how web-based information resources aimed specifically to less advantaged populations in large urban settings may be better marketed and utilized. In addition, an innovative aspect of the study was that we also evaluated a hardcopy version of a resource guide, and considered the complementarities of making available both online and hardcopy versions to users.
This evaluation found that the WCC Citywide Guide in New York City is considered to be a valuable reference tool by users. The most surprising finding was that users tended not to be primarily from the Guide's intended market-low-income and impoverished NYC residents-but rather social workers and other service professionals who work with these populations.
The evaluation's key findings are worth consideration by other non-profit or service managers and organizations that wish to create a tool or resource aimed at less-advantaged populations, especially when it is to be used in large urban settings with high poverty rates. In many cases, social workers are the primary channel from which clients and families of clients can receive helpful resources, so in ensuring the efficacy and usefulness of the resource guide or tool available, it may be best-or at least the idea deserves careful consideration-to also market the tool to social workers and other similar public service workers (Acosta, 2013).
Two other implications of our evaluation may be of value to other organizations. First, webbased resources are certainly valued, but organizations may be missing an opportunity if they do not also offer some form of printed reference. Lastly, the experience of this evaluation's uncovering of some unexpected results clearly has implications for other organizations. In order to determine the extent to which an organization's reference tool is useful for its intended market, an evaluation based upon a combination of a survey of users, information gathered by service providers, and focus group may reveal some unexpected surprises.