Bridging ICTD research and policy-making: notes from a systematic review on MSMEs in the low- and middle-income countries

ABSTRACT There is criticism that information and communication technology and development research community contributes less to policy-making. The present article shares an attempt to fill this gap. Systematic review is a structured way to collate, review, summarize, and use the published research for policy-making. A systematic review on the impact of information and communication technologies on the growth of urban micro, small, and medium enterprises in the low- and middle-income countries indicates a tiny impact and a lack of rigorous causal studies in this focus area. Quantitative meta-analysis did not find statistical significance for the impact. From an initial result of 24,000 plus hits, a set of 10 research studies were finalized for data extraction and analysis. The article suggests areas for future research and calls for rigorous research and reporting.


ICTD, policy, MSMEs, and systematic review
A highly read article in this journal lamented that "ICT4D [information and communication technologies for development] are at least interested in achieving socioeconomic impact, but they pay lip service to the processes that might make this possible" (Harris, 2016, p. 186). It also nudged the Information and Communication Technologies & Development (ICTD) "to take practice and policy influence more seriously, by engaging more closely with the users of their research, by encouraging more and better communications with the public, especially through the use of ICTs" (Harris, 2016, p. 187). Earlier, similar concerns were raised by the editor of this journal as well (Qureshi, 2015). The present research article shares findings from an attempt to fill this gap. The study adapted a novel methodology, systematic review, to collate the findings of the ICTD research in the area of urban microenterprises in the developing world to aid policy-makers. Unfortunately, the review did not find significant support for its desired outcome and calls for rigorous research and reporting in ICTD research.
Originated from the field of medicine or healthcare, systematic review is a replicable review process through which literatures are searched, filtered, and analyzed (Higgins & Green, 2011). For a systematic review, the researcher who is familiar with the field proposes a protocol, which sometimes is peer reviewed, and undertakes the actual review. The protocol looks for support for evidence-based policy-making with a clear research question. Then the extant literature is searched for studies that are related to the research question. The search for the relevant material is objective, exhaustive, and replicable and is not restricted to the resources available to the researchers. A set of exclusion (or inclusion) criteria delineated from the research question are used to discard studies that are not connected to the question. The final studies sieved out of the exclusion process are studied in depth. In the process of analysis, the impact parameters in all studies will be combined to form a single analysis. For example, standardized beta regression coefficients will be used in relation to the standard deviation and sample sizes to calculate the effect size. The entire process (Waddington et al., 2012) is described below.
Systematic review is also a novel methodology and is useful when there is previous research available on a subject; but there is uncertainty about the conclusions or an accurate picture related to past research is required to promote the development of new methodologies or a new way of asking questions (Petticrew & Roberts, 2006). The main objective of a systematic review is not to build theories or explore the theoretical frontiers, but to collate and combine the evidence in a scientific manner to be used in decisionmaking. It is becoming popular in the policy studies arena (for instance, Bilotta, Milner, & Boyd, 2014).
The research question raised for the present review is as follows: Does access to business-relevant information through the networked devices enhance the internal efficiency and business growth of the urban micro, small, and medium enterprises (MSMEs) in low-and middle-income countries?
MSMEs have been a constant focus of the ICTD literature (for instance, Donner & Escobari, 2010;Ghobakhloo & Tang, 2015). The question under study is important as MSMEs are treated as instruments of national development by the developing world governments (Stein, Goland, & Schiff, 2010), especially to reduce urban poverty (for instance, Ghanem, 2013). Also, ICTs with decreasing hardware costs and enhanced networking abilities are able to connect the MSMEs and the global market opportunities (Consoli, 2012). The findings from this review are expected to help policy agents. For example, a national government could take a decision on whether to roll out training on e-commerce programs to microentrepreneurs after finding evidence of the impact of e-commerce platforms' use on the growth of small businesses.
The concepts present in the research question are operationalized as below. The review treats any electronic tool through which a user can receive and send information to other users as a networked device. The MSMEs that receive business-relevant information through the networked devices would use them to achieve better internal efficiency and business growth of the enterprises. The business-relevant information is operationalized as any information that is used by the MSMEs for business-related activities. As a nominal definition, the review has set 250 employees as the outer boundary to define an MSME. This is adapted from the European Union, for which a medium enterprise is one with less than 250 employees and an annual turnover of fewer than 50 million Euros. The review focused only on studies based on low-and middle-income countries defined by the World Bank.
The internal efficiency of the MSMEs is inferred by multiple factors. Some of them are as follows: less travel or reduced transportation time, inventory management, employees' coordination and communication, customer management, vendor relationships, and accounting systems. For business growth, multiple facets are consideredincrease in sales, turnover, number of employees, number of customers, number of suppliers, expansion of work premises, business networks, new partnerships, inflow of referrals for business, and so on.
The review recognized the presence of confounding factors in the causal linkage between business-relevant information on internal efficiency and business growth of MSMEs through the use of networked devices. These factors were included in the analysis. Some of them are characteristics of MSMEs (age, size, industry domain, etc.), characteristics of owners (age, education, ICT literacy, etc.), policy environment (financial incentives for internationalization, skill development facilities, etc.), gender, and national culture-specific factors such as paid employment preferences and poor women labor participation rate. It was decided to undertake a subgroup analysis as a part of the review. The research question is presented in Figure 1.
In simple words, the systematic review includes studies that exclusively meet the inclusion criteria set by the protocol. A pioneering paper in the area might be excluded if it does not satisfy the conditions. The findings of the study are generalizable for the area bounded by the protocol of the systematic review. The present review is for low- and middle-income countries for which the findings are generalizable even if the filtered final studies are from few countries. This argument flows from the positivist tradition of the systematic review which could be critiqued from alternative methodological traditions.
The present review differs from an extant study that used systematic review (Donner & Escobari, 2010). The earlier study focused only on mobile phones, did not differentiate between rural and urban focuses, and did not conduct a quantitative meta-analysis as part of the review. The present one included all networked devices, focused only on urban studies, and performed a quantitative meta-analysis. Two studies (Donner, 2007;Esselaar, Stork, Ndiwalana, & Deen-Swarray, 2007) are included in both the reviews. The present review argues that descriptive observations about the impact of ICTs are not being substantiated in quantitative meta-analysis, which is not undertaken in the earlier review.

Methodology
A list of keywords aligned with the research question was prepared. It included the following: *SMEs or MSMEs or enterprise* or business* or microenterprise* or microbusiness* or entrepreneur* or microentrepreneur* or self-employ* or owner* or businessman or businessmen or businesswoman or businesswomen or "self-help group*" or cooperative* or "social enterprise*" or Start-up* or incubators or "born global*" (Laptop* or computer* or PC or Internet or landline* or telephone* or mobile* or phone* or cell or cellphone* or smartphone* or CSCs or telecenter* or telecentre* or WiFi or WLAN or GDPRS or messaging or digital or ipad* or iphone* or apple or android or windows or broadband or wireless or wireline or CDMA or SMS or text* or MMS or Facebook or LinkedIn or network* or Intranet or "discussion list*" or contacts or "online forum" or "discussion thread*" or "online feedback" or ICT or ICTs or "communication technolog*" or "information systems").
The keywords were used to search the following databases: Acad Search, Business Source, Econlit, SCOPUS, and Web of Science. In addition to the above databases, the following institutional databases and collections, the second set, were searched using the search option in the respective databases or through Google search engine using keywords such as: Infodev, World Bank, DIME, JOLIS, IMF, DFID's Research for Development, IDRC's Digital Library, IDEAS, J-PAL, ELDIS, British Library of Development Studies, Millennium challenge, USAid, FAO, UK Theses Ethos, US/Canada Dissertations, and SSRN. Furthermore, a third set of databases, ACM Press, IEEE Xplore Digital Library, and AIS Electronic Library, were also searched using the search option. The literature in multiple databases was scanned until June 2014. A total of 24,225 hits resulted from this scanning.
The search results were uploaded in EPPI Reviewer 4.0, a web-based application to manage and process the review (Thomas, Brunton, & Graziosi, 2010). The following criteria were used to either include or exclude the studies for the review: the study is conducted in urban localities, the study is conducted in the listed low-and middle-income countries, the study contributes to the understanding of MSMEs (maximum of 250 employees and an annual turnover of less than 50 million for an enterprise), the study has numeric data, MSMEs use at least one networked device, MSMEs process business-relevant information, the study is published in English, and the study is published in 2000 and after. Those studies which met the above criteria were included for further analysis.
In the first stage, the title and abstract of 24,225 citations were read. The studies that did not meet the criteria were excluded. For studies where details were inadequate to decide inclusion, full papers or reports were downloaded. In the second stage, full text for 203 studies was downloaded and the methodology section was read. In few cases, the authors of the papers were contacted for clarifications. After reading the full papers, those which met the above criteria were included in the final analysis. For the final stage, a set of 10 studies remained for full-text screening and data extraction. In the first stage, there were five researchers who screened the data. In the second stage, each of the studies was read by two researchers independently.

The final papers
In the final 10 papers, the period of work was 2007-2014 and the locations of the studies are split between the Indian and African continents. All the studies adapted cross-sectional survey as a method for data collection. Although there are adequate numbers of probability sampling techniques, only three studies followed three stages of random sampling. Two studies (Jahanshahi, Gashti, Khaksar, & Pitambar, 2011;Wamuyu & Maharaj, 2011) reported stratified sampling, but did not provide adequate details on the sampling process. Except for one study (Esselaar et al., 2007) whose sample size is large, 3691, the final studies are small-scale surveys, with the sample size ranging from 100 to 560. The rationale for the sample size and decision-making on the number of respondents to be interviewed is not reported in any of the studies. In one study (Jahanshahi et al., 2011), details about the nature of respondents are not reported. None of the final papers followed the mixed design. All the final studies followed either regression models or regression-based structural equation models. Brief summaries of the papers are presented in Table 1.
The major findings that emerged from the final papers are as follows: . Except for one study (Donner, 2007), all used self-reported data about the use of ICTs and used a composite index. The nature of business-relevant information is not reported directly by the studies. For instance, the studies report that businesses had used mobile phones to call and receive calls from the customers, not about the information processed. . The networked devices found to be used by the MSMEs are limited. Four studies exclusively focus on mobile phones and four other studies show mobile phones being used in tandem with other networked devices. In three studies, a single index that combined other devices such as fax computer and landlines is used. Only two papers mention about advanced networked devices such as database management systems, e-document management software, and e-commerce applications. . Overall, seven papers give clarity on the nature of the networked devices and how they are used in cause-and-effect relationships. . Half of the studies discuss the impact of the networked devices on internal efficiency.
o The numbers of business calls increased with the longer use of mobile phones. o ICT usage and possession predict the labor productivity in the MSMEs. Four categories of SMEs are presented: less than five workers, 5-10, 11-20, and 21-50.

Structural equation model
Mobile usage leads to increase in organizational performance o Operational support, strategic development and process improvement, and operational performance are improved by the networked devices. These outcomes are measured using a composite index. . Eight papers proved that there is a change in the growth of enterprises due to ICTs.
o After the purchase of mobile phones, business-related call partners become new entrants with half of them being customers, inferring an increase in the number of customers.
o Higher ICT expenditure resulted in an increase in turnover of the enterprises. o Profits increased with the use of mobile money. o ICTs are positively influencing business growth, measured in terms of market share, profits, profitability, and increase in customer base. . The subgroup or disaggregated analysis is clearly presented in only one study.
o The formality of the businesses influences the relationship between ICTs and business growth. . The results are not uniform across the studies. Business growth was found to be higher among women-owned enterprises in one study (Chew, Ilavarasan, & Levy, 2012), but the reverse was found in another (Frederick, 2014). The level of education of the respondents is a positive predictor of growth in one study (Chew, Ilavarasan, & Levy, 2013), but negative in another (Donner, 2007).
Quantitative meta-analysis is performed to derive a pooled effect size from an individual set of studies to achieve a combined evidence. Six studies focused on the business for the growth of the MSMEs and three studies on internal efficiency.
. The meta-analysis showed that the impact on business growth is positive, although the subtotal effect size is not statistically significant, at .051 with a confidence interval from −0.510 to 0.613. This is lower than the recommended small effect, 0.2 (Higgins & Green, 2011). . The meta-analysis of three studies on internal efficiency also showed a positive effect, but is not statistically significant, at .321 with a confidence interval of −2.740 to 3.382. . Egger's test (P = .002) indicated the publication bias in the final studies. The studies are not evenly scattered in the Funnel plot as well. o Out of the final studies, six studies reported that selection of participants was random and are rated as low risk of bias. However, only one has shared sufficient information about the methods and procedures that were followed in the study. The rest of the studies have followed non-random selection of participants in the survey, such as convenience or purposeful sampling techniques.
o None of the studies reported about handling dropouts and missing data.

Suggestions for future research
The systematic review started with 24,000 plus results and ended with an analysis of only 10 cross-sectional survey-based studies. The review indicates that the ICTD domain or the information systems domain still lacks adequate causal studies that link the ICTs and access to business-relevant information and growth of MSMEs. As a policy-maker, I would be reluctant to use these inputs in decision-making, as the rigor in the methodologies appears to be weak. However, as a researcher, I benefit from the gaps for future research identified by this review. The evidence-based policy-making is greatly benefited by the randomized control trials (RCTs) (Banerjee & Duflo, 2009), called as RCTs or experimental designs. Not belittling the survey designs which are ex post facto driven, the causal relationships are best captured by the experimental designs (Babbie, 2013). The domain under examination, the impact of ICTs on MSMEs, is likely to gain more credibility if the studies follow more appropriate study designs to capture the cause-and-effect relationships (Walsham & Sahay, 2006). Although RCTs are getting popular in the development studies, it is surprising not to find a single RCT study for the research question raised by this review.
Future studies should use rigorous probability sampling techniques. In this review, attempts to scrutinize the methods to increase the generalizability or representativeness of the sample are met with unsatisfactory results.
The location of the studies is predominantly Indian and African continents. The list of low-and middle-income countries in the World Bank contains about hundred names, which are not presented in the review. There is a need for understanding the impact of ICTs on MSMEs in these countries.
Among the ICTs, mobiles phones are the dominant focus of the studies. Some of the ICTs such as personal computers predate the mobile phones which cannot be used for all business functions. Future research can look at the use of other ICTs, in a disaggregated manner, and its impact on the internal efficiency and business growth of MSMEs. There is the likelihood that medium-sized enterprises are using ICTs other than mobile phones.
It is still unclear the nature of business-relevant information that is processed by the networked devices. Each of the possible business-relevant information and their processing and their linkage with the internal efficiency or business growth of MSMEs can be investigated in future studies.

Concluding remarks
It is desired by the experts that the ICTD community should make efforts to translate their research insights into practice for developmental outcomes. The language and medium of the research outputs also remain inaccessible to the users and policy-makers. A systematic review, adapted from the field of medicine, is an endeavor to collate all the relevant literature, analyze, and present it for the consumption of policy-makers. The systematic review presented in this article, albeit in a focused area of MSMEs, indicated that the extant research falls short of rigor of the desired level. There are possibilities that rigorous research is present in other subareas of ICTD, for instance, health or education, and in other languages. These can be the limitations of the present review. It will be useful to undertake a systematic review in the electronic governance area, as it has a significant volume of literature and is one of the important areas of ICTD space. Such a review shall either validate or reject the observations of this present work. Nevertheless, the present review offers food for thought for the ICTD community.
The extant research should be able to meet the quality benchmarks of the reviewers, who can be either from the ICTD community or policy-makers themselves, if there are structured attempts to use the research outputs for policy-making. This article did not aim to denounce the enormous efforts put forward by the ICTD researchers, but points out the possible areas of improvements. It also pleads that adequate details should be reported in the research outputs, in terms of either sampling or disaggregated analysis of different groups. Apart from policy-makers, future researchers, especially, junior scholars will benefit from the shared information.