Evolution of co-production in the information age: crowdsourcing as a model of web-based co-production in Korea

Abstract Revisiting the concept and typology of co-production, this study discusses web-based co-production and crowdsourcing of public services, in particular, how information communication technology changes the nature of co-production and how citizens and governments collaborate in designing and producing public services. With the advent of social media and continued development of web technologies, for example, open government initiatives taken by many countries facilitate web-based interaction between citizens and governments. Linking traditional co-production with web-based co-production, we propose and discuss four different types of web-based co-production characterized by the roles of citizens and governments in the design and delivery of public services. We provide illustrations of each type through selected cases and then discuss policy implications.


Introduction
Co-production has been presented as a new model for public-service provision as governments are faced with various challenges of efficiency, quality of public service, participation and so forth. Co-production in this context is government and other actors, including citizens, businesses and non-profit organizations, working together to produce public services (Alford, 2015;Bovaird, 2007;Brandsen & Honingh, 2015;Brudney & England, 1983;Linders, 2012;Meijer, 2012Meijer, , 2016Osborne & Strokosch, 2013;Ostrom, 1996;Parks et al., 1981;Pestoff, 2006;Sicilia, Guarini, Sancino, Andreani, & Ruffini, 2015;Thomsen & Jakobsen, 2015;Verschuere, Brandsen, & Pestoff, 2012). This study examines crowdsourcing as a model of web-based co-production in the age of information and also investigates how individual citizens contribute to the design and delivery of public services by reporting public-service needs, offering policy ideas and developing public-service apps based on public data provided by governments.
Citizens are no longer passive recipients of public services; instead, they often become proactive actors in producing public services. Pursuing an alternative design for the public-service system, Cameron (2007) emphasized that the traditional government-dominated public-service system is no longer effective and needs to be replaced with a new alternative design for public services by responsible citizens, professionals and institutions. The active role of citizens is often highlighted and demanded by government officials or politicians who attempt to reach out to citizens and non-governmental actors to strengthen the public-service capacity of governments by utilizing the resources of citizens and non-governmental actors to collect information, as well as to plan and deliver public services. Citizens' involvement in public services is no longer an option, as many social problems have become 'wicked' (Rittel & Webber, 1973) in that they are so complicated and severe that they cannot be resolved by the traditional mode of public-service delivery or by simple outsourcing practices.
Citizens' role in public-service delivery has become more feasible and active with the advancement of web and mobile communication technologies through which citizens can easily interact with governments and become involved in policy-making processes and public-service delivery. These technologies allow citizens to be part of a web-based co-production, which is defined as a joint and collaborative web-based mode of production of public services by the government and its citizens. The mode of co-production advances as e-government evolves and uses more open, interactive, integrated and collaborative mechanisms through which governments communicate with citizens and promote public participation in designing and delivering public service via various communication channels. While some studies examine how citizens participate in public-service co-production via simple communication interactions such as reporting potholes and broken streetlights, there is a dearth of literature on the theories and practices of web-based co-production of services beyond e-reporting like the 311 system (Clark, Brudney, & Jang, 2013).
Recognizing the significance of web-based co-production, this study discusses an analytical framework and examines selected cases that illustrate the nature and content of web-based co-production. In particular, the co-production pursued as part of the reforms undertaken by the Korean Government for openness, information sharing and collaboration in the public sector is analysed. For example, one of the key projects in Korea is to make the data held by local and central governments open to the public and to encourage citizens and businesses to develop public-service apps using these public data.
Web-based co-production is often facilitated by the advancement of e-government along with Web 2.0 technologies which promote information sharing, openness, connectivity and interactivity between governments and citizens. Web-based co-production is a type of crowdsourcing in that it takes advantage of collective intelligence and citizen participation when making public policy decisions and providing services. In this study, we first survey the scope and nature of web-based co-production based on a conceptual framework of co-production. Then, we survey various practices of Korean web-based co-production as a case study to illustrate the nature of web-based co-production as well as its prospects and challenges.

Co-production as an alternative design for public service
Co-production is not a new concept. As Thomsen and Jakobsen (2015) noted, the concept was discussed and introduced to the public administration discipline in the late 1970s and early 1980s. For example, Parks et al. (1981) highlighted the role of citizens as co-producers of public services in their early works on co-production. Since the 1980s, many scholars and practitioners have examined the nature of co-production and proposed it as a new alternative model for the design and delivery of public services. In his conceptual work, Bovaird (2007) confirms Alford (1998)'s assessment that the interest in co-production shifted to marketization at the peak of the new public management wave, then resurged in the mid-1990s. The scope of co-production becomes even wider by considering not only citizens but also non-profit organizations and businesses as partners who can collaborate with governments in delivering as well as planning, designing, commissioning, managing, delivering, monitoring and evaluating public services (Bovaird, 2007).
The scope and focus in the definitions of co-production are somewhat different among scholars. There is no single concept of co-production, though many refer to Ostrom's (1996) definition which emphasizes contributions to the production of goods or services by individuals belonging to other organizations. Similarly, in searching for the state of the art in the co-production research, Verschuere and others (2012) offer a narrowly defined and simple concept of co-production as 'the involvement of individual citizens and groups in public service delivery' (p. 1083). Based on a thorough survey of concepts of co-production in various studies, Brandsen and Honingh (2015) offer a revised concept of co-production as 'a relationship between a paid employee of an organization and (groups of) individual citizens that requires a direct and active contribution from these citizens to the work of the organization' (p. 431).
Introducing the concept of co-production in public administration, Brudney and England (1983) summarized two different theoretical perspectives on co-production: political and economic. The political and public administration perspective tends to understand co-production in the context of citizen (particularly beneficiary) participation in the provision of services. In contrast, referring to economic theory, particularly the concepts of consumer production, the economic perspective considers co-production as the combination of both regular and consumer production in public-service delivery. On the basis of these two perspectives, Brudney and England (1983) suggested that co-production is 'a multi-faceted concept, and that regular producer-consumer nexus should be placed within the context of types of co-production' (p. 61). Further, co-production is defined by 'the degree of overlap between two sets of participants -regular producers (e.g. service agents, public administers) and consumers (e.g. citizens, neighborhood associations)' (p. 63).
In addition to the government's willingness to collaborate with citizens, citizens' ability and motivation to participate in co-production are also important. Conducting a field experiment-based study, Thomsen and Jakobsen (2015) find that government often take a proactive approach in providing policy-related information helps increase citizens' co-production ability and motivation to engage in co-production, which eventually enhance the level of citizens' co-production activities. However, it should be also noted that co-production does not always lead to positive results. While many scholars view co-production as a useful alternative way to design and deliver public services (Bovaird, 2007;Brudney & England, 1983), as well as attain public values (Alford, 2015), Williams, Kang, and Johnson (2015) challenge the positive perspective of co-production by proposing the possibility of co-contamination. They argue that co-production might undermine public values when resources are misused by service providers, service users or both. 1 Broadly speaking, co-production refers to any kind of involvement of citizens and non-governmental actors in different stages of policy-making and public-service delivery. Co-production refers to any forms or processes in which citizens (as individuals, groups or crowds) and public-service providers (i.e. public organizations, professionals of non-profits or private organizations) collaborate in planning and delivering public services. It is thus presented as an alternative mode of public-service production to traditional public administration (a bureaucracy orientation with government as a monopoly) or to new public management programmes (market orientation, contracting out and other market-based management styles) (Sicilia et al., 2015).
Analysing co-production cases in the Lombardy region in Italy, Sicilia et al. (2015) suggested four different types of co-production: co-planning, co-design, co-delivery and co-evaluation. These types were identified on the basis of the public-service cycle of planning, design, delivery, and evaluation in multi-level and multi-actor governance settings. These four stages of the public-service cycle can be largely categorized into design and production. The design stage is basically a pre-service stage, which includes planning and design of the service, during which various dimensions of a specific public service are considered, including the quality and quantity of public services, the methods of public-service delivery, and the target population, among others. The design stage is followed by the production stage, wherein the actual public service is produced and delivered to the targeted population.
In light of various forms of public-service delivery, Bovaird (2007) proposes nine different types of service professional-user relationships by focusing on how service professionals, users and communities are involved in the planning and delivery stages. 2 The nine typologies are composed of the interaction of three alternatives of public-service planning (professionals as sole service planners, service users/community as co-planners, and no professional input into service planning) and three alternatives of public-service delivery (professionals as sole service deliverer, professionals/users/communities as co-deliverers, and users/communities as sole deliverers). As an extension of the typologies that Bovaird (2007) proposed, similar typologies can be proposed based on alternative forms of involvement of government and citizens in public-service design 3 as well as delivery as summarized in Table 1 where the shaded cells represent co-production of public services.
Full co-production basically occurs when the government and citizens are both involved in the design and delivery stages, whereas a self-organized provision of public services occurs when citizens initiate a particular public service without any involvement of governments. With the initiation and involvement of governments in the public-service design, voluntary co-delivery of public services with the government is a mode of co-production (e.g. voluntary neighbourhood watch in collaboration with the police). Citizens can also play 1 in the process of co-production, citizens might distort information or misuse resources to protect their individual or community interests, such as when a volunteer neighbourhood guard killed a black teenager as a result of his racial bias (Williams et al., 2015). 2 the two dimensions of planning and delivery of public services are proposed for theoretical clarity. the dimensions could be further classified into the full range of the public-service production cycle including planning, commissioning, designing, managing, delivering, monitoring, and evaluating (Bovaird, 2007).
only a limited role in the design stage by providing policy opinions like policy suggestions or policy referendums. Examining variations of co-production particularly focusing on proximity of tasks to core services (core versus complementary tasks) and degree of citizens' involvement (design versus implementation), Brandsen and Honingh (2015) also propose four different types of co-production: complementary co-production in implementation (students' assisting the university in organizing welcome events for incoming students), complementary co-production in service design and implementation (parents' helping and organizing extracurricular activities), non-complementary co-production in the implementation of core services (children's education in which students follow strictly defined lessons, yet their input is still crucial to effective learning), and non-complementary co-production in the design and implementation of core services (post-graduate training modules in which entrants, together with instructors, define their own learning objectives and learning activities).

Web-based co-production in the information age
On the basis of a conceptual framework of co-production, this section presents a discussion of the evolution of web-based co-production facilitated by advances in information and communication technologies (ICTs). In particular, the development of mobile communication technologies, especially web technologies, allows citizens to obtain digital information almost anytime and anywhere and to communicate and interact with anyone, including governments. Particularly, web 2.0 technologies developed in the social media environment offer new platforms of interactions between governments and citizens (Clark et al., 2013;Meijer, 2012Meijer, , 2016 with various modes of collective intelligence, information sharing, collaboration and long-tail customization among others.
Web 2.0 technologies appear to be pervasive, and they directly and indirectly reinforce the social demand for openness, interactivity, participation and sharing information. The advent of social media facilitates the social environment wherein interaction and sharing information take place not only among citizens but also between governments and citizens. citizen delivery of coplanned or co-designed services

Self-organized provision
With the increase in the number of active mobile social media users, citizens can further engage with governments in exchanging information and data as well as use various apps designed to provide public information and public services via mobile phones. As of January 2016, the global web index Slideshare reported that there were 3.4 billion Internet users, 2.3 billion active social media users and 1.9 billion active mobile social media users out of a total world population of 7.3 billion. 4 Advances in ICT affect the mode of co-production of public services in various ways by offering citizens more channels of communication with governments. Web technologies increase the amount of public information available to citizens, and online interactions between citizens and governments enhance the openness, interactivity and transparency of governments.
With the rapid increase in the number of mobile phone subscriptions, many governments have begun to pay attention to the potential of mobile apps for the provision of public information and public services. Governments often develop their own apps for public services or enable and encourage individual citizens to develop public apps using public information and data.
This web-based co-production allows the public to play more proactive and extensive roles in both the design and delivery stages of public services. Similar to the typology of co-production, web-based co-production can be specified in terms of the roles of governments and citizens or communities in the design and production stages.
Acknowledging the prospects of web 2.0 technologies in the provision of public services, it is fair to mention that there is also a dark side of web 2.0 applications due to concerns about the protection of privacy and 'forced co-production' (e.g. when Microsoft or Google collect data on the use of webpages by citizens) as well as the growing digital divide among those who have different access to various information communication technologies. 5 It should also be noted that the impact of web-based co-production is not always positive. In their study of the 311 system in Boston, for example, Clark and his colleagues (2013) suggest that poorer neighbourhoods refer to new information technology-based coproduced services such as the 311 service less often than more affluent communities.
As an extension of the typology of co-production presented in Table 1, we propose four different types of web-based co-production based on the actors and their roles in the design and delivery stages (see Table 2). In fact, web-based co-production is closely associated with the evolution of e-participation, 6 which has advanced along with the development of the web and Internet technologies. Recently, e-government has begun to emphasize citizen participation and interactivity by promoting collective intelligence, information sharing, openness and participation, and personalized e-government. E-participation as a supplement to e-government particularly focuses on e-information, e-consultation and e-decision-making which are defined by the United Nations (2014) as follows: e-information that enables participation by providing citizens with public information and access to information upon demand, 2) e-consultation by engaging people in deeper contributions to and deliberation on public policies and services, and 3) e-decision-making by 4 the figure is from http://www.smartinsights.com/social-media-marketing/social-media-strategy/new-global-socialmedia-research/. 5 i acknowledge a great suggestion by one of anonymous reviewers about the negative aspects of Web 2.0 and coproduction in the information age. 6 e-government is often broadly defined by including back-office applications and front-office applications of icts as well as e-participation, while e-participation particularly refers to online citizen participation in various stages of public policy-making processes.
empowering citizens through co-design of policy options and co-production of service components and delivery modalities. (p. 63) Among these three elements, e-consultation and e-decision-making are part of webbased co-production in that citizens as users of public services participate in different stages of public-service production. For example, e-consultation refers to citizens' webbased (online or mobile) engagement in the public sphere, wherein various policy or public service-related ideas are exchanged and policy suggestions are made. E-decision-making is more specific in that citizen participation is targeted at particular policies or public services, and its inputs are part of policy or public-service design. E-decision-making refers to citizens' active web-based (online or mobile) participation in designing public policy or public services by sharing their ideas and perspectives on alternatives for specific policy problems and public-service needs. The Red Tape Challenge by the British Government is an example of e-consultation in which citizens' inputs and suggestions were invited as part of the government's red-tape reform initiative; some of these suggestions were eventually adopted by the government.
Similar to e-consultation, e-suggestion is also a web-based co-production adopted widely in local government services, such as the 311 system for requesting various local government actions for non-emergency matters including pothole or streetlight reports in the US (Clark et al., 2013;Linders, 2012). E-suggestion (often referred to as e-reporting for complaints and suggestions) is a combination of crowdsourcing design and government sole production 7 : citizens play a critical role by providing information and public-service requests, while 7 the 311 system might be considered to be a crowdsourcing (citizen-sourcing) monitoring rather than co-design in that it only allows citizens to report complaints and service needs to governments but does not necessarily offer policy suggestions to governments on how to solve the problems (linders, 2012). Broadly speaking, however we consider e-reporting as a type of co-design in that reporting any problem or complaint to government naturally includes citizens' demands and suggestions for appropriate solutions. governments are the message receivers as well as doers in terms of producing actual public services responding to the citizens. E-suggestion (e-reporting) like the 311 system basically considers citizens as street-level bureaucrats who identify general public-service needs and report back to governments so that necessary actions can be taken by governments. 8 Citizens and groups of citizens can proactively become involved in co-production by delivering public services only, with government participation in public-service planning not delivery. Crowdsourcing delivery of public services became feasible when governments began to make more data accessible to the public, not only to facilitate the citizens' right to know but also to encourage citizens and groups of citizens to take advantage of the available public data and develop public-service apps. Using the available public data, technology-savvy citizens or businesses began to produce public-service apps by themselves (government design and crowdsourcing delivery) or with governments (crowdsourcing and government co-delivery). In government design and crowdsourcing delivery, individuals or groups of citizens actually produce the public-service apps, whereas governments only play a design role by simply providing the information or resources (APIs) used for public-service apps. Crowdsourcing and government co-delivery is a fully web-based co-production in which governments and citizens are jointly involved in producing public services. Accordingly, there are four different types 9 of web-based co-production, as indicated in Table  2. The following section illustrates these four different types of web-based co-production with selected examples from Korea which has been recognized as one of leading countries both in e-participation performance (United Nations, 2014) and open data initiative for crowdsourcing (OECD, 2015(OECD, , 2017.

Type 1: crowdsourcing co-design
Crowdsourcing co-design refers to active participation of individuals or groups of citizens in designing policies or public services along with governments which also play the part of designer rather than producer by exchanging ideas, information and perspectives with citizens or non-governmental actors on policies or public services prior to their implementation. Governments often establish a web-based system so that both governments and citizens can participate and contribute to shaping the content of policies and public services via online policy suggestions.
For example, in 2006 the Seoul Metropolitan Government (SMG) introduced a comprehensive crowdsourcing system called Chunmansangsang Oasis 10 (which translates to 'the Oasis for Ten Million Imaginations'), through which the SMG invites city residents to share their policy ideas and civil proposals with government officials in general areas or on specific policy issues pre-identified by the government. The online policy-suggestion function in Chunmansangsang Oasis established in 2006 is a crowdsourcing co-design that 8 it should be noted that the term e-petition is often used to differentiate from e-suggestion (e-reporting). the two concepts are somewhat different in that an e-petition is often made for oneself regarding personal-level complaints about government actions, whereas an e-suggestion is not necessarily made for the report per se but for the general public. 9 linders (2012) proposes a similar but more complex typology based on two dimensions (stages of co-production and modes of provider-beneficiary. He suggests three stages of co-production (design, execution, and monitoring) and three different modes of provider-beneficiary (citizen sourcing, government as platform, and do-it-yourself-government), which leads to nine types of co-production. We recognize that the classification has a great merit of specifying the two dimensions in details. it should be also noted that the typology is somewhat too complicated and there is a possibility of weakening the notion of government-citizen joint production in some of the proposed types such as self-monitoring like 'online citizen testimonial systems' (p. 449).
allows citizens to participate in designing specific policies with the government. Citizens can access the portal via the web or an app and propose policy suggestions, which are later reviewed, assessed and elaborated by government officials. Once a policy suggestion is proposed, it is then further discussed or revised within 10 days through online discussions, online voting, or expert review by researchers and retired government officials. After this maturation period, any new policy suggestion that earns the support of more than five citizens based on online citizen voting is transferred to the relevant administrative unit. The policy suggestion is then reviewed and evaluated within 20 days in terms of its feasibility, budget, legal basis and so forth. The review and evaluation are often conducted in offline meetings in which government officials, policy experts and citizens refine the suggestion.
To promote active citizen participation in making policy suggestions, the SMG offers small cash awards equivalent to USD $50 and $200 for those suggestions that are partially or fully adopted by the government, respectively. Eight hundred and sixty-four out of 167,000 policy suggestions have been actually adopted by the government after through reviews and assessment (Roh, 2017). Chunmansangsang Oasis and other related online and mobile channels were integrated into a new system called 'Eungdapso 11 (translated as responding, answering and communicating)' in 2014 as an integrated online management system for civil complaints and suggestions. Under the system, the government's responses to the citizens' requests shorted average process time from 3.8 days in 2013 to 2.8 days in 2014. 12 This crowdsourcing co-design approach seeks to invite citizen participation in solving public policy problems, opening the doors wide to citizens in the design and planning stages by allowing them to offer any suggestion or solution for policies and public services presented by governments. 13 However, while the number of policy proposals has increased, the quality of these policy proposals and the follow-ups on the proposals remain limited and often problematic. Though the experiment itself is not fully effective at this point considering these limitations, it still shows the direction and possibility of the benefits of crowdsourcing. In fact, SMG has recently announced that it launches a far more advanced version of co-design policy platform so-called 'Direct Democracy Platform for Citizens' in September of 2017 and upgrade the crowdsourcing co-design system.

Type 2: crowdsourcing design and government delivery
Type 2 is a simple version of web-based co-production that allows citizens to offer any inputs, such as complaints and pothole reports, via online or mobile channels. Government 11 its URl is www.eungdapso.seoul.go.kr. 12 the information is found in the following site. http://english.seoul.go.kr/policy-information/key-policies/diversifying-communicationchannels/eungdapso/. 13 Similar practices have been adopted by other countries such as U.S. and U.K. the obama administration also undertook an experimental crowdsourcing initiative for public policy problems. in a website called challenge.gov, various government departments identified their internal policy problems and then asked citizens to offer their ideas and solutions. For example, the office of the national coordinator for Health information technology announced the Reporting Patient Safety events challenge, asking for multi-disciplinary solutions to facilitate patient safety events, with cash awards totalling US$70,000. this was an attempt to move the problem-solving method from traditional insourcing (problems solved by governments) and outsourcing (problems solved by contracted entities) to crowdsourcing (problems solved by anyone). Under cameron, the UK Government also emphasized crowdsourcing co-design. For example, it attempted to reduce and eliminate red tape by using a crowdsourcing co-design approach called the Red tape challenge, which was launched in 2014 as a cabinet office initiative. the initiative's website invited any citizen to openly discuss burdensome regulations. the opinions, comments, and debates on specific rules and regulations were reviewed by related agencies and used as material for reform. the guiding principle was that the regulations questioned by citizens would be removed or improved unless they were justified by the relevant agencies. a recent report suggests that 3095 regulations have been removed or improved. agents review the information and take necessary actions for policy implementation and service delivery by themselves. The e-People system, an official web-based crowdsourcing system administered by the Anti-corruption and Civil Rights Commission in Korea, is a compelling example of web-based crowdsourcing design and government delivery of public services. e-People is an official web-based crowdsourcing system which receives and administers citizens' complaints, policy proposals and policy debates via the web. It is often called Kukmin Sinmungo, which means 'the people's big drum' in Korean. Sinmungo was a citizen-appeal system that operated in the Chosun Dynasty in which ordinary citizens beat a drum when they appealed to or complained against the government. Adopting this citizen-focused tradition, the Korean Government established the e-People system as an integrated web-based crowdsourcing system through which citizens could easily send their petitions, policy suggestions and any other public service-related reports to 47 central government agencies, 244 local governments, 195 educational district offices, 144 embassies and many other quasi-government institutions.
According to the statistics from the official website, the number of personal complaints and appeals tripled from 696,715 in 2009 to 1,903,019 in 2015, while civil proposals increased from 84,026 to 103,408 in the same time period. Personal complaints and appeals can be made whenever citizens have specific complaints about any actions taken by administrative organizations. Once the complaints or appeals are filed, the cases are supposed to be transferred to a responsible administrative organization and handled within 14 days (excluding holidays). According to official statistics, 14 the processing time was substantially shortened for simple 15 and complex petitions from 7.8 days to 5.5 days and from 14.4 days to 7.6 days, respectively, between 2006 and 2012. 'My Petition' in the e-People system allows the person who files a complaint to check its status and offer his or her evaluation and feedback about the results, which helps to improve satisfaction by shortening the handling time.

Type 3: government design and crowdsourcing delivery
This type is a web-based co-production in which citizens or non-governmental actors generate public services and governments are only involved in the planning and design stages by providing citizens with the necessary information for public services. Public data-based service apps developed by citizens or businesses are a good example of crowdsourcing production. With the initiative of Government 3.0 introduced by the Park Geun Hye administration in 2013, the South Korean Government promoted the values of openness, participation, sharing, and collaboration by making data and application programming interfaces (APIs) open to the public, including those who are interested in developing apps for the public interest. As of 15 March 2016, 731 apps have been developed using public data and APIs in various areas including culture and tourism, public health, public safety, transportation, environment and weather, labour and employment, science and technology, agriculture and fisheries, education, and so on. For example, IamSchool is an app developed by the Iam Company to provide class or school announcements and school activity-related information 14 the statistics are found on the homepage for e-People (Kukmin Sinmungo): www.epeople.go.kr. 15 Simple petitions address illegal or inappropriate administrative procedures and systems that infringe individual rights or cause unnecessary inconvenience and administrative burdens. complex petitions request changes in administrative systems or procedures.
to parents so that they can check on lunch menus, assignments and class activities in real time. Using the public information provided by the Ministry of Education, school district offices, and schools, the app provides information to parents who are interested in checking their children's school-and class-related information on their mobile phones.
With continued efforts by the government, the number of public-service apps increased from 147 in 2013 to 731 as of 15 March 2016. 16 This suggests that the government's active involvement in the provision of public data, along with its policy support, stimulates technology-savvy individuals and groups of citizens to develop public service apps; these are, in fact, the same mechanisms through which various public services are delivered to citizens (app users). These public-service apps are often developed by individuals or groups of citizens based on public open data and APIs provided by the central government, local governments, and public institutions, including public corporations or quasi-government organizations. 17 For example, 'Public Parking Lot Finder' 18 in the City of Pusan is an app developed by an individual citizen to locate public parking lots available around the city along with such information as parking fees, location, operating hours, contact information and so on. The GPS-based app was developed based on the open APIs and open data provided by the City of Pusan. One of the first public-service apps developed by an individual citizen is Seoul Bus, which provides arrival time for particular stops. The app was developed in 2009 by a talented high school student who took advantage of public transportation information offered by local governments. In 2009, it was one of most popular apps in Korea. The app service was terminated by local governments that raised copyright issues, but was resumed because of overwhelming popular demand.
Recently, many technology-savvy citizens, especially young people, have begun to organize public-service app development activities not as individuals but as groups. They look for potential public-service opportunities to merge public and private data, which are often complementary. For example, similar to Code for America, several local volunteer civic hacker groups organized under the umbrella community of CodeNamu 19 (codenamu.org) to develop public-service apps for citizens in the City of Seoul (codeforseoul.org) and the City of Incheon (codeforincheon.org). For instance, Code for Seoul developed an app called Anshimi (translated as 'safety') based on public data about antibiotic prescription information from clinics and hospitals that allows citizens to check which clinics or hospitals are good if they want to avoid over-prescription of antibiotics. Members of Code for Incheon developed a GPS-and mobile-based app called Share Your Light Mobile which allows citizens to communicate with each other and authorities for obtaining necessary assistance in case of a power outage. 16 the number of public service apps in each year is calculated on the basis of the description of the 731 public apps found in the official open data homepage (www.data.go.kr). 17 the total number of public service apps using public data and aPis provided by public institutions is 371, whereas the total number of public service apps using those provided by the central government and local governments is 247 and 157, respectively. 18 For more information, see https://itunes.apple.com/kr/app/busan-gong-yeongjuchajang/id1179981910?mt=8#. 19 code for america, established by Jennifer Pahlka in 2009, is a leading civic hacking group that aims to improve the quality of government by developing web-based and app solutions for civic problems. it collaborates with more than 100 local governments and community organizations for technology-based solutions for civic problems (www.codeforamerica.org).

Type 4: government and citizens co-delivery
Similar to full co-production in traditional methods of co-production, crowdsourcing co-delivery is a web-based co-production in which citizens and governments form a collaborative structure that actually produces public services together. This is different from the crowdsourcing co-design approach in that citizens and governments deliver public services together beyond the exchange of ideas and information in planning and designing public services. For example, governments and citizens work together to create integrated public-service apps and make them available to the public.
Seoul Mobile Platform (mplatform.seoul.go.kr) is an excellent example of crowdsourcing co-delivery. Along with its strong open-data initiative, the SMG offers a mobile platform on which interested citizens can participate in developing apps on the basis of open data and APIs available to the public. Since 2015, the SMG has organized an annual public-service app competition called Win the SMG!, which challenges individual citizens to develop a public-service app to provide better public services than the SMG. The app development is a joint production by both the SMG and citizens in the following three aspects. First of all, the SMG allocates some budget to promote active participation by technology-savvy citizens in developing public-service apps. Secondly, the SMG makes more open data and APIs available in the areas of transportation, pollution and tourism among others. Lastly, the SMG offers mentoring and open-source training to those who are interested in public-service app development. Thanks to the joint efforts by the SMG and citizens, many public-service apps have been developed such as Find Seoul Flea Market and Find Seoul Trailer Information for those who are interested in flea market events and city trailer information in Seoul.
Similarly, the central government also promotes public-service app development in collaboration with individuals or groups of citizens who are interested in technology-based solutions for civic problems. Many local and national government agencies support hackathons, 20 which are competitions for public-service app development to promote citizens' involvement in technology-based civic solutions. For example, the Ministry of Science, ICT, and Future Planning in Korea works with individual citizens or volunteer groups such as CodeNamu to develop public-service apps by using open data.
One of these public-service apps is National Assembly Toktok (Gukhoe Toktok), 21 an online legislative platform through which citizens and legislators can work together. Citizens offer their ideas on new legislation through the platform, then the ideas are transferred to lawmakers in relevant standing committees when 1000 citizens support the legislative idea. The app allows citizens to check which members of standing committees do or do not agree to support the idea and helps both citizens and participating lawmakers organize working groups to proceed with the legislative ideas.
Another example is My Products Standards, which is an app that provides specific quality information on industrial standards and certification of particular products, including the Korea Industrial Standard mark (KS), Good Recycled mark (GR) and New Excellent Product (NEP) certification. This app provides citizens with product quality information-based APIs developed by both public agencies and a web portal company. The product quality information is obtained from the Naver Dictionary system developed by Naver, the most popular 20 the term 'hackathon' was coined by combining the two terms 'hacker' and 'marathon' . 21 For more information, please visit toktok.io.
search engine in Korea, as well as product-quality APIs developed by various public agencies such as the Korean Agency for Technology and Standards and the Korea Environmental Industry and Technology Institute.
With growing interest in open-government and open-data initiatives, more public-service apps are developed as a joint product of governments and citizens, which is characterized as crowdsourcing co-delivery. Many public-service apps use public open data and APIs rather than private business data because it is more difficult and complicated to obtain and use private business data than public data. However, demand will continue to grow for public-service apps based on both public and business data, which requires solutions to the property and privacy issues of business data and systems.

Conclusions
Advances in ICT have transformed the way governments interact with their citizens. This has accelerated the transition from traditional governance modes to crowdsourcing in both design and delivery stages as a new mode of public-service production. Governments face new opportunities in the course of engaging their citizens in the social media and Web 2.0 environment, where more openness, interaction, participation, information sharing, and collaboration are demanded by the public, as highlighted by various open-government initiatives (Tauberer, 2014). In particular, social media and Web 2.0 technologies advance e-participation, which alters traditional co-production into crowdsourcing practices by engaging unidentified and multiple citizens or volunteer groups of citizens in both the design and delivery of public services via web and mobile communication channels.
As the selected cases illustrate, web-based co-production has largely developed into four different types of crowdsourcing: (1) crowdsourcing co-design, (2) crowdsourcing design and government delivery, (3) government design and crowdsourcing delivery, and (4) crowdsourcing co-delivery. Governments often promote web-based co-production as an extension of e-participation through which governments not only provide public information (e-information) to their citizens but also demand more informed, empowered, proactive and responsible citizens who can actively participate in policy-making processes (e-consultation and e-decision-making) and in the delivery of public services. Web-based co-production is expected to expand, owing to continued development of ICTs as well as the availability of more digitized public data and public-service APIs developed by governments. There are more crowdsourcing design and co-design practices than crowdsourcing delivery and co-delivery practices because citizens' role in the design stage is much more feasible than in the delivery stage simply because the delivery of web-based public services requires technological expertise.
However, crowdsourcing delivery and co-delivery are expected to expand with the increasing demand for better and more comprehensive web-based public services which require a higher degree of collaboration and information-sharing between the public and private sectors. Governments are often eager to engage more technologically savvy volunteers in web-based public-service delivery through either technical mentoring and training programmes or promoting public-service app development competition programmes such as hackathons along with more open data and APIs. On the other hand, volunteer civic hacking (solution) groups are actively involved in searching for public-service apps for civic problems. Following the Code of America established by Jennifer Pahlk in 2009, individual members of civic hacking groups have been active in developing public-service apps for transparent and democratic operation of governments and for participative and effective public-service delivery at the local and national levels.
While web-based co-production continues to evolve, its success is still questionable. Unless the quality of citizen participation in policy-making and public-service delivery is ensured, crowdsourcing design and co-design practices are not necessarily effective. Crowdsourcing delivery and co-delivery practices are also somewhat limited, simply because citizens do not always have enough technological expertise to implement web-based app solutions for civic problems. The quality of open data also constrains web-based co-production practices because these activities largely rely on the data and APIs provided by governments.
We find that the nature of web-based co-production is often limited to e-consultation or app-based e-information, whereas the targeted areas are limited to several areas of high citizen interest, such as cultural activities and tourism, weather, transportation, and so forth. We believe that the evolution of web-based co-production will continue with an increasing demand for crowdsourcing delivery and co-delivery rather than crowdsourcing design and co-design.
Strong open-data initiatives will remain critical to the success of crowdsourcing design or co-design because the quality of public information and public-service apps is often problematic. Public trust in the government (the quality of public information and the utility of public-service apps) is closely associated with the quality of citizen participation and open government. The quality of civic hacking groups and their work eventually determines the quality of crowdsourced public services. Many public-service apps developed by civic hacking groups or individual citizens do not necessarily meet public expectations. While hackathons have become more popular and helpful in strengthening civic hacking activities to resolve community problems, many of their app solutions are not ready to be used (Tauberer, 2014).
To advance web-based co-production in the social media environment, governments need to meet the burgeoning demand for public information and public services by facilitating or administering the development of public-service apps that are available to the public for their own needs. Governments need to make a greater effort to offer incentives that motivate their citizens to become more proactive and responsible actors in designing and delivering web-based co-production. On the other hand, governments are also expected to nurture the technological capacity of civic groups through technical assistance or training programmes that will facilitate effective crowdsourcing delivery activities. Both the quantity and quality of citizen participation, as well as the social and information connectivities in the social media environment, will determine the success of future web-based co-production and crowdsourcing approaches.

Disclosure statement
No potential conflict of interest was reported by the author.

Notes on contributor
M. Jae Moon is professor of Public Administration at Yonsei University. Moon is an elected fellow of National Academy of Public Administration (NAPA) and also currently vice president of Korean Association of Public Administration. He was editor-in-chief of International Review of Public Administration. Earning PhD from Syracuse University, he previously taught at the University of Colorado at Denver and Texas A & M University. His research interests include comparative public administration, e-government and public management.