Big data and organizational design – the brave new world of algorithmic management and computer augmented transparency

Big data and sophisticated algorithms enable software to handle increasingly complex tasks, such as detecting fraud, optimizing logistics routes, and even driving cars. Beyond technical tasks, algorithms enable new ways to organize work. In this article, I suggest a distinction of optimizing-oriented and open-ended systems leveraging big data and examine how they are shaping organizational design. The optimizing-oriented systems, typically based on numerical data, enable smarter control of well-defined tasks, including algorithmic management of human work. Open-ended systems, often based on textual data or visualizations, can provide answers to a broad range of managerial questions relevant to effective organizing, thereby enabling smarter and more responsive definition of tasks and allocation of resources and effort. Algorithms processing conversations that naturally take place in organizations can form ‘computer augmented transparency’, creating a host of potential benefits, but also threats. These developments are leading to a wave of innovation in organizational design and changes to institutionalized norms of the workplace.


Introduction
Advanced algorithms, including systems identified as 'artificial intelligence' (AI), promise to increase productivity by harnessing computers and data for increasingly complex tasks, faster and cheaper than humans (McAfee & Brynjolfsson, 2014).While AI gained notoriety during recurring hypes accompanied with waves of bankruptcies in the 1970s, 1980s, and 1990s, billions of dollars were invested into AI-based startups in 2015.The current crop of artificial intelligence companies all exploit 'big data' , processing gigantic datasets through various analytics technologies, most prominently deep learning.With merely a hint of irony, AI and more generally smart algorithms really look to be for real this time around.
While the most obvious application of advanced algorithms is to replace human talent with cheaper computers, more interesting developments involve entirely new functionalities built around humongous data sets that humans could never process.It is time for organization theorists to attend to the potent effects that big data is having on organizing and management (Gerry & Lin, 2017).Strategy and information systems scholars have examined how consumer data shape strategy and business models (Constantiou & Kallinikos, 2015;Zuboff, 2015).Economists have identified AI as a general purpose technology set to shape large number of industries (McAfee & Brynjolfsson, 2014).This will influence work and the value of skills (Autor, 2015), with predictions of polarization in productivity so that consequently 'average is over' (Cowen, 2013).Data, it is argued, push organizations to become more responsive and more interconnected.Yet, these technologies can also directly shape the internal workings of organizations and management practice (Huber, 1990;Zammuto, Griffith, Majchrzak, Dougherty, & Faraj, 2007).
Typically, organizational design is conceived as structures, processes, policies (including incentives), and roles that help an organization carry out its strategy (Daft, 2012).In this article, I follow Garud, Kumaraswamy, and Sambamurthy (2006) to consider organizational design to broadly include elements that enable coordination among individuals.Since the culture and identity of an organization can facilitate or hinder coordination, they are integral elements of organizational design (Nadler, Tushman, & Nadler, 1997).

Big data and advanced algorithms
The growth of data analytics and more recently artificial intelligence applications is largely based on the increasing ability of organizations to represent the real world digitally, also known as 'virtualization' (Dodgson, Gann, & Phillips, 2013).Sensors of various kinds have become cheap and ubiquitous.In commerce and services, mobile phone and web-based services create real-time records of customers' and users' behaviors.Inside organizations, information systems create traces of all employee tasks, behaviors, outputs, and interactions.
Big data are now enabling innovations in organizational design, as technical solutions for analyzing large streams of real-time data are becoming more powerful and, perhaps more importantly, far more affordable.While big data and analytics are already widely used, they are also enabling new forms of organizing (Garud et al., 2006).Algorithms are at the core of the world's most valuable companies (Amazon, Facebook, Google, Netflix and Uber, to name just a few).Research on the effects of information systems on organizational coordination has a long history (Daft & Lengel, 1986;Huber, 1990).Authors have explored, for example, how algorithms and big data fit in the existing structures of the organizations and the new roles they create (Davenport & Harris, 2007).Dozens if not hundreds of textbooks exist on data analytics in business.Yet, big data and advanced algorithms remain a constant source of innovations (Gerry & Lin, 2017) that reverberate into organizational design.
To make sense of these developments, I suggest an analytical distinction between optimizing-oriented systems and open-ended systems.While various more complex categorizations exist (Chen, Chiang, & Storey, 2012), I believe the distinction helps understand the impact of big data and smart algorithms on how organizations are designed.
At the time of writing this article, nearly all applications of big data were focused on optimizing.Numerical and textual data are analyzed to control processes in ways that minimize or maximize pre-defined outputs, such as costs or revenues.These systems process large amounts of data, often in real time, to make choices that are either more efficient (based on some pre-defined criteria) or timelier than those made by humans.
In contrast, open-ended big data systems are far more recent and remain underdeveloped.They are typically based on semantic analyses of text masses (e.g.tweets or customer feedback) or visualizations of numerical data.They do not optimize any particular process in organizations, but rather distil useful insights from large volumes of data (Fayyad, Wierse, & Grinstein, 2002).Google Search is a prime example of an open-ended text-based big data application, which can even be considered artificial intelligence; Google now tailors search results based on an interpretation of the user's intent and use context (so-called semantic search).

Optimizing-oriented systems and algorithmic management of work
Big data enable organizations to more reliably predict and control their key processes.While initial big data applications were technical (e.g.optimizing material flows or targeting of advertising), new applications relate now to organizing and work.As a consequence, 'algorithmic management' of work is emerging inside and between firms (Lee, Kusbit, Metsky, & Dabbish, 2015).The term captures the new reality where algorithms track the performance of employees or contractors, optimizing decisions concerning their tasks and future employment.Algorithms are taking over scheduling work in fast food restaurants and grocery stores, using various forms of performance metrics and even mood, for example to assign the fastest employees to work in peak times.In algorithmic management, computers do not facilitate governance (as in evidence-based management), but instead governance itself is made obsolete through total control provided by data (Zuboff, 2015).This is already evident in 'gig economy' firms, such as Uber and Deliveroo, where algorithms make underperforming employees/contractors redundant automatically without human involvement. 1 This algorithmic management, or Scientific Management 2.0, if you wish, shifts power from a hierarchy of managers to larger cadres of professionals who master analytics, programming, and business.Management is no longer a human practice, but a process embedded in technology.Consequently, organizational learning will increasingly be embodied in IT-driven processes.
As management routines shift from humans to technological systems, the room for creative adaptation is likely to decrease (Pentland, Feldman, Becker, & Liu, 2012).As companies grow larger, the systems typically become increasingly complex and more rigidly coupled to established organizational design.Recognizing this, companies are likely to introduce artificial intelligence first for processes that are relatively rigid or independent from other routines.
Given the increasing centrality of information systems for organizing, organizational scholarship must study when and how artificial intelligence becomes coupled with or decoupled from other aspects of organizational design.Companies that lack the internal resources to develop artificial intelligence systems themselves, i.e. nearly all logistics, manufacturing, and retail companies, seem to have significant trade-offs in adoption of numerical big data systems.To benefit from the productivity promised by artificial intelligence, the companies effectively need to adopt organizational structures, processes, and even goals conceived by the providers of the IT systems.As optimal designs are likely to keep evolving through market competition, the companies who benefit the most from algorithmic management processes will likely be those that develop them internally (such as Amazon and Uber).

Open-ended big data systems and computer augmented transparency
Much of big data is text, such as messages.Students of information systems have long recognized the crucial ability of information technology to not just automate but to also informate (Zuboff, 1985), providing feedback with new insights into what is going on within and outside the organization.Numeric data have been processed through algorithms (from ERM systems to Excel) and visualizations (dashboards).In contrast, software has had a limited role in the processing of textual data, such as emails and customer complaints.This is changing.
Sophisticated natural language processing has the potential to create an ever-increasing visibility to the activities and knowledge of organizations, effectively creating 'computer augmented transparency' to skills, ideas, and opinions of employees as well as to attention allocation and ongoing issues (as predicted in Huber, 1990).This has the potential for organizations to increase their responsiveness and flexibility, assuming related changes to organizational design.Concept extraction and topic extraction algorithms can provide machine-readable codes to messages and documents, enabling easier access to knowledge.Google has developed a gigantic 'knowledge graph' of relationships across concepts, capturing their meanings.Machine learning algorithms now understand the flow and ebb of conversations, e.g.detecting who are the powerful individuals in control of conversations (Nguyen et al., 2014).Classification algorithms can group employees, customers, projects, tasks, and even meetings into distinct buckets to allocate managerial attention and resources.As computers are increasingly reliable in transcribing speech with negligible costs, all conversations that take place in organizations will soon be accessible to artificial intelligence. 2 AI can provide 'computer augmented transparency' , providing managers and (hopefully) employees answers to a broad variety of questions about knowledge work being done in the organization.Which customers are unhappy with our product?What are the topics our development team spends most time discussing in meetings?Who in our firm knows the most about usability testing?Who has been involved in discussions about the new product design?Although managers are unlikely to be talking to an artificial intelligence assistant any time soon, it seems likely our information systems will be able to provide answers to questions such as these in a few years' time.To take advantage of such developments, companies will rethink their structures, processes, and culture.
How will such artificial intelligence change organizations?Organizational structures will likely become less rigid, as employees are able to spot more transient opportunities for useful collaboration.Greater transparency of knowledge work will increase the span of control, allowing a single manager to monitor and mentor a greater number of subordinates.Once information systems can widely distribute information concerning individuals and their work and also organize that information so as not to drown people in information, experts can organize their problem-solving efforts more effectively without managerial involvement.That is, we may see a renaissance of self-organized work.
Unfortunately, employers may also use the technology to monitor their employees ever more closely, expanding Scientific Management 2.0 to creative professionals.The technology creates threats that would need a book rather than a paragraph to cover.There are countless ways in which the transparency created by processing of speech and text can reveal not only what the individuals know and work on, but also their sentiments towards tasks and individuals.In the worst case, technology creates more symbolic work.As firms become able to mine emails, employees may treat their every act of communication as a performance for the machine.

Conclusions and future research agenda
Big data and algorithms are already changing the way companies organize.New fast-growing corporations have developed algorithmic management to control employees and subcontractors, creating Scientific Management 2.0 in code, if not in discourse.While earlier information systems have created unprecedented transparency to numerical data and outcomes, new smart algorithms are expanding this computer augmented transparency to knowledge, attention allocation, and sentiments of employees.The volume of real-time, high quality digital data has increased dramatically, and it is inevitably leading to richer virtual representations of organizational activities and knowledge.Because information and knowledge are central sources of power, the data will be used to optimize and reconfigure what organizations do and how.The next decades will witness increasing 'digitalization of management' .
Advanced algorithms and the ability to process large volumes of real-time data enable companies to build 'routine smartness' in their operations, thereby reshaping the existing elements of organizational design.The structures and processes within organizations will be augmented by optimizing and open-ended information systems, leading to new kinds of roles and most likely widespread cultural change.We are likely to see shifts in the institutionalized norms of the workplace.Beyond the obvious questions related to changing elements of organizational design, these developments suggest a number of future research directions.
First, we need a better theoretical understanding of rigidities and flexibilities created by smart big data systems (Kane & Alavi, 2007).Do companies that adopt optimizing-oriented information systems become more rigid and exploitation focused?If so, how can such developments be avoided or mitigated?In what ways can companies build flexibility into algorithms or the other elements of organizational design?Under what conditions will openended big data systems increase exploration?Can they also support exploitative innovation?At this stage, such questions would seem best served by qualitative research.Given the dominant role of organizing and social practices in the use of technology, it seems worthwhile to study the same software solutions in multiple organizational settings to understand and; how systems and organizational design elements together shape organizational outcomes.
Second, we need to understand how the digitalization of management may be changing the norms and culture of professional knowledge work.Does the increased ability of technology to collect, process and synthesize data lead to a diminishing divide between the private and public spheres inside organizations (Brin, 1998)?In most organizations, both unique information and knowledge of other employees' secrets are significant sources of power.Can companies impose transparency on the upmost levels as well?How can companies avoid transparency translating into toxic politics?Will successful organizations adopt norms of radical equality and tolerance, where attempts to use the information about the knowledge, opinions, and efforts of coworkers for solely private benefit or political games will be considered illegitimate and condemned?Are companies more successful when transparency is extended to the work and activities of the upper echelon?Any changes to workplace norms are likely to take place at the institutional level, with companies in specific industries adopting similar approaches.
Third, we need to understand how companies will manage their external boundaries if their internal transparency is increasing.While internal transparency can significantly improve the functioning of organizations, secrecy towards external parties (customers, competitors, suppliers, and even investors) is a source of significant organizational advantage.How can organizations make their operations more transparent to their employees (and selected external partners) and still secure their secrets from external parties?Is greater internal transparency inevitably associated with greater transparency externally?
The adoption of big data systems and algorithms for internal management creates an obvious need for a new wave of critical studies (Zuboff, 2015).Algorithmic management and computer augmented transparency are not only shaping the objective conditions of workers, but they will also inevitably shape the identities of employees -the way members of an organization conceive themselves and their work.Inductive research will uncover the dynamics and effects of the digital management practices that we cannot foresee from the outside.
Several open questions related to organizational design innovation in response to big data are summarized in Table 1.The table serves more as a summary of potential research directions than a survey of the present states.There are unlikely to be clear-cut answers to any of the questions, as the effects of big data will be contingent on a number of contextual factors.Thus, perhaps, the correct way to phrase the research agenda would to ask: what are the conditions under which big data and smart algorithms will lead to one or the other outcome?
There is room for much optimism.The optimizing-oriented systems will remove many routine jobs and, by making better decisions, enable workers to focus on the most meaningful tasks and issues.The open-ended big data systems will help employees apply their skills to problems and projects that they can make the most difference to.They have the Will open-ended systems lead to greater autonomy and entrepreneurialism of professionals?
or will optimizing lead to an increased value of management roles in improving and enacting choices made by the optimizing algorithms?
or will they increase the supervision and control of knowledge workers?

Culture
Will optimizing algorithms make employment more transient and substitutable?
Will greater transparency lead to subversive ways to resist the managers?or will feedback loops generated by systems allow ever-increasing specialization within the company?
or will greater transparency lead to tolerance and more equal relationships across hierarchical levels?
potential to identify numerous valuable opportunities for collaboration inside companies, and broaden the horizons of knowledge workers.Natural language processing might finally deliver the expected benefits of knowledge management, eliminating redundant work where problems that have already been made are remade and solutions that have already been developed are reinvented with much effort.Few of these benefits can be realized without significant changes to organizational design.And much can go terribly wrong with the digitalization of management.

Notes
1. See. for example, 'When your boss is an algorithm' by Sarah O'Connor in Financial Times (8 September 2016).2. At the time of writing, Google Speech API charges merely $0.024 per minute to convert speech to text.While it mistakes around 5-8% of the words and cannot yet identify speakers, the pricing is indicative of how affordable, and thus ubiquitous, text-based systems are likely to become.

Table 1 .
open questions concerning the organizational design related to the adoption of optimizing and open-ended big data information systems.