Humans, Machines and the Design Process. Exploring the Role of Computation in the Early Phases of Creation

: Computational tools and the methodologies used in the early phases of creation often have conflicting requirements related to structure and systematicity. This paper describes an exploration into the increasing role that computation could play in the early stage of the design process. Insights from case studies at the design consultancy IDEO identified several key design activities and process variables used throughout the projects. Breaking these down into the contributing knowledge and creative ‘functions’ allowed computational representations to be created and a creative prompt tool developed. Further considerations on the possible features for the next technologies developed for these design activities concluded that a ‘bricolage toolbox’ of small, flexible and personalisable modules will enable the more ad hoc approach that is present in the early phases of creation.


Introduction
Since the mid-twentieth century, computation has become an increasingly important tool for designers. From the computer-aided machining tools that sought to "replace an individual's embodied engagement… with repeatable and controllable digital process" (Llach, 2015) to the Computer-Aided Design (CAD) tools that parameterised our drawings (Sutherland, 1963) to scientific research into a more computational epistemology of design (Simon, 1996), researchers have been striving to more closely integrate the logic and machinery of computation into the creative process. Gero (1990) in particular uses language familiar to the computational world when he writes that designing "can be modeled using variables and decisions made about what values should be taken by these variables." Applying this model to computational tools of the past, where automation of design tasks was the key goal, one could say that designers defined both the variables and the values for designs; the computers helped repeat their execution precisely. Today, designers can input key variables of a design into CAD tools that iteratively test huge numbers of different values to quickly output many alternatives with optimal characteristics defined by certain goals (Papanikolaou, 2012).
But how do designers initially define these design variables to assign values to? Are there any overarching categories of variables that guide the design process? What is the role of computation in this early phase of creation and how can it be improved? This paper describes research carried out at the product design consultancy IDEO that investigated these questions and contributes to this growing discussion by presenting a framework of design process variables and a new computational tool in which they may be used.

Integrating Computation into the Design Process
" [Design] intentions may be very vague at the outset, then may evolve and sharpen as the design process unfolds" (Mitchell, 1993) "Algorithmic representations require an explicit awareness of underlying assumptions and details" (Stiny & Gips, 1978) The above quotes highlight the dilemma presented when applying computation to the early stage of the creative process. The automatic generation and optimisation of many new values for design variables by today's computational design tools is possible because an explicitly understood and code-able algorithmic logic has been found that can define equations relating the different variables to each other (Alexander, 1966;Loukissas, 2012;Stiny & Gips, 1978). The early creative process on the other hand is an exploratory cycle of experimentation and evaluation (Schön, 1983) that meanders between phases of discovering variables, reframing concepts, envisioning solutions and creating designs (Mendel, 2012). There are no explicitly definable algorithms for the design yet; the variables are still emerging from considering a range of unexpected alternatives (Mitchell, 1993). Through a process of reflection-in-action, the designer continually tests their early 'hunches' and evaluates the experiments with respect to how well the design variables contribute to the overall reframing of the design problem (Alexander, 1967;Schön, 1983).
To generate truly paradigm-shifting 'creative' designs, this exploration relies on "the introduction of new variables into the design process, variables which were not originally considered by the designer or design system" (Gero, 1990). Such new variables can be found in many ways: reinterpreting the existing design by mutating its features, reframing it by considering analogies several levels of abstraction away from the original context, recombining ideas in surprising new ways to create a new requirement, or from first principles (Gero & Maher, 1993;Minissale, 2013;Schön, 1983).
These discrepancies between the explicit requirements of computational design and the nebulous experimental journey of early creative explorations highlight the challenges when applying computational design tools to this stage of the process. This is presented very clearly in a diagram by Bernal, Haymaker, and Eastman (2015), shown in Figure 1. It maps out the different human and computational tools available for the 'actions' of the design process; from early explorations, where tacit knowledge is used to discover, reframe and define the design variables, to the more explicit envisioning and creation of solutions where values are assigned. Fewer darker squares are present in the early activities of the design process indicating a lack of computational tools available, which Bernal et al. (2015) posits is due to the fact that the explicitly-defined hierarchical data structures required for computer programs are limited in their ability to support the more heuristic, abstract thought processes and ad hoc methodologies present in the variable definition stage.

Researching the Variables in the Design Process
What are the variables that are explored and discovered in the early stage of the design process? Eisenhardt's (1989) methodology to build theories from case studies was used to investigate this question in both the literature and real design projects.

Review of existing models of the design process
To inform the case study research, 88 existing design process models (Dubberly, 2004) were semantically analysed to identify any correlations in the variables used ( Figure 2). Of the many structures and activities present in the models, a set of general design variables did not emerge. However, the words used in the models could be divided into two simple categories: • Activities: the processes that are used to develop the design such as 'identify', 'synthesise' and 'evaluate', and often refer to the specific knowledge and tools used, e.g. during the 'rapid prototyping' process, knowledge about different scenarios can be applied to the design and mocked up using a 'videography' tool • Artefacts: the conceptual or physical design components created throughout the process, such as 'specifications', 'concepts' and 'models', and act as inputs to and outputs from a design activity, e.g. a list of 'essential problems' can be used as an input to a brainstorming process that outputs a range of 'conceptual variants' This framework allowed most of the design process models to be analysed using these categoriesthe processes, knowledge, and tools used during the activities, and the input and output artefacts created-and was used to guide the collection of information during field research of real design projects at a design consultancy.

Field research at IDEO design consultancy
The above framework was used to study four product design projects carried out by the Cambridge (MA) studio of product design consultancy IDEO during a six-week research residency in 2016. The case studies selected were current or recent projects that included a range of design outputs (research-inspired concepts, physical objects, digital communications websites and apps). These were chosen to understand the variables used across a range of design disciplines and to provide insights from seeing the projects develop in real-time.
The research methodology for these case studies was guided by semi-structured interviews and ad hoc discussions with twelve designers (eight male, four female) from a range of disciplines -four industrial designers, one interaction designer, one communications designer, two design researchers, two design engineers, one business designer, and one developer. At least two designers of different disciplines were interviewed for each project, allowing a multi-faceted view into each case study.
Participants were asked to discuss the development of the projects they worked on from initiation to final deliverable. Interviews were relatively free form to allow the designers to describe their own interpretation of the design process, but also included questions to prompt discussion around the framework categories, such as: 'What design artefacts, e.g. word maps, images, models, etc. were made throughout the process?', 'What was an input question or idea for a particular stage of the process and what output artefacts did it lead to?'. Questions about the design activities they used for their discipline were also asked, including: 'What processes did they use during the different design activities?' and 'What knowledge or tools, e.g. CAD software or prototyping media, did they need to achieve this?' The design artefacts that were created throughout the projects, e.g. design research videos, mood board images that led to early concept designs, physical form prototypes that led to design principles, etc. were also reviewed with the participants and their purpose in the design activities discussed.

Identifying the Variables in the Design Process
Using the 'activities' and 'artefacts' framework to guide the collection of information during the case studies allowed the often ad hoc design processes to be considered in a more structured way. This section describes the resulting breakdowns of the design activities studied and identifies further subcategories of design process 'variables' used throughout the projects. Project specific information has been removed for IDEO confidentiality purposes. Data gathered from interviews and design materials has been represented using the categories of the framework: input artefacts, e.g. 'extreme design themes', leading to output artefacts, e.g. 'extreme concept mood boards', using design knowledge, e.g. 'design elements appropriate for the project context', with specific processes or tools, e.g. 'image search tool such as Pinterest'. Interpreting the designers interview responses, the knowledge used in the design activities was further broken down using Veneselaar's (1987) model of the four types of design knowledge (Ahmed, 2005)-declarative (or factual), procedural, situational and strategic (or implicit).

Breaking Down the Design Process
These maps reveal that the projects followed the oft described experimental cycle of concurrent analysis, design, and evaluation (Lawson, 2006;Schön, 1983). However, the phases were often very overlapping during early exploration and experimentation activities and so dependent on the project brief and discipline of the designers involved that a generalisable process could not be realistically modeled, e.g. the very large multi-disciplinary project shown in Figure 3 included an extensive design research phase that the smaller more industrial design-focused project in Figure 4 did not. The ad hoc approach to selection of activities in this early phase agrees with Minissale's (2013) posit that the designer's process is one of "making-do with materials-a bricolage-rather than an inflexible systematicity." A comment from one design researcher described this diverse multi-disciplinary exploration method well: "Design right from the beginning, in whatever medium is most natural".
Despite this lack of a generalisable overall process, many of the design activities-such as 'metaphorical inspiration' or 'extreme themes'-were present across all projects. The use of certain knowledge and tools also appeared to be general across projects, with qualitative strategic knowledge used earlier in the design process and the more computational procedural knowledge towards the latter stages, concurring with previous research (Gero, 1990).

The Variables of the Design Process
Despite no coherent overall process model being identified, the above analysis of the activities carried out and artefacts created revealed several similar sub-categories-or design process 'variables'-that the designers applied to specific project information in order to guide their experiments. As one of the industrial designers described: "Every designer has their 'moves' that they work through to test out ideas".
These variables are: • Artefact types: When considering 'metaphorical inspiration' or carrying out 'consumer prototyping', the designers considered the many different types of artefacts that could be related to their final designs, ranging from objects, images, services, etc. depending on the openness of the brief and the designer's discipline. • Inspiration sources: There were many sources that inspired the designers while carrying out 'interviews and observations', considering the 'key insights and tensions', and creating 'extreme design elements', ranging from a word mentioned in a design research interview, or experiential elements from an immersive field trip, or a shape or pattern that fascinated them. • Experience adjectives: The designers often used expressive adjectives to describe the experiences they wanted their designs to evoke while semantically exploring the 'key insights and tensions' and creating the 'extreme themes'. These adjectives are layered with meaning and can relate directly to the design attributes of the image or object or be as abstract as the feelings evoked by a brand. • Design attributes: While creating 'design principles' and prototypes of their creations, the designers used their knowledge of 'archetypical design elements' and 'design logic' to experiment with many different attributes that could be relevant to the final product, e.g. the patterns of shapes to create image, or the forms and materials that build an object, or the touch-points that constitute a brand experience. • Media types: The designers experimented with different media types during 'existing artefact teardowns' and when creating 'extreme design mood boards' or early prototypes. They often had their 'go-to's', but mixing it up-using code instead of paint, or vice versa-often helped them to find new inspiration for their designs.
These design process activities, artefacts and variables that emerged from the case study research were used by the author as an initial prompt to consider how these more explicit elements of the design process might inspire the development of new computationally-driven creativity tools.

Computational analogies for activities in the design process
The framework and variables described above allowed the ad hoc activities of the early design process to be broken down into a more explicitly defined logic. This section considers how this almost computational understanding of the design process could be used to develop future design tools by actually 'codifying' these logical design procedures. Figure 5 shows an example of this by 'translating' the four types of design knowledge used in the design activities into example code for data structures. Figure 6 takes this further by considering what computational functions may be analogous to some of the activities in the design process. Here, Gero and Mahler's (1993) model of the modes of discovering new design variables-through reinterpreting, reframing, and recombining elements from existing designs-was used as a framework of 'functions' through which the design knowledge can be transformed.
These examples intend to show that by considering the design process from a more computational point of view we can develop a language for the knowledge and processes of design that can be understood by machines. Using this philosophy, a computational tool that uses the 'recombine' function defined above was developed and is described in the following section.

design(human)design
Recombining existing design elements in unexpected ways to discover new variables is a particularly human skill and enabling computational tools to "trigger unpredictable inferences" is a key area of development identified by Bernal et al. (2015). The IDEO designers confirmed this need as many expressed that "inspiration comes from random but purposeful inputs" and they often liked situations that provoked their creativity in surprising ways. These insights led to the development of design(human)design, a physical and digital creative prompt tool that uses the 'recombine' function defined in Figure 6 to present designers with a structured but serendipitous selection of design variables to inspire creative design ideas. Inspired by similar tools that use chance and ambiguity as a provocation mechanism such as Dadaist poetry (Gaver & Dunne, 1999) and Eno and Schmidt's (1975) Oblique Strategies, design(human)design creates a randomly populated sentence from words relating to the five design process variables discussed above: "Design {an artefact type} inspired by {inspiration sources} that is {experience adjectives} through {design attributes} using {media types}". The physical instantiation of design(human)design is a deck of cards for designers to use as a creative game to prompt new design ideas. Designers randomly select a card for each variable from a set of examples, e.g. 'an object' or 'an image' for the artefact design variable etc. and create their design inspired by the sentence constructed (Figure 7). Blank cards are also included for designers to add their own examples. A prototype of the design(human)design cards were tested during a one-hour workshop held at IDEO (Figure 8). The designers liked the unexpected provocations created by the random card selection, but sometimes selected specific examples of the variables while allowing randomness in others. They also wanted to create their own variables related to their specific project or discipline.
Building on this feedback that designers wanted to selectively randomise some of the variables and personalise the design variable words, an interactive design(human)design website was created at designhumandesign.media.mit.edu (Figure 9). A data structure similar to that described for declarative knowledge in Figure 5 is used to store a list of examples for the five design variables which are then randomised using the 'recombine' function. This design variable example list is stored in an online spreadsheet that can be modified by the designers to reflect words that would be most inspiring, e.g. collected from the design research stage of their project. Initial feedback from designers using the design(human)design website to inspire specific projects described it as a fun addition to the design process that would be particularly valuable in promoting expansive lateral thinking and idea generation after a phase of more in-depth research. The artefact type, inspiration source and experience adjective variables especially helped to juxtapose disparate research topics and inspire unexpected design directions, while the design attribute and media type variables were useful at guiding the new ideas to a more focused concept. However, despite the authors providing relevant examples, the designers found it challenging to populate the online design variable list with words that were relevant but not too general or specific from their research. Suggestions for improvements included the addition of unexpected analogical words and images related to the design variables selected, and a methodology and computational tool to help populate and continually modify the design(human)design database directly from documents in their research. These features are currently being developed by the authors and further more in-depth user studies conducted in the next few months.

Discussion on considerations for future computational design tools
Breaking down the activities carried out in the early design process enabled a consideration of how computation could play a greater role in the unstructured experiments that lead to the identification of new variables and creative designs. In comparison to the explicitly-defined hierarchical structures required in existing tools, the next technologies developed for this more ad hoc process need to provide a 'bricolage toolbox' of smaller more flexible tools that designers can use as their creative process dictates.
Building on these insights, some considerations for future computational design tools for the early creative phase can be proposed:

Small and modular:
The case studies showed that designers used many different design activities as and when appropriate for their experiments. The lack of a defined structure in this variable definition stage contrasts to the more systematised affordances of most computational design tools. New tools must provide a toolbox of these smaller and modular design tasks that can be used in the ad hoc style suited to the exploratory nature of the early design process. 2. Flexible and situated: Definition of the design variables highlighted that, while there are some similarities in the activities and variables considered across design projects, each process is situated related to its unique context and individuals working on it. Tools that designers use must therefore be flexible to integrate the varying information in these different situations, e.g. inputting new design variable examples into the design(human)design database.

Clear logic and interaction:
Any new information inputted to these situated tools could be provided directly by the designers or collected by another computational tool. Either way, the methodology for collecting the information used by these tools and the logic applied to it to generate prompts must be clear so that the designers can understand their interactions with the tools. 4. Personalisable: During the study, the IDEO designers were also asked to consider the role that increasingly intelligent design software could and should play in the design process. A key insight identified was the role that the designer's personality, individual style, experimental attitude or even just self-confidence plays in the final designs created. How might these tools learn our design idiosyncrasies and influence our interactions with them and resulting creations? As well as being flexible to the context of projects, these future tools need to include features that can adapt to the designer's sensibilities and style.
With these proposed features, the authors hope to contribute to the development of future computationally-inspired tools that can help define unexpected variables and provoke creative designs, as well as influence their continued refinement of the design variables, tool features and overall methodologies through which a computational epistemology can be discovered for the early creative process.

Conclusion
The role of computation in design is expanding to include more and more of the activities a designer carries out during their process. However, the affordances of current tools require designers to explicitly define design variables and adhere to systematic procedures which conflicts with the more ad hoc tool and methodology selection used in the early stage of the creative process. This paper considered how computation might be integrated into the early stage of the design process more effectively and contributes to this growing area of research by investigating how design activities might be broken down and translated into more computationally-parsable attributes and tools that can use them.
Analysis of existing literature and field research at the design consultancy IDEO identified some key categories of activities, artefacts and design process variables that designers frequently use, e.g. using 'metaphorical inspiration' activities to help identify different 'inspiration source' design variables. Considering the design process through the framework of these categories allowed it to be broken down into a more explicitly defined logic, from which code snippets were written as examples of an internal representation of the design process for a computational design tool, e.g. conditional functions to execute situational knowledge or randomising recombine functions to link design elements in new ways. The use of this code was demonstrated in the design(human)design creative prompt tool, which provided designers with random juxtapositions of words from their own research in order to help them generate new design ideas.