Fiction and Physicality: a designerly approach towards complexities of emerging technologies

: Rapid technological progression results in exciting new ways of interacting with our world whilst simultaneously limiting our experiences. Due to the pervasiveness of emerging technologies, designers are constantly faced with complexities and challenges, which necessitate the use of various tools and methodologies. This paper combines inspiration from the fields of aesthetics of interaction (Overbeek 1999), somaesthetics (Shusterman, 1999), design ethnography (Salvador, 1999), design fiction (Bleecker, 2009) and speculative design (Auger, 2013), to explore a designerly way of overcoming the complexity of implementing technologies into our daily life. We propose a holistic design approach to envision possibilities for emerging technology, integrating the physicality of human bodies with technological materiality. Further, we present a plausible narrative, containing visionary aspects and the investigated methodologies, alongside a series of design concepts that drive the storyline and form the basis for examining social implications, design and future contexts, and improving the way in which designers handle the limitations of a technology driven design approach.


Introduction
The continuing emergence of new technologies and their influence on our daily life -as well as upon design practice -can lead to issues with adoption, integration, interaction, and the overall experience of technology.We strive to change alongside the novelty of the technologies that we are presented with, learning new techniques and experiencing complex modalities.The process and 2 ongoing adoption of technology into our lives brings about the question: What does it mean to humanize technology?For designers, this question raises a crucial perspective in considering the transformations that emerging technologies present, and understanding these new challenges (Grudin, 2008, Sellen et al., 2009).The rapid growth and development of technology, its influence on human behaviour, and the way we interact with each other demands various challenging design considerations.New technology has previously provided us with exciting and novel ways to interact and extend our experience.This creates new notions of aesthetics and affordances for designers, but this complexity in emerging technology has also brought limitations alongside experience.
Dealing with this complexity has always been challenging, although design tools and methodologies have helped provide the means to deal with such challenges.For example, while various implementations of shape-changing technology try to address emergent interaction qualities, the complexity of the technology and implementation difficulties overshadows the importance of the interaction qualities and their implications in societal transformation (Rasmussen, 2012).
It is crucial to address the challenges that designers face in considering a new approach capable of overcoming technological difficulties, and emphasize interaction qualities and creative opportunities.Consequently, this work explores challenges associated with implementing emerging technologies and investigates new design approaches via exploring and adapting existing design tools and methodologies.

Challenges and Potentials of Technology
As our relationship with technology transforms, design researchers and practitioners are urged to rethink the underpinnings of their research and practice (Grudin, 2008, Sellen et al., 2009).Thus, these underlying values are the main aspect of consideration for designers faced with the technological complexities and societal transformations.Sellen et al. (2009) proposed five transformations that result from technological progress, two of which are of interest here: the growth of techno-dependency, and, the growth of creative engagement.
The former is indicative of our practice of technological reliance, and how we evolve around this technology on a daily basis.Sellen et al. also argue that this dependency provokes an essential question: "…how might they alter the skill sets of the people for whom we must design?".To consider this, designers need to understand the role of human skills in the technological integration of meaningful interactions.This often positions technology as a driving agent rather than a medium with which to expand creative potentials, therefore the challenge is to implement a design approach that understands the human skills and perceptions first, to then drive technological implementation.
The second transformation, the growth of creative engagement, highlights the potential of emerging technologies as instruments for creativity, where propagation of new kinds of tools enable people to engage in creative practices (Sellen et al., 2009).The implication of this transformation not only manifests in playful and self-expressive aspects of the creative space, but also indicates how such technologies could be integrated into design practice.The maturation of new technology could provide users and designers a new tool of expression, where it enables new interaction possibilities (Follmer et al., 2013) and opens up new design space (Rasmussen et al., 2012).Further, it encourages designers to adopt a new design approach that enables them to explore these aspects of design.
Overall, a particular technology can be seen as a phase of evolution in time (Grudin, 2008) -as we progress, the products of these phases face obsolescence and are ultimately replaced, resulting in limitations of the technology-oriented approach, leading to short term product developments (Lazaret al. 2015) and stalling at the novelty stage (Tanenbaum & Tanenbaum, 2015).This notion of "stalling" can be applied when examining research around the central topic in current project.We look to the design and development of shape-changing interfaces, where the drive is toward the "Ultimate Display" (Sutherland, 1965), and many interesting prototypes and ideas are abandoned early on as we strive toward an ideal technology.
To embrace this ideology, we must begin to think in multiple dimensions.We therefore explore here the validity of a blended approach to design shape-changing interfaces, using design fiction as an ethnographical tool (Lindley, Sharma, & Potts, 2014), alongside the aesthetics of interaction (Dunne, 1999;Overbeeke et al. 1999) and somaesthetics (Shusterman, 1999), enabling us to reconsider the role of human skills in technologically driven design implementations.

Related Work
The discussion and fictional narratives within this paper can be placed amongst several disciplines.Design fiction and speculative design play a major role in setting the stage for creating this unique ethnography of shape-changing interfaces, and borrow in turn from design methodology, somaesthetics, and the aesthetics of interaction.These are outlined below.

The Aesthetics of Interaction
The aesthetics of interaction is concerned with how experience of a product should consider users' cognitive, perceptual-motor and emotional skills, where it provides rich interaction (Overbeek et al., 1999).This stance provided designers with new ways of exploring objects, other than just visual aesthetics, and situations through bodily experience (Petersen et al., 2004), and suggests that we approach technologies from the standpoint of already knowing ourselves, and applying ourselves against the feelings, motions and textures of devices and surfaces that we are given, therefore, suggests that the aesthetics of interaction helps us "establish the way that an object allows for interaction, the way it influences our experience of value" (Hummels and Overbeek, 2010)-good design can tap into this experiential act, complementing the view that humans are experiential creatures who have the capability to change with technology.

Somaesthetics
The field of Somaesthetics developed from a pragmatic approach towards aesthetics of experience in the context of analyzing and implementing interactions, to consider various aspects of human movements (Shusterman, 1999), and at the same time illustrate the importance of experiential qualities in design made clear by the study of affordances and the aesthetics of interaction.Human movement can be seen as an under-utilised and instrumentalised design consideration (Höök et al., 2016), despite its value as an explorative perceptual tool for participants, and as technology has developed, we have in turn evolved new ways of moving to interact with devices (i.e.swiping or pinching touchscreens).However, the relationship between human movement and technology can also be constricting in nature.

Anticipatory Ethnography (Speculative Design / Design Fiction)
One of the biggest challenges designers face when dealing with emerging technologies is envisioning a future where they will be closely working with them.Due to this uncertainty, designers often take the easier approach by implementing what is 'Possible', instead of considering the 'Preferable' (Dunne and Raby, 2013).Candy (2010) suggests future perspective -where one views the future as a predictable continuation of the present -but also as a future from the perspectives of 'Probable, Preferable, Plausible and Possible'.For designers, the imagination space of a 'Preferable' future is intriguing (Dunne and Raby, 2013).
Speculative design and design fiction employ imagination and practical knowledge to create and/or critique future scenarios that fall within varying degrees of likelihood (Coulton & Lindley, 2016).Speculative design proposes an approach where designers create grounds for discussion and others can share the vision of their future through experiencing the designs.For the purposes of this paper we will use the term design fiction.
Design fictions do not merely tell stories, but also worlds (Sturdee et al. 2016).Within them we look not only at diageses, but at the discursive space around such prototypes.A key notion, "perceptual bridge" (Auger, 2013), addresses the discursive engagement that exists between our perception of the world and the fictional element of the concept.Management of such speculation determines the richness of the perceptual bridge, therefore it is crucial to consider what informs the use of technology, aesthetics, behaviour, interaction and function of each artifact, by opening up the discussion to various stakeholders throughout the design process.Lindley, Sharma & Potts (2014) suggest that design ethnography can support and inform design fiction in as much as it supports uses, investigations and cultural appropriation.This anticipatory ethnography relates to the process of creating design fictions, their audience and their content.In essence, traditional design for technology might be seen as reactive, whereas design fiction can be seen as preemptive.Design fiction can inform future technology by disseminating artifacts and stories to a wide audience, creating a circular exchange of information.However, the act of creating fictions is not sufficient to embody the experiential world alone -complementing design fiction with other methodologies means that we should be able to explore richer experiences.We therefore propose a blended, holistic approach, which incorporates elements of all of the above methodologies as our primary investigative method.This work explores the challenges of designing with emerging technology as a technology-centric design process, utilising a holistic design approach in the form of a design workshop.It addresses the complexities of creating new ideas of aesthetics in the context of future design, by providing insight as to how designers might adapt their current practices.This future design practice should enable designers to overcome such challenges as technological dependency, understanding of creative opportunities, and provide a way to deal with challenges arising from integrating emerging technologies.Thus we seek to investigate: how do we facilitate future-creatives to think and design alongside emerging technology, and further provide designers with appropriate tools and methodologies for the future design space?

Exploratory Workshop
The following section describes a workshop setup that employs the aforementioned theoretical background as a guideline for design exploration.
The participants were six Industrial Design students who were interested in exploring design approaches within the context of designing for shape-changing interaction, e.g.devices, which interact with you via their own physical movements.Participants were given a short introduction and overview of the workshop and the intent of the activities, before moving onto the three sessions outlined below.

Session 1: Interaction Perception
Here, participants were oriented around human perception in design, through notions like affordances (Gibson, 1979;Norman 1999;& Hartson 2003), feedback/feedforward (Wensveen et al. 2004), the aesthetics of interaction (Overbeeke et al. 1999) and especially focusing on Interaction Frogger framework (Wensveen et al. 2004).They were also introduced to a set of method cards that were developed for the workshop (Hur et al. 2015), and both of the latter were used as guidance throughout.The cards represented the notion of Interaction Frogger, showing a series of clear examples of interaction with various daily artifacts.
Once participants became familiar with the notion through presented example artifact, they were asked to either pick an artifact from the cards, or from their everyday life.Afterwards they were encouraged to walk through the perceptual information exchange that happens during the interaction with the artifact, using the method cards.After an initial examination, participants gathered to discuss how each information exchange made up the overall interaction experience and the detailed implications of designing embedded information.At the end of the session, participants shared a detailed analysis of the interaction and associated perceptual qualities to the rest of the group.

Session 2: Bodily Movements
In session 2, the discussion moved toward perceptual information flow and overall experiences, which could be physically expressed through bodily movement.Participants were paired within the group (to consider physical discomfort due to close encounters with unfamiliar people) and presented with a short movement exercise, followed by a brief introduction to Laban Movement Analysis (LMA) theory.Participants were encouraged to warm up for movement, and help their partner to move, these movements were then used as a method learning the fundamental methodology behind LMA concepts.The remainder of the session consisted of bodily movement explorations, expressions and creations -Choreography of Interaction (Klooster et al. 2005) -in accordance with the interaction analysis from the previous session.Participants were encouraged to go through various iterations of presentation with their movements, including feedback and adjustment.The goal of the session was to analyse the existing interactions through experience, and redesign these to form an expressive and intuitive movement-based interaction.Additionally, in order to explore the interaction further with their bodies, participants alternated roles: either as the artifact itself or the user.Participants were encouraged to be creative in exploring, expressing, sensing and perceiving physical interaction between them.Despite the freedom of the exercise, human bodies present physical limitations in mobility and scalability, thus participants were encouraged to use provided physical props to explore limitations and potentials.Toward the end of the session, participants were asked to perform first their existing interaction, then their newly created and redesigned interaction.The remaining participants were then encouraged to discuss and "try out" the movement with the performers.Each pair was then encouraged to reflect upon their bodily experience and discussion to adjust and tweak the interaction they had witnessed.

Session 3: Fiction Through Experience
Following refined concepts of movement, interaction, and quality of experience, participants were encouraged in the final session to think about the artifacts in a wider context.Interactions explored up to this point had only been concerned with one particular event that would represent the artifact, therefore each pair was asked to create a context in which each artifact would reside.This was intended to encourage participants to consider the artifact-level context where they would be forced to expand the limited interaction experience into the wider context of other functional experiences.
Once the context was defined, they were asked to create a short description or scenario to detail the context in which each of the newly designed artifacts would be utilized.These were then presented to the rest of the group and discussed in detail.
Afterward, participants were encouraged to come up with cross-contextual description, thinking about how each artifact could expand its functionality, purpose, and experience.To initiate a story compiling process, participants were asked to identify common qualities across scenarios, then analyze and seek out the common contextual settings where these artifacts could coexist in an ecology.These crossovers were then refined and merged into a single storyline to describe the interactions and experiences that would enable these artifacts to co-exist in a continuous experiential world.As a final task, participants were asked to reflect on the outcomes from each session, looking at possibilities to re-iterate and/or refine, for example examining an individual artifact scenario to re-explore the interaction with bodily movements.

Outcomes
The workshop exploration documented here resulted in three distinct forms of outcome: Movement Expressions, Artifact Concepts, Concept Fictions, each representing unique design perspectives of the current approach.

Movement Expressions
Three artifacts were analyzed with bodily movements: Turntable, Electric Food Mixer, and Car Windshield Wiper, each with two sequences of movement.The first choreography represented the existing interaction, with the second representing the reinvented interaction.Initial choreography tended to emphasize one or more aspects of natural couplings, e.g.Car Windshield Wiper emphasised Dynamics and Expression (where the relationship between the repetitive motion of the wiper and a relatively static control arm movement created a passivedynamic relationship, putting the user at a distance -see Figure 1.), whereas Electric Food Mixer choreography emphasised Location and Direction (where the location of manipulation points was specified with posture, and direction of action/reaction was richly expressed through haptic feedback -see Figure 2.).This tendency may have arisen because interaction aspects were introduced as "manipulation points" that could therefore change the overall experience of the interaction, thus these expressions therefore indicated the important aspects of interaction in each artifact, significantly influencing the overall experience.For the second choreographies, emphasized aspects were explored and manipulated to find alternative implementations.These movements resulted in changes in the overall expressions, interactions, and experiences.Most significant were the changes in Car Windshield Wiper, where the static relationship between taking action on the control arm and the subsequent reaction of the windshield wiper was dramatically changed to very dynamic movement relationship between user's pumping action and artifact's blowing reaction (Figure 3.).The remaining choreographies kept their core movements, such as the locking and releasing movements of mixer arms in Electric Food Mixer, or the delicate movements of tone-arm in Turntable, instead changing other supporting movements to influence the overall expression and movement relationships (Refer to video links below).An overview of the choreographies for each artifact can be seen in Table 1.
Table 1.Overview of Choreographies (refer to the links below for each choreography).

Artifact Concepts
The movement choreographies were expanded into design concepts where each interaction and experience was elaborated into a whole artifact consisting of additional functionalities and an overall purpose.These concepts initially followed the existing functionalities of the artifacts, e.g.Electric Food Mixer was primarily a food mixer, even after its second choreography.After reflection, these initial artifact purposes were re-examined from an artifact level perspective, enabling participants to realize changes in their definition of the artifact, or envisage it serving an entirely different purpose.
Exploration of Turntable started with the function of playing a vinyl record and serving a purpose of reproducing sound.Analysing the choreographies that expressed the dynamics of a record player lead participants to manipulate, translate and later consolidate these qualities into a futuristic portable Memory Player, which mechanically recorded and played memories of users.Here, while the mechanism of playing recorded materials was maintained, the application of the mechanism and its implementation changed the overall definition of the artifact.More dramatic changes in overall artifact purpose were seen in Electric Food Mixer, where investigation into direct/passive and near/remote (related to location and dynamics) lead participants to propose an Automated Food Management device, incorporating both passive and active interaction.Meanwhile, the Car Windshield Wiper group explored further the dynamic nature of action and feedback, which lead them to consider the nature of attention, this lead to introducing a new concept for an attentionseeking Rear-view Mirror.

Dynamic Windshield Wiper
Overall, each artifact-concept translated the essential qualities of the interaction experience from movement to artifact-level purpose, making participants think about how these experiential qualities could be implemented as design artifacts, and what relationship to them we would have as users.

Concept Fictions
Fictional narratives were created to elaborate upon the context around the novel devices described in the previous section.
Electric Food Mixer (now an Automated Food Management device called Wiz-M) imagines a social context of "food crisis" where trading food is prohibited, forcing people to produce their own, and where storing and maintaining food becomes challenging: "The Wiz-M is an autonomous mixer that walks through your fridge to keep your food fresh."It not only stirs food when it needs to be kept consistent and fresh, but also keeps the moisture of food optimal to prevent drying up or early oxidation.
Memory Player tells the story of a device that does more than just playing someone's memory."It made it easier to highlight your favorite moments in your own memories, which was difficult to do using the wink control on your lens (another memory control interface).Imagine you need to wink through 10 years of footage, it was awful".Here, the device is introduced as a solution to a technical challenge in using physical interfaces that they have explored in the interaction.
Car Windshield Wiper explored the wider context of peripheral assistance in automobiles: a rearview mirror called Wally."The Wally resides inside a modern vehicle with a look of self-consciousness, constantly seeking to see if anyone in the vehicle requires its attention.It loves the attention and also likes to give attention to people."The mirror's function as a safety device has changed, it is repurposed as a communication device inside the vehicle.Wally functions to assist passengers of the car, and the story expands on how it is implemented the implications.
Each story enabled participants to expand the scope of exploration and the refined artifact concept on a social level, but also enabled participants to explore the wider contexts and how each artifact fit within them.The stories were subsequently compared and compiled into a single storyline demonstrating the ecology in which these artifacts thrive, rely on each other, and share context.
Two elements were highlighted as significant across all stories, Unique Future Perspectives, and Autonomous Artifacts (with highly physical interactions).Examining these elements, participants then discussed the wider context where the overarching story might take place, identifying social, economical and cultural settings.These were used as a common backdrop for the three individual stories.From Autonomous Artifacts came the term 'artifreacture' where people interact with devices like living creatures.The final story focuses on four individuals with distinct artifreactures, their relationships and experiences -also detailing how these artifreactures are perceived, placed, treated and used, and the overall implications of their existence.(Refer to the Main Story, find the link below).

Discussion
The current study explored a potential design approach to address future and current challenges of designing with emerging technologies.We have tried to address two particular challenges that Sellen et al. (2009) proposed; overcoming technological dependency through focusing on human skills rather than technology itself, and expanding creative potentials through technological possibilities.
We aimed to do this through the utilization of three key design considerations: analytical aspect of perceptual consideration, experiential aspect of bodily exploration, and contextual aspect of narrative expansion.Theoretical knowledge reflected in Aesthetics of Interaction, Somaesthetics and Design Fiction was emphasized through the transformative process of turning this knowledge into design methodology (Hur et al. 2015).These three elements played a key role in developing a complex ecology of artifacts and contextual settings as a result.
These three theory-derived elements provided design tools to develop the presented concepts and expand the design context through the process.The participants started with perceptual consideration where they were concerned with perceptual qualities of an-object-to-a-user interaction level, where they had opportunity to understand detailed sequences of interaction that makes up the overall experience of an artifact.These insights provided focus points for the participants to generate experiential choreographies where these perceptual aspects were tested and emphasized.These experiential elements were then expanded into artifact concepts and transformed into general concepts of artifacts.The most significant transformation or translation happened when using this accumulated design knowledge in wider contextual fiction.By setting the wider context and history in the development of each artifact and societal, technological and personal (specific to each character and person of interest) perspective, the story created a common background for how these unique artifacts could be integrated into people's lives.The story does not only cover the societal impact or contextual and wider impact but also showed personal implications of each implementation and integration (Refer to the Main Story, find the link below).
The compiled story enabled the participants to see the story unfolding not just in the perspective of one artifact or a single user, but as an overall experience and the consequent relationships between different physical interactions.The conventional design for interaction puts the scope of interaction to an object and a user (Lenz et al., 2014).Further, it also treats the interaction as one sequential event.However, in reality, the interaction with our surrounding artifacts is something we do continuously moving from one to another, and even interacting with multiple devices simultaneously, which designers often overlook and is something most design approaches do not consider.
The current approach provided participants with an opportunity to consider various levels of design perspectives, where they started from the smaller scale of an object and a user interaction consideration and gradually building design details up to a larger contextual scale (Figure 4.) using the three previously mentioned design elements as tools.The intention of expanding interaction level of concepts through re-examining the functional level of artifact's purpose and re-defining the artifact itself was to re-imagine the artifact through newly created functional interaction.In some cases, the newly created interaction introduced totally different functionality that would change the definition of the artifact as seen in the example of the Memory Player.This change of artifact definition and what purposes and roles these artifacts take in our daily lives seems to be more dynamic as these objects become increasingly technologically advanced.One of the aims of the current approach was to raise the awareness of this dynamic nature that artifact definitions have.Further, it would provide designers with an opportunity to examine the overall purpose of each artifact and how their intended interaction with the artifact fits into the manifested purpose.
The current study is an exploration towards an ever evolving process of design and integration of changes and technology.The discussed design process and results demonstrate new potentials for dealing with design challenges in the future.Especially with holistic insights into various implications of the design implementations and technological applications, the results from the proposed approach provide a foundation for further design exploration and actualization rather than being the final design product in itself.

Note:
All videos of choreographies are available online at (password required: designfornext): Snippets of Three Concept Fictions https://medium.com/@submitted/three-artifacts-4df8727ab720 Compiled Main Story https://medium.com/@submitted/curious-cases-of-human-artifact-ecology-4a5e9c59610e About the Authors: Yeup Hur is a PhD candidate at Eindhoven University of Technology.
Trained as an Industrial Designer with a focus in tangible interaction, he explores human body movement as a medium to create richer interaction, through collaborating with choreographers, artists and designers.
Miriam Sturdee is a PhD candidate at Lancaster University, holding an MRes, an MFA in Visual Communication, and having previously studied psychology.These relate to her current interests in HCI and design, focusing on illustration, sketching techniques, and design for shapechanging interfaces.
Miguel Bruns Alonso is Assistant Professor at Eindhoven University of Technology, focusing on aesthetic experiences emerging from the continuous interactive loops between users and products with perceptive and programmable material qualities such as haptic and shape-changing interfaces.
Panos Markopoulos is Professor at Eindhoven University of Technology, focusing on ambient intelligence, end user development, interaction design and children, rehabilitation technologies and design for behaviour change.
Jason Alexander is Senior Lecturer in HCI at Lancaster University, with a particular interest in developing novel interactive systems to bridge the physical-digital divide.His recent work focuses on the development of shape-changing interfaces, haptic feedback, and gestural interaction.

Figure 1 .
Figure 1.Car Windshield Wiper interaction expressed with whole body movements, emphasising Dynamics and Expression.

Figure 2 .
Figure 2. Electric Food Mixer interaction expressed with whole body movements, emphasising Location and Direction.

Figure 3 .
Figure 3. Redesigned Car Windshield Wiper choreography with emphasis on Dynamic aspect of interaction.

Figure 4 .
Figure 4. Design Perspective Scales mapping with current workshop activity sessions