Using Design Fiction to Inform Shape-Changing Interface Design and Use

: Shape-changing interfaces are tangible, physically dynamic devices which enable user-experience beyond 2D screens. Within Human Computer Interaction, researchers are developing these from low-resolution, low-fidelity prototypes, toward a vision of a truly malleable world. The main focus is in producing and testing hardware, and basic user interactions, which leaves the question unanswered: what are shape-changing interfaces good for? In response, we propose the use of design fiction to investigate potential applications for this technology: to create and analyse artifacts relating to future use-scenarios for shape-change. Whilst research within shape-change often proposes future use-cases for prototypes during discussion, they are seldom in a form that presents them as everyday artifacts. Here, we present and discuss a printed game-play instruction manual for a truly high resolution shape-changing game entitled First Hand , which aims to draw parallels between current gaming practices and the tangible nature of shape-changing interfaces.


Introduction
As we progress beyond the desktop computer or tablet, the idea of the interface moves from flat, planar devices towards interactive surfaces that afford the user a physically tangible experience.Shape-changing interfaces are a branch of this ideal, weaving application with touch, sound and image in a physically dynamic, temporal user-environment.However, this area of inquiry remains far from the high-resolution interfaces we see within science fiction (Troiano, 2016), and researchers are left to ponder the implications of their prototypes, should technology allow the blending of surface and material into a seamless, malleable interface (Ishii, 2015).Discussion around future application areas is a common theme, with suggestions encompassing gaming (Ye & Khalid, 2010, Khalilbeigi 2011), data physicalization (Jansen et al. 2015;Taher et al. 2016), robotic pets (Nojima et al. 2013), and multipurpose clothing (Follmer, 2012).Additionally, there are studies which examine the potential of shape-change from the users' perspective: e.g.Sturdee et al. (2016) employed a participatory design process which showcased a shape-changing prototyping tool in order to gain perspectives on the desires for this new technology from a public participant base.There are also a growing number of papers which examine the deeper implications of this technology, such as emotionality (Kwak et al. 2014) and temporality (Vallgarda, 2007).
Combining the scope of the current state-of-the-art into meaningful avenue of enquiry is a difficult proposition, as is considering devices that can operate beyond the current application design.To enable us to take the necessary leap forward, we employ the technique of design fiction to create a plausible product based on the current trajectory of the field, which can then feed back into current research practice: exploring both the viability of the methodological approach, and generating meaningful results for shape-change.
Design fiction is not a new concept, although it has only recently become popular in HCI research (Lindley & Coulton, 2015;Sturdee et al. 2016), and refers to the creation of artefacts with which we can explore and analyse future scenarios.Design fiction tells worlds -not just stories (Sterling, 2013), blending narrative, films, comics, and ephemera to immerse the reader in a possible future (Coulton et al. 2016).Design fiction describes the area, though the artifact discussed within this paper is classed as a diagetic prototype (i.e. an object existing within a piece of narrative art), exploring the gaming scenario as a use-case for shape-changing technology.We propose First Hand: a game concept that allows the user to "play god" and either shape planetary life, or the planet itself, as a single player or in a Massive Multiplayer Online Role Playing Game, presented in the form of a gameplay instruction manual, with further credence given via an article written about the game launch and the corresponding technology required to run the program.

Shape-Changing Interfaces
Shape-changing interfaces are an exciting part of Human Computer Interaction, creating a bridge between our planar, screen-based computers, with devices and surfaces more commonly seen in science-fiction (Troiano, 2016), but not yet a tangible reality.Shape-changing interfaces can be defined as devices that utilise physical 3D surfaces that can move independently, react to varying stimuli, accept and respond to user input and display information.Exemplars in the field include Transform by MIT (Ishii et al. 2015); Emerge (Taher et al. 2016) which is concerned with the challenges surrounding data physicalisation (see Figure 1); Sachiko Kodama's dynamic ferrofluid sculptures (2008); and, Jamming interfaces (Follmer et al. 2012) which examine the interplay between controllable stiffness and fluidity.These are but a few of the prototypes in the field, which offers a rich tapestry of tangible, interactive and expressive interactions.Alongside this expanded interaction potential comes the challenge of how designers begin to create applications that may span different materials, resistance, temporality and even emotionality (Kwak 2014;Rasmussen 2012).This multi-dimensional area goes beyond the current scope of interaction design, requiring a new approach to ensure that design keeps up with hardware.

Design Fiction in HCI
Design Fiction is a fairly recent research trend, but has already been incorporated into several areas within HCI.Tanenbaum (2014) makes the case for design fiction in HCI and interaction design, by suggesting that it can be methodology, communication tool, and motivation or inspiration for design -allowing us to explore requirements prior to the build process.Most relevant to this paper perhaps, are Linehan et al's Alternate Endings (2014) which looks at contemporary HCI research and a long term view of the technologies they depict -challenging the short-term, utility driven work that is seen the field; and Lindley & Potts (2014) work on prototyping using design fiction.(Taher et al. 2015) is an actuator-based interface allowing the user to explore multi-dimensional data.

Design Fiction, Shape-Change & Games
The gap between technologist and designer can be broad, but there are those within the field of shape-change who are already embracing the concepts explored in detail here.Ishii et al. (2012) proposed a material called Perfect Red, which was part of a project exploring a vision-driven design process of shape-changing interfaces, and explores the idea of a clay-like material that also has computational attributes (such as snapping to geometric shapes, or merging distinct pieces).
The future of gaming has long been seen in science fiction films (e.g.eXistenz, Lawnmower Man), and is a popular speculative topic, but not often addressed in research.Game designers are quick to adopt new technologies to explore their entertainment potential, e.g.eye-tracking (Vidal et al. 2015) or mixed-reality (Simeone, 2015), but do not focus on what is not yet available, as much as they do on improving current technology.Design fiction as a methodology has not yet been applied to the design space of shape-changing games, despite the popularity of gaming as a theme in research.

Why gaming?
Gaming as a theme for the diagetic prototype was chosen for several reasons.Primarily, Sturdee et al. (2015) found that during a public ideation workshop investigating shape-changing interfaces, the most popular category of idea was games and entertainment.Second, several papers concerning shape-change are either based around gaming (Ye & Khalid, 2010) or suggest gaming as a future direction for research (e.g. Lee et al. 2008;Khalilbeigi et al. 2011;Kildal, 2012;Makino & Kakehi, 2011).Others in HCI also suggest that gaming, and gameplay, plays a vital part in shaping the future of the interface (Isbister, 2011).Central to the principle of shape-change, is the fact that the interface is physically tangible, dynamic and directly graspable in a way that eludes current game-ready hardware.Thus we focused on building the game concept around the interface novelty, rather than adapting an existing game for the potential interface.Features that are collectively unique to shapechanging interfaces (in this application) include: Physicality -Tangible surfaces and spaces upon which the user can exert force to change the physical shape.
Dynamicity -The surface of the interface can react physically to stimuli (be it environmental or programmed) and is not passive.
Sculpting -Using hands or tools to manipulate the surface in meaningful ways.
Multi-sensory interaction -Shape-change can harness a wide variety of inputs and outputs.
Option to incorporate hybrid gaming environments -Shape-change can be combined with existing technology to enhance the player experience and add multiple dimensions to the interface.
Although additional features are possible, for the purposes of the creation of the design fiction we focused on a "wish-list" which would be necessary for a rich user-experience within the game world.These features informed and were expanded on during the design process.

The Making of First Hand
The creation of First Hand took place over several stages.Initially, an exploration of current and past games was made which fit around the theme the researchers had envisaged (e.g.Populus, Civilization, Spore, Elite), utilizing the knowledge of experienced gamers and bibliographical research conducted online.The initial theme itself was borne out of the idea that shape-change gives us infinite possibilities and sensory experience, and to have that kind of power to manipulate our environment might be akin to being omnipotent (albeit within the confines of the interface).
Novelty was an important factor in the design of the game and its play, thus we combined both the idea of "playing god" and the concept of a second stage involving space exploration and an overarching theme of intergalactic domination to differentiate the game further from its predecessors -not relying simply on the tangible nature of the interface to carry the idea.The multiple-stage gameplay allows the shape-changing interface to show off numerous features, whilst maintaining a consistent narrative (e.g.planet formation, species evolution, species saturationexploration of space and settling new worlds.Sketching even the basics of the game idea threw up questions that required answering before finalizing the diagesis.These are discussed in the following section.

Analysis of the Diagetic Prototype
Using design fiction to create an artefact or prototype means that traditional evaluation cannot be employed (Blythe, 2014).HCI user-studies usually involve a working prototype with which the user can perform set tasks and the researcher can gather data as to the efficacy of the device or application in use.In comparison, a design fiction or diagetic prototype might be examined via multiple methods.Here, we view First Hand via anticipatory ethnography (Lindley et al. 2014) -a technique designated to "operationalize the practice (of design fiction) in industry contexts" -by looking at the process of creation (e.g.what insights were gained during the making process, how were design decisions made?) , and the study of the content itself (what insights can be made by the viewer/reader upon completion?), then document the findings from the content/thematic analysis (the potential for audience interaction is explored within the discussion).

Themes, Ideas and Implications
Creating the First Hand: Quick Start Guide produced many points of interest, e.g. the need for tool based interaction.Some points were "solved" during the creative process (i.e.decisions were made by the designer), and others emerged as themes discovered after the game manual was completed (see Figure 2 for examples -full manual available online at shape-change.org/design-fiction).Following completion, the content was explored via thematic analysis, and three distinct categories emerged: interaction, hardware, and conceptual (i.e.non-tangible concepts or themes not directly relating to interaction or gameplay), containing a total of 19 features.These features were generated by comparing current working prototypes with the proposed hardware capabilities of the novel interface, looking specifically at novel (and currently undeveloped) interaction styles and application design:

INTERACTION:
• The Riffler: Examining sculpting as an interaction in gameplay quickly gave rise to the need for tool-based interaction for fine detail when sculpting creatures and landscapes.This tool became an addition to the interface design, named the Riffler (a tool used for shaping fine detail in a number of materials).• Rotation of suspended items above another layer: Editing the lifeform/planet necessitates a 360 degree view, hence items being edited need to be rotated in space and remain in place when pressure is applied.• Painting/drawing on shape-changing surfaces: Mark-making on dynamic 3D surfaces is akin to painting 3D objects, but there must be an algorithm which dictates how that surface is managed during topology changes -i.e.surface area vs perceptual volume • Using buttons and menus: Despite the added dimensionality of shape-change and tangibility, it was still deemed necessary to have toolbars which could be "raised" and interacted with to change between editing modes, layers and screens.• Animation by recording movements: This feature can be linked to robotics or physiotherapy in terms of manipulating items in the desired manner and "saving" or adjusting them to the most favourable movement.• Physical "Undo": This works in the same way as it would in an text or image editorphysical changes can be reversed, step by step.• Adding gestural interaction: Gestural interaction can be used to execute large "area" commands -i.e.close layer, close game.• Moving solid objects within space: Objects within the game environment must retain a rigid position in space, and be moved only within the limits of the game program.

HARDWARE:
• Distance from baseline: Device-based gameplay cannot be infinite due to limitations of contained mass in the interface, range, and safety aspects.Therefore we postulate that there must be a limit as to how far matter can be projected or moved from the central processing unit.In gameplay, this manifests itself as a smoothing of surfaces when maximum zoom and distance from baseline are reached, or as a maximum distance that spaceships or other objects can be placed from the processing unit.
• Representing liquid and solid concurrently: Variable rigidity must be possible to differentiate between liquids and solids in the interface -e.g. a liquid must be pliable and flow around objects with a greater density.• Placement of pre-formatted 3D objects: Items can be dragged from tool bars and have a pre-set 3D form, i.e. limbs in creature mode or volcanos in terrain editor.• Switching between solid and holographic layers: Gameplay requires the function to work between solid (i.e.tangible) and projected (non-tangible) layers, and choose which is solid at a particular point in time.
• Multiple solid-state shape-changing layers: The game requires solid objects to be present at various distances from the baseline at any one time -e.g.spaceships floating above planets.• Transitions between modes/views: Transitions might need to occur instantaneously (switching between planet and space/creature view) or gradually during interaction (zooming in and out).

• Exporting physical items for 3D printing:
The tangible nature of the sculpting process might give form to works of art that can then be exported in a fixed state.• Physical "life force" bars: Game dynamics such as health, interventions, technology etc. might be represented by physical objects which varying in size, shape and colour.

CONCEPTUAL:
• Ethics and safety: Temporality and physicality give rise to the potential of physical harm, either via hardware malfunction or safety settings, or via a third party acting upon the player.
• Managing MMPORG on a physical level: Levels of attachment to lifeforms or planets may be higher than those on planar screens due to the time taken to create them and the emotional attachment present in physicality, therefore resulting damage caused by third party players may cause additional distress or consequences that might be explored.
• Proficiency at real-world sculpting/painting translating to gameplay: Relating to the previous, proficiency at sculpting creatures and planets can directly translate into artistic practice outside of the game-world.
These features address various aspects inherent in the design of shape-changing interfaces, from an application perspective.Their potential contribution to the field is considered within the discussion.

Summary
The creation of design fiction on the subject of shape-changing interfaces is an novel approach that has great potential value to application and further prototypes development.This can be further capitalised upon: via application of the findings to existing research practice; examining the practice as artistic endeavour; considering the limitations and potential improvements of the approach; via discourse around how the work presented here could be further expanded to allow wider audience access; and finally, used with existing user-based studies to create a blended methodology.

Implications for Shape-Change
Via thematic analysis of the diagetic prototype, we identified 19 novel features or ideas across 3 categories -some being recently under consideration in the research context, which supports the value of the design fiction technique, although not at the level of detail present in the fictional account.Interactive qualities in shape-change are currently the most comprehensively explored: examples from Interaction include Vallgarda et al's notion of tool use in shape change ( 2016), which has parallels to the Riffler; moving solid objects within predefined space e.g data-plotting using ultrasound (Omirou et al. 2016); adding buttons and menus to interfaces (Taher et al. 2015) or gestural interaction (Hardy et al. 2015).
Some features from Hardware are largely absent from current research, not because they cannot be explored, but because they have not yet been approached, e.g.smoothing of surfaces at the limits of devices (distance from baseline), or painting and drawing on shape-changing interfaces.Others, like transitions between modes, require an advance in technology that is beyond current practice -most interfaces employ a single materiality, or the transition is dictated by the hardware (i.e.actuator speed/elasticity in TableHop - Sahoo et al. 2016).Other themes here are between the aforementioned states of enquiry, such as how to switch between solid and holographic layers (jamming only works for physically present surfaces (Follmer et al. 2012), although there is potential to use ultrasound/sonic manipulation in tandem with projections (Long et al. 2014).Also, the advent of 3D scanning means that there is the potential in the near future to scan time-points of an interface and export this data to print.This combination of tangible making and computational intervention/output has also been seen in prototypes such as ReForm (Weichel et al. 2015).The Conceptual themes give rise to the potential for philosophical enquiry, encompassing emotionality (Kwak & Frens 2015, Rasmussen et al. 2013), personal safety and boundaries, and even the long proposed idea that playing computer games has a benefit in developing real-world life skills.The idea that utilising shape-changing interfaces can bridge the gap between the digital and physical world in a meaningful context is perhaps one the most exciting propositions arising from the analysis.The magazine article (Figure 3.) also approaches some of the more conceptual themes from a societal point of view.
Given the promising identification of features and ideas from the use of the diagetic prototype, it is hoped that the overlap in themes and challenges seen between the design fiction approach and that of traditional technologists can bridge the gap between disciplines, and offer up a new design space for enquiry.This might be achieved by disseminating research and presenting work at large scale events, as well as via collaboration with other researchers at the design stage of shape-changing interfaces (e.g. using the identified features as components in the development of new prototypes).

Diagetic Prototypes as Artistic Endeavour
A design fiction can exist as a standalone artifact within a research area, however, we might explore how it works as a blended methodology to encourage interaction and discussion.In many cases, the making of diagesis is not simply a method of inquiry, it is a creative act that has meaning for the researcher beyond the act of investigation.For some, it affords the opportunity to bring creative practice to their work, for others it adds an interdisciplinary angle.Lindley et al. (2014) suggest the importance of the act of creation, and it is this process that adds dimensionality to the research.HCI research has scientific rigour, but the human aspects are sometimes lost (Bannon, 2012).By employing design fiction as artistic practice in addition to research practice, we can appreciate research on a deeper level.

Limitations
Design fictions and diagetic prototypes are often co-created (e.g.Tsekleves & Darby, 2016), but are also often the product of one practitioner.This leaves the value of the artifact open to bias, be it during the process of creation, or whilst presenting analysis.The easiest way to address this bias would be to require the artifact to be exposed at some level to a non-expert audience: examined, reworked and re-presented for consideration.
There is also the possibility of creating a "bubble" around certain scenarios or artefacts -something that can be addressed via a process of world-building (Sturdee et al. 2016) -rather than relying on single objects or individual ideas.These worlds can contain multiple ideas, artefacts, stories and imagery, by many different researchers or participants, thus expanding the view around the initial diagetic prototype in a detailed and meaningful manner.It might be suggested that the initial prototype sets the stage for the associated explorations, thus limiting further creativity, but by building these worlds together we can inspire -rather than limit -each other.

Future Work
We investigated the potential of using design fiction as a tool to inform the design of shape-changing interfaces via a diagetic prototype, however the next steps require adoption of such methodologies within the field, to supplement and enhance current thinking and prototype design at an early stage.For example, the researcher might better imagine the context and direction of their work, or the user may explore how prospective devices might fulfill their needs.The challenges in this approach lie within adoption in the research setting, and in taking on board the innovations it produces.
First Hand is a comprehensive enquiry into the future potential for shape-changing games, however, this work could be extended further from the initial artefact and article to include further items to support the "world building" proposed by Sturdee et al. (2016).Such items might include physical artefacts such as game cartridges, 3D printed items depicting the game environment (e.g.creatures in various stages of evolution, or world types).
An additional extension would be to create a WOZ (Wizard of Oz) study, where the ideas and implications borne from the workshop and creative process could be tested.WOZ testing involves creating realistic prototypes that participants do not realise are being controlled by a hidden researcher, e.g.Rasmussen et al.'s shape-changing bench (2013).WOZ methodology would create a bridge between existing user-testing on shape-changing interface prototypes and design fiction, and we might also further explore the process of creation, moving a step closer toward realising the kind of interaction design needed for shape-change.Working around the design fiction can also be approached using lo-fidelity methods, such as sketching and creating illustrated user-scenarios.Whereas these lack the realism of WOZ, they can be quickly co-created with researcher and user to analyse and discuss the research.First Hand provides a jumping-off point for research in either direction, and a focus of inquiry.
Finally, design fictions of the type explored here may also have benefit in other fields where novel technology is still in development, such as brain-computer interfaces (e.g.Vallabhaneni et al. 2005) or advanced gestural interactions (e.g.Grossman, 2004).Design fiction can also evolve alongside technology, creating ever more complex investigative possibilities, and could enable us to "catch up" with and reflect upon our past fictions to compare speculation with actuality.

Final Thoughts
First Hand is the first piece of research utilising design fiction to explore shape-changing games, with consequences for the wider field.We show that the creation of design fiction on the subject of shape-changing interfaces is an exciting, inclusive approach that has potential value to the field (e.g. to direct technology development or user studies), as well as for the design and application of other emergent technologies.We envision this approach being used for the creation of shape-changing prototypes and their applications, to create a blended practice of artistic and scientific inquiry.Additionally, we hope to inspire researchers to embrace this alternative practice as a methodology when creating these exciting devices. 3

Figure
Figure 1.Emerge(Taher et al. 2015) is an actuator-based interface allowing the user to explore multi-dimensional data.

Figure 2 .
Figure 2. Page examples from the First Hand Quick Start Guide

Figure 3 .
Figure 3. Article created to add depth to the game-world imagining scenarios in which business plays a role in furthering the research into shape-changing interfaces.