Conceptualising Kinaesthesia – Making Movement Palpable

Abstract Methods for observing, registering and understanding movement have become increasingly sophisticated given the advancements in data capture, simulation and analysis, however there is still much to learn when questioning the kinesthetic properties of movement and how they relate to intersubjective phenomena and social flow. Our project, Dancing in Data, aims to reconnect with the sensuous, experiential nature of movement, to visualize data in a move away from the conceptual and concrete towards a more subjective, expressive medium. Rather than perceiving data as remote and dispassionate, we present an opportunity to address a gap in the research that considers movement as autotelic and flowing in response to a heightened bodily and kinesthetic awareness. Our research suggests that technology can play a valuable role in making movement palpable, giving rise to a richer understanding of the sensations of movement as a foundational aspect of the way in which we learn, create and communicate.


Introduction
The Dancing in Data project draws attention to the temporality and subjectivity of movement as a physical, social phenomenon, and has developed technology that brings into being variables within movement in order to question current normative perspectives on social engagement and reciprocity. Despite requiring a highly concrete, physical body to exist at all, movement remains a kinesthetic experience, potentially shaped by social and cultural codes to determine meaning. In this respect, we hope to form a new critical discourse that allows for the discovery of meaning and value within movement that may not 'fit' within conventional social and cultural codes.
Our motivation for the pursuit of an alternative vision for social interaction, and its potential augmentation within digital culture, has been informed by our applied research over the past five years. Individually, we have been designing technology applications with groups of neurodivergent (primarily autistic people and those with learning disabilities) people (Walker, 2014) as well as professional dancers. Collectively, this work has amassed a body of evidence to suggest that an augmented experience of movement may play a unique role in identifying pathways to social agency and developing efficacy when brought into consciousness through technology. During the design process we have resisted analysing the cultural value, skill or desirability of the newly created movements, as this would risk excluding movements that cannot be easily rationalised, but appear to have value to the individual generating the movement.
As an alternative approach, we have worked with graphical representations of full-body movement as an explorative method of accessing what would otherwise be an internalised body perception, construct or schema (Head & Holmes, 1911). In the way that body schema is understood to be an unconscious process used primarily as a means for organising one's body in relation to objects and the spatial environment, in our examples, within our projects we have aimed to visualise the configuration of limbs, the shape of the body surface and the integration actions and intentions (Holmes & Spence, 2004). Theorising by Milton (2013) and research by Pellicano and Burr (2012) have suggested that autistic people may utilise learnt schema to a lesser extent than non-autistic people. Research on movement qualities has shown that training in perceiving movement is pivotal for how movement is engaged in human-computer interaction (Mentis & Johansson, 2013). Similarly, in neuroscience findings point to that human perception and understanding of movement is conditioned by whether the person themselves have performed the movement (Cross & Ticini, 2012). In our projects, it was thus fundamental not to try and impose meaning upon the movements that were being created. Using simple camera and projection technologies, we have rendered movement visible in the here and now, the goal has been to draw attention to the shapes and traces of movement in order to prompt the user to modify, disturb and direct their actions. In making palpable what would otherwise be unseen nuances and subtleties, the Dancing in Data project will build on our earlier practice-based research in order to question the current application of design data modelling tools, and to contingently develop a discourse that brings movement into a social and relational consciousness (Milton, 2014(Milton, , 2014a(Milton, , 2014b).

Movement as meaning
Gesture has long been of interest as a highly observable mode for examining intentionality and expressive communication. This performative role of movement has been explored by social scientists; Goffman (1959) argued for our acts -our presentations to the world -as performances central in social interplay, whilst Birdwhistle (1971) extensively mapped the non-verbal signs and movements as a way to interpret the role of gesture as communication. From developmental psychology, Stern (2010) emphasises the wealth of non-verbal knowledge that reside in what he calls 'vitality' i.e. in the way we move, in movement itself. Butler (2002) positioned movement as part of the construction of self; closely examining how we through our acts construct our identities and in particular our gender. Noland further extends Butler's decisive position on movement, by arguing for movement also has an agency in and of itself: "Kinesthesia allows us to correct recursively, refine, and experiment with the practices we have learned" (Noland, 2009, p.4). Thus, Noland argues that by performing movement we are continuously exploring and extending our culturally learned practices. And it is this line of argument we take on in our work with digital technology, by allowing movement as a creator, generator and as experience which in turn form novel and individual practises that go beyond what may be imagined prior to the enacting moment.
Movement is traditionally assumed as more or less controlled by our conscious or unconscious mind. However, the notion that the brain decides over the body is challenged by anyone who has ever tried to juggle: the simple instructions that are so easily grasped by our mind, quickly make way to the acknowledgement that a multitude of tiny, yet decisive movements are required to sense and correct in order to successfully keep balls up in the air (Hummels et al., 2007). These nearly imperceptible adjusted movements and swift perceptions are crucial in our performance not only in juggling but in all interpersonal communication (Stern, 2010), yet for movement-based digital interaction the "methods to extract such information from the digital data stream are still in their infancy" (Bevilaqua, 2007, p.27). Additionally, in everyday life, we continuously re-tune our movements to new ways of seeing, making meaning and connecting through appropriating digital technology (e.g. Coyne, 2010). This leads us to suggest that there is a design space or a creative scope for digital movement perception yet to be explored. Next, we will use our design cases: Somantics and Sync to shape a discourse built on the phenomena itself, through iteratively triggering generative movements. Thus, we exemplify performed movement data as a kinaesthetic phenomenon, allowing for an alternative, explorative and experiential view on the role and scope of physical movement.

Our (divergent) digital designs on movement
We propose that through collaborating with, and observing, people with an alternative or nonnormative experience of movement, we have an opportunity to examine movement from an interdisciplinary perspective. Drawing on performance ethnography (Conquergood, 2006) as a broad methodological framework we are uniquely positioned to consider theories from the performing arts, social anthropology, design and human computer interaction. In ethnographic terms, the development of Dancing in Data will rely on the centrality of the relationship with participants whose individual movement patterns will bring the research into focus. We are thus proposing to extend our earlier research with autistic people and expert dancers to permit us to explore the magic in extra-ordinary or more unconsciously expressive movements -so that they may be perceived, touched and experienced. In particular, by approaching technology as not merely mapping, miming and registering, but as a tool for exploring and extending, shifting and shaping the way we move, we disengage the hegemonic interpretations and assumptions that are presumed 'good', 'useful' or seen as meaningful. This is particularly pertinent to consider for designers, who calibrate sensors for how movement is to be seen or not seen, made meaningful or not meaningful -and thereby made acceptable or non-acceptable. This matters as our surroundings are increasingly digitised today.
By using movement data to visualise movement abstractly, yet in real time, a novel experience of self may be created. Our interest lies in whether generating awareness with digital means could alter or optimise somatic modes of attention (Csordas, 1993), thus augmenting the crucial connection between affect and movement.

Materialising movement
As stated, Dancing in Data will use data modelling tools as a means to materialise and reconnect us with the sensuous, experiential nature of movement -opening up possibilities for a subjective and expressive relationship with data. Our ideas have been directly informed by earlier research in which we have materialised movement through a process of cooperative design and performance with expert dancers and autistic people. In both these cases, the potential to see the movement as it morphed and progressed in time and space appeared to be liberating, enabling the mover to experience their own actions, and those of others, in a playful manner.
The Sync and Somantics research projects adopted cooperative inquiry study methods (Druin, 1999), using observational and participatory techniques within an explorative, experimental approach.
Cooperative design has emerged from the field of HCI research to incorporate participatory design and situated action (e.g. Suchman, 2007). The design process evolves through context specific activities, and the iteration of low-tech and high-tech prototyping (Druin, 1999;Guha et al., 2013). Within our studies, we created a series of prototypes that made movement palpable, bringing it into consciousness through a range of linear effects. The simplicity and uncluttered nature of the visual feedback afforded participants the freedom to experience and explore movement without the need for verbal explanations, thus supporting openness and a failure free environment for facilitating active engagement. Real-time observation, video documentation and retrospective video analysis, (Baranek, 1999;Payne & Payne, 2004) were crucial elements for the data generated by movement to be understood. The development of the data visualisation tools went through rapid iterations, informed by time intensive, cyclic, exploratory body-based exercises. As such, our designs and evaluations are based on many hours of work by the participants and many hours for the researchers, and key informants, observing, altering and observing again until one could observe a shift in use, a shift in understanding, a shift in culture (e.g. Sklar, 2008;Coyne, 2010). The nature of the query -as having movement itself lead our understanding of movement -excluded the frequently used questionnaire or written summaries. By observing and filming, the movement development stayed as movement, loosing as little as possible in the translation to research findings.
Taking these iterative prototyping methods forward as an approach to materialising movement we suggest that there are opportunities to develop novel digital visual dynamics to re-examine and reframe movement principles, visualisations, meanings, relations and engagements (Hansen, 2015). Most significant however, is the context in which movement has been explored and observed, and how it has iteratively informed both the design of the technology and the ways it could contribute to new research.

Making movement palpable: Somantics and Sync
This section of the paper briefly describes the design of Somantics and Sync, which we reference in order to draw attention to the role the participants (e.g. dancers and autistic children) played in shaping the development of a variety of digital tools for experiencing movement.

Somantics
The overarching goal of Somantics was to encourage expressive language among hard to engage autistic students, aged between 14 and 16 years. As many of the students were non-verbal, this made the desire to experiment with gesture-based devices even more compelling. Before creating any interactive prototypes, we experimented with a variety of low-tech ideation methods. We began with two design workshops, before introducing simple prototypes that were iterated over a period of six months on the basis of regular observations and teacher feedback.
The design of Somantics took place at two venues, a special school and residential college providing education of students with diagnosis of autism. Four students, three boys and one girl, aged 10-11 years, participated in our school workshop. The corresponding group at the residential college comprised of six students aged between 19 -22 years, four boys and two girls. We paced the sessions to ensure that students did not feel overwhelmed and we were aware that we should not set tasks that needed with too many instructions, which could lead to confusion and feelings of failure.
The first activity invited the students and their teacher to watch four short abstract films, Colour Box by Len Lye (1953), Allegretto by Oskar Fischinger (1936), Free Radicals, by Len Lye (1958, and La Merle by Norman McClaren (1959). The films used visually harmonious shapes and colour. The purpose was to observe the level of concentration and to prompt ideas without intense questioning. After each film had played we asked the students to draw anything that came to mind, we also participated in this activity and used our own drawings to prompt them to talk about their drawings. The main objective of exercise was simply to gauge whether the abstract sequences were interesting enough to capture the student's attention and imagination. Whilst it has been shown that animated characters can assist autistic children in relating to emotions, (Golan et al, 2010, Holmgaard et al 2013, Suskind, 2016 we were curious to see how they responded to movement as a form of expression when there was no anthropomorphised character to convey emotion. We were also interested to discover whether the animation would trigger a diversity of expressive responses. This was a relatively short activity; the films were no more than a few minutes in length. The children concentrated well and showed no signs of distraction, they seemed to enjoy the drawing activity and talked happily about their drawings. We were sensitive to the fact that the children may become upset when questioned about their ideas and so we captured the whole session on video and kept all their drawings for future reference. The next activity was designed to explore full body movement. We used a large studio space and projected Oskar Fischinger's films, Spirals, (1926), andMotion Painting No. 1, (1947). Both films include patterns of spiraling circles, the latter introducing further flowing lines that appear to be hand drawn. Without instruction, the students put their hands on the wall as if directly drawing the lines and colours. They also interacted with each other in a synchronised manner using the both arms and feet, visibly changing the rhythm of their movement in response to the kinetics of the film.
The drawings and the body movements inspired us to create a series of storyboards that depicted the traces and flow of movement in relation to space. As these required imagining a scenario which was not yet in existence we asked art teachers for feedback, which informed the development of prototypes that used blob tracking and optical flow to capture and replay graphical representations of movement as lines, shapes or colour (figure 1).

Figure 1. Somantics -Graphical effects ©Wendy Keay-Bright
We then designed five more applications using video mirroring, introducing ghosting, scanning, kaleidoscopic and painterly effects, that could be animated by the body in motion. The visibility of the mirrored body was deliberately kept at a low, almost transparent resolution; instead the focus was on high contrast effects that could amplify the results of interaction (figure 2). Some students appeared to enjoy seeing their bodies mirrored in motion, for others the video had the opposite effect, and they showed no interest, preferring the directness of the graphical responses.

Figure 2. Somantics -Video mirroring ©Wendy Keay-Bright
Following the workshops we introduced the prototypes to a different group of students at the school, five boys, non-verbal and hard to engage, aged 14-16. Regular access to the residential college was difficult to organise, although in the latter stages some testing occurred. The boys explored the Somantics prototypes for approximately an hour per week over a period of six weeks. They attended each session for approximately 20 minutes at a time, occasionally in pairs but usually individually. Even though the projections were large, the boys were not naturally inclined to explore them. They did not automatically understand the relationship between their movements and the projected effects, and each one had different sensory preoccupations. However, their teachers began to join in these sessions, providing encouragement, and over time each student was able to detect that their bodies were causing the interface to change. They began to show a preference for different visual representations and we were able to build a profile of movement preferences based on the continuity of the responses over time. Whilst the levels of engagement varied greatly, much of this was dependent on what else was going on, whether something had disturbed them or made them happy.
We experimented with a range of observational methods -field notes, general observer feedback and inter-observer analysis of video taken from two or three angles simultaneously -in an attempt to capture each child's engagement. Given the diversity of the autism spectrum we were particularly interested in any occurrences that did not correlate with the predicted developmental trajectory. Whilst it is impossible to accurately interpret the experiences of many autistic students, we believe the nuanced bodily communication facilitated by Somantics served as their voice. From their earliest inception in anthropology, ethnographic techniques have offered insight into non-literate populations, thus offering a large knowledge base for understanding gesture and movement; it was our challenge to capture and interpret this through the design. As a method of cooperative inquiry, video analysis proved most productive. We invited teachers and therapists to review video footage. In order to avoid the reductionism possible of over-reliance on objective coding and statistical analysis of video we annotated 'key moments' as the data was only meaningful when viewed in the context of the teacher's narrative, which focused on the student, not the technology (Walker et al, 2012).
The most important refinement was to ensure that we designed an interface for choosing from the different Somantics applications that could be used by the students without intervention from teachers. A simple icon based interface was tested and the software became part of a regular activity across the school for students of all ages and abilities. The openness of the design allowed teachers to link Somantics into curriculum subjects, such as creative writing or science. Therapists used the software for speech and language and occupational therapy.
We released Somantics as a free download application for Mac, PC and iPad, since this time the software has been used in many different settings -in special schools catering for pupils of all ages and abilities, in therapy centres, adult learning centres, speech and language and drama settings (Keay-Bright, 2013). In a series of studies of published in 2016 (Guldberg et al, 2016) teachers in schools across the UK described how they used video storytelling to capture pupil engagement with Somantics. Rather than rely on statistical or diagnostic data the video provided an authentic voice for pupils, each of whom explored Somantics in different ways, using expressive movement.

Sync
The motivation for developing Sync came out of how technology now affords an increasingly detailed and personal collection of movement data, and it is essential to query the performative role this data could play. By engaging dancers and choreographers in the development of visualizing movement data, the borders of what was possible and the apparent differences to actual movement were quickly sought out in the search for new expressions. Sync highlights movement dynamics over posture or function i.e. emphasising how we do something as opposed to what we do. Sync does this by visualising the velocity or the rate of change in the position of a body part (see Figure 3 and 4). Based on the performers extensive body-based practice, we were able to open up the examination of the role of this digital material, that is movement data, and in particular the creative and decisive decision taken in data handling (Hansen, 2013). We found that working with real-time 1:1 images created an engagement whereby we could question the pre-emptive requirements of computation to decide on values, as the dynamic visualizations allowed variation and individuation. It brought up questions as to why and how certain movement qualities were highlighted over others. These discussions gave access to the ways in which movement data is different to actual movement and the role our perception plays in our understanding and experience of movement: "We always see less than is there […] We also always see more than is there" (Bleeker, 2008, p.18). The dancers' movements were informed by what they saw visualised back to themselves generated by themselves in real time. It created an attachment to the visualisations akin to what young dancers develop to the mirror in their rehearsal halls, which they subsequently have to unlearn early in their career.
The dancers also immediately sought out the boundaries of what could be captured -what could be registered -from the physical perimeter of the sensors to the ability to represent micro movement or fast paced phrases of movement. This probing and prodding of a new expressive repertoire however, always ended with a playful messing around, whereby new completely unmotivated and undirected movement gave new visuals to again examine.

Why this kind of 'making strange'?
Our motivation is to explore and to find ways to visualise and articulate how novel digital expressions are internalized. When dynamic visualisations of movement data are re-introduced to the body from which it was created in its own unique way, it will be perceived according to a person's own personal reasons for moving and with the unique capabilities and motivations of the particular person. In the above examples, the captured data was visualised in real-time for the participants to be able to see their own movements, and for those who may be caring for, educating or seeking to choreograph such movements. The overarching intent was to detect possibilities for social engagement or individual experiences that might otherwise have been unnoticed: "We suggest that the uneven skills profile in autism depends on which interests have been fired into monotropic superdrive and which have been left unstimulated by any felt experience" (Murray et al., 2005, p.143).
This led to positioning playfulness as an explorative method. For autistic people, movements that signal engagement can be seen as successes in and of itself (Milton, 2014). For dancers, there is a continuous development of refining and extending the borders of their expressive movement repertoire. Both these aspects, the engagements and the refinement, are in part achieved by merely trying movements out, through repeating and rehearsing movements. This can be seen as a material exploration akin to sketching with a pencil or modelling with clay (Hansen, 2013). Such material explorations are at the core of creative practices, and we argue, are crucial in building a discourse on technology, interaction and movement. Only when we make this connection can we examine movement as the condition that informs the larger issues of identity, personality and quality of life (Milton, 2014ba).

Reflections on design and digital interaction
We have challenged how movement can be made to matter and made relevant for a range of people, from those considered neurodivergent to those considered movement experts, and we have shown that movement data has a role to play in a discursive development across technology, movement and interaction.
For the design field, our discourse allows for extending playfulness and the creative scope of what movement data can be. Today we increasingly interact with our digitally, connected surroundings through our movement, in turn these movements are registered, calculated and categorized according to invisible lines of codes; algorithms (e.g. Thrift, 2008). These algorithms determine what movement matters; what movement is important -even what is 'normal'. This comes about as machines require us to perform this or that movement in this or that way; in order to give us our ticket or to 'automatically' open a door to let us through. Such everyday innocuous procedures are governing much akin to how social convention through people's feedback has kept our movements appropriate and regulated in a social choreography (e.g. Hewitt, 2005) before we interacted with and through digital technology (Suchman, 2007). By extending the discourse to include the notion that movement itself has agency, we may identify new research trajectories for the next technological advances.
Digital technology allows for new ways to gain an understanding of neurodiversity, in turn enabling an appreciation of what makes us human and what creates culture. As we have shown, by developing digital design tools, we need to re-examine motivations and methods, in order to include rather than unwittingly exclude. As expectations are challenged by people not moving as expected, new expressions and relations may be found. Sheets-Johnstone proposes that we need to first "make the familiar strange" to think anew when we draw on the dynamic body (2009, p.379). Digital movement, as we have argued and illustrated, is an unchartered terrain wherein we do not yet know how to realise nuances of meaning, social relations and developments. Approaching digital interaction in terms of reciprocity and interpretation, we allow for both a strange-making of our represented selves in digital interaction (e.g Loke & Robertson, 2013), but also to individually be able to re-construct ourselves digitally and physically by novel means. In turn, such an approach will challenge social conventions and restrictive norms in individualisation and the experience of self through movement. We have argued that by approaching technology through a movement-centric, playful, neurodivergent engagement, we are able to find ways to build a rapport of a more tacit (Milton, 2014b(Milton, , 2014c and individual nature whereby movement remains under the ownership of the 'mover'.
We suggest that aspects of our aim to digitally model kinaesthesia, will allow a humility and humanity to exist (Milton, 2014b(Milton, , 2014c in and through digital movement, as our surroundings are increasingly connected and interactive. This, we have argued, happens through reconfiguring how people may interact with technology by contributing new theoretical and practical knowledge necessary for advancing movement-based interaction. Critically, such an explorative, visual, enactive and material approach to movement data allows for representations of variation and scope, and discussions on what and how movement matters.