Harmonics: Towards Enlightened Evaluation

This presentation aims to explore ways to enable designers, researchers and practitioners to share, discover and critique alternative and creative evaluation techniques. It draws on case examples of piloted enlightened evaluation approaches derived from a partnership between the Glasgow School of Art and Highlands and Islands Enterprise, a regional development agency with a remit of the Highlands and Islands of Scotland. Due to the complex initiatives, collaborations and partnerships involved in such a process, measurement frameworks that more appropriately support understanding and learning than traditional approaches have been being piloted. Despite a demand for more flexible techniques, there is a significant gap in understanding and knowledge regarding the most effective approaches, techniques and tools for enlightened evaluation.


Introduction
The Creative Futures Partnership (CFP) is a partnership between The Glasgow School of Art (GSA) and regional development agency Highlands and Islands Enterprise (HIE) and aims to make a wide range of teaching and research opportunities available to students, businesses and communities from across the region. Due to the complex initiatives, collaborations and partnerships involved in such a process measurement frameworks have been piloted that support understanding and learning. The intent has been to help develop measurement frameworks that are flexible, adaptive and iterative. This paper begins by noting the rise in emphasis on the importance of formal evaluation within publically funded activities and some of the shortcomings of traditional, mainly quantitative techniques, for capturing impact, particularly within complex situations. Following this, the paper then sets out the specific context for this project, a partnership between a Higher Education Institution (HEI) and economic and community development agency before introducing the guiding principles of negative capability and method assemblage. The next section introduces the case study of the Harmonics project and some of the approaches piloted in the project so far including 'narrative tracking' and 'evaluation through making'. Following this case examples (Yee, 2010) are introduced to show how such approaches have been trialled including: events aimed at sharing, critiquing, learning from work in progress 'Gatherings' and 'Dailies' and the collective production of a newsletter/ zine entitled 'Fieldnotes' detailing the work of the CFP in progress. Finally, before concluding the paper, reflections are shared.

Context
Within the context of the rise of evidence-based policy making (Black, 2001;Davies, Nutley and Smith, 2000) there has been a pressure for academia and policymakers to demonstrate the impact of their activities (Meagher, Lyall, and Nutley, 2008;Schlesinger, 2013). There has been an increasing trend towards evidence based policy making in the UK gaining particular currency from the time of New Labour onwards (Black, 2001;Davies, Nutley and Smith, 2000;Sanderson, 2000;2002). This trend is associated with efforts to modernise government encapsulated within the Blairite phrase 'what matters is what work' (Sanderson, 2002). Current evaluative techniques within public and private sector are based within positivistic or scientific methods of review and progression (Denzin, 2009). Increasingly, however, specialists are identifying the shortcomings of utilising pre-determined outcomes and indicators in a tangible way as a framework for evaluation. If positivistic scientific data is solely relied on, it is impossible within disciplines such as design to articulate the full range of impacts for example those relating to processes as well as outputs and outcomes. There is a need for responsive and effective evaluation of complex, non-linear, uncertain, adapting and changing environments. This, we argue, as a corollary suggests a role for more flexible and qualitative measurement frameworks. Despite this demand, there is a significant gap in understanding and knowledge regarding the most effective approaches, techniques and tools for enlightened evaluation.
Whilst such approaches may work, to a limited extent, for bounded and defined projects in stable contexts, they fall short when evaluating complex situations (FSG, 2014) and neglect to identify benefits and impacts that emerge over longer time periods. As Donaldson (et al, 2011: 1) explains "The cost of investing in communities of influence is typically more immediate and visible to them than the benefits. Whereas the costs usually appear up front, many of the benefits take months or even years to materialize". Further to this, traditional indicators are limited in within the realm of understanding the value of creative approaches, interventions and practices Learning is also a critical aspect of evaluation and thus the aim of our exploration of evaluation was "not to simply look back and learn but also to look forward and spot opportunities for improvement" (Donaldson et al, 2005: 192). As it is aspiration of the CFP to develop a framework for continuous learning and improvement and similarly, those involved need to make the invisible visible. This requires understanding the following: the emergent and adaptive nature of the work; having a core evaluation team that includes a senior decision maker and meets regularly; tracking developments over time; breaking material down into useful pieces; noticing new language emerging (Donaldson, 2013;Donaldson et al 2005Donaldson et al , 2011.
In this sense we proposed a framework that is agile, adaptable, and able to work with members of the partnership in multiple ways. This new framework allowed the creative artefacts, narratives, and iterative processes to be measured alongside the more quantitative statistic so important in public fund use. This framework is what we called, Harmonics.

Harmonics and Enlightened Evaluation
Harmonics was conceived of as an overarching research and evaluation project, which would promote the learning and sharing of impact that the CFP has had on the Highlands and Islands region. This, it was agreed, would entail the development of meaningful methods that capture and convey the effects and outcomes of a complex partnership. Prior to demonstrating impact or value, it was recognised that it would be necessary to make sense of complex interactions amongst stakeholders and collectively identify, map and embed enlightened indicators for cumulative impact over time. This process, it was acknowledged would also need to reveal any aspects of the CFP, which were not working to inform future decisions. The aim was for transparency in how the work was assessed and, of course, evidence of effects and outcomes from the CFP to be shared with diverse audiences. The ultimate aim would be to develop an evaluation framework, which would evidence impact and highlight experiences to cultivate imaginative conversations with policy makers. In this engagement with the unknown the concept of negative capability struck a chord. As the partnership was conceived of as new and evolving, it was likely that the effects would not be evident within quantitative data for some time and 'softer' quantitative measures were considered important to show the direction of travel or trajectory of change. These measures were envisaged to take place in sync with the interventions to allow for adjustments. As the partnership was developed with an acknowledgment of complexity and iteration, it was felt that new methodological approaches should be explored and developed. It was further suggested that the messier and less tangible data of the CFP could be better understood through a method assemblage.

Negative Capability
In the fields of academia, business, and government, 'success' is often measured by knowing and being sure of that decision. It is important to be sure that taxes are used in a 'proper' way and that a car driven on the motorway is safely and securely made. This approach of striving to order and know is suggested to limit perspectives and options for growth and understanding.
By engaging with the concept of negative capability, it is possible to seek alternatives in an open and iterative way for working with the unknowable. Negative capability is thought of as 'when man is capable of being in uncertainties, Mysteries, doubts without any irritable reaching after fact and reason. ' (Keats 1958(Keats (1899. Keats' description of being in uncertainties without effecting a premature closure of the creative process is one of the defining characteristics of the imagination. It is the ability to deal with complexity, paradox, and ambiguity in processes that have uncertain contexts and outcomes. It is interesting to observe that this capacity is increasingly of value in a world in which the contexts and fields of operation of academic disciplines, governments and businesses are expanding. The CFP works with this uncertainty as a founding element of Harmonics. It embraces this ambiguity and prepares future makers, leaders and educators to deal with complex processes that the CFP must currently deal with and in the future. Living within the uncertain makes for a powerful approach in the ever-increasing realm of interdisciplinary partnerships, like ours. In the end, it is perhaps allowing the positive and negative capabilities to coexist (Edmonstone 2016) where not knowing, as well as, knowing can bring about the most innovative approaches and provide a foundation for our enlightened evaluation.

Method assemblage
Method assemblage is another approach that deals with the unruliness of the CFP partnership. It embraces that unknown, since existing methods are often considered poorly suited for modern social science research. Law (2004) states that method does not simply act as a tool to innocently discover and show reality; rather methods participate in the enactments of reality. In that case, our reflective and evaluative work is as much a part of the research as the research being evaluated. Law calls into question our current need for just one methodological point of view when many of them tend to filter out 'unwanted' noise. It is that 'unwanted' noise that is important to 'gather', and important to refuse the singular narrative. It allows variability in the research and the collection of a variety of methods to be grouped and bundled together.
Furthermore, 'it makes manifest what is otherwise invisible. It extends the fields of visibility, and crafts new realities, and at least sometimes, it also does something that is even more artful. This is because it makes space for ambivalence and ambiguity ' (Law 2004: 90). Thus like 'negative capability' it maintains that space of uncertainty whilst working to determine the outcomes and narratives within. Therefore, method assemblage forms a foundation in dealing with the messiness in the CFP without overly trying to organise it, looking not just at the patterns and wholes, but also at the gaps, the tangents and detritus that make our partnership rich and varied.

Methods
Drawing on the guiding principles and approaches of negative capability and method assemblage, we piloted a number of key methods that have aided our aspiration to make the invisible, visible: evaluation through making and narrative tracking.

Evaluation through making
Given that one of the partners of the CFP is an art school, it was an aspiration that the evaluation visually shows the impact of the partnership through artefacts. In anthropology and art history this material cultural study forms the foundation of understanding human behaviour (Berger 2009). The artefacts are studied for their particular story that they tell, the form they make, or the meaning in relation to the person. In this sense, we can evaluate the progress of things through the artefacts that are created.
For example, a postcard created and received after an encounter is the material representation of not only the encounter, but can be used as an evaluative measure in how the encounter took shape, and what it meant for the people who developed that event. The card becomes embodied with the encounter. Therefore, the making of those things constitutes a means of evaluation and an ecology of things is thus developed through the co-production of artefacts. What this all means for Harmonics is that in our evaluative processes we are thinking iteratively through the things we make, like postcards, 3D printed artefacts, or a booklet and can use them as part of our evaluative process making. Furthermore, this 'vital materialism' (Bennett 2009) covers the material nature of CFP, meaning the material processes that we make constitute the politics of that exchange (Wylie 2007). Therefore, the things that we make not only express the embodiment of the relationship, but the power and the labour required in its making of the partnership that we can use to understand those complex stories within our artefacts.

Narrative tracking
The second approach used to track those ongoing relationships focused not only on what was embodied in the artefact but also what was spoken. It was the intention was that the evaluation processes developed through Harmonics would tell the story of the CFP weaving narratives and vignettes together to explain, exemplify and locate change, effects and impacts. This involved consideration of how to capture the impact of the CFP through the narratives of those involved. Whilst this is a critical part of the evaluation process there can be tensions and challenges in getting buy-in from stakeholders within the context of highly monitored and measured ideas of success. Despite a bewildering choice of methods for collecting stories, it could be asserted that there is no single 'right' way.
A careful consideration of what and who is tracked required further involvement and discussion, but we began to establish a base framework that took into account how storytelling can be useful. Thus between narrative tracking and evaluation through making we are able to uncover a vast example of stories and forms not currently 'measureable' through standard ways. In the following section we explore three case examples that begin to apply these methods.

Case examples
This section will introduce three case examples. The first is that of the Harmonics Gatherings, a series of encounters where open discussion across the two groups is encouraged and second the Dailies, which aimed to evaluate work in progress. Finally, the third case is that of the 'zine 'Fieldnotes

'Gatherings'
The 'Gatherings' were initiated as physical gatherings bringing together HIE and GSA staff working across the CFP projects in facilitated and co-run events. This was anticipated to offer a space for thoughtful and enlightened reflection through critical commentary, critiques of work and informal conversation. The 'Gatherings' were major and minor events and there were three significant Harmonics 'Gatherings' over the course of one year. At the September gathering GSA reviewed possible approaches to enlightened evaluation with the help of a range of experts. Focusing on the CFP projects underway and using the material artefacts gathered on trips to the Western Isles and Shetland as well as other activities, participants reflected on the journey taken so far and shared their experiences. At the second 'major' November meeting HIE and GSA met 'en masse' to consider Harmonics for the first time. The work concentrated on: what had been the experience of HIE and GSA staff of the partnership to date; and, what preferable futures might be springing from the partnership. GSA and HIE people participated with frank and enthusiastic engagement and energy for the partnership was high. A clear question was foregrounded by participants: What is the 'partnership space', as opposed to GSA and HIE spaces? As one participant described "There is a huge amount of creativity in both organisations and it's how we join forces and meld that creativity for the wider socio and economic development of the Highlands and Islands".

'The Dailies'
'The Dailies' were inspired by a practice from the film world, where the director and others involved in the production of a film watch the rushes or takes at the end of a day. They do this, in order to see what is taking shape, what is usable, what can be discarded, what needs another take, what amendments are needed in the way of production design, sound design, lighting design, make up and costume design, location or set design, etc. Rather than the daily rhythm of filmmaking, a seasonal rhythm was proposed for our CFP Dailies.
In terms of the form of the events piloted, people in the middle of a project were asked to volunteer to explore their work with whoever else was invited to the session. Rather than displaying or presenting their work, they were asked to show what was 'in the making'. This meant that they gathered together whatever bits and pieces they were working with, unfinished artefacts, snippets and models, disappointing and promising trials, ideas glimpsed and so on. They would then begin to engage those present with the possible stories the materials offer, reflecting spontaneously on the tracks and trails of their practice and activity. Those present were intended to actively aid this process by listening and looking attentively, noting and offering responses, asking questions, pointing to lacunae in the stories or leaps made or details missing. Finally the session would turn to what practical judgements about next steps, possible directions and so on.
Two Dailies have been hosted through the CFP during 2016. In the March session insights were shared across four key projects: The Teaching Studio, and in particular Winter School; The Design Network; The Experience Labs and Workshop through a series of pinups. These pin-ups involved providing a representation of a work -in progress and 'in the making' -where visual materials (photos, pictures, artefacts, etc.) were combined with text summaries (titles, quotes, slides). Everyone was encouraged to bring questions for discussion, challenges they have experienced or overcome, particular themes/ insights emerging from their work. The second set of Dailies were held in July and included representation from the PGT and PGR student cohort. As well as members of the research cohort of PhD and MRES candidates, MDES students gave presentations about their group projects, which began in January 2015. A representative from the programme discussed her team's project, Bike Hub, sparking wide interested from the audience. Bike Hub proposes the creation of a series of shelters throughout Barra. Young people would be encouraged to take ownership of the Bike Hubs and decide what they are used for. Inspired by interviews with young people and Barra Youth Café, the suggestion from the MDES students is that they could become maker spaces or meeting places. The team of three was given the theme of environmental design and the topic of health and wellbeing during Winter School. During their research the group decided to focus on Barra and became interested in how wellbeing was linked to creativity and having a good relationship with place and landscape. Field research conducted in Barra brought the trio into contact with teenagers on the island and the local youth café. Interviews revealed that cycling was a favourite activity. In the audience during the student presentation was HIE's Head of Strengthening Communities in Stornoway. She encouraged the students to take their idea back to the islands to ask the young people what they think. She said that the young people on the island could potentially take it forward for inclusion in the Barra community plan. She added: "I think the young people could feed this into the plan as a priority for them. We fund a local development officer in the area who could then support the young people to deliver that project." This highlighted the importance of involving partners from both organisations in the process of evaluation and information sharing. The value of making use of local knowledge and connections was strongly illustrated by the Barra Bike Hub that HIE could help turn into reality.

'Fieldnotes'
Though we live in a digital age, the desire for physical artefacts where stories can be shared is still needed and desired. Inspired by the 'zines' of the 60s and 70s, the decision was made to create a Harmonics 'zine'. 'Fieldnotes' was intended to act as a vehicle for sharing the initial experiments in enlightened evaluation and innovation dissemination approaches. Conceived of as 'notes from the field' whilst initiated by the GSA Harmonics team, HIE staff were involved both as contributors and editors.
This short, 16-20 page A5, booklet reported on activities that had been undertaken during the previous quarter, upcoming events, reflections on the work of the CFP and the evaluation process. The first 'Fieldnotes' booklet was produced in December 2015 and three further issues were disseminated on a quarterly basis. They covered the ongoing work from the Gatherings to the Dailies. Their purpose evolved from sharing of projects, to making a space for disseminating particular papers written for journals or to provide a forum for developing a particular idea. This was evident in the fourth issue of Fieldnotes where a PhD student wrote an explorative piece about the Dailies. Here she explained how the Dailies were a useful space in sharing her research to a wider audience. Though difficult to show work in progress, it allowed her a time to simply share. The impact of the Fieldnotes has been to help make the work of the CFP visible across the partnership. It gives a tangible, yet thoughtful way of expressing the ongoing relationships within the CFP.

Reflections
Reflecting upon the three case studies it can be seen that our work at this primary stage showcases a developing narrative between two different institutions. It has progressed from an idea of storytelling to something more tangible and able to be grasped and assessed in the larger picture of the CFP. The beginning stages of this partnership have not been without their difficulties. Particularly in communication, our two institutions speak to each other where assumptions continue to be made. As we have expressed, Harmonics does not intend to replace existing methods of evaluating, but rather it complements and gives rise to new 'holistic' ways of understanding our complex partnership. It shines light in areas that usually do not get attention and gives a voice to those that are not usually asked.