Semiotics and design: Towards an aesthetics of the artificial

Abstract Semiotics is the theory par excellence of the artificial and therefore should have a substantial role in understanding designed phenomena. By tracing the relation between design and semiotics at the level of the distinction between the analytic and the synthetic (or artificial), this paper argues that semiotics struggles to explain the environmental element of design so central to post-artefactual accounts of design. The analytic method of semiology is suitable for understanding existent semiotic structures but less so at modeling alternate signifying systems—or systems that alter, transform and self-interpret, that is, environments. The paper argues that to understand such milieus a turn to the aesthetic is necessary. By aesthetics it is meant the simultaneous mapping of the environment, the articulation of the environment and the counterfactual element of any design process. More particularly the paper will focus on recent developments within social semiotics to argue that such a framework must move beyond the constraints of analytical spatial and visual grammar to take into account not only multimodal texts but planning, systems and services. It will conclude by arguing that ultimately design and aesthetics are the same phenomenon, not in the sense that design is the study and application of aesthetic principles to useful objects or experiences, but in the sense that it is the organization of the counterfactual elements of artificial—designed—environments.


Introduction
Semiotics has always been the premier "science" of the artificial-after all, the sign is only ever meant to stand in for something else (aliquid stat pro aliquo), even if it is another sign. The sign is not the designated itself (the real, the referent or the idea); in fact, Roland Barthes famously argued that myth is constituted when the artificial nature of the sign, its ever-tenuous link to the object, is forgotten and taken as "natural" (1972). Any theory of the sign is therefore a theory of constructed meaning-of designed meaning. Furthermore, the fundamental principle of most linguistics, particularly structuralist linguistics, is that signs constitute an artificial system that can be studied autonomously. That system or structure can be said to contain its own laws and its own logic independent of any particular context of embodiment or use (langue over parole). Whatever the case, the semiotic approach to the study of communication, that is, to organised meaning-making, has had a profound influence on design methodology and pedagogy. This influence includes the semiotic, information theory and cybernetic-based theory at Ulm of Max Bill, Max Bense, Tomás Maldonado, Horst Rittel and Abraham Moles (Lindinger 1991), the pedagogical (and political) alternatives to the Beaux-Art approach in, particularly, France and Italy in the late 1960s and early 70s, product semantics and indeed "post-semiotic" design semantics (Klaus Krippendorff, 2006). In turn, within the discipline of semiology designed phenomena have held a special place of analysis, corresponding with the broadening of the structuralist project beyond purely linguistic and textual phenomena. For example, there is Umberto Eco's longstanding relation with architecture and urbanism (1972), Roland Barthes' studies of fashion (1985) and Jean Baudrillard's analysis of the "system" of objects (1996). There is Jean-Marie Floch's "industrial semiotics" (2000), Anne Beyaert-Geslin's work on semiotics, design and aesthetics (2012), Kress and van Leeuwen's studies of the "grammar" of visual culture (1996) and, more recently, the increasing acknowledgement of spatial design and the built environment within social semiotics and multimodal discourse analysis (van Leeuwen 2005, Ravelli andMcMurtrie, 2016). If design is a signifying practice, common to all semiotic-based design theory is the search for a language, or at least a grammar, of design.
The scientific status, however, of any such language is contentious. Science investigates and describes "what is" (nature in its broadest sense), and the form of its logic is propositional and its methodology analytical. But at least since the argument offered in Herbert Simon's Sciences of the Artificial (1996Artificial ( [1969) the epistemological domain in which design is located, the artificial, is that of the emergent, of "what will be" and to a certain extent of "what ought to be". "Everyone", he famously stated, who "designs… devises courses of action aimed at changing existing situations into preferred ones" (p. 111). Differently put: understanding the grammar of a designed object or experience must always be done in tandem with how that object or experience may be altered; how it can be re-designed, reformulated, or begun anew. This is quite different from other forms of knowledge and inquiry. To understand design-the creation and communication of the artificial-is not just to understand representation(s) and their associated meanings but planning, invention, mapping, prototyping, etc., that is, all forms of disegno. Design creates models of the to-be as much as it attempts to model the existent. Unlike scientific discovery, design research intentionally aims to alter the status quo and therefore all design has an element of the counterfactual to its language.
Furthermore and crucial to our argument the field of design research today is environmental, that is, at its most basic level, circumstantial and interactive not artefactual. This contextual dimension, as we shall see, has been difficult for traditional semiotic theory to grasp.

Semiotics and design
2.1 Objectivist and constructivist, analytic and synthetic approaches to meaning in design To organise the brief history of the relation between design and semiotics, I would like to consider a distinction that Klaus Krippendorff employs to, in fact, criticise the semiotic approach to design-"objectivism" and "constructivism" (1992, andreformulated in Krippendorff 2006). The distinction more or less maps on to what we have begun to distinguish between the analytic and synthetic; even between science and design according to Simon. Conventional semiotics for Krippendorff is objectivist, that is, it attempts to describe a reality that ultimately stands outside the process of its description. In objectivism, categories of thought are devised to accurately as possible describe the phenomena under observation and the central category of semiotics, the sign, Krippendorff believes, demonstrates this conviction that reality "possesses observer-or culture-independent structures, objects, codes and laws waiting to be discovered, enciphered and described" (1992, p. 25). This discovery and description is necessarily analytic; it divides, labels and catalogues. He also claims the "laws" of semiotics are thought to be ultimately independent of context and of any particular application; for example, in the case of syntax, the rules governing relations between signs are often sought in formal languages-mathematics and geometry in particular-which do not provide a place for "creators, users or observers" (p. 27). In this vision of semiotics, there is a tendency to standardisation, an inherent dualism (material sign and immaterial concept) and, at worst, an intellectual imperialism (p. 29). Constructivism, on the other hand, finds reality neither outside the observer, nor solely and solipsistically "inside". Rather, there is a co-authorship, a "circular process of perception and action or of conceiving and making things" (p. 25). Though Krippendorff does not explicitly use the term, the constructivist notion of meaning making and therefore of signification is environmental. Meaning arises for constructivism not through the connection between a sign and a referent (or non-connection as the case may be), but through participation, that is, social practice. This practice, Krippendorff argues, is a form of understanding, by which he means not the application of categories or principles to phenomena to determine whether they match or not, but the ongoing, recursive process of constructing a relationship with the world (p. 33). The "semantic turn", as he calls it, is the "awareness that the human world is created by human involvement" and not by a foreign "world of signs" (2006, p. 275). While we will later question the humanism of this claim, the charge that classical semiotics is overly analytical is worth pursuing especially as it has a bearing on whether the discipline can support a thinking of environments (rather than structures). Nevertheless, to briefly preface our investigation: analytics in design has always occupied a privileged place. The basic units of design have been able to be isolated and then arranged, whether they be the units of colour systems or the grid(s) of graphic design, or the basic forms of building in the modular dream of modernism, or the pattern language of, say, Christopher Alexander (1977). But what of the resultant synthesis? What of composites whose parts interact, adapt and alter in ways not explained by reverse engineering (e.g., systems, services, planning)? These are environments in which the flow of communication and information are essential and by nature multi-modal. Moreover, such environments are responsive.

Analytic structuralism: Barthes and Eco on urban design
In his paper of 1967, "Semiology and Urbanism", Roland Barthes reminds us that all (human) space is and always has been "signifying space" (p. 191). The ancient notion of habitat, the oekoumène, for example, was thoroughly semantic and based on structuring oppositions (hot and cold, known and unknown, men [sic] and monsters, etc.). This is quite different, he argues, from contemporary cartographic thinking that represents urban space by exactitude of position (what something is, is where it is located), which, according to Barthes, is akin to a form of "censorship". For Barthes, the relation between centre and periphery, for example, is an active structural principle of differentiation that enables signification (meaning) and therefore precedes "urban distribution based on functions and usage" (p. 192). In 1967, he sees little evidence that urbanists are addressing this key constituent of signification, with the exception perhaps of urban planner Kevin Lynch (1960); but, Barthes argues, identifying the elements of the city-"paths, enclosures, districts, intersections, points of reference", etc.-does not necessarily make them semantic. While semiology too is first and foremost analytical-it breaks signifying phenomena into units and then groups those units into classes-it goes further, determining rules of combination and transformation of those units in order to identify meaningful and meaning-generating patterns and events (p. 196). Lynch's elements are ultimately lexical rather than grammatical, more "gestaltist than structural" (193). Otherwise phrased, there is significant discord between "objective geography" and signification: two parts of a city may be geographically bound (they may "intersect") but they may also signify radically different urban experiences so that, on the plane of signification, they are not linked at all. The lexical gives a glossary for the city but not a syntax. "The city", Barthes instead claims, "is a discourse, and this discourse is actually a language: the city speaks to its inhabitants, we speak our city, the city where we are, simply by inhabiting it, by traversing it, by looking at it" (p. 195).
Though Barthes recognises the need for semiology to account for the synthetic nature of the city he struggles to find a way to articulate it. He admits that the units of componential analysis used to describe phenomena may always prove elusive-for semiotics, the signified slips from under the signifier, and so the relationship between material expression and the concept is ever fluctuating, never fixed. He attempts to locate the synthetic element-the "sentence" that is the city-in what he calls the erotic dimension of the urban. By erotic he means the encounter with the stranger that the city constantly and by definition offers up, and which he describes as a form of reading (p. 201). Alongside its distribution of functions, the city is also a libidinal and textual system; or, rather, libidinal because textual. Here Barthes is attempting to overcome the more analytical aspect of a structural approach with an idea of interaction, in this case, the encounter.
In his essay published in 1972 edition of Semiotica, "A Componential Analysis of the Architectural sign /column/", Umberto Eco also initially investigates an analytical approach to the semiological units that compose architectural artefacts. Any architectural element is a process of signification rather than simple stimulation (a step responded to by raising a foot); for example, /staircase/ consists of the articulation of a few morphemes recognised as a "machine for ascending". But the same set of morphemes may not actually be used as stairs, therefore Eco argues that the "communicative aspect predominates over the functional aspect, and precedes it" (p. 213). The architectural unit or object is a sign before it is an object. Nonetheless, he maintains the distinction between primary functions (denotation) and secondary functions (connotation) is still relevant to architectural study, as long as the term function is not taken too literally. Indeed Eco points to an "aesthetic fallacy" which separates building-functionality-from architecture-"aesthetic autoreflectiveness" (p. 215); by the latter he means the tendency to privilege the "poetic function" (Jakobson) as the primary language of architecture (p. 219). Eco makes the basic point that any semic feature of an architectural, spatial language is realised in a morphological feature. His analysis of the semic feature /column/ is thus: Figure 1. Umberto Eco, A componential analysis of the sign /column/, Semiotica 1972 Eco indicates that this componential process is potentially endless, or at least every semic analysis "must continually return to the problem of semantically defining its own instruments" (p. 232). Here at least is recognition of the recursive nature of the artificial: that is, the incessant feedback loop between the morphological and semantic levels (between the plane of expression and the plane of content). Componential analysis operates well enough when reading existent architectural texts or units like a column but it is much less comfortable putting those units back together or understanding those units as a dynamic whole. Eco admits as much; the last words of the article declare, "… if and how the system is to be constructed has yet to be demonstrated", where by system it is meant the organisation of the sememes, this unlimited process of semiosis. But it is not only the artefact or the unit (the -eme) that is to be semiotically decoded. If design is also counterfactual, then the operations of planning and the creation of schemata for future creation and future use must also be taken into account. A core trait of design is the mapping of the existent to alter what is, and therefore the mapping of the yet-to-exist (this is the role of the imagination in design, see Folkmann, 2013). This "echo chamber"-this movement, alternation and vibration between verb and noun-form, active and passive, genesis and structure, etc.,-is difficult to grasp conceptually and, it appears, semiotically. This is because the field design describes and intervenes in-the artificial-is simultaneously analytic and synthetic.

Social semiotics and design
Social semiotics is based in systemic functional linguistics (Halliday, 1978) and includes multimodal discourse analysis, that is, the theory and practice of scrutinising the meanings that evolve out of the use of multiple semiotic resources in texts (or textual aggregates: discourses), whether print, visual, electronic, sonic or spatial, etc. It also investigates "cross-functional systems" such as colour. The primary purpose of social semiotics, as the name suggests, is to understand the use of signifying systems. Social semiotics professes to focus less on the infrastructural properties of such systems (their "laws") as on the deployment and modification of those systems by the participant. Social semiotics is therefore constructivist in Krippendorff's terms rather than objectivist. Indeed the field prefers the idea of "resource" to that of sign as its central category. Resource emphasises the act of selection from a given range of meaning making "tools" to communicate, while "sign" tends to be overly formal and asocial (van Leeuwen 2005, p. xi). Dominant in media and communication studies, and though it addresses both spatial and visual communication, the approach, however, has not focused explicitly on design.
Social semiotics identifies three levels, or metafunctions, of meaning in any complex text: representational, interactional, and organisational. Representational meaning is what something is about, interactional meaning is the range of social, affective connotations, and the organisational is the manner in which meaning is derived by the configuration of phenomena (different arrangements of the same pieces of furniture, for example, may indicate a class is to be held or an interview will be taking place) (Ravelli and McMurtrie 2016). Design maps onto, or can be mapped by, each level: iconographic or indexical meanings that the symbolic language of graphics employ (representational); interior design that creates moods, ambiences, or hierarchies (interactional), and systemic meanings generated by configurations of information, for example, way-finding, diagraming, and assemblages (organisational). This last metafunction also, crucially, is intended to take into account that something can be rearranged, and therefore meaning can be altered; that is, it is a function to understand configuration and synthesis.
In particular, visual communication studies and media studies have been well served by the social semiotic, systemic functional linguistic approach. Not only does the image "naturally" lend itself to semiotic analysis-every image seems to stand in for something else regardless of its actual level of iconic verisimilitude-images are relatively static texts, or at least the layout of text and image is relatively stable. For example in any newspaper, magazine or internet page, the left in terms of reading practice will appear as the "given", the pre-existent, and anything materialising to the right will appear as "new" (difference to the given therefore always emerges as sequential, implying a narrative). The upper area of a layout will be the realm of the "ideal", the lower section, the "real", a zoning corresponding to and activated by deeply embedded cultural practices of reading vertical or perpendicular space (this would accord with Krippendorff's quasi Chomsky-esque notion of "understanding"). Interestingly, if not problematically, social semiotics effectively reverses the semiological order-the idea or concept, e.g. the "ideal", far from being constantly displaced in relation to the signifier is always realised in a material choice or selection. Placing text or an image in the top zone of a layout, for example, will have radically different "meaning" to placing it in the lower half. This choice or selection is crucial to the participatory dimension of meaning generation in social semiotics, even if the range from which the selection is made is finite and indeed systematised.
In many respects social semiotics seems to respond to both the broader criticism leveled at semiology by Krippendorff-that its categories ultimately exist outside the participation required to make meaning and the understanding required to use and design artefacts-and the limitations we have seen in structural semiology: the struggle to embrace the synthetic nature of discursive structures and, moreover, the notion of design as generating alternatives to what is given. Nonetheless, the componential approach still governs social semiotics. Any mode or text to be investigated is initially handled analytically: after research, an inventory of resources-the individual items that can be selected to make meaning-is drawn up (van Leeuwen 2005, p. 6f). These inventories ultimately form a "system network" characterised by either-or choices or both-and choices at each "level". For example in formalising the notion of framing-the relative "disconnection of the elements of a visual composition" (p. 7)-in the context of magazine advertising, van Leeuwen arrives at this map: Figure 2. Theo van Leeuwen, system network diagram: framing, from Introducing Social Semiotics 2005 Beyond the visual, the same cataloguing process can be made of, say, spatial phenomena: Such inventories point to the fact that the units of semiological analysis are not atomistic; they are "bivalent". Further, they are not static but directional. Nonetheless, and as is clear in the diagramming, there is little indication of recursion, feedback, or even misreading (misdirection). Without this notion of recursion or environmental interaction it is very difficult for social semiotics to move beyond the artefact, no matter how much attention is paid to intersemiosis and to multimodality (the latter is conceived as an aggregate composition of modes, rather than a new state), and therefore to account for the full breadth of the designed, that is, of the artificial. For example, what of services and service design that rely on the full participation of its users to be "realised", and often in ways completely without prediction, that is, counterfactually, or at least in ways which divert from a scheme of selection? More importantly still, what of milieus that modify themselves, that are composed of non-human agencies? Much contemporary design, especially architectural, spatial and urban design, is now essentially this form of environmental design (for a history see Anker 2010): those environments are once spatial, informational, temporal and recursive. Furthermore, as the concept of "framing", for instance, is realised by a chosen semiotic resource, we have an odd Platonism reappearing-Krippendorff's "two worlds" that haunt the semiotic projectinsofar as the material element becomes an instantiation of the conceptual element, a kind of inverted indexicality, or at least an inverted referential structure. What is more, a materialist or material studies view might easily reverse the polarity, as it were: instead of separation being realised by "empty space, furniture arrangements, etc." (p. 16), the experience or understanding of empty space may have engendered the more abstract, cultural notion of "separation".

From ecology to environment: the artificial
The analytical methodology inherent in semiology has struggled with conceptualising environments. As we have seen Krippendorff identifies two approaches to the study of designed phenomena, the "objectivist" and the "constructivist". The former is constrained by a "two-worlds" ontology while the latter is practice-orientated, that is, meaning is created through interaction defined by evolutionary understanding, rather than by logically equating ideas with representations. While taking into account the theoretical accuracy of this criticism of the ontology of semiotics (its inherent Cartesianism), the notion of design semantics transpiring predominately through understanding is also problematic, first and foremost because an environment is not constituted solely by or through human activity. Environments self-organise. They regulate and transform according to the processing of (new) information, and the social practices within them-or constituting them-are part of this "process". Environments are at once technical, biological and communicational. In the Semantic Turn, Krippendorff, who studied under the cybernetician W. Ross Ashby, does in fact offer an account of an ecological approach to artefacts. But that ecology is only activated by human use: the "crucial difference between ecologies of biological species and of artifacts: Biological species interact on their own terms; artifacts interact on human terms" (2006, p. 195). So opposed to the dualist metaphysics inherent in semiology, Krippendorff reconstitutes it here as the relative autonomy of nature and human activity. Yet far from an issue of deconstructive subtlety, it is the very fact that the two spheres, for good or ill, are now so merged they must be understood by the broader concept of the "artificial" (Fry, Dilnot, Stewart, 2015)-such too is the presumption of the concept of the "Anthropocene" (see Turpin 2013). No two-world theory works anymore because there are no longer two distinct worlds to theorise.

The aesthetic turn
To date, Beyaert-Geslin's studies notwithstanding, aesthetics has had a limited role in semiotics and indeed in social semiotics. Rather than being associated with concrete meaning, it describes the domain of taste, sensibility, or personal predilection. Thus the aesthetic appears as either utterly parole-like-part of the moment and indeed arbitrariness of the individual choice from a broader language (say individual items of clothing selected from the language or grammar of costume, Barthes 1967, pp. 26-27)-or utterly langue-like, as the autonomous system of colour relations, for example, or the formal relations between spatial typology (Ching, 2007), the range of textures, or sets of sounds, etc., that is, a non-user system in Krippendorff's terms.
Nonetheless, it seems to me that aesthetics is the discourse which may be the most adequate for the conceptualisation of the artificial precisely because it has never been a discourse strictly devoted to analysis (nor then to generating concepts which simply categorise phenomena); but, rather, one devoted to articulation. Aesthetics explores relations between things, how they are organised, how they "fit", how they materially converse. The aesthetic is also the domain in which concepts are experimented with, worked out, turned inside out and exposed to the counterfactual. Aesthetic activity is the process of counterfactual "communication". In terms of design studies, or designing, this is the point where art and design intersect-not in the produced artifact about which statements can be made about relative beauty or value (though this may occur), but in the propositional space, so to speak, of the model, that is, the space in which the future of the artifact, service or experience has not been fully actualised. The model is first and foremost a "resource" for experimentation, dispute, and reorganization, not a prototype for scaling up and building. To employ another social semiotic term, the aesthetic-or aesthetic "thinking"-is aligned with the "organisational" metafunction. This means not just how an actual interior is arranged, for example, but its possible arrangement. Aesthetic thinking, aesthetic research, is the "redesign of design" (un redessin du dessin), and art takes the form of a "metadiscourse of design" (un métadiscours du design) (Beyaert-Geslin 2012, p. 15).

Conclusion
If classical semiotics excludes human agency (or understanding) and is overly componential; if social semiotics is abridged by its reliance on system networks without response (e.g. feedback); and if human-centred design semantics understates the essential non-human elements of an environment, is there a discourse capable of understanding the artificial (and, indeed, is it a question of "understanding")? As Simon surmised, a science of the artificial is required which takes into account the alteration of the status quo as essential to its epistemology, but as we have seen perhaps it is not a new science of the artificial that is vital but rather an aesthetics of the artificial. What is clear at least is that the two fields may now no longer be separated in any meaningful way because: 1. The distinction between artificial and natural is no longer effective, 2. The observer and observed are environmentally imbricated, 3. The counterfactual occurs at least simultaneously to the factual (the factum: the made); the "to-be" precedes the real, as it were; and so, 4. There is only an open series of models, not a closed referential system between model and real.
When it comes radically artificial environments-those effectively "un-analysable"-the science of the artificial merges with aesthetics of the artificial. Not explicitly named in Simon's characterisation of the artificial-synthetic, imitative, adaptive, and imperative (p. 5)-aesthetic discourse encapsulates those definitions, while also modeling and figuring the hidden dimensions of our constantly evolving and responsive environments.