This research considers the possibility that videogames and their technology can be used as the basis for the creation of a sound installation, namely through the modeling of audio architecture in three dimensions, starting from a sound map. The development process is the following: firstly a map is defined in a game engine. Sound assets are placed in such a way as to automatically simulate a more or less realistic perception dependent on the listener’s position in relation to the sound hotspots. Secondly, this virtual map is superimposed proportionally over the empty exhibition space, so that the map and the site are identical in terms of dimensions and sound propagation.
As an artist, author-composer and video game sound designer, creating soundscapes is at the heart of my everyday practice and concerns. In my sound pieces, I tell stories to the listener; they are very much linked to daydreaming and interrogate questions concerning imaginary travel, memory, wistfulness and the productive free play of inner thought. These soundtracks have an air of vulnerability and demand a degree of openness and concentration from the listener. My field of research expands on the questions of how sound can animate a physical space but also the viewer’s interior space and how to stimulate his/her imagination – which also requires shifting from cultural to personal understanding of sound.
More recently, because of my experience working with video games, I have been wondering how sound can favor the user’s immersion, as content but also through spatialization, and how sound can work as a feedback of the player’s actions. This, in turn, means taking into account real time process. When designing sound for a video game, one of the keywords is immersion: audio needs to reinforce the credibility of the virtual world and therefore the player’s experience, but also the gameplay – the concept and the system’s rules.
These questions led me to conceive my soundtrack as a 3D map where the listener could physically wander, rather than creating a linear stereo soundscape. I developed these ideas during a three-month residency with Locus Sonus in 2014 that lead to the creation of the installation Mapping the Iceberg. I consider this work as the expansion of a video game environment (that’s why I will refer to the user as a player): firstly, the audio surroundings become the interactive part between the player and the space; secondly, the listener becomes his own avatar by moving in the 3D model.
My installation superposes a 3D, virtual sound map scaled to an empty space. The player can explore the audio scene by walking freely in the space and has access to the sound through headphones. The headphones solution was chosen for two reasons: firstly, they improve the player’s immersion by masking the surroundings – experience shows that participants feel more “present” in the virtual world when listening to sound through headphones rather than via loudspeakers. Secondly, the audio signal is not altered by the acoustics of the exhibition space and thus preserves the original properties of each recording.
Technically, the map has been designed using Unity 3D by Unity Technologies, a game engine enabling the creation of interactive 2D and 3D content. Each audio source is set in 3D architecture; a few walls are simulated by using a lowpass filter in order to reinforce the feeling of enclosed spaces. We track the player with an infrared LED situated on the headphones that is detected by a camera with a wide-angle lens and an infrared filter; a custom application written in C++ (developed by Simon Chauvin) extrapolates this data to represent the player’s position in the exhibition space, so that Unity 3D can retrieve and use this information to trigger the appropriate sound sources. In order to experience the tridimensional soundscape, we use the gyroscope and the accelerometer features of a mobile phone, also set on top of the headphones; thus, when the player moves his head, this directionality is translated in Unity 3D and the listener can perceive the position of the sound source in the 3D space. He/she will sometimes be able to hear a sound above by tilting their head towards a source. Beyond this, a day/night parameter allows us to present a soundscape evolving through its daily cycle.
The 3D model represents an imaginary universe created from field recordings collected in the city of Aix-en-Provence. The staging of the audio architecture is influenced by the nature of the sound sources themselves, acting like an animated ecosystem: I consider each sound as living matter as it is a portion of time and space that I documented. The way I structured my sound map is the result of a balance between complementarity and opposition of the sounds: each source is connected to the other – and so are their 3D cones – but, at the same time, they are different enough to make the player aware that the soundscape is evolving. The installation simulates a fictional scenario implanted in a neutral setting: each sound works as part of the scene and simulates the presence of people, nature, music, etc. like a movie soundtrack with which we can interact by physically wandering from one point to another.
A particular form of audio mobility is represented in the world by a pause function: if the player moves away from a sound source, it pauses and will start again when the player returns. So, for example, the player will be able to catch a conversation exactly where he/she left it. This choice is intended to favor a notion of soundmarks through identification and memorization of the sound sources.
The installation may at first seem like an audio guide but the experience is closer to audio navigation. The player discovers the existence of an interactive soundscape in which he/she can play – in the manner of video game playing – where sounds are triggered, becoming a score controlled by the player. In this way, he/she is both directing and performing his/her own story. The player’s body is also engaged in the process of discovering the piece: he/she will have to adapt their motion to the scale of the model but also to his/her own experience of listening to the space. It is for this reason that I propose that we can consider the player’s motion as a choreography. Some will be tempted to run, others will walk very slowly; some will hardly move at all – maybe only their heads; yet others will repeatedly take one step and stop. So no player executes the same process and there will always be different stories created from the same initial audio recordings. This aspect of the work relates to the domain of interactive narrative, as the ability of an interactive experience to tailor the narrative for each player.
The work also refers to the player’s mental model of a physical space: how will the player define his/her own imaginary map depending on his/her listening experience? Will he/she memorize each sound source’s position? Will he/she get a global view of the map through an understanding of the way each microcosm communicates with the other? Or will he/she interpret the space by remembering his/her own path, like a choreography? My intention is that the installation and its structure relate to the way we perceive a city: arguably, rather than remembering our itinerary as a map, we create imaginary beacons punctuating our daily paths (at least in terms of audio perception). It is this type of perception that I seek to induce in the exhibition space: a transitory, ephemeral, organization of connected personal landmarks. I also relate it to web browsing and the mental process of remembering one’s way through the architecture of websites.
When confronted with the issue of how to organize the sound sources, the methodology I adopted has been created through a very intuitive process. Indeed, how does one connect two sound sources? Because of repetition or because of difference? The map has been established on the schematic of a center and its periphery, taking into consideration the acoustic properties of the sound sources. So binary oppositions emerging from classification are inappropriate since, firstly, the sounds are spatially related to several different “neighbors” and, secondly, they dialogue with one another in different ways; they are transitions between polarities: indoors/outdoors, density/expanding of the spaces, condensation/distention of listening.
The schematic of the map is one of concentric rings: the city-center is in the middle and the periphery the exterior. The center is represented by enclosed spaces. Around the center, transitive spaces connect – the inside and the outside. And then the periphery, which sounds like countryside with high reliefs.
The classification of sound sources into categories has been set up as follows:
– Confined spaces: small closed spaces with reverberation: Saint-Sauveur Cathedral’s cloister; Aix’s Music School; a café. Most of them are silent during the night since they are closed.
– Transient spaces: tight spaces with reverberating sounds: the Passage Agard – a very narrow, half covered street that looks as if it cuts two houses crosswise; a radio reverberating from an open window into a small street; the City Hall courtyard.
– Open spaces: quite large spaces with little reverberation: the marketplace from Place Richelme and also the one from Avenue du 8 Mai, with North-African origins; the Cours Mirabeau; parks; petanque strip; a secondary school exit; an accordion reverberating in downtown streets.
– Bars and cafés: either open or closed spaces that change between day and night time because of the density of their population. Different reverberations.
– Marketplaces: open spaces; the marketplace from Place Richelme shifts during night time to café terraces according to custom in Aix-en-Provence. Different reverberations.
– Sketches: what I consider as sketches – snippets of conversation where people unintentionally stage themselves. Different reverberations.
– Soundmarks: typical sounds from the city, very easily recognizable. Different reverberations.
– Nature: parks, country-side. Almost no reverberation.
In conclusion, in this installation audio mobility requires from the player both physical and imaginary spatial transfers. The exhibition space and the proposed representation are very much inter-connected, one makes sense because of the other: an imaginary map is understood while experiencing the real space. As the map and the exhibition’s space converge, a third space emerges from the player’s imagination.
Cauquelin, Anne. 2002. Le site et le paysage. Paris: PUF.
Cauquelin, Anne. 1989. L’invention du paysage. Paris: Edition Plon.
Gallet, Bastien. 2005. Composer des étendues (l’art de l’installation sonore). Genève. École des Beaux-arts de Genève.
Leonardo Music Journal, 23. Cambridge: The MIT Press, 2014.
Leonardo Music Journal, 47. Cambridge: The MIT Press, 2014.
Calvino, Italo. 2013. Les villes invisibles. Paris: Gallimard.
Unity 3D by Unity Technologies: http://unity3d.com
 The pattern of audio diffusion defining its directionality.
I am very grateful to Simon Chauvin, PhD candidate at Thales/CNAM in Game Design/Interactive Storytelling/Artificial Intelligence, for having programmed the whole installation and for running the tests. I would like to thank also Guillaume Stagnaro for his help and advices concerning the technical equipment. And Locus Sonus and my fellow residents for support during this process.