David Dowhaniuk: [Personal Nature]: A Virtual Reality Picnic for Visually Unresponsive Persons and Their Caretakers

Trees from behind

image by A Hernandez

[Personal Nature] is a media project which focuses on a specific scenario where persons are visually unresponsive, leaving communication and interpersonal connection difficult. Researchers at Holland Bloorview Kids Rehabilitation Hospital have been sonifying the data from the autonomic nervous system as a means for enhancing the perceived personhood of an individual with profound multiple disabilities (Blain-Moraes et al. 2013). By expanding on this idea [Personal Nature] takes on a different aesthetic approach, a way of creating a sense of space specifically that of a busy park. The work considers aspects of sonification and music therapy methods as well as philosophical concerns.

Introduction

The hospital soundscape is a complex one. On first entry it can seem quiet, but as one spends time there, the beeps and whirs of machines, intercom pages for doctors and other staff, as well as patients and their visitors, multiply into a surprisingly loud environment. Some research has put Hospital Critical Care Units at volume levels as high as 70 DB, similar to a busy restaurant, which is double the level found in a typical residential neighbourhood (Baker 1984). There is strong evidence that the hospital soundscape can be an ambient stressor, which becomes problematic when “hospital patients lack the authority, competence, and/or energy to reduce ambient stressors” (Topf 2000, 524). The soundscape and its effects are amplified for patients suffering decreased consciousness (DC)[1] or those with profound multiple disabilities (PMD),[2] a situation where communication channels are already strained. Insufficient communication is often compounded with difficulties sleeping and sensory deprivation as well as a lack of empathy between medical staff and the patient (Aldridge, Gustorff, and Hannich 1990). “Many activities in an intensive care situation appear to be between the unit staff and the essential machines, i.e. subjects and objects. To a certain extent, patients become a part of this object world” (Aldridge, Gustorff, and Hannich 1990, 345). I want to make clear that this is not an intentional shift but one that comes with a busy routine. Patients can often hear and generally have higher levels of awareness[3] and wakefulness[4] than loved ones perceive; music therapy is often used as a tool to facilitate family involvement and communication (Kennelly and Edwards 1997).

To combat this subject/object dichotomy researchers at Holland Bloorview Kids Rehabilitation Hospital are using the biometric data of patients’ autonomic nervous system (ANS) (heart & breathing rates, temperature and moisture on the skin) and turning them into an audio display or biosongs. This research was conducted on the basis of evidence which “suggests that physiological signals have the potential for creating a sense of connectedness or intimacy when revealed to another person” (Blain-Moraes et al. 2013, 3). After conducting pre and post intervention interviews with caretakers of three children with PMD the results showed “an improvement in their interaction with and perceptions of the person with PMD” (Blain-Moraes et al. 2013, 1). The intervention is thought to increase a sense of personhood towards the person with PMD (Blain-Moraes et al. 2013) or a reintroduction to the subject world.

The biosong intervention also positively affected the relationship with the families involved. For instance, the paper cites interviews with Thomas’[5] parent pre-and-post intervention: (pre)“I talk to him about random things. ‘Do you recognize me? I am your [parent]’”(Blain-Moraes et al. 2013, 6); (post) “I think that Thomas knows that there is a presence of a loved one… just like a boy who jumps up when he sees his [parent]… I’m quite sure, because whenever I am near, [the biosong is] … loud” (Blain-Moraes et al. 2013, 7). These interview results show that with amplification non-verbal communication cues can convey wakefulness and/or awareness. Once this minimal form of dialogue is generated there is opportunity for increased feelings of connectedness that would otherwise not be articulated.

[Personal Nature] aims to build upon the research outcomes from the biosong project while providing a different aesthetic experience. Reflecting on my experience of visiting loved ones in the hospital I realized that in those times in my life I wanted to be anywhere but there, preferably outdoors for instance in a park having a picnic. [Personal Nature] is an audio virtual reality (AVR) environment which uses audio recordings of birdsong, children laughing, water lapping at the shore and the sound of wind gently blowing past as well as rustling leaves. Ideally the system provides AVR by creating an environment which is conductive to open communication (a park/picnic setting) while amplifying and transmitting the ANS data into information, creating a sense of presence for the person with DC or PMD.

State of the art

Music is a form of artistic and personal expression and is traditionally associated with a variety of manmade instruments (violin, drum, piano, etc.). Composer Edgard Varèse defined music as “organized sound,” he considered music to be “sound as living matter” and wanted a more open definition to describe his work (Wikipedia 2014a). This characterization lead to Pierre Schaeffer’s “musique concrète” where recordings of real-world sounds were incorporated into compositions. Sonification, as an information tool which often attempts to replicate the more traditional forms of music, creating atonal sounds due to the nature of data. To combat this, Vickers and Hogg proposed sonification concrète, the method of using real world sounds within sonification (Barrass and Vickers 2011). By mapping the data to trigger clips to play while changing the components of the sound with the data (left/right, volume, speed etc.), the information comes to life through sound. The specific point of sonification concrète is that real world sounds are often used naturally for sonification (the sound of a kettle about to boil or water reaching the top of a glass) (Barrass and Vickers 2011). John Neuhoff describes how evolution has affected our neurological and cognitive architecture making us exceptionally sensitive to real world sounds; our brains process these sounds differently than erratic beeps and buzzes (Neuhoff 2011). This makes it also more palatable to listen to as there is no approximation of music in the classical sense. For [Personal Nature] the use of sonification concrète is to create the soundscape of a city park. Through the use of: birdsong, background chatter and children playing, water lapping at the shore, wind and trees; the user’s ANS data will be amplified.

Using this methodology for people with DC or PMD, the intent is to make the data clear and listenable while providing additional potential benefits for the loved ones of the patients, the patients, and the caretakers. The hope is to be able to “remove” those persons from the often stressful hospital environment. As a form of AVR, the listeners will be transported to an open park. There has been an increasing amount of scientific research into the benefits of nature on health. Ulrich published in 1984 that post-surgery patients whose hospital window faced a nature scene had shorter hospital stays than those who’s faced a building or parking lot (Ulrich 1984). Similarly patients on ventilators in mild comas have been shown to display reduced anxiety after listening to nature recordings than the control group whom had no intervention (Saadatmand et al. 2013). In the same vein, nature walks have shown to reduce mild depression and increase cognition compared to those who went on urban walks (Berman et al. 2012). Julian Treasure, a leading researcher in retail and public soundscape interventions, states that natural sounds (such as wind, water and birds) “makes us feel comfortable because it has always been there”(Treasure 2011, 100).

Music Therapy plays an important role in [Personal Nature]. This is a rich and well developed form of art therapy; it is used to work with patients of many abilities and maladies. Specifically, I am referring to work on active music therapy. This method is already often used on visually unresponsive patients. The therapist monitors the data from the patient (heart and breathing rates) to control the musical qualities of a piece of improvised music involved in the therapy session (Formisano et al. 2001). Upon reviewing videotaped sessions, one research study has shown that those with brain injuries in prolonged coma showed improvements in inertia[6] and/or psychomotor agitation from this form of therapy (Formisano et al. 2001). Essentially, [Personal Nature] is a digital version of this method but the decisions have been made in advance. By this, I mean that I will have curated the specific sounds and set the parameters but the data will control the rhythm of the soundscape. [Personal Nature] does not intend to replace music therapy, but these methods were considered in its making; it may also work as a tool within music therapy sessions.

Background

[Personal Nature] is balanced between art/design and analysis using scientific and medical data. The approach is highly focused on listenability and aesthetics. Although aesthetics is a well-known term, it is necessary to provide a specific definition when considering [Personal Nature]. Aesthetica was a term re-appropriated into modern language by German philosopher Alexander Baumgarten in 1739, he used the term to describe the “taste or sense of beauty” rather than the ancient Greek usage “I perceive with my senses” (Reiss 1994). The key for using Baumgarten’s definition is that he maintained that taste was a matter of the bodily reaction to beauty or disgust rather than an intellectual one. Beauty was now in the mind of the observer or specifically in the act of cognition, not in the object itself (Reiss 1994).

To revive and expand Baumgarten’s aesthetic definition, American philosopher Richard Shusterman proposed the discipline of somaesthetics in 1999. For him Baumgarten’s definition was neglecting the body within the sensing of beauty, essentially it was to connect somatic practices (for example T’ai chi ch’uan, Zen meditation) with Baumgarten’s aesthetic focus on the perfection of sensory perceptions (Shusterman 1999). He writes of the quickening of the breath or goose bumps in relation to experiencing emotions or sensing beauty. For me this is the philosophical counterpart to the biological ANS. The subtle automatic or instinctual reaction to stimulus; in this case, a caretaker, a loved one, or the medical staff.

Henri Lefebvre’s rhythmanalysis is another essential concept in the creation of [Personal Nature]. For Lefebvre “[e]verywhere where there is interaction between a place, a time and an expenditure of energy, there is rhythm” (Lefebvre 2004, chap. 1). These rhythms encompass everything from the smallest atoms to the ever expanding universe. There are three main sources of rhythms provided by Lefebvre and they are as follows:

  1. “repetition (of movements, gestures, action, situations, differences);
  2. interferences of linear processes and cyclical processes;
  3. birth, growth, peak, then decline and end.” (Lefebvre 2004, chap. 1)

He describes the different rhythms of the body all with their own time frames; from the quickly repeating heart rate to the slower rhythm of the kidney. He even has a classification for the physiological and psychological rhythms which he calls secret rhythms. For Lefebvre, rhythm is logical and mathematical yet heavily linked to the visceral and the body’s vitality, for here “the secrets and the answer to strange questions” are held (Lefebvre 2004, chap. 1).

Another main idea from rhythmanalysis is Lefebvre’s distinction between the present and presence. For Lefebvre, presence includes dialogue (time, speech and action). In contrast, the present only includes an exchange and acceptance of exchange. The present is commerce whereas presence is poetic (Lefebvre 2004). The important element is that time is to be considered; the present is people walking down the street whereas presence is people walking down the street through time.

Considering Lefebvre’s concepts in the context of a medical environment, a summary of a patient’s health data taken at the end of each day (or even multiple times during the day) does not provide any indication of presence because it is usually condensed into a single set of numbers rather than the fluctuations that happen over even a short amount of time. Deeper in the case of the visually unresponsive persons is that the patient is present in the room; however, they usually have minimal presence. There may be the beeping, buzzing or whirring of the machines or even a tear forming in the eye of the patient, but otherwise they can be lost to time. By amplifying the ANS, the minimal response can be made obvious making the user’s body their moderator over time or presence. This can been seen in the research conducted at Holland Bloorview Kids Rehabilitation Hospital were the amplification of this response through audio created a greater sense of personhood in the caregiver-patient relationship (Blain-Moraes et al. 2013).

By taking the user from the present to presence [Personal Nature] or biosongs makes the patients’ body and subconscious intuition expressive (however limited) and apparent through time. Presence takes value, creation and situation into account which is far beyond the present (in its relation and exchange). When considering [Personal Nature], I want to be clear that presence is still many steps away from agency; however, providing more medical information in a curated fashion can increase chances for dialogue, no matter how minimal.

Implementation

Data for this work was obtained through cooperation with the LiveIT lab at the Holland Bloorview Kids Rehabilitation Hospital. There I was provided access to Thought Technology’s Biofeedback system which includes non-invasive sensors for: heart pulse rate, breath rate, skin temperature and galvanic skin response (GSR). [Personal Nature] will be composed as follows:

Heart Rate – Birdsong/Children: Peaks in the heart rate data trigger a randomized clip of either birdsong or children playing/laughing. The greater the peaks distance from below zero (louder on the left side) and above zero (louder on the right side); this is to amplify the influx in the heart rate and create a sense of place.

Breath Rate – Water: The breath rate controls the speed and volume of a looped audio recording of water gently lapping at the shore. As the breath quickens so does the pace of the water lapping at the shore which also becomes louder.

Skin Temperature – Wind: As the temperature rises and falls so does the volume and speed of looped recording of wind rustling leaves.

GSR – Wind: Using a prebuilt open source wind generator the data from the GSR changes different aspects of the pitch and tone. The sound replicates wind as it moves past the ear.

The system is built using Max 6; a visual node based programming platform. The values of the sensor data are highly manipulated as to scale them to useable values in the sonification process. This is a part of the amplification process as it is the fluctuation in values that are important not the specific numbers themselves.

Conclusions and Future Work

This work aims, not to heal in the traditional sense, but to provide comfort and presence for often but not purposefully objectified persons, their caretakers, and loved ones. As this is a media project, the aim is to provide a means for communication no matter how minimal, a dialogue which offers a greater chance for connectedness. Considering Baumgarten’s sensual aesthetics and the bodies reaction to human contact, the sensed beauty of a loved one speaking and touching us, exposed through the ANS. The automatic reaction may be invisible but can now be audibly represented though time.

[Personal Nature] is really at the starting point. Currently sound is the major focus and offers a wide range of location options to replicate. For instance, beach or rainforest environments or even a replication of the home environment with all the hums of electronics and sounds of loved ones. However, I see the chance for a full experience of perception. By this, I mean I see room for visualization and haptic options. By moving in this direction, I believe there will be a more immersive experience allowing for an even deeper connection for the patient.

As sensor technology is continuing to mature and become more affordable, I also see growth in mobile iterations of this project. By using Bluetooth and smartphone technology the options for gathering and interpreting data grow immensely. For example, the current wearable sports and training technology is a market that [Personal Nature] may add to other novel possibilities.

Finally, as an artist I see great potential for performative and installation iterations as well as the ability to explore a variety of emotions while recording the data. Being able to create a space sonically, visually and haptically using such personal and raw expression excites and inspires me.

References

Aldridge, D, D Gustorff, and H J Hannich. 1990. “Where Am I? Music Therapy Applied to Coma Patients.” Journal of the Royal Society of Medicine 83 (6): 345–46. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1292679&tool=pmcentrez&rendertype=abstract.

Baker, Carol F. 1984. “Sensory Overload and Noise in the ICU: Sources of Environmental Stress.” Critical Care Nursing Quarterly 6 (4): 66–80.

Barrass, Stephen, and Paul Vickers. 2011. “Sonification Design and Aesthetics.” In The Sonificatiation Handbook, edited by Thomas Hermann, Andy Hunt, and John G. Neuhoff, 1st ed. Berlin, Germany: Logos Publishing House.

Berman, Marc G, Ethan Kross, Katherine M Krpan, Mary K Askren, Aleah Burson, Patricia J Deldin, Stephen Kaplan, Lindsey Sherdell, Ian H Gotlib, and John Jonides. 2012. “Interacting with Nature Improves Cognition and Affect for Individuals with Depression.” Journal of Affective Disorders 140 (3). Elsevier B.V. 300–305. doi:10.1016/j.jad.2012.03.012. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3393816&tool=pmcentrez&rendertype=abstract.

Blain-Moraes, Stefanie, Stephanie Chesser, Shauna Kingsnorth, Patricia McKeever, and Elaine Biddiss. 2013. “Biomusic: A Novel Technology for Revealing the Personhood of People with Profound Multiple Disabilities.” Augmentative and Alternative Communication 29 (2): 159–73. doi:10.3109/07434618.2012.760648. http://www.ncbi.nlm.nih.gov/pubmed/23484489.

Feist, Gregory J., and Erika L. Rosenberg. 2012. Psychology: Perspectives & Connections. 2nd ed. New York, New York, USA: McGraw-Hill.

Formisano, R, V Vinicola, F Penta, M Matteis, S Brunelli, and J W Weckel. 2001. “Active Music Therapy in the Rehabilitation of Severe Brain Injured Patients during Coma Recovery.” Annali dell’Istituto Superiore Di Sanità 37 (4): 627–30. http://www.ncbi.nlm.nih.gov/pubmed/12046234.

Kennelly, J, and J Edwards. 1997. “Providing Music Therapy to the Unconscious Child in the Paediatric Intensive Care Unit.” The Australian Journal of Music Therapy 8: 18–29. http://ulir.ul.ie/handle/10344/2737.

Lefebvre, Henri. 2004. Rhythmanalysis: Space, Time and Everyday Life. Translated by Gerald Moore and Stuart Elden. 1st ed. New York, New York, USA: Continuume.

Neuhoff, John G. 2011. “Perception, Cognition and Action in Auditory Displays.” In The Sonificatiation Handbook, edited by Thomas Hermann, Andy Hunt, and John G. Neuhoff, 1st ed. Berlin, Germany.

Reiss, Hans. 1994. “The ‘Naturalization’ of the Term ‘Ästhetik’ in Eighteenth-Century German: Alexander Gottlieb Baumgarten and His Impact.” The Modern Language Review 89 (3): 645–58.

Saadatmand, Vahid, Nahid Rejeh, Majideh Heravi-Karimooi, Sayed Davood Tadrisi, Farid Zayeri, Mojtaba Vaismoradi, and Melanie Jasper. 2013. “Effect of Nature-Based Sounds’ Intervention on Agitation, Anxiety, and Stress in Patients under Mechanical Ventilator Support: A Randomised Controlled Trial.” International Journal of Nursing Studies 50 (7). Elsevier Ltd: 895–904. doi:10.1016/j.ijnurstu.2012.11.018. http://www.ncbi.nlm.nih.gov/pubmed/23245705.

Shusterman, Richard. 1999. “Somaesthetics: A Disciplinary Proposal.” Journal of Aesthetics and Art Criticism 57. http://www.jstor.org/stable/432196.

Topf, M. 2000. “Hospital Noise Pollution: An Environmental Stress Model to Guide Research and Clinical Interventions.” Journal of Advanced Nursing 31 (3): 520–28. http://www.ncbi.nlm.nih.gov/pubmed/10718870.

Treasure, Julian. 2011. Sound Business. Management Books 2000 Ltd. http://www.thesoundagency.com/wp-content/uploads/2011/09/SB2-sample-chapter.pdf.

Ulrich, RS. 1984. “View through a Window May Influence Recovery from Surgery.” Science 224 (4647): 420–21. http://www.ideal.forestry.ubc.ca/frst524/09_ulrich.pdf.

Wikipedia, Contributors. 2014a. “Definition of Music.” Wikipedia. Accessed May 27. http://en.wikipedia.org/wiki/Definition_of_music#Organized_sound.

———. 2014b. “Rhythmanalysis.” Wikipedia. Accessed May 27. http://en.wikipedia.org/wiki/Rhythmanalysis.

PDF version of this article

Footnotes

  1. This can include those suffering varying levels of wakefulness and awareness such as: coma, vegetative and minimally conscious states. ^
  2. This can be described as those with “little or no understanding of verbal language and symbolic interactions or capacity for self-support” (Blain-Moraes et al. 2013, 2). Many also have sensory impairments particularly brain related vision deficiencies (Blain-Moraes et al. 2013). This can be described as those with “little or no understanding of verbal language and symbolic interactions or capacity for self-support” (Blain-Moraes et al. 2013, 2). Many also have sensory impairments particularly brain related vision deficiencies (Blain-Moraes et al. 2013). ^

  3. Awareness refers to “monitoring of information from the environment and from one’s own thoughts” (Feist and Rosenberg 2012, chap. 6). ^
  4. Wakefulness denotes the “degree of alertness reflecting whether a person is awake or asleep” (Feist and Rosenberg 2012, chap. 6). ^
  5. A child with PMD’s ^
  6. Refers to a reduced psychomotor initiative when referring to those in long-term comas (Formisano et al. 2001). ^

Leave a Reply

Your email address will not be published. Required fields are marked *