Art, technology and the Internet of Living Things

Abstract

Intelligence augmentation was one of the original goals of computing. Artificial Intelligence (AI) inherits this project and is at the leading edge of computing today. Computing can be considered an extension of brain and body, with mathematical prowess and logic fundamental to the infrastructure of computing. Multimedia computing—sensing, analyzing, and translating data to and from visual images, animation, sound and music, touch and haptics, as well as smell—is based on our human senses and is now commonplace. We use data visualization and sonification, as well as data mining and analysis, to sort through the complexity and vast volume of data coming from the world inside and around us. It helps us ‘see’ in new ways. We can think of this capacity as a new kind of “digital glasses”. The Internet of Living Things (IOLT) is potentially an even more profound extension of ourselves to the world: a network of electronic devices embedded into objects, but now with subcutaneous, ingestible devices, and embedded sensors that include people and other living things. Like the Internet of Things (IOT), living things are connected; we call those connections “ecology”. As the IOT becomes increasingly synonymous with the IOLT, the question of ethics that is at the centre of aesthetics and the arts will move to the forefront of our experience of and regard for the world in and around us.

Introduction: what can we learn from the natural Internet of Living Things?

Tacit knowledge is that which is learned from experience rather than taught in traditional academic ways. It is considered implicit knowledge and know-how that is shared in everyday life. What can we learn from the natural Internet of Living Things? One possibility is that tacit knowledge and ecological systems are linked, along with cultural and natural diversity, and that our survival depends on them. Here we trace a series of artistic explorations that aim to expand our tacit knowledge with technology using implicit interaction systems. Computers form an increasingly transparent interface between the inside and outside world, and our human senses and brain, due to miniaturization and speed of computing, and smart materials including digital-biological systems, and bio-informatics. We need to intensify our sensory and ecological awareness, and expand our tacit knowledge in ways that are fundamentally ethical. These goals are the focus of the Sorensen’s recent creative work. Vibeke Sorensen’s artwork aims to reshape technology so as to enhance our awareness of and compassion for the natural world, providing new opportunities for respectful and subtle emotional, intellectual, and physical engagement (Fig. 1).

Fig. 1.

One of these networks takes in raw energy, transforms it into life, cools the Earth, cleanses water, absorbs CO₂, and releases oxygen. The other does the reverse. Photograph of Sao Paulo, Brazil by Ana Paula Hirama—Flickr: Passeio de Helicóptero em São Paulo, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=21405720. Photograph of Lower Lewis River Waterfall, https://www.peakpx.com/en/hd-wallpaper-desktop-orwom

The way we form and employ technological connections to our world as we engage with and reshape it can become a sensitive 2-way process of feedback, including bio-feedback, in order to be ethical and responsible. Tacit knowledge from everyday life experience can interact with technology to bring that which is unseen and otherwise undetected to our everyday senses. Sorensen uses implicit interaction strategies, in particular physical computing, data visualization and sonification, to provide new kinds of input to technological systems that welcome people from all backgrounds, by making it more inclusive and transparent. Importantly, it also connects us to the wider living world.

One example of tacit knowledge is interaction with hinged boxes, and how to open and close them. Most people have done this since early childhood and hence do not need instructions to learn how. Hence boxes can be considered “implicit” interaction objects. If a digital sensor is inserted into such a box to detect its state, open or closed, it can become an input device for a real-time digital graphics and multimedia system. This was done in Morocco Memory II (1999), making the box an implicit interactive input device, which due to its ubiquitous nature was intuitive for people from a wide range of backgrounds and cultures to engage, especially those from material cultures.

Morocco Memory II was an interactive architectural installation that consisted of a small house, a table and 2 chairs, and 6 aromatic wooden boxes with Moroccan spices housed in them, as well as fabrics with sensors. Each box contained an embedded system, a small circuit board set inside it, under the spices. The circuit detected the state of the box and then wirelessly transmitted this information to a computer via a receiver under the table which was hard wired to a computer. This allowed people to pick up the boxes and carry them about as they moved inside the house. Because of the aromas emanating from the spices in them, the boxes not only triggered the electronic system, but also impressions and memories associated with them in the minds of the users. This was further enhanced by the sense of taste of the spices (as they could be eaten) and the sense of touch of other materials in the installation some of which also served as input devices.

This work explored dynamic memory and narrative by interactively recombining and re-contextualizing electronically mediated memory fragments, both personal and cultural, and correlating them with the physical-digital boxes and their internal spice smells, as well as tactile textiles in the small wood and satin house. The mediated memory elements included 66 lexia (1–2 paragraph stories), more than 3000 still images, and hundreds of movie, sound, and musical recordings. The physical materials included not only the spices and seeds, but the aromatic wood of the boxes themselves, as well as a variety of scratchy and soft wools in rugs and pillows on the floor, and the silky satin walls. The boxes were wireless and therefore could be carried, and the spices inside them could be “rolled” in the fingers by users, and even eaten. Some of the fabrics on the walls and floor, and those on the table, had embedded sensors so that touching them also served as inputs to the system. There was also the experience of the modulating coloured light that filled the space and fell directly on the skin of the user. The light changed colour and intensity based on movement of the user inside the installation. The result was an immersive multimodal physical-digital experience for dynamic digital remembering and re-collection of memories stimulated by all five of the human senses (smell, touch, taste, sight and hearing). Contributors and users included people from around the world including Morocco (Fig. 2).

Fig. 2.

In Morocco Memory II (1999), a multimedia architectural installation, seeds and spices were placed inside 6 wooden boxes fitted with custom embedded systems as part of a real-time interactive process. When the boxes were opened and closed by a user, the system kept track of the number of boxes and order of choices, and responded by triggering navigation of story elements comprised of more than 10,000 texts, sounds, images, and movie fragments. Aromas from the spices mixed and triggered memories in the mind of the user, which informed the users’ choices. The result was a re-combinant multimodal narrative that due to the large number of elements resulted in an emergent system in constant transformation. Morocco Memory II was installed at the Interactive Frictions Exhibition as part of the Labyrinth Initiative on Interactive Narrative, Fisher Gallery, University of Southern California, June 5–18, 1999. Photographs by Vibeke Sorensen.

To link and correlate the digital memory fragments with the physical-digital system, non-linear and associative connections and interaction strategies were programmed into the software using Pure Data (Pd/GEM) so that the data could be navigated in dynamically changing ways, which allowed for different kinds of narrative and poetic structures to emerge. Multiple directories of associated data were created in branching tree structures, and intersections and gaps formed the basis of the non-linear data organization for it. A link in a branch could become a gap, or a gap could become a bridge. There could be cuts or leaps, or a range of transitions based on associated qualities such as a word or text, image, sound or the state of interaction by the users. Moreover, opportunities for people to cooperate rather than compete were provided by having 6 boxes, any of which could open and close in any order, and at any time and by any number of people. To move to a new combination of elements shaping a story, the users could either interact individually or in collaboration with other users, thus creating a collective narrative.

The digital media materials included thousands of documentary images, music and sound recordings, animations and videos, and short texts about Morocco that were organized according to different relationship categories. These materials were gathered from many sources, contributed by people with experience of Morocco that the author encountered over many years. It was based on the concept of the “open book” and “open journal” where anyone she met during her travels could write or draw in it. Memory can be primary, secondary and tertiary, and the additive meaning that stories and memories told by individuals become the shared and collective memories of the people who hear them. This system was created to be additive in a similar way. The software employed combinatorics, randomness and weighted probabilities in the logic in order to free the system from stasis and repetition and to provide surprise so that each story and interactive experience was unique. Because of the number of elements, and endless number of possible combinations, it became an emergent system. The overall metaphor for the installation was the concept that the mixing of smells is like the mixing of memories (Fig. 3).

Fig. 3.

Seeds are actual genetic memory, part of human life and culture, intricately connected with our survival and memory. The seeds in Morocco Memory II were spices that provided scent and taste. Photographs by Vibeke Sorensen

Unlike many artworks and museum and gallery exhibitions, this piece had an active role for aroma, and touch was encouraged. The entire body and groups of people were inside the work, and all of their senses were active elements without which the piece did not function. In this way, it foregrounded people and backgrounded technology and implicitly welcomed and celebrated traditional world cultures.

The author’s next architectural installation was entitled Sanctuary (2005) and was even more implicit or tacit, using liminal systems such as water bowls with coiled wires under them to detect the proximity of people in the installation. Presented at Gallery 1-1-1 at the University of Manitoba, Winnipeg, capacitance sensors were placed in the earth below the roots of plants to detect the touch of people, as well as in chairs and the floor to detect their presence in the space. The system was in a “dream” or “subjective” state when no people were in it. When people entered the installation it “woke up” and transitioned into a “documentary” or “objective” mode (Fig. 4).

Fig. 4.

Sanctuary (2005) installation at Gallery 1-1-1, University of Manitoba, Winnipeg, Canada. Plants, embedded systems, capacitance and custom hardware were included as primary elements of a real-time responsive environment. Any human touch of plants provided signals that the plant cells responded to in biofeedback with the system. Signals from the hands were sent through the leaves to the stems. A metal wire was placed into the dirt near the roots detected the plant’s signals, which was sent to the computer and converted into sound, light and navigation pathways for the mixing and display of lights, images, music and movies. There was also a bowl with water in it that detected proximity of users, and copper coated benches that sent signals to the system when people sat down or arose. The system traversed multimedia content in intersecting tree structures and included thousands of original recordings of spiritual and natural sanctuaries around the world. Sorensen received a 2001 Rockefeller Foundation Fellowship in Film/Video/Multimedia to support the creation of Sanctuary. Additional support was provided by the Midi@rte Laboratory, School of Fine Arts (EBA) at the Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil. Photograph by Vibeke Sorensen

If people sat and rested, it would return to the “dream state.” If people touched the plants in different ways, the system would respond with different effects. This was the most challenging part as the author did not want the users to consider the plants as “game devices” and touch them violently merely for effect. The question of why plants respond the way they do to different kinds of touch` led to discussions with biologists and plant physiologists on this topic. Some plants react to touch by triggering new growth and others by dying depending of the nature of the touch. In Sanctuary, people were usually gentle with the plants which survived and grew (Fig. 5).

Fig. 5.

When no people were inside the Sanctuary installation, the system continued to run in a “dream” state that associated and layered mediated memory fragments as a montage. In this case, the photographer was detected near the water bowl (not seen) which “awakened” the system and responded with images of the Amazon river in Brazil. Small pubescent or “fuzzy” plants networked to the system can be seen on the low table in the middle. Photograph by Vibeke Sorensen

Technically, it was surprising that putting a small wire in the dirt of the plant below its roots, and attaching it to an external capacitance sensor connected to a computer, made it possible to detect a signal from a person touching the leaves of a plant. The simple use of a small wire and sensor led to questions about how this could work. If the wire was placed near the plant in the open air, and a person touched the leaves, no signal was passed from the hand nor the plant to that sensor. However, when the wire was placed in the dirt below the roots at the bottom of a pot, a signal from the hand was detected. Sorensen proposed that the signal propagated from the site of touch to the wire through water in the plant. However, plant physiologists informed her that cell to cell communication in a plant is necessary but does not work that way. However, the system did work and therefore the artwork was able to bring this phenomenon to the attention of biologists.

Many kinds of plants were tested for this project, and the result was that smaller and bushier, and surprisingly pubescent plants or those that were “fuzzy”, were excellent, as were plants with larger leaves. The concept that emerged was that leaves are similar to antennae, collectors of light and other signals such as those passed to them from our hands when we touch them, and the signals move through the plants’ structures to the roots, in reverse and distributed tree structures. The light in the installation came from the projections only, as the University of Manitoba would not allow interactive lights installed at the roof of the structure to be turned on due to fear of fire. However, the remaining light was sufficient for the plants to provide biofeedback to the system.

Sanctuary took shape as a response to “virtual reality”. Rather than a closed box isolating people from nature and the physical world, this work was an open box that required nature and the physical world to be interfaced in order for it to run. It was also a collective sanctuary, with many images, texts and movies from natural and spiritual sanctuaries in many countries and cultures around the world, that Sorensen recorded over a 5 year period. The implicit elements in the installation were the architecture, chairs, bowls, tables and plants. Like Morocco Memory II, it was programmed using Pure Data (Pd/GEM). The music and sound was by Professor Shahrokh Yadegari of the Department of Music at the University of California, San Diego.

As these installations show, we can expand both our tacit knowledge with digital technology, and our technology with tacit knowledge when they are developed with ethics and aesthetics; in particular by integrating physical media and material culture with digital technology. With physical computing and embedded systems ethically informed by daily life across cultures and nature, we can have a “smarter” environment of things, the Internet of Things (IOT) and even the Internet of Living Things (IOLT). In combination with Artificial Intelligence (AI), ubiquitous digital technology can adapt to us and to the environment, and help us to live in greater awareness of and harmony with it. But it takes careful consideration of the consequences to make it ethical and life affirming. In this way, technology can offer new possibilities for expanding and enriching our senses and our experience and consideration of the natural world while providing new ways to interact with the vast data associated with it. And vice versa—tacit knowledge and strategies from our interaction with the physical world and nature can help us to develop our technology in more responsible and ethical ways.

Crucially, human beings must explicitly learn empathy. Children cannot directly feel the pain that other people—and other living things—are feeling. The first requirement is to sense others and contemplate them with compassion. It is known that mirror neurons in the brain help with this.

We also know this from our own experience of the physical world (Ramachandran and Oberman 2006). When we see other people, we empathize and literally begin to experience their pain or joy. This also helps to promote understanding and cooperation, rather than competition. However, in most computer systems including commercial applications, the paradigm for interaction is the remote control of devices both at home and from afar using the “master-slave” metaphor to describe it. Remote control of people from afar, politically, was what colonial regimes did, and called Imperialism. Historically, revolution and law were needed to bring ethics and justice to political and governmental systems. Countries where slavery is outlawed, and social and racial equality are protected by law, are generally considered “enlightened”. The common use of the term “master-slave” in technology, however, facilitates the normalization of the concept and practice, which tragically still exists around the world today as modern slavery even in countries where it is outlawed. Should we change the metaphor of “master-slave” in technology, too? Yes! Goals include replacing top-down with bottom up and lateral system structures, and increased focus on human rights, animal rights, and ecology.

As we begin to experience and invent the Internet of Living Things (IOLT), it is helpful to reflect on the ways the problem of empathy has been approached in deep time. The earliest textual recognition of human rights is thought to be the Cyrus Cylinder, dating from the 6th century BC and discovered in the ruins of Babylon from the reign of King Cyrus the Great of Persia (Kuhrt 1983). An influential source for Asia were the “Edicts of Asoka” issued by the ruler of the Mauryan empire of India between 272 and 231 BC. The Edicts consist of 33 inscriptions on pillars, boulders and cave walls, including one which abolishes the slave trade. Asoka describes his conversion to Buddhism which was the inspiration for eloquent texts affirming human rights and care for the living world:

“Ten years (of reign) having been completed, King Piodasses (one of the titles of Asoka: Piyadassi or Priyadarsi, “He who is the beloved of the Gods and who regards everyone amiably”) made known (the doctrine of) Piety to men; and from this moment he has made men more pious, and everything thrives throughout the whole world. And the king abstains from (killing) living beings, and other men and those who (are) huntsmen and fishermen of the king have desisted from hunting. And if some (were) intemperate, they have ceased from their intemperance as was in their power; and obedient to their father and mother and to the elders, in opposition to the past also in the future, by so acting on every occasion, they will live better and more happily (Figs. 6 and 7) (Translated by G.P. Carratelli,1925).”

Fig. 6.

“…everything thrives throughout the whole world. And the king abstains from killing living beings, and other men and those who are huntsmen and fishermen of the king have desisted from hunting. … in opposition to the past also in the future, by so acting on every occasion, they will live better and more happily.” Khandahar Rock Edict of Emperor Asoka, 268–232 B.C.E. Photograph: https://commons.wikimedia.org/wiki/File:AsokaKandahar.jpg

Fig. 7.

“Formerly in the kitchen of King Devanampriya Priyadarsin (one of the titles of Asoka) many hundred thousands of animals were killed daily for the sake of curry. But now, when this rescript on morality is caused to be written, then only three animals are being killed (daily), two peacocks (and) one deer, but even this deer not regularly. But even these animals shall not be killed (in future).” First Kalsi Rock-Edict of Emperor Asoka, 268–232 B.C.E. Translation: Hultzsch, Inscriptions of Asoka. New Edition. In: Corpus Inscriptionum Indicarum vol. I. Oxford: Clarendon Press, 1925. Photograph by Mishra.prashant89—Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=51442997

New explorations in Singapore

Vibeke Sorensen went to Singapore in 2009, where she was influenced by the rich heritage of the region, including the history and cosmologies of the people there. Singapore has a tradition of religious harmony and therefore her recent work is connected with this.

In 2013, she produced Illuminations, a 30 meter long, glowing, illuminated folding screen (Fig. 8). Also an interactive installation, it combined the Asian panorama with computer animation rear-projected onto 12 screens as part of a real-time visual music system. The entire work was based on Asian concepts of space-time and cosmologies of One-ness, represented by a rainbow shaped, curved arch layout of the screens, the use of Tibetan Singing Bowls, and plants. A Tibetan medicine mandala which was also on the cover of the book A Clear Mirror of Tibetan Medicinal Plants (1999) by Dawa Galypo (Fig 9.) was consulted for selection of the plants, which included ginger. During the six weeks of the installation, people of all the religions of Singapore often returned daily to meditate and pray. Surprisingly, some reported that the abstract nature of the images welcomed everyone. Towards the end of the 6 weeks, they asked Sorensen to keep the installation running permanently. But that was not possible due to the scheduling of the gallery.

Fig. 8.

Illuminations (2013) Interactive architectural installation by Vibeke Sorensen. An illuminated folding screen, 30 m wide and arched, it included living plants where their O₂/CO₂ was detected as part of a multimedia biofeedback system. Plant biorhythms interacted with real-time visual music that included Tibetan singing bowls. The plants “sang” melodies that affected changes in colours, textures, 3D shapes, and movement, which in turn affected their growth and biological rhythm. Photograph of Christine Veras playing the singing bowls, by Vibeke Sorensen

Fig. 9.

From the Illuminations installation. Selection of plants for biofeedback was informed by the traditional Tibetan Buddhist medicine mandala on the left, which was also on the cover of the book, A Clear Mirror of Tibetan Medicinal Plants (1999) by Dawa Galypo

Illuminations used 13 networked computers, with 1 for each of the 12 screens, and 1 for the music and interaction. There were 12 data projectors, 12 custom 2-sided screens, 3 Tibetan singing bowls, custom furniture including stands for the Tibetan bowls and Ottomans for people to sit on, and circuits embedded into them. The sensors in the furniture detected qualities of the users’ playing of the bowls and the Ottomans were activated when people sat or rose up. In addition, there were live plants with sensors that measured their O₂/CO₂ output. All were interfaced to the real-time system, which featured biofeedback from the plants that altered the animation and music in response to their signals, and which in turn affected their responses, biological rhythms, and growth. The chronobiology, or natural rhythms of the plants, followed the Circadian rhythms of the 24 h cycle of light and dark during day and night. The sensors not only from the plants, but also the singing bowls and Ottomans, were interfaced with Arduino and custom circuits to the main computer where they were input into a Pure Data (Pd/GEM) programme generating the live computer music, and managing the networking, data sharing, signal processing, and graphics for the other 12 computers, each of which generated unique (though related) real-time 3-D computer animation for their respective screens. The digital electro-acoustic music system was tuned to the 3 Tibetan singing bowls played by the users in the installation using contact microphones under the bells. Signals from the bells included amplitude (loudness), frequency (pitch), and duration of the playing and the sound. The embedded systems in the 3 Ottomans placed next to the singing bowls also sent data to the system so that it was aware of people sitting on them, whether or not those people were playing. When all 3 bowls were “singing” together, producing a sustained complex harmonic “chorus”, the animation transformed into a huge moving rainbow traversing all 12 screens across the entire 30 meter span of the installation. This was further reflected by wall sized mirrors at either end of the arched rainbow, creating a 90 meter span. The motion of the rainbow was that of a gently flowing river (Fig. 10).

Fig. 10.

From the Illuminations (2013) installation at the ADM Gallery, NTU Singapore. Some of the plants are visible on the bottom left. Photograph by Vibeke Sorensen

In addition to the Tibetan singing bowls, the soundscape included granular synthesis of waveforms using Pure Data (Pd), programmed by Sorensen. In order to carefully follow the plants, Sorensen mapped the CO₂/O₂ data to specific voices in the music composition programme which allowed her to “hear” the plants. Both the music and animation produced in the Pd/GEM system were realized using waveforms and shared data (Fig. 11). The music was based on the pentatonic scale often used in music of the region. Because of the many interlocking cycles and epicycles of interrelated frequencies and amplitudes, the entire visual-music work became a huge polyrhythm following the Circadian cycles of the plants. The final work was a cross-kingdom (plants, animal and people) co-embodiment of the physical computational system. There was biofeedback from the plants, as they not only responded to the changing light and sound conditions produced by people in the installation, but even at night when people were gone, the plants interacted with the multimedia sound-light system. The sensor data showed a daily spike in CO₂/O₂ an hour before dawn that had no obvious trigger, which Sorensen and her team speculated could be the plants’ biological clocks. A plant expert warned that the plants might die within a week as they were in a darkened gallery with only light from the projectors. And over a 6 week period, the plants may need to be replaced weekly. Instead, the plants flowered and grew new roots. None withered or died during the installation, only afterwards when they were returned to normal light conditions.

Fig. 11.

Plants with sensors and interactive Pure Data (Pd/GEM) programme used for the Illuminations (2013) installation at ADM, NTU Singapore. Biofeedback from the plants was mapped to real-time computer music and animation using CO₂/O₂ sensors, Arduino and Pure Data (Pd/GEM) software. Photograph by Vibeke Sorensen

Illuminations used 13 networked computers, with 1 for each of the 12 screens, and 1 for the music and interaction. There were 12 projectors, 12 custom screens, 3 Tibetan singing bowls, and circuits embedded into custom furniture to sense and network them. Sorensen conceived the work, and composed the music and the animation, doing the Pure Data (Pd)/GEM programming for both. Custom hardware and software was contributed by Marilia Bergamo of the School of Fine Arts, Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil. Networking was done by Nagaraju Thummanapalli, and custom furniture was designed by Fabrizio Galli and fabricated by Ong Kee Sing, of the School of Art, Design and Media, NTU Singapore.

Do plants feel? Some, like the Mimosa Pudica, respond dramatically to touch and pull back and curl their leaves. Nyctinastic plants such as clovers and lotus plants, move their leaves and petals according to light and time of day. Similarly, the plants in Illuminations responded to the environment, a visual-music biofeedback system, and flowered. Living things that feel are by definition sentient. With sensor technology and physical computing systems, it is possible to detect many of their physical qualities and translate data emanating from them into forms we can perceive, making us more aware of their sentience.

Big Data and the Mood of the Planet (2015)

A new phase of digital media artwork was inspired by a Pre-Columbian archaeological site in Tiwanaku, Bolivia which marked the celestial movement of the heavens with life on Earth (Fig. 12). On the winter solstice in the Southern hemisphere, 21 June, the sun comes up exactly in the middle of the Gate of the Sun, a stone portal, casting light onto people there, mostly of indigenous tribal descent including Aymara, as well as Shamans who brave the cold to be there at this auspicious time of the beginning of a new agricultural year. They raise both hands in ritual to take in the Sun’s energy, and at the same time chant animal sounds in unison, thus connecting them with the Universe and other living things. The archaeological site at Tiwanaku consists of a temple structure with several portals and open squares, and like most temples around the world, each portal marks a point in time and space, including observation of the Sun, Moon and Milky Way, and correspond with changes in life stage of people passing through them.

Fig. 12.

The Mood of the Planet (2015) installation was inspired by a Pre-Columbian archaeological site in Tiwanaku, Bolivia which connects the movement of the heavens with life on Earth. On the winter solstice in the southern hemisphere, June 21, the sun comes up in the middle of a portal, casting light onto people present who raise their hands in ritual to take in the Sun’s renewed energy and put them into harmony with nature and the Universe. Photograph of Vibeke Sorensen in Tiwanaku on 21 June 2002 by Heitor Capuzzo of UFMG, Brazil

The Mood of the Planet (2015) is an interactive sculpture, a dynamic architectural installation that has as its center-piece a large ‘arch’ or ‘doorway’ that emits coloured light and animates in reflection of the live emotions expressed by people all around the globe communicating through networks such as Twitter. It rethinks the term “public art” in the context of global social transmodal transmedia. Laid out as a portal in a temple, the “arch” or “doorway” is iconic and implicit in that people everywhere go in and out of spaces using some kind of device to close and open, or transgress, boundaries. The arch and doorway as a metaphor also references developmental transformation, the metaphoric passing from one state to another, of growth and change that is analogous to the transformative effect that global communications technologies have upon our collective human condition. The arch also signifies human transformation of the environment, today both physical and digital, just as this iconic form has been used across cultures throughout human history. This “doorway” is reflected within a wall-mirrored room where it is repeated into a tunnel-like shape, an infinity of doorways that exist as an endless cycle, or echo, of past and future in space and time, and collapsing into the eternal present. A wooden pathway traverses the room through the doorway, and connects the two mirrored walls and thus creates the “infinite” pathway to walk upon (Figs. 13, 14 and 15).

Fig. 13.

The Mood of the Planet installation was laid out as a portal or arch, like Tiwanaku, and represents a passage to greater understanding of our existence on the Earth and in the Universe as a metaverse where people of all walks of life are increasingly interconnected to each other and the vast physical-digital environment, like the Asian concept of One-ness. Diagram copyright 2015 Vibeke Sorensen

Fig. 14.

The Mood of the Planet under development. The larger image on the right shows the inside of the arch, including “digi-tiles” made of crushed recycled glass, LEDs, and custom electronics. The 3 small vertical images on left show the “tunnel” created by reflections in mirrors on either side of the arch sculpture. Photographs by Vibeke Sorensen

Fig. 15.

The Mood of the Planet (2015) installation in the ADM Gallery, NTU Singapore. Photograph by Nagaraju Thummanapalli

The arch sculpture was made of 30 building blocks of “digi-tiles”. Consisting of crushed recycled glass and custom electronics, they emit coloured light, and the colours and shapes change in real-time based on a real-time analysis of keywords representing human emotions, emanating from sources such Twitter and from all across our planet. The current mood of the people of the Earth through colour and motion thus become an immersive presence, a dynamic rainbow that bathes the interactor in light.

Patterns of light and colour were based on the 6 basic emotions that psychologists agree are universal, including happiness, sadness, anger, fear, disgust, and excitement. Colours and textures were associated in the following way:

• 1. Happiness and pleasure: green/gold

• 2. Sadness: blue/grey

• 3. Anger: red/black

• 4. Fear: white/black

• 5. Disgust: brown/yellow

• 6. Surprise: patterns and multiple colour combinations.

Anecdotal feedback from users around the world were positive, and included comments that although abstract and considering that colour-mood relationships across cultures are not universal, the combination of sound/music, movement, and colour provided clarity about the moods. Viewers often observed and interacted for long periods of time, following the changes as they were updated.

Anonymous Twitter messages were analysed continuously and the results were updated on the arch every 1 min, by dividing up into proportions distinct colours and animating patterns reflecting the balance of the 6 moods. For example, if the mood was only happiness, then green/gold patterns would display for the entire 1 min. If half were sad and half were happy, then 30 s would be blue/grey and 30 s would be green/gold. The shapes and patterns were abstract, and the animation motion was modulated to enhance the emotion (i.e., slower for sadness, very fast for excitement, etc.). Passage through the arch would trigger rainbow colours.

The endless reflections in the mirrors represented a passage to greater understanding of our existence on Earth and in the Universe, due to the explosion of information and new ways to apprehend it with digital technologies. Billions of people of all walks of life are increasingly interconnected not only with each other but with the vast emerging system that these networks and Big Data provide, immersing and reflecting them in it, similar to the Asian concept of One-ness (Like several of Sorensen’s other works).

The Mood of the Planet was first installed in the NTU ADM Gallery as part of the Everything is Data exhibition curated by Dr Laurene Vaughan of the Royal Melbourne Institute of Technology University (RMIT Melbourne, Australia) that took place from 14 August to 26 September 2015. This was followed in May 2016 by the Mood of New York (Fig. 16) and the Mood of Singapore as interactive animated sculptural ”paintings.” In 2018, Mood of Singapore was installed in the lobby of the Experimental Medicine Building of NTU Singapore, and in 2020, Mood of the Planet was installed at the main entrance to the NTU ADM building.

Fig. 16.

The Mood of New York (2016) installation, at the Experiments in Art and Digital Technologies (EADT) Exhibition in 2016, a 2 m × 0.5 m LED custom display with original software, animation and music by Vibeke Sorensen. PureData (Pd/GEM) programming by Vibeke Sorensen. Custom Python programming and hardware development by Nagaraju Thummanapalli, and with assistance from Dennis Low Peng Thiam. Photograph by Suzanne Ball

The Internet of Things (IOT) and Internet of Living Things (IOLT): Digital Amulet and In Other Wor(l)ds

The Mood series was followed by works concerned with the Internet of Things (IOT) and the Internet of Living Things (IOLT). In 2017 Sorensen created Digital Amulet: Smart Necklace, a 3D printed wearable multimedia technologies collar, or necklace of digital beads, that has several modes including a biofeedback system. It contains multiple networked Android displays with touch sensors, real-time weather data, sound, and animation. Changing animated colours and music respond to heart rate and body temperature, as well as live reporting from networked sensors located in different biomes of the Earth. Data is continuously updated from the wearer’s body as well as from global environmental monitoring agencies such as the U. S. National Oceanic and Atmospheric Administration (NOAA) so that the wearer could be aware of the changing conditions of our planet, in particular global warming. The design was inspired by Ancient Egyptian collars which included beads symbolizing their gods. The tacit or implicit metaphor is beads, not only for adornment and identity, but as a form of mnemonic memory common across history and cultures (Kelly 2015). This work is multicultural, both due to its design references as mentioned, but also because the iconography is based on nature, and the animation changes with the wearer’s experience of environmental conditions from different ecologies. Further, the IOT becomes the IOLT because the networking of human body sensors provides the potential to transmit personal data, with or without the knowledge of the wearer, as a form of invasion of personal privacy which risks surveillance of the wearer/user. In this case, transmission of the wearer’s data was strictly limited to the local network of the wearer. This work was premiered at the ACM International Symposium on Wearable Computing/UbiComp, ISWC 2017 in Maui, Hawaii, USA. Sept 11– 15, 2017.1

The Digital Amulet: Smart Necklace provoked more questions about our place in nature, beginning with the air we breathe. Air is fundamental to life but in crisis due to global warming caused by polluting human activity. Scientific and popular media inform us of the tragic effects such as entire flocks of birds falling out of the sky due to starvation as their food source could not survive the heat, and whole schools of fish washing up dead on sandy ocean beaches because of the loss of the plankton that they eat due to temperature shifts. Increasingly, global warming and related atmospheric changes are affecting people closer to where they live. Huge forest fires have burnt down towns and killed innumerable animals and plants in their wake, and flooding from the melting of glaciers is raising sea levels and water temperatures, and changing the coastlines of countries around the world (Fig. 17).2

Fig. 17.

Digital Amulet: Smart Necklace (2017). Photograph by Vibeke Sorensen

How might we tangibly perceive and experience such changes when they occur far away from where one lives? Remote sensing of data, physical-digital media and tacit knowledge offer possible technologies to enable people to become aware of distant changes not only through news reports, but by physical experience and immersion of the entire body. Sorensen approached this challenge with In Other Wor(l)ds (2018), an IoT installation consisting of six 2-sided large scale (5 m × 6 m, 60 K × 40 K pixels), semi-transparent tapestries embedded with custom digital technology affected in real-time by the Earth’s changing environmental conditions (i.e., wind and weather data). Networked from 5 different planetary biomes, the tapestries were hung in layers so that the audience could walk through and in between them, and tangibly “feel” the data on their skin. In this way, the actual air temperature and wind speed from the 5 biomes on Earth could be experienced through touch. Networked physical fans translated the real-time wind conditions of the biomes to blowing air surrounding the tapestries, causing them to move, simulating the wind in those regions. The air and fabric could then be felt directly on the skin of the viewers. It is a reversal of the dematerialization of physical data that Virtual Reality (VR) and Augmented Reality (AR) typically provide for immersion inside a projected 3D space. Instead, it translated environmental data into the material domain of physical textiles and touch in poetic ways so as to provide a tangible sensory experience of it. This work was inspired by the Chinese philosophy of Chi and Schoenberg’s quartet “I feel the air from another planet”, and suggests that that the energy of the Universe and the Earth’s environmental signals envelope us like the air we breathe, and the winds of other places move the fabrics of our lives. And like curtains in a window, they touch us physically, intellectually and emotionally in constantly shifting breezes (Herbert 1955).

Technically, the images printed onto the tapestries were computed using turbulence algorithms running in software such as Adobe AfterEffects and PureData (Pd/GEM). Like Morocco Memory II they incorporated poetry (text) as well as Earth based data. Each image printed was part of an animation sequence computed at 60,000 × 40,000 pixels which took more than 1 h per frame to calculate. Some moments in the sequences look like glass, a liquid turned solid, some like threads in motion in the wind, others with colours and shapes that flow like water, and all in dialog with the idea of air flow, not only as conceptual-poetic image “text-tiles” or “text-isles”, but with each other and the viewer/interactor as a whole. Each was embedded with networked, animating LEDs and custom electronics that constantly updated the air temperature, wind speed and direction from the biomes while the fans moved the air around them.

The images on the textiles are primarily explorations into non-objective, but very real, turbulent conceptual and emotional states connecting people with remote natural environments, which by means of physical installation are externalized into the local environment. They employ live natural data (wind speed, direction and air temperature) and human touch, displayed as moving light inside the tapestries and in the movement -and temperature- of the physical fabrics. For this installation, the tapestries are considered layers of space–time, and experiential visual-tactile bridges for reflection on interior and exterior worlds we inhabit, as well as immaterial and material experience of the global environment. The winds of other places are the winds that touch our bodies, and move the tactile textiles that brush against our skin, like curtains blowing in the wind. The tacit or implicit metaphor is the curtain in the window.

In Other Wor(l)ds employed 6 two-sided layers of super high resolution (60 K × 40 K pixels) digital images custom printed onto fabric as tapestries, each 6 m long by 5.3 m wide. Embedded LEDs in the fabrics displayed real-time air temperature and wind speed that came from networked sensors in five of the Earth’s major biomes (Fig. 18)(Table 1).3

Fig. 18.

In Other Wor(l)ds (2018) installed at the Beyond Festival at ZKM in Karlsruhe, Germany. Digitally printed tapestries with embedded LEDs and networked fans. 6 layers of large (6 × 5.3 m) 2-sided fabrics were created to display real-time temperature and wind data from 5 global environmental biomes as an immersive multimodal experience. (2 of the 6 layers were exhibited at ZKM.) Colour was coded to temperature (blue to white = cool to cold, and yellow to red = warm to hot). Wind speed was mapped to the speed of movement of coloured light across the LEDs. An online app allowed viewers both to track and select the biomes. By translating live sensor data into moving coloured lights and air movement, the fabrics could move in the wind like curtains in a window and brush against the skin, with LEDs bathing the viewer in coloured light. Photograph by Vibeke Sorensen

Table 1.

Relationship of colours to biomes in In Other Wor(l)ds

Biomes	Colours
Dry Desert (i.e., Phoenix, AZ; Cairo, Egypt; Xian, China; Las Vegas, NV)40 C	Yellow
Tropics (i.e., Singapore; Denpasar, Bali; Havana, Cuba) 30 C	Red
Arctic (i.e., Helsinki, Finland)20 C	Magenta
Savannah (i.e., Sofia, Bulgaria; Johannesburg, South Africa)10 C	Blue
Rainforest (i.e., Manaus, Brazil) 0 C	Grey

Embedded LEDs in the fabrics displayed real-time air temperature and wind speed that came from networked sensors in five of the Earth’s major biomes. The LEDs changed colour according to the remote temperature, and their movement according to the remote wind speed. The author confirms that all data generated or analysed during this study are included in this published article.

In addition to the installation, a website and app allowed the audience to select the biomes and track the changing data on their own devices, and in this way become even more connected and aware of the environments being monitored. Linking this to wearable technology, such as the Digital Amulet: Smart Necklace, could further connect the work in biofeedback with the wearer’s body, as an IOLT.

Ethically, the IOLT was carefully designed to not constrict, deceive or destroy life. In this case, no animal or plant died as part of the installation. The question of perception of the feelings, the sentience of specific living things being tracked, was a paramount concern. While there are many examples of human induced suffering of innocent animals facilitated by technologies that produce perceptual barriers for the people perpetrating it, there are also good examples of ethical uses of animal tracking. For example, radio bands worn by migratory birds allow scientists to the monitor population movements and losses, so as to address their condition in relation to global warming. The emergence of networked small sensors, including miniature ingestible ones used in medicine, for example, provide even greater potential for detecting the state of living things, as a global IOLT.

The Punan, the Rainforest, and the Tree Dress

Co-author J. Stephen Lansing, an anthropologist, is using the same technology to address human migration, working with the Punan people of Borneo, indigenous hunter-gatherers in Indonesia, in an effort to help them save their rainforests that are rapidly disappearing due to logging and palm oil plantation expansion (Fig. 19). To assist, Sorensen created unobtrusive belts embedded with GPS trackers for the people to wear, with their permission, so as to record and document their movements in the forests, to help the Nature Conservancy negotiate their legal claims to their ancestral homelands (Fig. 20).

Fig. 19.

Anthropologist J. Stephen Lansing with Punan people in Borneo, 2019. Photograph copyright J. Stephen Lansing

Fig. 20.

37 GPS tracks from 27 different Punan adults in their forests wearing Sorensen’s belts with GPS trackers, collected over 3 four-week periods (5 blue shades—collection 1, 15 red shades—collection 2, 17 purple shades—collection 3). Image courtesy of Dr. Guy Jacobs

Inspired by Punan and their traditional bark cloth and in solidarity with their plight, Sorensen created the Tree Dress (2020), returning to the importance of plants and in particular rainforest trees, their sentience and our collective human relationship to them (Figs. 21 and 22). The Tree Dress consists of digital panoramic photographs of a living tropical tree in Singapore which were printed onto sustainable silk, wearable technology, and embedded systems. (Singapore was chosen because Sorensen was living there during the COVID-19 pandemic lockdown in 2020 when travel was forbidden.) LEDs and circuits inserted into the dress displayed the continuous measurement of the O₂/CO₂, temperature, humidity, and light conditions of the tree in real-time. An “Internet of Living Things” (IOLT) garment, the computer system connecting the dress and the tree converted real-time gas emission data from the tree and its environment into sound (sonification and music), with visualization altering the colours of light emitted from LEDs in custom Arduino circuits. An app was created that allowed the wearer and networked participants to track the data, and be constantly aware of the tree’s condition. This work communicates not only the real-time scientific data but translates it into a poetic representation through wrapping onto the human body. The wearer can symbolically identify with a specific tree, and in this way “become one” with it. The tree’s enveloping bark is metaphorically “un-wrapped” and through digital media embedded into the sustainable silk, “re-wrapped” as a “second skin” for the person wearing the dress. By wearing a one-to-one representation of the tree’s “skin” and a light (LED) “necklace” displaying its “breathing”, it was intended to catalyze empathy with human beings and the trees they depend upon, and to help bring attention to this delicate and crucial relationship. As Nagaraju Thummanapalli, who provided technical assistance said, “you are taking the tree for a walk.” To which Sorensen replied, “and people are meeting the tree in a new way.” The Tree Dress thereby attempts to provide an alternative way to expand our awareness of the condition of the rainforests of Southeast Asia, and the human impact of their destruction.

Fig. 21.

Punan woman wearing bark cloth in the rainforest of Borneo, Indonesia, 2019. Photographs by J. Stephen Lansing

Fig. 22.

The Tree Dress (2020). A tree in the rainforest in Singapore was photographed using panoramic digital techniques, and the images were digitally stitched together and printed onto sustainable silk fabric. The same tree was tracked in real-time using networked sensors at the site of the tree. Data was transmitted wirelessly to a remote receiver, and custom Arduino/Lilypad soft circuits sewn into the dress converted it into changing colours displayed on the LEDs embedded in the dress, with data sonification and music played in wirelessly connected small speakers. A web app allowed people to track the tree and the Tree Dress online in real-time. Photographs by J. Stephen Lansing and Vibeke Sorensen

Conclusion

The transition from the Internet of Things to the Internet of Living Things is underway, and we are only beginning to grasp its implications. As the late Korean video artist, Nam June Paik said decades ago, “the real issue implied in art and technology is not to make another scientific toy, but how to humanize technology and the electronic medium.”4 Ethics and aesthetics are deeply entwined, and exploring the implications of human rational thinking that the computer is expected to enhance requires something more than rationality alone. The ability to elicit feeling in others, the experience of empathy and awareness of sentience, has always been central to aesthetics as it is realized in art and music. Modern Western aesthetics owes its origins to Kant, and it is instructive to recall that for Kant, Geist gives life to aesthetic forms. In this context, Geist is the faculty of presenting aesthetic ideas: that is, representations of the imagination which induce much thought but which cannot be rendered completely by definite thoughts: that is, by rational concepts alone. No such concept is adequate to Mind, the aesthetic idea. Instead there is a self-maintaining process by which art gives life to Mind. This is most clearly exemplified by music, which is not related directly to rational conceps, though there are many connections including mathematics. But who dares to find only rational concepts in a Mozart symphony (Herbert 1955)?

The order of ideas in music resembles the order of rational ideas, and vice versa. And rational music can be experienced aesthetically. But no solely rational concept can be experienced independent of the Mind which holds it, nor be adequate to musical and aesthetic ideas as internal conceptions. Narrative in the Mind is the meaning we give to the dynamic synthesis of the senses and memory, from which ideas in the Mind arise, and which in music is activated by hearing it. The internal aesthetic ideas permeate our entire being, an experience shared by others and connect us in ways that transcend purely rational concepts to give deeper meaning to our lives. With ethics and aesthetics, the IOLT can do this too.

Data availability

The author confirms that all data generated or analysed during this study are included in this published article.