Virtual Interactive Puppetry

"YONG-SHIN-GUD (calling-dragon-spirit) is my digital translation of the ancient

puppetry. This virtual interactive puppet performance consists of the artist's

performance with live music, storytelling and 3D motion graphics, and the audience's

participation. "Yong-Shin-Gud" is a Korean word, named for a special shamanic ritual

that evokes dragon spirit. Same with original meaning, my live puppetry is a special

ceremony to embody the dragon spirit in multimedia performance, shaping the

constant process of transformation and spiraling phases of interaction [4].

Yong-Shin-Gud starts with 3 components: the virtual world, the real space of the

performance room, and a sound activated puppet, "Virtual Shaman," who mediates

between the virtual and the real, in the same way of Korean Shaman ritual, "Gud",

composed of the binary pairs and Shaman. The virtual sound activated puppet

eventually acquires the ultimate trans-state of shaman, by spiraling interactive

dialogues with the real puppeteer. Yong-Shin-Gud explores the most intimate and

spiraling connections inherent in human-computer interaction (HCI), using a form of

traditional puppetry and shamanic ritual.

Unlike traditional puppet, virtual puppet in Yong-Shin-Gud explores the more

intuitive way of puppet interaction. Virtual puppet is motivated by sound, not by

string, rod or hand triggers. Any sound input coming into the microphone, such as

musical instrument sound and storytelling motivate the mouth, body and facial

expression of 3D virtual puppet on screen, in real time. The virtual puppet constantly

speaks and sings back to the puppeteer, like real-time echoes and mirror reflections.

Whereas the traditional puppeteer requires having a certain trained skill to

manipulate the puppet, my virtual puppet provides us immediate knowledge of how

the system works. In this complete hands-off experience, the puppeteer transformed

to the puppet in a second, with strong attention to what puppet speaks about and

how. Anybody can play with virtual puppet instantly, telling his or her story through

the virtual puppet's mouth.

Virtual Shaman: sound activated puppet comes after the primitive expression of

Korean traditional puppet. Lacking realistic details, virtual shaman's appearance

connects us with the remote time of totem pole [5]. It stimulates the viewer's

perception of the figure in question and encourages the act of imagination based on

the abstracted signs of life [6]. Virtual puppet leads viewers to imagine. The viewer's

imagination completes the figure and life of the virtual puppet. It is like the eastern

traditional painting with blank space that stimulates viewers involved more into the

painting, and allows them to complete the story of painting.

Virtual Shaman was created by using open source, Antoine Schmitt asFFTXtra in 3D

interactive application. The asFFT Xtra computes a power spectrum using a FFT (Fast

Fourier Transform) on the input sound samples. This live sound spectrum was

applied to the 3D puppet's mouth, body and texture with different type of sensitivity

and controls. Most of all, real time mouth interaction has a significant role to make

virtual puppet alive. The sign system of speech is vital importance to puppetry,

allowing people to imagine the puppet as having life [7]. Even though primitive up

and down mouth response, it makes virtual puppet look exactly lip-syncs with the

puppeteer's voice and musical instrument play. However, what truly makes the

puppet alive is not the technical control over the puppet's physical movement, but

the user (puppeteer)'s spiritual engagement and "the willing suspension of disbelief"

towards the presence of the virtual puppet. The audience's willingness makes the

leap from the perception of representative abstractions to the imagination of life [8].

The audience chooses to imagine and complete the puppetry. This willingness only

exists when the distinction between puppet and puppeteer is abolished. It came true

by performing the ritual of interaction.

YONG-SHIN-GUD shapes the ongoing real -time lip-sync process and eventually

erase the line between puppet and puppeteer. As yin and yang continuously

exchange their position side by side without hierarchy, the real puppeteer becomes

the virtual puppet and the virtual puppet becomes the real puppeteer in cycles. This

dynamic demonstrates my perspective of the body as a strong spiritual entity that is

continuously transformed into our binary reflections, seen as objects, images, and

puppets. The puppetry starts with a simple physical motivational element (the lip-

syncing) with which to interact; and then the whole interactive process becomes

more and more powerful by the puppeteer's increasing spiritual engagement with the

puppet. This back and forth between physical motivation and spiritual activation

constitutes the magical part of puppetry: The puppeteer's voice motivates the

puppet's mouth. The puppet lip-syncs with the puppeteer. The puppeteer becomes

the puppet and talk through the puppet. The puppet's voice motivates the

puppeteer's mouth. The puppeteer lip-syncs with the puppet. The puppet becomes

the puppeteer and speaks through the puppeteer. The puppeteer's voice motivates

the puppet's mouth

For this process, my project requires that the performer watch carefully her puppet's

continuous response on screen, whereas it is not necessary for the sound performer

to watch the image unfolding on the screen in the most of audio-visual interactive

performances. The performer should face her puppet on the screen, as the puppet's

response will gradually motivate the performer's musical or storytelling inspiration.

The audience should be located between the performer and the puppet, thus placed

physically, and spiritually, between this spiraling inner dialogue of the puppet and

the puppeteer.

It is magical how much the puppeteer merges into the puppet on the membrane of

screen. Without seeing the puppet, the puppeteer can't possibly play the music with

the same energy. The melody she plays is not generated by her brain and

consciousness-it's not her melody at all. Rather, the melody is created between the

puppet and her. Her sound inspires the puppet and its movement inspires her in

return. They continue to inspire each other and play a melody that is neither solo nor

duet. The puppeteer falls in love with her puppet during this cycle. The more she

loves the puppet, the more the puppet loves her. The more her puppet loves her, the

more she loves her puppet. As a Shaman begins to build its relationship with an

object, so too does the puppeteer build its relationship with the virtual puppet. The

relationship begins in a microscopic way, but it soon expands step by step, becoming

a gigantic scale that fills up the entire world. The puppeteer communicates with the

puppet, with themselves, with the audience, with the spiritual world and with the

whole universe. The Inner space communicates with the outer space. The energy of

ritual spirals up and up.

In Korean traditional puppet play, puppeteers all belongs to the lowest social class

who were oppressed mistreated or even persecuted [9]. Puppet plays were vehicles

for their satire and criticism in society. I think contemporary people also need

vehicles for free speech. No matter what classes they are in, they have been much

socialized and oppressed by invisible structure of society. Transformed to the puppet,

they can discover to speak something that was not revealed in his/her everyday life.

It can be criticism or free imagination that doesn't appear in their daily social

patterns. Puppet can be functioned as a revolutionary interface to bring people's free

speech and imaginative storytelling in this contemporary time.

Following this tradition, Yong-Shin-Gud is the public puppet performance forum to

bring people's true voice and imagination. It awakes our forgotten tradition of oral

storytelling and reinstalls it in our visually oriented cyber-culture. "Once upon a time,

long long time ago . . .. " This is the opening sentence given to audiences when they

interact with the Virtual Shaman before and after the artist's musical and storytelling

performance. It is the secret interface to open the door of inner communication and

improvisation. People's storytelling coming out of the virtual puppet's mouth is

unexpected. It is unexpected hybrid created between their puppet and themselves.

Virtual shaman helps people to tell and also listen their story.

"��Puppet could tell the story. Puppet could tell him how to tell stories. So the big

man and the puppet could tell the stories together. And this is how the big man

learned to tell the stories��" ---a part of stories told by an audience through the

virtual puppet.

The Virtual Shaman is designed for anybody, not only for the trained artist and the

technician. It uses simple technology that works perfectly in every environment.

Everyone who enjoys puppetry can come to the microphone and be a storyteller, a

puppeteer and, eventually, his/her puppet. The dynamic dialogue between puppet

and puppeteer brings forth the energy of free imagination and improvisation in this

performance forum. As in the shamanic ritual, such a spiraling interaction ultimately

erases the line of distinction between all the binary pairs, such as artist/audience,

reality/virtuality, mundane/spirit, east/west and puppet/puppeteer. YONG-SHIN-

GUD, as a contemporary shamanic ritual, frees all those who are oppressed in our

predominantly structure- oriented society.