Flong

Printed from www.flong.com/texts/interviews/interview_mitp_messa/
Contents © 2020 Golan Levin and Collaborators

Golan Levin and Collaborators

Interviews and Dialogues



Tmema Interview for Processing: A Handbook by Reas & Fry

Tmema (Golan Levin and Zachary Lieberman), December 2005.

What is Messa di Voce?

Messa di Voce is an audiovisual system in which the speech, shouts and songs produced by two vocalists are augmented in real-time by custom interactive visualization software. The installation touches on themes of abstract communication, synaesthetic relationships, cartoon language, and writing and scoring systems, within the context of a sophisticated and playful virtual world.

Our software transforms every vocal nuance into correspondingly complex, subtly differentiated and highly expressive graphics. These visuals not only depict the users' voices, but also serve as controls for their acoustic playback. While the voice-generated graphics thus become an instrument which the users can perform, body-based manipulations of these graphics additionally replay the sounds of the users' voices thus creating a cycle of interaction that fully integrates the visitors into an ambience consisting of sound, virtual objects and real-time processing.

Messa di Voce lies at an intersection of human and technological performance extremes, melding the unpredictable spontaneity of the unconstrained human voice with the latest in computer vision and speech analysis technologies. Utterly wordless, yet profoundly verbal, Messa di Voce is designed to provoke questions about the meaning and effects of speech sounds, speech acts, and the immersive environment of language.

Why did you create Messa di Voce?

Messa di Voce grew out of two prior interactive installations that we developed in 2002: "RE:MARK", which explored the fiction that speech could cast visible shadows, and "The Hidden Worlds of Noise and Voice", a multi-person augmented reality in which the users' speech appeared to emanate visually from their mouths. These installations analysed a user's vocal signal and, in response, synthesized computer-graphic shapes which were tightly coupled to the user's vocal performance. After making these pieces, we had the feeling that we hadn't taken full advantage of everything we had learned about analyzing vocal signals. And although RE:MARK and Hidden Worlds were reasonably successful with a general audience, we wanted to step up to a much greater challenge: could we develop voice-interactive software that could somehow equal or keep pace with the expressivity of a professional voice artist?

We invited the well-known experimental vocalist/composers, Joan La Barbara and Jaap Blonk, to join us in creating the Messa di Voce performance. Although Joan and Jaap come from very different backgrounds — she works in 'contemporary art music', while he comes from a background in sound poetry — both of them share a practice in which they use their voices in extremely unusual and highly sophisticated ways, and both use a visual language to describe the sounds they make. The software was really designed in collaboration with them; there are even specific sections or modules of the software that were directly inspired by improvisation sessions that we held together. Once the performance was finished, we realized that some sections could only ever be performed by trained experts like Joan and Jaap, but that other software modules could actually be experienced by anyone uninhibited enough to get up and yell or sing. We gathered up five or so of these — about a third of the original concert software — and that's how we re-developed Messa di Voce into an installation. We're proud that these software pieces could be used to good effect by expert vocalists, but even more pleased, in a way, that children can enjoy them too.

What software tools were used?

We developed Messa di Voce in C++, using the Metrowerks Codewarrior development environment. Some of the sound analysis was accomplished with Intel's commercial IPP library. We also incorporated a large number of open-source sound and graphics toolkits, including OpenCV, OpenGL, and portAudio.

Why do you choose to work with software?

Because software is the only medium, as far as we know, which can respond in real-time to input signals in ways which are continuous, linear or non-linear as necessary, and — most importantly — conditional. The medium that we're interested in, to borrow a quote from Myron Krueger, is response itself, and only software able to respond in such a rich manner and with such a flexible repertoire. Is this answer too simple?

Why did you write your own software tools?

This is always a clear decision for us: because there isn't any other software, written by someone else, which does what we want it to do — and most importantly, which does it in the way that we imagine it could be done. In the specific example of Messa di Voce, although a significant aspect of the project is entirely conceptual (the idea of visualizing the voice in such a way that the graphics appear to emerge from the user's mouth), an equally important dimension is the quality and degree of craft that is applied to the creation of the work, and which is evident in its execution. Although the idea of Messa di Voce could have been implemented by any number of other artists (and indeed, systems illustrating related ideas have been created by others, such as Toshio Iwai, Josh Nimoy, Mark Coniglio, and Steven Blyth), we'd like to believe that nobody else could have created it with the character and texture we did.

That said, it would be a mistake to believe that we wrote Messa di Voce completely from scratch. As we mentioned earlier, we made extensive use of both commercial and open-source software libraries in order to develop it. It's not even clear what "completely from scratch" would mean for our project, unless we were to somehow construct our own CPU and develop our own assembly language for it! We incorporated features and functionality from the other software libraries whenever we didn't know how to do something ourselves, or could simply save time by doing so. Our work was built on the efforts of probably thousands of other people.