Thursday, July 03, 2003

Ambient Music as Interface

I'm going to tie together two recent threads here: ambient interfaces and game design elements. And the knot that ties them together is music.

I remember, more than ten years ago, playing Wing Commander, a sci-fi flight-simulation shoot-'em-up. It was notable at the time for its extremely cinematic experience (more so than the later movie, some noted). And although most comments were aimed at the video sequences that interspersed the actual interactive parts of the game, even the flying/shooting segments could be cinematic as well. This was primarily through the music.

Now Hollywood has known for a long time that music can heighten the emotional impact of visual sequences; we're now so accustomed to it than in a strange way, the absence of music is seen (or rather, heard) as being not realistic. But of course the music must be carefully tailored to the scene. (There's an exhibit in the Museum of Science in Boston that shows a scene with one of five different soundtracks, and you're supposed to guess what happens next based on the mood. It can be quite different, even though the video sequence is the same in all cases.)

What Wing Commander did so effectively is dynamically generate appropriate music for what was happening that moment. If you were cruising along from planet to planet, it played, well, cruising music. When the enemy showed up, the music became more urgent and tense; when your shields started failing, the music was more threatening; and when you finally blew away the last bogie, the music turned triumphant. The transitions were barely noticeable.

Now what's interesting about music is that while we do hear it and perceive its emotional message, it doesn't take up a great deal of our conscious attention. I didn't have to listen to the music, much less try to interpret it; I just knew, often subconsciously. And it's a good thing I didn't need to think about it, because I had a lot of other inputs to be paying attention to and commands to issue. Music is sort of the aural equivalent of Code-Division Multiplexing Access wireless protocol; a way to share more information in the same box of air.

Could we use this technology to convey more complex information? Hopefully, it's applicable in situations other than an attack by cat-like aliens. It could be dynamically generated, or it could just be selected from a large library. Imagine letting a computer select your music from your legally-obtained collection of digital music files, and it picks a certain kind of music when your portfolio is down (Yesterday, Eleanor Rigby), and a different kind when your portfolio is up (Twist and Shout, Eight Days a Week).

How precise could this channel be? How much bandwidth would it really have, in terms of shades or nuances of information? ("Your technology stocks are down, but biotech is doing well.") For a science fiction take on this (and because it's an amazing book), check out Jonathan Lethem's Gun, With Occasional Music. And Mark, if you're reading this, I want my copy back.