Making music is an arduous process, as any musician will testify. The SoundTracker project enables anyone to experience the process of music making, but in a less demanding and more intuitive way – moving in space.
Like a player studying his instrument, first each person will explore his movement, and understand how it affects the music he creates in the space. And like a band whose members learn to play together, the participant will play with his temporary partners and will have to adapt his movement, and thus his playing, to theirs, in order to achieve harmony.
The SoundTracker project tracks players inside a room, using Android based tablets. A 3D mapping of the individuals is extracted, which is then used to generate music.
Each person plays a role in the group, and his movements in the room (i.e. - his location) dictates how the instruments he controls sound
Our aim is to allow any person to get a sense of playing an instrument, both by himself and as part of a group, in the easiest way possible - by simply moving.
We attempt to maintain the balance between controlling the music in order to achieve harmony and giving the players more freedom at the risk of creating a cacophony.
Our first and foremost task is to track the players inside the room.
We use Qualcomm's AR SDK in the FrameMarkers android app in order to track the markers that the players wear. The application sends that data to a server which then processes it.
Each camera provides a pose matrix for the markers it detected in regards to the camera itself.
The Banjii project transforms those matrices into a 3D mapping of each player, cross-referencing and smoothing the various inputs and translating them into a single 3D image.
A band is not homogenous, and neither is the music generated by SoundTracker.
As in a band, there are both backup roles and leading roles. The Cintie project transforms the coordinates of the
players (i.e. - their movement) differently for each role.
In order to help the players recognize the affect of their movement on their instrument, the Cintie module generates surround sound, following each player around the room through the speakers, playing his instruments in the speaker he is closest to. It achieves this by using the OpenAL open source project (modified by Uri to work with Java)
Since our tablets are hanging from the ceiling, rebooting them in case the program crashes can be a hassle. In order to tackle this problem, we created Doggie, a small watchdog app that starts up FrameMarkers on startup and reloads the application if it crashes.
In 2012, Google Israel and the Fresh Paint art exhibition project held an "unconference" at Google's offices, in an attempt to
bring the developer community and the art community together and create digital art for the upcoming Fresh Paint 5 exhibition. During this "unconference",
various artists presented their ideas and asked developers to join them in bringing their vision to life. Some artists had a well formed idea and only needed help
implementing it, while others has a more abstract notion of what they wanted to achieve, and were looking for the developers' input as well.
Soundtracker, on the other hand, was born in a different manner. During the "unconference", Uri came up with the idea of translating motion in a room into sound.
We got the organizers' blessing and began working on it. Later on, Fresh Paint introduced us to Assaf Twina, who provided the musical knowledge we required and who created the tracks which SoundTracker plays.
SoundTracker was displayed in Fresh Paint 5 between May 14-19, 2012.