RESEARCHES

In this study, the concept of Visual Auditorization (VA) for converting visual sensor information into audio presentation information is proposed for real-time interaction. The concept of VA is to present visual sensor information as audio information by collecting the position, velocity, and acceleration information of a target in an actual environment through a visual sensor, converting the information from virtual audio information from the target, and to output this information to an audio presentation device. To realize this concept, a high-speed vision system fully capable of processing audio information in a frequency band of approximately several kilohertz to tens of kilohertz is important.

To demonstrate real-time VA, a virtual musical instrument system was developed. The system proposed in this research consists of a high-speed megapixel vision, a computer for producing a virtual musical instrument model, and an audio presentation device such as a MIDI sound source. By replacing the model, various virtual musical instruments can be realized. Some virtual musical instrument models are introduced below.

• Virtual guitar

The virtual guitar model generates guitar sounds according to human hand movements. This model only notes the right-hand movement strumming virtual guitar strings and automatically determines the scale for the left-hand movement. When a marker at the tip of the right-hand thumb of the player, a sound is generated at the moment when the coordinate of the marker passes the position of the virtual guitar string, and its volume is determined from the passing velocity.

• Virtual xylophone

The virtual xylophone model generates a sound depending on the positional relationship of the two mallets held by the player and the virtual xylophone in space. In this model, makers are set at the tips of the two mallets. When the height coordinate of a marker enters a specified area, a key strike is assumed. This model determines from the coordinates which key was struck and generates the corresponding sound. Since the two mallets are processed independently, different key sounds can be generated simultaneously.