Smart Vision & Robotic Sensing
Smart Innovation Program, Graduate School of Advanced Science and Engineering
Virtual Musical Instrument Using High-speed Vision
To demonstrate real-time VA, a virtual musical instrument system was developed. The system proposed in this research consists of a high-speed megapixel vision, a computer for producing a virtual musical instrument model, and an audio presentation device such as a MIDI sound source. By replacing the model, various virtual musical instruments can be realized. Some virtual musical instrument models are introduced below.
- Virtual guitar
The virtual guitar model generates guitar sounds according to human hand movements. This model only notes the right-hand movement strumming virtual guitar strings and automatically determines the scale for the left-hand movement. When a marker at the tip of the right-hand thumb of the player, a sound is generated at the moment when the coordinate of the marker passes the position of the virtual guitar string, and its volume is determined from the passing velocity.
- Virtual xylophone
The virtual xylophone model generates a sound depending on the positional relationship of the two mallets held by the player and the virtual xylophone in space. In this model, makers are set at the tips of the two mallets. When the height coordinate of a marker enters a specified area, a key strike is assumed. This model determines from the coordinates which key was struck and generates the corresponding sound. Since the two mallets are processed independently, different key sounds can be generated simultaneously.