• AlloSphere & AlloLib:
The AlloSphere is a three-story full-surround, multimodal, immersive facility in the Media Arts and Technology at the University of California, Santa Barbara to represent large and complex data, including immersive visualization, sonification, and interactivity.
AlloLib is a cross-platform suite of C++ components for building interactive multimedia tools and applications.
- Developing and improving the overall system
(graphics/acoustics/interface/network in C++)
• Multi-user interactive system in virtual 3D spaces using the gesture of smartphones:
- Developed general-purpose gesture-based smartphone 3D interface (C++).
- Developing a multimodal 3D interface with pattern recognition using signal processing and machine learning.
• Music source conducting based on gestural control using machine learning:
Reproduction of acoustic signals based on the interpretation of conducting gestures using controllers with accelerometers and gyroscopes.
- Developed and performed machine learning and signal processing based interactive-musical system
(MATLAB, Python, C++).
- Designed and made a baton controller prototype.
- Improved the work using the smartphone with the gestural sensors.
• Machine learning
- Performed experiments on the application of various machine learning techniques
with Tensorflow, Python, Kaldi, and MATLAB
- Applications: reverberation time estimation, acoustic models, jointly trained neural network, bandwidth expansion, and gesture interpretation.- Contributed distributive research with Samsung electronics.
• Summer internship in Nokia Bell Labs. Experiments in Art and Technology (EAT)
- Acoustic signal processing and machine learning task
• Reverberation time estimation using machine learning:
Obtaining room acoustic information from sound sources received by microphones.
- Conducted an intensive study with a single & multi-channel based algorithm
using a deep neural network. (Published in IEEE IC-NIDC and Acta Acust united Ac)
- Developed an estimation algorithm for dereverberation and acoustic model
- Contributed distributive research with LG electronics.
• Crosstalk cancellation:
Reproducing audio to utilize binaural audio signals with two loudspeakers by
offsetting the crosstalk components
- Developed crosstalk cancellation algorithms based on HRTF with low sensitivity.
- Participated in a national project exclusively; Information Technology Research Centre: Center for Ultra Realistic Audio Technologies (ITRC: CURAT)
• Multi-channel audio spatialization:
Synthesis of audio signals with directivity using multiple loudspeakers.
- Developed an adaptive multi-channel audio spatialization algorithm (MATLAB)