Interactive application (2020)

Tech: DDSP, Unity, C#, FMOD
Field: AI, Audio Generation, Sound Design, Interactive Media

The Neural Network Particle Choir is a research project, carried out together with the wonderful people at Kling Klang Klong, dealing with how a neural network can be trained and used to create complex and realistic audio.

The DDSP: Differentiable Digital Signal Processing system by Jesse Engel, Lamtharn (Hanoi) Hantrakul, Chenjie Gu, Adam Roberts was used first to train a model and later to perform a timber transfer from the model’s sonic qualities to external audio recordings in order to create new realistic choir voices.

In-depth explanation on the DDSP can be found here

This project made use of the demo examples available in the project’s git repository here

In the project’s last phase, selected audio files was employed through FMOD studio into Unity in order to create an immersive interactive application. Here, the user - through a first person view - is able to explore and experience the choir as generative spatial audio within a moving particle system.

The particles represents the different choir voices from which audio will be emitted if two or more particles are connected with each other.
A connection will only happen if the particles are within a certain distance. Which audio file that will be played is determined by the numbers of particles that it’s connected to.