Dreaming Species… neural net

What would it sound like for humans and animals to sing together?

Would it be possible to create a new form of voice that stages that relationship, in all its complexity?

What would that even sound like?

Dreaming Species, and the wider project it emerges from, turns to another form of intelligence, artificial intelligence, as a way to broker a new frame for these questions. Not to find a definitive answer, but to play with a new way of listening.

The music for this project has been created in part through the use of neural synthesis. Neural synthesis is a computer based approach to generating content (e.g. text, image, MIDI, raw audio, or even video) using Artificial Neural Networks.

A Neural Network, or neural net, is a system made up of artificial neurons, modelled closely on the operations of cells in biological systems such as the brain. Neural nets are typically built from several layers, each made up of one or more neurons. Each layer may frame several internal or ‘hidden’ layers, with a single such layer consisting of several hundred or perhaps even thousands of cells.

In order for a neural network to learn it must be trained, which involves repeatedly exposing the network to a set of training data, that may take several hours, or even days. In this project the training data was made up of voice examples from animals and humans singing.

I collaborated with Dr Christopher Melen to explore the possibility of creating a human/animal hybrid voice. Dr Melen was the Research Software Engineer at PRiSM (the Centre for Practice & Research in Science & Music at the Royal Northern College of Music) between 2019 - 2023. The neural net at PRiSM was developed as an open source software tool, by modernising and reimplementing a pre-existing neural net known as SampleRNN. SampleRNN generates audio based upon learning any corpus of sound or music, which is perfect for working with actual recordings of animal and human voice.

This article by Dr Melen provides further insight into SampleRNN.

Training

Creating an AI voice from human and animal sources takes place over a number of stages:

Stage One: creating a training data set

Collection of audio examples of a wide range of animal voices, along with existing examples of my own music.

Stage Two: first training with SampleRNN

Dr Melen generates initial material, resulting in 70 x 30 second audio responses from SampleRNN (examples).

Stage Three: responding to SampleRNN

Listening to SampleRNN’s material, making selections of most interesting elements and composing music based on motifs heard.

Stage Four: creating a second training data set

Collaborating with sopranos Peyee Chen and Héloïse Werner to record them singing the material SRNN had proposed, the music I had composed, and the animals from the first training data.

Stage Five: second training with SampleRNN

Dr Melen generates secondary material, with longer ten minute responses (examples).

Stage Six: taking the voices forward

Composing with the material created by SampleRNN, Peyee, Héloïse and the animals to create soundworlds for two inter-twined projects:

Dreaming Species: a digital project that asks initial questions

Oracle: a live performance for six voices that answers them
This article offers a more in depth description of the training process.

Machine Learning for Music

Developments in machine learning for music are moving quickly. Composing with SampleRNN is akin to using a particular sounding instrument, created at a particular moment in the development of the tool. Things sound different already.

Here are some musicians and writers who inspired me as I learned:

Holly Herndon & Mat Dryhurst
dadabots
Jennifer Walshe
Sam Salem
K Allado-McDowell
Pauline Oliveros

Credits

prism-samplernn
Led by Sam Salem and Christopher Melen
Initiated by Sam Salem
prism-samplernn code by Christopher Melen

A PRiSM Collaboration also involving David De Roure, Marcus du Sautoy, and Emily Howard.

The RNCM Centre for Practice & Research in Science & Music (PRiSM) is funded by the Research England fund Expanding Excellence in England (E3).

Melanie Wilson
creator of Dreaming Species