How Does the NSynth Algorithm Generate New Sounds?

Updated

NSynth uses a deep neural network to learn the acoustic characteristics of individual sounds, then synthesizes a completely new sound from those learned qualities rather than blending or mixing the originals. The result can be something that is, in Google Magenta's own example, "part flute and part sitar all at once"—a single new timbre that inherits traits of both sources without being a mashup of the two recordings.

What Makes NSynth Different from Blending

The distinction matters if you are trying to understand why NSynth Super sounds unlike a sampler or a crossfade tool.

Approach What happens to the audio Result
Blending / combining Two or more recordings are mixed or crossfaded You hear both sources layered or fading between each other
NSynth synthesis A neural network learns each sound's characteristics and generates new audio from them You hear one new sound carrying acoustic qualities of the sources

With blending, the source recordings remain audible as separate ingredients. With NSynth, the algorithm produces audio that did not exist in either input—the sources contribute qualities, not layers.

How the Algorithm Works, Step by Step

  1. Collect source sounds. Sounds are recorded and fed into the model. In the NSynth Super experiment, 16 original source sounds across a range of 15 pitches were recorded in a studio.
  2. Learn sound characteristics. A deep neural network learns the characteristics of those sounds rather than storing them as playable samples.
  3. Synthesize new sounds. The algorithm generates entirely new sounds using the acoustic qualities it learned from the originals.
  4. Precompute for performance. Because generation is computationally heavy, the outputs are calculated ahead of time. For NSynth Super, that produced over 100,000 new sounds, which were then loaded into the instrument.

That last step is the practical bridge between the algorithm and an instrument: the model does the generation work in advance, and the player explores the results in real time.

How This Plays Out in NSynth Super

NSynth Super is the physical interface built around this algorithm—an open source experimental instrument created by Magenta in collaboration with Google Creative Lab. It gives musicians access to the precomputed sounds from four source sounds at a time:

  • Each dial is assigned four source sounds.
  • Turning the dials selects which source sounds to explore between.
  • Dragging a finger across the touchscreen navigates the new sounds that combine the acoustic qualities of those four sources.

So the algorithm's job is to generate the sound space; the instrument's job is to let a musician move through it. The experience prototype was shared with a small community of musicians to understand how they might use it in their creative process.

What to Keep in Mind

  • It is an experiment, not a product line. NSynth Super is described as part of Magenta's ongoing research into how machine learning tools can help artists create art and music in new ways. The prototype pictured on the site was a limited experience build.
  • The sound set is fixed at build time. Since sounds are precomputed, the palette you play is the one generated from the recorded sources—not an infinite live synthesis.
  • "New sound" is the point. If you want to layer or morph familiar recordings, conventional samplers do that directly. NSynth is for timbres that sit between your sources in a way mixing cannot produce.

For musicians, the takeaway is that NSynth treats sound design as a learned property rather than a editing operation: you choose sources, and the network invents the territory between them.