MusicLM

MusicLM: Generating Music From Text |paper|dataset| Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, Christian Frank Google Research Abstract We introduce MusicLM, a model generating high-fidelity music from text descriptions such as "a calming violin melody backed by a distorted guitar riff". MusicLM casts the process of conditional music generation as a hierarchical sequence-to-sequence modeling task, and it generates music at 24 kHz that remains consistent over several minutes. Our experiments show that MusicLM outperforms previous systems both in audio quality and adherence to the text description. Moreover, we demonstrate that MusicLM can be conditioned on both text and a melody in that it can transform whistled and hummed melodies according to the style described in a text caption. To support future research, we publicly release MusicCaps, a dataset composed of 5.5k music-text pairs, with rich text descriptions provided by human experts. Audio Generation From Rich Captions Caption Generated audio Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. 1 2 3 Long Generation Text prompt Generated audio melodic techno Your browser does not support the audio element.swing Your browser does not support the audio element.relaxing jazz Your browser does not support the audio element. Story Mode The audio is generated by providing a sequence of text prompts. These influence how the model continues the semantic tokens derived from the previous caption. Text prompts Generated audio Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Text and Melody Conditioning By adding melody embeddings to the conditioning, we can generate music that respects the text prompt while following the provided melody. melody prompt → text prompt ↓bella ciao - humming Your browser does not support the audio element.bella ciao - jingle bells - whistling Your browser does not support the audio element.mozart symphony25 - whistling Your browser does not support the audio element.ode to joy - humming Your browser does not support the audio element.fingerstyle guitar Your browser does not support the audio element.jingle bells - marimba Your browser does not support the audio element.twinkle twinkle little star - piano Your browser does not support the audio element.when the saints go marching in - strings Your browser does not support the audio element.a cappella chorus Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.electronic synth lead Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.guitar solo Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.jazz with saxophone Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.opera singer Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.piano solo Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.string quartet Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element.tribal drums and flute Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Painting Caption Conditioning Painting title and author Painting image (from Wikipedia) Painting description Generated audio Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. 1 2 10s Audio Generation From Text Instruments Caption Generated audio acoustic guitar Your browser does not support the audio element.cello Your browser does not support the audio element.electric guitar Your browser does not support the audio element.flute Your browser does not support the audio element. 1 2 3 4 Genres Caption Generated audio 8 bit Your browser does not support the audio element.ambient Your browser does not support the audio element.berlin 90s house Your browser does not support the audio element.big beat Your browser does not support the audio element. 1 2 3 4 5 6 7 Musician Experience Level Caption Generated audio beginner piano player Your browser does not support the audio element.intermediate piano player Your browser does not support the audio element.professional piano player Your browser does not support the audio element.crazy fast professional piano player Your browser does not support the audio element. 1 2 Places Caption Generated audio beach in the caribbeans Your browser does not support the audio element.escaping prison Your browser does not support the audio element.gym Your browser does not support the audio element.opera Your browser does not support the audio element. 1 2 Epochs Caption Generated audio club in the 50s Your browser does not support the audio element.club in the 60s Your browser does not support the audio element.club in the 70s Your browser does not support the audio element.club in the 80s Your browser does not support the audio element. 1 2 Accordion Solos Caption Generated audio accordion death metal Your browser does not support the audio element.accordion edm Your browser does not support the audio element.accordion piano Your browser does not support the audio element.accordion rap Your browser does not support the audio element. 1 2 Generation Diversity We test the diversity of the generated samples while keeping constant the conditioning and/or the semantic tokens. Same Text Prompt Text prompt: Motivational music for sports Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. 1 2 3 4 5 Same Text Prompt and Same Semantic Tokens Text prompt: Motivational music for sports Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. Your browser does not support the audio element. 1 2 3 4 5