AI Music Generation

AI Music Generation

Overview

Two open-source AI music generation frameworks, each with different strengths:

Decision:

• AudioCraft when you need text prompts only, sound effects, melody conditioning, or stereo audio infrastructure
• HeartMuLa when you have specific lyrics and want a full song with tags (like Suno)

Section A: AudioCraft (MusicGen / AudioGen / EnCodec)

Quick Start

pip install audiocraft
# Or via HuggingFace transformers:
pip install transformers torch torchaudio

Text-to-Music (MusicGen)

import torchaudio
from audiocraft.models import MusicGen

model = MusicGen.get_pretrained('facebook/musicgen-medium')
model.set_generation_params(duration=8, top_k=250, temperature=1.0)

wav = model.generate(["happy upbeat electronic dance music"])
torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)

Using HuggingFace Transformers:

from transformers import AutoProcessor, MusicgenForConditionalGeneration
import scipy

processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small").to("cuda")

inputs = processor(text=["80s pop with bassy drums"], padding=True, return_tensors="pt").to("cuda")
audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
scipy.io.wavfile.write("output.wav", rate=model.config.audio_encoder.sampling_rate, data=audio_values[0, 0].cpu().numpy())

Model Variants

Melody-Conditioned Generation

model = MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=30)

melody, sr = torchaudio.load("melody.wav")
wav = model.generate_with_chroma(["acoustic guitar folk song"], melody, sr)
torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)

Text-to-Sound (AudioGen)

from audiocraft.models import AudioGen
model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=5)
wav = model.generate(["dog barking in park with birds chirping"])
torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)

GPU Memory Requirements

Section B: HeartMuLa

Hardware minimum: 8GB VRAM (--lazy_load true), recommended 16GB+.

Installation

git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
uv venv --python 3.10 .venv
. .venv/bin/activate
uv pip install -e .

# Fix dependency conflicts
uv pip install --upgrade datasets transformers

Patch Source Code

Patch 1 — In src/heartlib/heartmula/modeling_heartmula.py, in setup_caches, add RoPE reinitialization after reset_caches:

from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
for module in self.modules():
    if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
        module.rope_init()
        module.to(device)

Patch 2 — In src/heartlib/pipelines/music_generation.py, add ignore_mismatched_sizes=True to all HeartCodec.from_pretrained() calls.

Download Checkpoints

cd heartlib
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123'

Generate Music

. .venv/bin/activate
python ./examples/run_music_generation.py \
  --model_path=./ckpt --version="3B" \
  --lyrics="./assets/lyrics.txt" --tags="./assets/tags.txt" \
  --save_path="./assets/output.mp3" --lazy_load true

Tags format: comma-separated, no spaces: piano,happy,wedding,synthesizer Lyrics format: use bracketed structural tags: [Verse], [Chorus], [Bridge], [Outro]

Key Parameters

Pitfalls

1. Do NOT use bf16 for HeartCodec — degrades audio quality
2. Tags may be ignored — lyrics tend to dominate
3. Triton not available on macOS — Linux/CUDA only
4. No GPU → extremely slow on CPU (30-60+ min per song)
5. RTX 5080 incompatibility reported upstream

Links

• Repo: https://github.com/HeartMuLa/heartlib
• Models: https://huggingface.co/HeartMuLa
• Paper: https://arxiv.org/abs/2601.10547