Audio Visualization

Audio Visualization in Remotion

Prerequisites

npx remotion add @remotion/media-utils

Loading Audio Data

Use useWindowedAudioData() (https://www.remotion.dev/docs/use-windowed-audio-data) to load audio data:

import { useWindowedAudioData } from "@remotion/media-utils";
import { staticFile, useCurrentFrame, useVideoConfig } from "remotion";

const frame = useCurrentFrame();
const { fps } = useVideoConfig();

const { audioData, dataOffsetInSeconds } = useWindowedAudioData({
  src: staticFile("podcast.wav"),
  frame,
  fps,
  windowInSeconds: 30,
});

Spectrum Bar Visualization

Use visualizeAudio() (https://www.remotion.dev/docs/visualize-audio) to get frequency data for bar charts:

import { useWindowedAudioData, visualizeAudio } from "@remotion/media-utils";
import { staticFile, useCurrentFrame, useVideoConfig } from "remotion";

const frame = useCurrentFrame();
const { fps } = useVideoConfig();

const { audioData, dataOffsetInSeconds } = useWindowedAudioData({
  src: staticFile("music.mp3"),
  frame,
  fps,
  windowInSeconds: 30,
});

if (!audioData) {
  return null;
}

const frequencies = visualizeAudio({
  fps,
  frame,
  audioData,
  numberOfSamples: 256,
  optimizeFor: "speed",
  dataOffsetInSeconds,
});

return (
  <div style={{ display: "flex", alignItems: "flex-end", height: 200 }}>
    {frequencies.map((v, i) => (
      <div
        key={i}
        style={{
          flex: 1,
          height: `${v * 100}%`,
          backgroundColor: "#0b84f3",
          margin: "0 1px",
        }}
      />
    ))}
  </div>
);

visualizeAudio() TypeScript Signature

// Required parameters:
type MandatoryVisualizeAudioOptions = {
  audioData: MediaUtilsAudioData;
  frame: number;
  fps: number;
  numberOfSamples: number;  // Must be power of 2: 32, 64, 128, 256, 512, 1024
};

// Optional parameters:
type OptionalVisualizeAudioOptions = {
  optimizeFor: "accuracy" | "speed";  // ONLY these two values, default "accuracy"
  smoothing: boolean;                  // default true
  dataOffsetInSeconds: number;         // required when using useWindowedAudioData
};

// Returns: number[] — values 0-1, left = bass, right = highs

Do NOT pass minFrequency, maxFrequency, type, or other non-existent parameters. The only valid optimizeFor values are "accuracy" and "speed" (NOT "frequency" or "waveform").

Important: When passing audioData to child components, also pass the frame from the parent. Do not call useCurrentFrame() in each child - this causes discontinuous visualization when children are inside <Sequence> with offsets.

Waveform Visualization

Use visualizeAudioWaveform() (https://www.remotion.dev/docs/media-utils/visualize-audio-waveform) with createSmoothSvgPath() (https://www.remotion.dev/docs/media-utils/create-smooth-svg-path) for oscilloscope-style displays:

import {
  createSmoothSvgPath,
  useWindowedAudioData,
  visualizeAudioWaveform,
} from "@remotion/media-utils";
import { staticFile, useCurrentFrame, useVideoConfig } from "remotion";

const frame = useCurrentFrame();
const { width, fps } = useVideoConfig();
const HEIGHT = 200;

const { audioData, dataOffsetInSeconds } = useWindowedAudioData({
  src: staticFile("voice.wav"),
  frame,
  fps,
  windowInSeconds: 30,
});

if (!audioData) {
  return null;
}

const waveform = visualizeAudioWaveform({
  fps,
  frame,
  audioData,
  numberOfSamples: 256,
  windowInSeconds: 0.5,
  dataOffsetInSeconds,
});

const path = createSmoothSvgPath({
  points: waveform.map((y, i) => ({
    x: (i / (waveform.length - 1)) * width,
    y: HEIGHT / 2 + (y * HEIGHT) / 2,
  })),
});

return (
  <svg width={width} height={HEIGHT}>
    <path d={path} fill="none" stroke="#0b84f3" strokeWidth={2} />
  </svg>
);

Bass-Reactive Effects

Extract low frequencies for beat-reactive animations:

const frequencies = visualizeAudio({
  fps,
  frame,
  audioData,
  numberOfSamples: 128,
  optimizeFor: "speed",
  dataOffsetInSeconds,
});

const lowFrequencies = frequencies.slice(0, 32);
const bassIntensity =
  lowFrequencies.reduce((sum, v) => sum + v, 0) / lowFrequencies.length;

const scale = 1 + bassIntensity * 0.5;
const opacity = Math.min(0.6, bassIntensity * 0.8);

Volume-Based Waveform

Use getWaveformPortion() (https://www.remotion.dev/docs/get-waveform-portion) when you need simplified volume data instead of frequency spectrum:

import { getWaveformPortion } from "@remotion/media-utils";
import { useCurrentFrame, useVideoConfig } from "remotion";

const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const currentTimeInSeconds = frame / fps;

const waveform = getWaveformPortion({
  audioData,
  startTimeInSeconds: currentTimeInSeconds,
  durationInSeconds: 5,
  numberOfSamples: 50,
});

// Returns array of { index, amplitude } objects (amplitude: 0-1)
waveform.map((bar) => (
  <div key={bar.index} style={{ height: bar.amplitude * 100 }} />
));

Postprocessing

Low frequencies naturally dominate. Apply logarithmic scaling for visual balance:

const minDb = -100;
const maxDb = -30;

const scaled = frequencies.map((value) => {
  const db = 20 * Math.log10(value);
  return (db - minDb) / (maxDb - minDb);
});