Audio Intelligence

Multimodal Audio Intelligence Skill

Overview

This skill describes how to utilize Gemini's native audio understanding. Unlike standard pipeline models that transcribe audio via a separate automatic speech recognition (ASR) system before processing, Gemini consumes audio waveforms directly. This allows it to capture acoustic details like tone, speed, emotion, speaker transitions, and environmental sounds.

When to Use This Skill

• Transcribing speeches, interviews, podcasts, or call logs.
• Performing audio-based sentiment analysis (detecting anger, sarcasm, or nervousness).
• Sound classification (detecting police sirens, glass breaking, background rain).
• Dynamic Q&A over long-form lectures or recorded meetings.

Quick Start (with runnable code examples)

import time
from google import genai

# Initialize the Gemini GenAI Client
client = genai.Client()

def analyze_audio(audio_path: str):
    print("Uploading audio file to Gemini...")
    audio_file = client.files.upload(file=audio_path)
    
    # Wait for file processing to complete
    while audio_file.state.name == "PROCESSING":
        print("Waiting for audio processing to finish...")
        time.sleep(3)
        audio_file = client.files.get(name=audio_file.name)
        
    if audio_file.state.name != "ACTIVE":
        raise ValueError(f"Audio upload failed with state: {audio_file.state.name}")
        
    print(f"Audio ready! URL: {audio_file.uri}")
    
    # Query Gemini to transcribe and analyze the audio
    response = client.models.generate_content(
        model='gemini-2.5-flash',
        contents=[
            audio_file,
            (
                "Perform three tasks on this audio file:\n"
                "1. Provide a verbatim transcription.\n"
                "2. Analyze the speaker's emotional state, tone, and pacing.\n"
                "3. List any significant background noises or sound effects."
            )
        ]
    )
    
    print("\n--- Audio Transcription & Analysis ---")
    print(response.text)
    
    # Cleanup
    print("\nCleaning up audio resource...")
    client.files.delete(name=audio_file.name)

if __name__ == "__main__":
    # Example audio path
    analyze_audio("sample_audio.wav")

Advanced Usage

Audio Question-Answering (Temporal Spotting)

You can query the audio timeline to locate specific speech cues or extract timestamps:

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        audio_file,
        "What is the speaker discussing at the 02:45 mark? Answer in detail."
    ]
)

Key References

• Google GenAI File API Audio Guide
• Gemini Audio Capabilities Reference

Dependencies

• google-genai>=0.1.1
• pydub (optional, for local audio conversion/segmentation)