Skip to main content

Speaker Diarization with Pyannote on Vast.ai

Speaker diarization partitions an audio stream into segments according to speaker identity-identifying “who spoke when” in multi-speaker recordings like meetings, podcasts, or interviews. This guide walks through using PyAnnote Audio for speaker diarization on Vast.ai.

Prerequisites

When to Use Diarization

Speaker diarization answers “who spoke when”-it doesn’t transcribe what was said. Use diarization when you need to:
  • Attribute transcribed text to specific speakers
  • Analyze speaking patterns (talk time, interruptions, turn-taking)
  • Split multi-speaker audio into per-speaker segments for downstream processing
For full transcription with speaker labels, combine diarization with a speech-to-text model like Whisper: run diarization first to get speaker timestamps, then transcribe each segment.

Hardware Requirements

Pyannote’s speaker diarization model is efficient and runs on modest hardware:
  • GPU: RTX 3060, 4060, or similar
  • VRAM: 6-8GB
  • System RAM: 8-16GB
  • Storage: 10GB minimum
  • CUDA: 11.0+
  • Python: 3.8+

Setting Up the Instance

  1. Go to Vast.ai Templates
  2. Select the PyTorch (CuDNN Runtime) template
  3. Filter for an instance with:
    • 1 GPU
    • 6-8GB VRAM
    • 8-16GB system RAM
    • 10GB storage
  4. Rent the instance
  5. Install the Vast TLS certificate in your browser
  6. Open Jupyter from your instances

Creating a Notebook

  1. In JupyterLab, click File → New → Notebook
  2. Select the Python 3 kernel
  3. Run the following cells in your notebook

Installing Dependencies

Install the required Python packages:
Install FFmpeg for audio processing:

Downloading Test Data

This example uses a sample from the AMI Meeting Corpus dataset:

Running Speaker Diarization

Initialize the Pipeline

Load the pretrained diarization model and move it to GPU for faster processing:

Process Audio

Run diarization on an audio file. The pipeline returns timestamped speaker segments:
Example output:

Analyzing Results

Calculate Speaking Time per Speaker

Example output:

Detect Overlapping Speech

Filter by Speaker

Extracting Speaker Segments

This utility function splits the original audio into separate files for each speaker segment, useful for downstream processing like transcription:

Playing Audio Segments

Verify results in Jupyter:

Additional Resources