Who: On-device Audio-Visual Speaker Diarization
On-device audio-visual speaker diarization for talking-head video: per-turn speaker IDs, face-track anchors, and millisecond timestamps.
Who is on-device audio-visual speaker diarization for talking-head video. One call returns Whisper-compatible turns with per-segment speaker IDs, face-track anchors, normalized face bounding boxes, and millisecond timestamps, suitable for cutting, captioning, and follow-the-speaker camera switching. It is video-first, combining audio and face tracking rather than diarizing pure audio.
Use cases
Follow-the-speaker editing
Cut or switch camera on speaker change with face context, for talking-head and multi-cam video.
Captioning
Attach speaker labels to Whisper-compatible transcript turns for captions and subtitles.
What it does
- Per-turn: speaker id, face track id, normalized face bbox, millisecond timestamps
- Whisper-compatible turns for captioning pipelines
- Video-first audio-visual, not pure audio
Specs
- Accuracy
- 9.87% DER on VoxConverse-dev (collar 0.25), vs 13.97% for the Silero+CAM++ baseline
- Size
- ~17 MB weights
- Platform
- Apple (Swift) first
Early access
Tell us what you are building and we will get you set up.