AI Real-Time Source Separation and Stems

Real-time AI source separation relies on deep neural networks trained on massive datasets of mixed and isolated audio. Models like Demucs, Spleeter, and Open-Unmix use spectrogram-based or waveform-based architectures to learn the statistical patterns of different sound sources. In a live context, the algorithm processes short audio frames (typically 10–50 ms) with low latency, outputting separate stems for vocals, drums, bass, and other instruments.

The key challenge is balancing latency and quality. For live monitoring, latency must stay below 5–10 ms to avoid perceptible delay. Current state-of-the-art models achieve this on dedicated hardware (e.g., GPUs or specialised DSP chips), but CPU-based systems often struggle. SSOUNDS engineers are evaluating FPGA-based accelerators that can run separation models with sub-5 ms latency, making them viable for front-of-house and monitor applications.

For live remixing, AI separation allows sound engineers to adjust levels of individual instruments or vocals in real time, even if the source is a stereo mix from a DJ or a backing track. This is particularly useful in festivals where multiple acts share a PA system but have different mix requirements. With AI, a FOH engineer can boost a weak vocal or reduce a muddy guitar without needing multitrack inputs.

In IEM systems, personal mixes can be generated from a single stereo feed. Each musician can independently control the balance of their own instrument, vocals, and other elements via a tablet or mixer app. This reduces the need for complex monitor consoles and multiple sends, simplifying setup for smaller tours and houses of worship. SSOUNDS is developing integration with Dante and AES67 networks to stream separated stems directly to wireless IEM transmitters.

Broadcasters can use real-time separation to isolate commentary from crowd noise, clean up remote feeds, or create multilingual stems for live translation. For example, during a sports event, AI can separate the announcer's voice from the ambient sound, allowing the broadcaster to adjust levels independently. Similarly, in live music streaming, viewers could choose a 'vocals only' or 'instruments only' mix.

However, broadcast latency requirements are less strict (typically under 100 ms), so higher-quality models can be used. SSOUNDS has tested cloud-based separation for streaming, but the added round-trip delay makes it unsuitable for live FOH. Edge computing on local servers offers a better balance for broadcast environments.

Despite rapid progress, real-time AI separation has notable limitations. The most significant is audio quality: separated stems often contain artifacts like 'bleeding' from other sources, metallic timbres, or loss of transients. These artifacts are more noticeable in complex mixes with many overlapping sounds. For critical live sound, the degradation may be unacceptable for main PA but acceptable for monitoring or broadcast.

Another challenge is computational cost. Running a high-quality separation model in real time requires a powerful GPU or dedicated AI accelerator, which adds expense and power consumption. Latency also increases with model complexity, forcing trade-offs. Additionally, the models are trained on specific instrument combinations and may fail with unusual arrangements or electronic music. SSOUNDS recommends using separation as a creative tool rather than a replacement for proper multitrack recording.

To integrate AI separation into a live sound workflow, the processing must occur at the right point in the signal chain. Ideally, the separation engine sits between the mixing console and the PA system, taking a stereo mix as input and outputting stems that can be routed to different outputs. SSOUNDS is developing a DSP plugin that runs on its network amplifiers, allowing separation to happen at the amplifier level with minimal added latency.

Engineers should also consider redundancy: if the AI processor fails, the system should fall back to the original mix. SSOUNDS systems include a bypass relay that preserves the unprocessed signal. As the technology matures, we expect AI separation to become a standard tool in the live sound engineer's arsenal, but it will complement rather than replace traditional mixing practices.