AI Real-Time Source Separation and Stems

At its core, real-time AI source separation uses deep neural networks trained on massive datasets of mixed audio and their isolated stems. Models like Demucs, Spleeter, or proprietary algorithms analyze the frequency, phase, and temporal patterns of the incoming mix to predict which components belong to each source. Unlike traditional filtering, which relies on fixed frequency bands, AI models learn the unique signatures of instruments and voices, enabling them to separate sources that overlap in frequency.

In live applications, the audio stream is buffered into short frames (typically 10–50 ms), processed by the neural network, and output as separate stems. The key challenge is achieving this with low enough latency for foldback (IEM) or broadcast use — typically under 10 ms for monitoring. Modern GPUs and dedicated AI accelerators (e.g., NVIDIA TensorRT, Apple Neural Engine) make this feasible, though CPU-based solutions still struggle with higher latency.

For live sound engineers, real-time stem separation opens up new creative possibilities. Imagine isolating the lead vocal from a backing track to add reverb or delay without affecting the band, or extracting a guitar solo to send it to a separate effects chain. DJs and electronic musicians can remix live performances by muting or soloing stems on the fly, creating unique mashups or extended versions.

In festival and concert settings, this technology can also be used to generate separate feeds for broadcast or streaming. For example, a broadcast mixer might need a clean vocal stem for TV while the house mix remains unchanged. SSOUNDS sees this as a potential value-add for its DSP ecosystem, allowing engineers to route separated stems to different outputs without additional hardware.

Another consideration is artifact quality: if the separation introduces audible glitches, phasing, or 'bleed' between stems, it can ruin the monitor mix. Current state-of-the-art models achieve near-transparent separation for well-recorded material, but challenging mixes (e.g., heavy distortion, overlapping vocals) still produce artifacts. Engineers must weigh the benefit of flexibility against potential degradation.

Despite rapid advances, real-time AI source separation is not yet a plug-and-play solution for every live scenario. The primary limit is latency: even the fastest models on high-end GPUs introduce 5–20 ms of delay, which can be problematic for monitoring or time-sensitive broadcast. For FOH (front-of-house) use where latency is less critical, this is acceptable, but for IEMs, every millisecond counts.

Artifacts are another concern. AI models can produce 'musical noise' — faint, unnatural sounds that resemble the original source but are slightly off. In a live mix, these artifacts can accumulate across multiple stems, degrading overall sound quality. SSOUNDS recommends using stem separation primarily for auxiliary purposes (e.g., broadcast feeds, recording) rather than as the sole source for critical monitor mixes until the technology matures.

Processing power is also a barrier. Running a neural network in real time requires a dedicated GPU or accelerator, adding cost and complexity to a sound system. Cloud-based solutions introduce too much latency, so all processing must be local. SSOUNDS is exploring integration with next-generation DSP chips that include AI inference cores, which could make stem separation a standard feature in future amplifiers and loudspeaker controllers.

As AI accelerators become more affordable and efficient, real-time stem separation will likely become a standard tool in professional audio. SSOUNDS envisions a system where the PA processor not only handles crossover and EQ but also performs source separation, allowing engineers to route stems to different speaker zones or create immersive audio experiences.

For example, a line array system could send the vocal stem to a dedicated center cluster while instruments are panned to left and right arrays, improving intelligibility and coverage. Similarly, subwoofers could receive only the bass and kick drum stems, reducing muddiness. This level of control is currently only possible with multitrack recording, but AI separation could make it achievable from a stereo mix.

SSOUNDS is actively researching low-latency models that can run on its proprietary amplifier platforms, with a focus on maintaining the audio quality and reliability that professionals expect. While we are not yet ready to announce a product, the potential for AI-driven PA systems is immense — and we are committed to leading the charge.