Researcher releases AI model converting stereo music to spatial audio

The researcher began working on Stereo2Spatial after developing an interest in spatial music listening and noticing the shortage of quality spatial remixes available for existing stereo recordings. Over approximately six months of on-and-off development, they built a flow-matching diffusion model—a generative approach that learns to transform one distribution into another—operating in the latent (compressed) space of a separately trained VAE called EAR-VAE.

The pipeline works as follows: the VAE encodes a stereo input track into a single latent representation, and separately encodes each channel of a 7.1.4 surround output (which includes side and rear surround channels plus height channels). The diffusion model learns to map from the stereo latent to the surround latent space. The core architecture follows a similar design to ImmersiveFlow, a prior paper the researcher found describing another stereo-to-spatial conversion project. However, the researcher made a key addition: memory tokens that persist across processing windows. In audio generation, processing long tracks requires breaking them into overlapping or sequential chunks; without memory, each chunk loses context about earlier audio, leading to discontinuities. The memory tokens enable the model to maintain state across these boundaries, resulting in more coherent long-form generation.

One technical challenge the researcher highlights is that the VAE was trained to encode and decode stereo tracks directly, not individual surround channels—placing the model outside the VAE's training distribution when it attempts to encode surround outputs. This mismatch is a constraint the architecture must handle carefully. The codebase was built from scratch, as the researcher was unable to locate working code from the ImmersiveFlow paper (links had returned 404 errors at the time of development). The model is now publicly available for use.

The release of Stereo2Spatial reflects a practical response to a real limitation in digital music consumption. While spatial audio formats (like Dolby Atmos and immersive binaural mixes) have grown in availability on streaming platforms, the vast majority of recorded music—especially older catalog—was mastered only in stereo. The researcher's motivation to build this model stems from enjoyment of spatial listening and recognition that most music lacks "real quality spatial mix."

The technical approach draws directly from prior academic work (ImmersiveFlow), but the addition of memory tokens across processing windows addresses a concrete problem in audio generation: maintaining temporal coherence and audio quality over long tracks. The researcher notes a constraint: the VAE was trained on stereo tracks directly, not on the individual channels of a 7.1.4 surround mix, placing the model in a domain-mismatch scenario that must be managed by the conversion architecture. This out-of-distribution design challenge is a key implementation detail that shapes how the final model behaves.

No comments yet. Be the first to share your thoughts!