Abstract

Speech enhancement (SE) using diffusion models is a promising technology for improving speech quality in noisy speech data. Recently, the Schrödinger bridge (SB) has been utilized in diffusion-based SE to resolve the mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB inference remains slow owing to the need for a large number of function evaluations (NFE) to achieve high-quality results. Although consistency models have been proposed to accelerate inference, by employing consistency training via distillation from pretrained models in the field of image generation, they struggle to improve generation quality as the number of steps increases. Consistency trajectory models (CTMs) have been introduced to overcome this limitation, achieving faster inference while preserving a favorable trade-off between quality and speed. In particular, SoundCTM applies CTM techniques to sound generation. The SB addresses the aforementioned mismatch and CTM improves inference speed while maintaining a favorable trade-off between quality and speed. However, no existing method simultaneously resolves both issues to the best of our knowledge. Hence, in this study, we propose Schrödinger bridge consistency trajectory models (SBCTM), which apply CTM techniques to the SB for SE. In addition, we introduce a novel auxiliary loss, incorporating perceptual loss, into the original CTM training framework. Consequently, SBCTM achieves an approximately 16x improvement in real-time factor compared with the conventional SB for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM enables time-efficient inference by limiting multi-step refinement to cases in which one-step inference is insufficient. SBCTM enables more practical and efficient speech enhancement by providing fast inference and a flexible mechanism for further quality improvement. Our codes, pretrained models, and audio samples are available here.

Audio Samples of Speech Enhacement