Open-Unmix - A Reference Implementation for Music

Source Separation

Fabian-Robert Stöter

1

, Stefan Uhlich

2

, Antoine Liutkus

1

, and Yuki

Mitsufuji

3

1 Inria and LIRMM, University of Montpellier, France 2 Sony Europe B.V., Germany 3 Sony

Corporation, Japan

DOI: 10.21105/joss.01667

License

Authors of papers retain

copyright and release the work

under a Creative Commons

Attribution 4.0 International

License (CC-BY).

Summary

Music source separation is the task of decomposing music into its constitutive components,

e.g., yielding separated stems for the vocals, bass, and drums. Such a separation has many

applications ranging from rearranging/repurposing the stems (remixing, repanning, upmixing)

Firstly, to accelerate academic research as Open-Unmix provides implementations for the

most popular deep learning frameworks, giving researchers a exible way to reproduce results.

Secondly, we provide a pre-trained model for end users and even artists to try and use source

separation. Furthermore, we designed Open-Unmix to be one core component in an open

ecosystem on music separation, where we already provide open datasets, software utilities,

and open evaluation to foster reproducible research as the basis of future development.

Background

Music separation is a problem which has fascinated researchers for over 50 years. This is

partly because, mathematically, there exists no closed-form solution when many sources (in-

struments) are recorded in a mono or stereo signal. To address the problem, researchers

ever, was hardly feasible as freely available datasets did not exist at that time. In fact, for

a meaningful evaluation, the ground truth separated stems are necessary. However, because

commercial music is usually subject to copyright protection, and the separated stems are

considered to be valuable assets in the music recording industry, they are usually unavailable.

Nonetheless, thanks to some artists who choose licenses like Creative Commons, that allow

sharing of the stems, freely available datasets were released in the past ve years and have

enabled the development of data-driven methods. Since then, progress in performance has

been closely linked to the availability of more data that allowed the use of machine-learning-

Nonetheless, even with this small amount of data, deep neural networks (DNNs) were not

only successfully used for music separation but they are now setting the state-of-the-art in

this domain as can be seen by the results of the community-based signal separation evaluation

campaign (SiSEC) (Liutkus et al., 2017; Ono, Rai, Kitamura, Ito, & Liutkus, 2015; Stöter,

Liutkus, & Ito, 2018). In these challenges, the proposed systems are compared to other

methods. Among the systems under test, classical signal processing based methods were

clearly outperformed by machine learning methods. However they were still useful as a fast

and often simple to understand baseline.

In the following, we will describe a number of these reference implementations for source

separation. While there are some commercial systems available, such as Audionamix XTRAX

STEMS, IZOTOPE RX 7 or AudioSourceRE, we only considered tools that are available as

but also includes other model-based methods. In 2016, the untwist library was proposed in

(Roma, Grais, Simpson, Sobieraj, & Plumbley, 2016). It comprises several methods, ranging

from classical signal-processing-based methods to feed-forward neural networks. The library

is written in Python 2.7. Unfortunately, it has not been updated since 2017 and many of its

methods are not subjected to automated testing. Nussl is a very recent framework, presented

in (Manilow, Seetharaman, & Pardo, 2018). It includes a large number of methods and

generally focuses on classical signal processing methods rather than machine-learning-based

techniques. It has built-in interfaces for common evaluation metrics and data sets. The library

oers great modularity and a good level of abstraction. However, this also means that it is

challenging for beginners who might only want to focus on changing the machine learning

to their own baseline implementations, thus showing relative instead of absolute performance

gains, so that other researchers cannot assess if a method performs as well as state-of-the-

art. Also, the lack of a common reference for the community potentially misguides young

researchers and students who enter the eld of music separation. The result of this can be

observed by looking at the popularity of the above-mentioned music separation frameworks

on GitHub: all of the frameworks mentioned above, combined, are less popular than two

recent deep learning papers that were accompanied by code such as MTG/DeepConvSep from

(Chandna, Miron, Janer, & Gómez, 2017) and f90/Wave-U-Net from (Stoller, Ewert, &

Dixon, 2018). Thus, users might be confused regarding which of these implementations can

be considered state-of-the-art.

track using learning-based algorithms. We therefore designed Open-Unmix to address these

issues by relying on procedures that were veried by the community or have proven to work

well in the literature.

Design Choices

The design choices made for Open-Unmix have sought to reach two somewhat contradictory

objectives. Its rst aim is to have state-of-the-art performance, and its second aim is to

still be easily understandable, so that it can serve as a basis for research to allow improved

performance in the future. In the past, many researchers faced diculties in pre- and post-

processing that could be avoided by sharing domain knowledge. Our aim was thus to design a

system that allows researchers to focus on A) new representations and B) new architectures.

“MNIST-like”

Keeping in mind that the learning curve can be quite steep in audio processing, we did our

best for Open-unmix to be:

In particular:

Results

are not looking for changes that only focus on improving the separation performance as this

would be out of scope for a baseline implementation. Instead, we expect many researchers

will fork the software as a basis for their research and the documentation explicates several

Salaün, Y., Vincent, E., Bertin, N., Souviraà-Labastie, N., Jaureguiberry, X., Tran, D. T., &

Stoller, D., Ewert, S., & Dixon, S. (2018). Wave-u-net: A multi-scale neural network for

Open-Unmix. doi:10.5281/zenodo.3382104

Stöter, F.-R., & Liutkus, A. (2019c, August). Open-unmix-pytorch umx-hq. doi:10.5281/

zenodo.3370489

Vincent, E., Araki, S., Theis, F. J., Nolte, G., Boll, P., Sawada, H., Ozerov, A., et al.

(2012). The signal separation evaluation campaign (2007-2010): Achievements and remaining

challenges, 92(8), 1928–1936. doi:10.1016/j.sigpro.2011.10.007