Test

  • First
  • Second
  • Third

Unsupervised
Machine Learning Paradigms
for the Representation of
Music Similarity and Structure

Unsupervised
Machine Learning Paradigms
for the Representation of
Music Similarity and Structure
Axel Marmoret, IRISA, PhD defense
Under the supervision of
Frédéric Bimbot and Jérémy E. Cohen
December 2, 2022

PhD,
in a nutshell

PhD,
in a nutshell

Frédéric Bimbot

Jérémy E. Cohen
Verse
Chorus
Verse
Solo
Chorus

Music audio signal

Music structure

Music

Audio signal

Symbolic representation

  • Human-intended representations
    • Score, tablature, sheet music…
  • Digital Format
    • MIDI, MusicXML, …

Music structure?

  • Segmenting a song in structural elements
Verse
Chorus
Verse
Solo
Chorus
A
B
A
C
B’
a
b
c
c
a
b
d
e
f
c
c’

Song organisation

(ground-truth)

High-scale (sectional level)

Low-scale

1
2
3
3
4
5
4
5
1
2
3
3
6
7
8
8
4
5
4
9

Boundary retrieval

Why structural segmentation?

For structure analysis
  • Music analysis/Musicology
  • Automatic summary
  • …
For indirect applications
  • Genre/music recognition
  • Recommendation
  • Cover detection
  • …
As a proxy task

Quantitatively evaluates the relevance of a music analysis model

How are sections estimated?

  • Two principal criteria [Nie+20]:

Homogeneity/Novelty

  • Locally stationary

Repetition

  • Repeated accross the song

\Rightarrow Notion of similarity and dissimilarity accross the song

[Nie+20] Nieto, O., Mysore, G. J., Wang, C., Smith, J. B. L., Schlüter, J., Grill, T., & McFee, B. (2020). Audio-Based Music Structure Analysis: Current Trends, Open Challenges, and Applications. Transactions of the International Society for Music Information Retrieval

Practical example

“Come Together” – The Beatles

Intro Verse 1 Verse 2 Chorus

Solo 1

Solo 2

Outro

Machine Learning?

Machine Learning?

List of boundaries

(e.g. [0, 3.1, 4.5])

Segmentation

algorithm

Machine Learning

Decision: determined by a human (coded)

Conventional algorithm

Decision: inferred from data (“learning”)

Supervised Machine Learning

Machine Learning algorithm

(learning phase)

…

  • Learning: data \rightarrow annotation
    • Requires an annotated database
      • RWC Pop [Got+02]
      • SALAMI [Smi+11]

deep learning [GS15]

  • State-of-the-Art
  • Time and energy consuming
  • Annotations: Hard to collect and subjective

Annotation:

No boundary

Annotation:

Boundary

Annotation:

No boundary

[Got+02] Goto, M., Hashiguchi, H., Nishimura, T., & Oka, R. (2002). RWC Music Database: Popular, Classical and Jazz Music Databases. ISMIR, 2, pp. 287–288

[Smi+11] Smith, J. B. L., Burgoyne, J. A., Fujinaga, I., De Roure, D., & Downie, J. S. (2011). Design and Creation of a Large-Scale Database of Structural Annotations. ISMIR, 11, pp. 555–560

[GS15] Grill, T., & Schlüter, J. (2015). Music Boundary Detection Using Neural Networks on Combined Features and Two-Level Annotations. ISMIR, pp. 531–537

Unsupervised Machine Learning

  • Based on the similarity of the data
    • Does not depend on annotation
  • Autosimilarity matrix:
    • similarity accross time
    • Most of unsupervised state-of-the-art techniques [Foo00, ME14, Ser+14]

[Foo00] Foote, J. (2000). Automatic Audio Segmentation Using a Measure of Audio Novelty. IEEE International Conference on Multimedia and Expo (ICME), pp. 452–455

[ME14] McFee, B., & Ellis, D. P. W. (2014). Analyzing Song Structure with Spectral Clustering. ISMIR, pp. 405–410

[Ser+14] Serrà, J., Müller, M., Grosche, P., & Arcos, J. Ll. (2014). Unsupervised Music Structure Annotation by Time Series Structure Features and Segment Similarity. IEEE Transactions on Multimedia, 16(5), pp. 1229–1240

How is structure evaluated?

  • Accordance of two sets of boundaries (in seconds)
    • Estimated boundaries (e.g. [0, 3.1, 4.5])
    • Annotated boundaries (e.g. [0, 4.7, 5.2])
  • Boundary: valid if close to annotation, with a tolerance t

t
Time

  • \tick{black} Annotation
  • \tick{lime} Valid estimation
  • \tick{red} Invalid estimation
  • Metrics: (t= 0.5s and t= 3s)
    • Precision ({P}_{t}) = \frac{\tick{lime}}{\tick{red} + \tick{lime}}
    • Recall ({R}_{t}) = \frac{\tick{lime}}{\tick{black}}
    • F-measure ({F}_{t}) = \frac{2PR}{P+R}

\left\vert \begin{array}{l} F_{0.5} \\ F_3 \end{array} \right.

(MIREX standard)

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

Barwise spectrograms

Barwise spectrograms

Barwise spectrograms

Barwise spectrograms

ftb

Spectrogram?

  • Features: frequency content (Fourier Analysis)

Spectrogram

(feature description)

\text{Ω} (time)

f

Frequency scale

(Log Mel Spectrogram)

Musical scale

(Chromagram)

Barwise scale

…

ft b

Hypotheses for barwise processing

  • Hypothesis 1: patterns tend to develop at the bar scale
  • Hypothesis 2: boundaries are located on bars

Bars are estimated via an external toolbox (madmom [Böc+16])

[Böc+16] Böck, S., Korzeniowski, F., Schlüter, J., Krebs, F., & Widmer, G. (2016). Madmom: A New Python Audio and Music Signal Processing Library. Proceedings of the 24th ACM International Conference on Multimedia, pp. 1174–1178

Barwise spectrograms

Original

spectrogram

f

\Omega

t

b bars

ftb

Time-Frequency-Bar Tensor (TFB)

btf Barwise TF matrix “Matricisation”

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

Convolutive Block-Matching (CBM) algorithm

Convolutive Block-Matching (CBM) algorithm
[MCB22a]
(with open-source code)
[MCB22a] Marmoret, A., Cohen, J. E., & Bimbot, F. (2022). Convolutive Block-Matching Segmentation Algorithm with Application to Music Structure Analysis. arXiv preprint arXiv:2210.15356 (Submitted to ICASSP 2023)

CBM algorithm

CBM algorithm
[MCB22a]
(with open-source code)
[MCB22a] Marmoret, A., Cohen, J. E., & Bimbot, F. (2022). Convolutive Block-Matching Segmentation Algorithm with Application to Music Structure Analysis. arXiv preprint arXiv:2210.15356 (Submitted to ICASSP 2023)

[0,4,8,16,20,25]

Barwise autosimilarity

btf X

bb A(X)

A{\left( X \right)}_{11}=s\left( {X}_{1},{X}_{1} \right) A{\left( X \right)}_{12}=s\left( {X}_{1},{X}_{2} \right) A{\left( X \right)}_{16}=s\left( {X}_{1},{X}_{6} \right) A{\left( X \right)}_{45}=s\left( {X}_{4},{X}_{5} \right)

  • Similarity between pairs of bars {X}_{i},\ {X}_{j}
    • Depends on a similarity function \text{s}\left( \ \right)

b
b

A(X)

Convolutive Block-Matching (CBM) algorithm

  • Idea: framing square blocks of high similarity in the diagonal
  • Principle:
    • score \gamma ({\text{S}}_{\text{i}}) for each segment (weighted average)
    • maximizing the segmentation (set of segments) via dynamic programming

{\text{Z}}^{*}={\text{argmax}}_{Z}\sum_{{S}_{i}\in Z} \gamma ({S}_{i})

[0,4,8,16,20,25]

Cosine autosimilarity

“Come Together”

  • \text{s}\left( {X}_{i},\ {X}_{j} \right)=\frac{\left\langle {X}_{i},\ {X}_{j} \right\rangle }{\left\| {X}_{i} \right\|\left\| {X}_{j} \right\|}
  • Standard autosimilarity in literature [Foo00]

[Foo00] Foote, J. (2000). Automatic Audio Segmentation Using a Measure of Audio Novelty. IEEE International Conference on Multimedia and Expo (ICME), pp. 452–455

CBM algorithm, in practice

  • Outperforms the unsupervised State-of-the-Art algorithms

Segmentation results, compared with the State-of-the-Art (RWC Pop dataset [Got+02])

[Foo00, ME14, Ser+14, GS15]

[Got+02] Goto, M., Hashiguchi, H., Nishimura, T., & Oka, R. (2002). RWC Music Database: Popular, Classical and Jazz Music Databases. ISMIR, 2, pp. 287–288

[Foo00] Foote, J. (2000). Automatic Audio Segmentation Using a Measure of Audio Novelty. IEEE International Conference on Multimedia and Expo (ICME), pp. 452–455

[ME14] McFee, B., & Ellis, D. P. W. (2014). Analyzing Song Structure with Spectral Clustering. ISMIR, pp. 405–410

[Ser+14] Serrà, J., Müller, M., Grosche, P., & Arcos, J. Ll. (2014). Unsupervised Music Structure Annotation by Time Series Structure Features and Segment Similarity. IEEE Transactions on Multimedia, 16(5), pp. 1229–1240

[GS15] Grill, T., & Schlüter, J. (2015). Music Boundary Detection Using Neural Networks on Combined Features and Two-Level Annotations. ISMIR, pp. 531–537

RBF autosimilarity

  • s\left({X}_{i}, {X}_{j}\right)={\text{e}}^{-\frac{1}{2\sigma}\ {\left\| \frac{{X}_{i}}{\left\| {X}_{i} \right\|}\ -\ \frac{{X}_{j}}{\left\| {X}_{j} \right\|} \right\|}_{2}}
  • Better represents the dissimilarity between bars
    • Higher “contrast”

CBM algorithm, RBF

  • RBF results outperforms Cosine ones
    • Due to higher contrast
  • Levels of performance close to those of the State-of-the-Art without supervision

Segmentation results, compared with the State-of-the-Art (RWC Pop dataset)

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

Machine Learning used for compression

  • Summing up each bar to a few important components
    • Compression must ensure accurate reconstruction
  • Objective: disambiguating the barwise dissimilarity
    • i.e. increasing the contrast (similarly to RBF)

Compression

btf bk

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

Nonnegative Tucker Decomposition

Nonnegative Tucker Decomposition
[KC07, SG18, Mar+20, Mar+22]
[KC07] Kim, Y.-D., & Choi, S. (2007). Nonnegative Tucker Decomposition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8
[SG18] Smith, J. B. L., & Goto, M. (2018). Nonnegative Tensor Factorization for Source Separation of Loops in Audio. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 171–175
[Mar+20] Marmoret, A., Cohen, J. E., Bertin, N., & Bimbot, F. (2020). Uncovering Audio Patterns in Music with Nonnegative Tucker Decomposition for Structural Segmentation. ISMIR, pp. 788–794
[Mar+22] Marmoret, A., Voorwinden, F., Leplat, V., Cohen, J. E., & Bimbot, F. (2022). Nonnegative Tucker Decomposition with Beta-Divergence for Music Structure Analysis of Audio Signals. GRETSI

Nonnegative Tucker Decomposition

Nonnegative Tucker Decomposition
[KC07, SG18, Mar+20, Mar+22]
[KC07] Kim, Y.-D., & Choi, S. (2007). Nonnegative Tucker Decomposition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8
[SG18] Smith, J. B. L., & Goto, M. (2018). Nonnegative Tensor Factorization for Source Separation of Loops in Audio. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 171–175
[Mar+20] Marmoret, A., Cohen, J. E., Bertin, N., & Bimbot, F. (2020). Uncovering Audio Patterns in Music with Nonnegative Tucker Decomposition for Structural Segmentation. ISMIR, pp. 788–794
[Mar+22] Marmoret, A., Voorwinden, F., Leplat, V., Cohen, J. E., & Bimbot, F. (2022). Nonnegative Tucker Decomposition with Beta-Divergence for Music Structure Analysis of Audio Signals. GRETSI

\mathscr{X}

(TFB Tensor)

f
t
b

\approx Wfr_f \mathscr{G}r_fr_tr_b Hr_tt Qr_bb \mathscr{X}=\mathscr{G}{\times }_{1}W{\times }_{2}H{\times }_{3}Q

Nonnegative Matrix Factorization (NMF)

X \approx Wk Vk X\approx \ WV

NMF for Music Transcription [SB03]
Example adapted from [Wu+22]

\approx Frequency Time W V

X\approx \ WV

[SB03] Smaragdis, P., & Brown, J. C. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 177–180

[Wu+22] Wu, H., Marmoret, A., & Cohen, J. E. (2022). Semi-Supervised Convolutive NMF for Automatic Music Transcription. 19th Sound and Music Computing Conference (SMC 2022)

Mixing templates

W G H

X=\ WGH X =

Barwise musical pattern: WGH

G

♩ ♪ ♩ ♩ H ♩ ♩ ♩ ♩ ♩ ♩ = W Hi-hat Snare Kick

Combining patterns

W G H

X=W\left( \sum_{i} {q}_{[i]}{G}_{[::i]} \right)H q

X =

Bars, expressed with patterns

W G H

Barwise spectrogram {X}_{[::0]}

\approx

{X}_{[::k]}=W\left( \sum_{i} {Q}_{[ki]}{G}_{[::i]} \right)H Q

NTD: Nonnegative Tucker Decomposition

\mathscr{X}

(TFB Tensor)

f
t
b

\approx Wfr_f \mathscr{G}r_fr_tr_b Hr_tt Qr_bb

  • All factors are nonnegative, {\text{r}}_{f},\ {r}_{t},\ {r}_{b} are hyperparameters (dimensions: order of magnitude of 20)

\mathscr{X}=\mathscr{G}{\times }_{1}W{\times }_{2}H{\times }_{3}Q

NTD in practice: chromagram

\mathscr{G}

\approx W \mathscr{G} H Q \mathscr{X} “Come Together”

NTD computation

{\text{argmin}}_{\mathscr{G}\geq 0,\ W\geq 0,\ H\geq 0,\ Q\geq 0}\ d(\mathscr{X}\text{, }\mathscr{G}{\times }_{1}W{\times }_{2}H{\times }_{3}Q)

  • Euclidean norm [Mar+20]:
    • Solved with HALS [GG12]

\hl{1-}{accent}{{d}_{euc}\left( x\middle| y \right)= {\left\| x-y \right\|}_{2}^{2}}

  • \beta-divergence [Mar+22]:
    • Solved with Multiplicative Updates [FI11]

{d}_{\beta }\left( x\middle| y \right)=\begin{cases} \hl{1-}{accent}{\frac{x}{y}-\log\left( \frac{x}{y} \right)-1} & \hl{1-}{accent}{\beta =0 \text{ (Itakura-Saïto)}} \\[1.2em] \hl{1-}{accent}{x\log\left( \frac{x}{y} \right)+\left( y-x \right)} & \hl{1-}{accent}{\beta =1 \text{ (Kullback-Leibler)}} \\[1.2em] \frac{{x}^{\beta }+\left( \beta -1 \right){y}^{\beta }-\beta x{y}^{\beta -1}}{\beta \left( \beta -1 \right)} & \beta \in \mathbb{R}\setminus \{0,1\} \end{cases}

[Mar+20] Marmoret, A., Cohen, J. E., Bertin, N., & Bimbot, F. (2020). Uncovering Audio Patterns in Music with Nonnegative Tucker Decomposition for Structural Segmentation. ISMIR, pp. 788–794

[GG12] Gillis, N., & Glineur, F. (2012). Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization. Neural Computation, 24(4), pp. 1085–1105

[Mar+22] Marmoret, A., Voorwinden, F., Leplat, V., Cohen, J. E., & Bimbot, F. (2022). Nonnegative Tucker Decomposition with Beta-Divergence for Music Structure Analysis of Audio Signals. GRETSI

[FI11] Févotte, C., & Idier, J. (2011). Algorithms for Nonnegative Matrix Factorization with the \beta-Divergence. Neural Computation, 23(9), pp. 2421–2456

  • 1st bar of Come Together
  • Hand-picked pattern from NTD
    • Euclidean norm
  • Kullback-Leibler divergence
  • Itakura-Saito divergence

Musical patterns W{\mathscr{G}}_{[::i]}H?

W{\mathscr{G}}_{[::i]}H Audio signal

STFT

Audio signal spectrogram

NTD

ISTFT*

(One musical pattern)

*Phase information: from the original STFT

NTD in practice: chromagram

\mathscr{G}

\approx W \mathscr{G} H Q \mathscr{X}

“Come Together”

Q matrix

  • Most bars appear as a sparse combination of musical patterns

Autosimilarities: Uncompressed vs NTD
(Cosine)

\Rightarrow Higher contrast with NTD (as desired)

Cosine RBF Uncompressed NTD

NTD segmentation results

  • Similarity function
  • Objective function
    • Kullback-Leibler (\beta =1)

Segmentation results, according to parameters (RWC Pop dataset)

NTD segmentation results

  • Cosine: NTD improves uncompressed results

Segmentation results, compared with State-of-the-art (RWC Pop dataset)

  • RBF: NTD does not improve uncompressed results
    • NTD dimensions are hard to fit
    • hyperparameters for CBM were fitted for uncompressed matrices

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

Linear and Nonlinear Barwise Compression Schemes

Linear and Nonlinear Barwise Compression Schemes
[MCB22b]
[MCB22b] Marmoret, A., Cohen, J. E., & Bimbot, F. (2022). Barwise Compression Schemes for Audio-Based Music Structure Analysis. 19th Sound and Music Computing Conference (SMC 2022)

Linear and Nonlinear Barwise Compression Schemes

Linear and Nonlinear Barwise Compression Schemes
[MCB22b]
[MCB22b] Marmoret, A., Cohen, J. E., & Bimbot, F. (2022). Barwise Compression Schemes for Audio-Based Music Structure Analysis. 19th Sound and Music Computing Conference (SMC 2022)

Barwise spectrograms Q

NTD

NMF

PCA

Autoencoders

PCA

  • Computes directions of maximal variance (“principal components”)
    • Not intepretable!
  • Projection on these principal components

Image by Nicoguaro - Own work, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=46871195

PCA segmentation results

  • Improves NTD results, at the cost of interpretability
  • Does not outperform best condition for uncompressed representations
    • Compression dimension is hard to fit
    • Parameters for CBM were fitted for uncompressed matrices

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

AE-NTD

AE-NTD
Autoencoder & NTD

AE-NTD

AE-NTD
Autoencoder & NTD

X_i^Tft Encoder Q_i^T \mathscr{G}_{(2)}^T \xrightarrow{\;a()\;} (W \otimes H) \xrightarrow{\;a()\;} \hat{X}_i^Tft

AutoEncoders

X_i^Tft Encoder
(\theta) Q_i^Tk Decoder
(\Psi) \hat{X}_i^Tft

  • Encoding: compression
    • Latent representation {Q}_{i} for each bar
  • Decoding: reconstruction
  • Both are based on matrix products

Objective function: {\text{argmin}}_{\theta ,\text{Ψ}}\ d({X}_{i},{\hat{X}}_{i})

NTD matricisation

  • NTD: \mathscr{X}\approx \mathscr{G}{\times}_{1}W{\times}_{2}H{\times}_{3}Q \mathrel{\htmlData{hlmark=iff}{\Longleftrightarrow}} \hlo{2-}{teal}{\mathscr{X}_{(2)}} \approx \hlo{3-}{orange}{Q}\,\hlo{4-}{purple}{\mathscr{G}_{(2)}}\hlo{4-}{purple}{{\left( W\mathbin{\htmlData{hlmark=kron}{\otimes}} H \right)}^{T}}

\mathscr{X}ftb

\mathscr{X}_{(2)}bft

\mathscr{X}_{(2)}

(Barwise TF)

\approx Q \mathscr{G}_{(2)} {\left( W\otimes H \right)}^{T}

“Matricisation”

Kronecker product

Barwise TF matrix

(“unfolding” of the tensor)

Latent
representation

2 matrix products

2 fully-connected
layers

NTD-AutoEncoder:
NTD as a 2-layer decoder

X_i^Tft Encoder
(\theta) Q_i^T \mathscr{G}_{(2)}^T \xrightarrow{\;a()\;} (W \otimes H) \xrightarrow{\;a()\;} \hat{X}_i^Tft

Initialization: NTD results, random (… or in between)

a(): nonnegative activation function (ReLU, sotfplus, …)

Objective function: {\text{argmin}}_{\theta,\mathscr{G},W,H,Q}\ d(X_i,\hat{X}_i)

Exploratory results

  • Segmentation:

        (RWC Pop)

  • Pattern {\text{W}\mathscr{G}}_{\text{[::}\text{i}\text{]}}H\text{:}

\Rightarrow Empirical trade-off between segmentation performance and interpretability

NTD (original) Decoder:
NTD Init 		Decoder:
Random init

Music

Barwise spectrograms

CBM algorithm Structure

Compressed representations Machine Learning

NTD

NMF

PCA

AutoEncoders

AE-NTD

Overview of segmentation results

  • Structure is exhibited in similarity and dissimilarity notions
  • Two strategies for better representations:
    • Kernel-based similarity function
    • Machine Learning (compression methods)
  • Machine Learning methods: trade-off between performance and interpretability
    • Methods present different pros and cons
  • NTD: uncovers musical patterns

Perspectives

Dimensioning compression
  • Dimensions required prior to Machine Learning
  • No obvious link between dimensions and segmentation results
Domain knowledge
  • Incorporating domain knowledge to enhance representations
  • Constraints
    • Supervision/semi-supervision
    • Dictionary learning
    • …
Extension of results
  • Different musical contexts
  • Results are only quantitative
    • Need for musical analysis of results

Unsupervised
Machine Learning Paradigms
for the Representation of
Music Similarity and Structure

Unsupervised
Machine Learning Paradigms
for the Representation of
Music Similarity and Structure

What could constitute structure?

  • Lyrics?
    • Around the World - Daft Punk
  • Timbre?
    • Come Together – The Beatles
  • Melody/Harmony?
    • Lambé An Dro – Matmatah

Example of pattern, with melody

Barwise point of view

  • Hypothesis 2: boundaries fall on downbeats
    • Valid for results of state-of-the-art on RWC Pop with madmom

CBM Results, according to features

Convolutive “block-matching” (CBM) algorithm

  • Principle:
    • associating a score for each segment [{\text{b}}_{\text{i}},\ {\text{b}}_{\text{j}}[ as a convolution

\gamma ([{b}_{i},\ {b}_{j}[)=\frac{1}{j\ -i}\sum_{k=0}^{i-1} \sum_{l=0}^{j-1} {A(X)}_{i+k,\ i+l}{K}_{k,l}\ +p(j-i)

  • maximizing their sum via dynamic programming

{\text{Z}}^{*}={\text{argmax}}_{Z}\sum_{{S}_{i}\in Z} \gamma ({S}_{i})

Dynamic Programming (CBM)

  • Dynamic programming, can be interpreted as a longest-path retrieving in a graph

b_1 : 0 b_2\,:2 b_3\,:6 b_4\,:10

2 6 1 8 1 4

Results, according to kernels

Audio – power discrepancies

  • Low-frequency coefficients are generally larger in magnitude than high-frequency coefficients
  • Components of low intensity can perceptually be as important as components of high intensity
    • perception of loudness in the human ear

By Lindosland http://en.wikipedia.org/wiki/Image:Lindos1.svg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=2565926

[FBD09] Cédric Févotte, Nancy Bertin, and Jean-Louis Durrieu, « Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis », in: Neural computation 21.3 (2009), pp. 793–830.

[FBD09] Févotte, C., Bertin, N., & Durrieu, J.-L. (2009). Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis. Neural Computation, 21(3), pp. 793–830

\beta-divergence

{d}_{\beta }\left( x\middle| y \right)=\begin{cases} \frac{x}{y}-\log\left( \frac{x}{y} \right)-1 & \beta =0 \text{ (Itakura-Saïto)} \\[1.2em] x\log\left( \frac{x}{y} \right)+\left( y-x \right) & \beta =1 \text{ (Kullback-Leibler)} \\[1.2em] \frac{{x}^{\beta }+\left( \beta -1 \right){y}^{\beta }-\beta x{y}^{\beta -1}}{\beta \left( \beta -1 \right)} & \beta \in \mathbb{R}\setminus \{0,1\} \end{cases}

(\beta =2:Euclidean\ norm)

For any \beta \in \ \mathbb{R},\ \lambda \in \mathbb{R},   {d}_{\beta }\left( \lambda x\middle| \lambda y \right)={\lambda }^{\beta }\ {d}_{\beta }\left( x\middle| y \right)

\beta =2\ \Rightarrow {d}_{2}\left( \lambda x\middle| \lambda y \right)=\ {{\lambda }^{2}d}_{\beta }\left( x\middle| y \right) \beta =1\ \Rightarrow {d}_{1}\left( \lambda x\middle| \lambda y \right)=\ \lambda {d}_{\beta }\left( x\middle| y \right) \beta =0\ \Rightarrow {d}_{0}\left( \lambda x\middle| \lambda y \right)=\ {d}_{\beta }\left( x\middle| y \right)

( Example: {d}_{\beta }\left( 100\middle| 10 \right) vs {d}_{\beta }\left( 0.1\middle| 0.01 \right) )

Results for CBM, scores aligned on bars
RWC POP dataset

Close

metric

Large

metric

Seconds Barwise aligned

Results for CBM, scores aligned on bars
SALAMI dataset

Close

metric

Large

metric

Seconds Barwise aligned

Why mixing AE and NTD?

Performance-Interpretability
  • Combining:
    • segmentation results of AutoEncoders
    • interpretability of NTD

Objective: interpretable patterns in the decoder

Expressiveness
  • Greater subspace of solutions with AE than with NTD
    • activations functions
    • encoder design
    • …
  • Ease of implementing constraints (ex: sigmoid in the latent space)

Objective: improve the NTD outputs, and follow state-of-the-art research

Technical tools
  • Using toolboxes (PyTorch, TensorFlow) with high-performance computing

Objective: benefiting from recent developments in Neural networks computation

PCA

  • Computes directions of maximal variance (“principal components”)
    • Not intepretable!

\hat{X}=US{V}^{T}

(\hat{X}: centered X matrix)

Image by Nicoguaro - Own work, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=46871195

PCA: Q=\hat{X}{V}_{[:k]}

btf bk X Q

Results on SALAMI

Fourier analysis

Kronecker product

References

[Böc+16] Böck, S., Korzeniowski, F., Schlüter, J., Krebs, F., & Widmer, G. (2016). Madmom: A New Python Audio and Music Signal Processing Library. Proceedings of the 24th ACM International Conference on Multimedia, pp. 1174–1178

[FBD09] Févotte, C., Bertin, N., & Durrieu, J.-L. (2009). Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis. Neural Computation, 21(3), pp. 793–830

[FI11] Févotte, C., & Idier, J. (2011). Algorithms for Nonnegative Matrix Factorization with the \beta-Divergence. Neural Computation, 23(9), pp. 2421–2456

[Foo00] Foote, J. (2000). Automatic Audio Segmentation Using a Measure of Audio Novelty. IEEE International Conference on Multimedia and Expo (ICME), pp. 452–455

[GG12] Gillis, N., & Glineur, F. (2012). Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization. Neural Computation, 24(4), pp. 1085–1105

[Got+02] Goto, M., Hashiguchi, H., Nishimura, T., & Oka, R. (2002). RWC Music Database: Popular, Classical and Jazz Music Databases. ISMIR, 2, pp. 287–288

[GS15] Grill, T., & Schlüter, J. (2015). Music Boundary Detection Using Neural Networks on Combined Features and Two-Level Annotations. ISMIR, pp. 531–537

[KC07] Kim, Y.-D., & Choi, S. (2007). Nonnegative Tucker Decomposition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8

[Mar+20] Marmoret, A., Cohen, J. E., Bertin, N., & Bimbot, F. (2020). Uncovering Audio Patterns in Music with Nonnegative Tucker Decomposition for Structural Segmentation. ISMIR, pp. 788–794

[Mar+22] Marmoret, A., Voorwinden, F., Leplat, V., Cohen, J. E., & Bimbot, F. (2022). Nonnegative Tucker Decomposition with Beta-Divergence for Music Structure Analysis of Audio Signals. GRETSI

[MCB22a] Marmoret, A., Cohen, J. E., & Bimbot, F. (2022). Convolutive Block-Matching Segmentation Algorithm with Application to Music Structure Analysis. arXiv preprint arXiv:2210.15356 (Submitted to ICASSP 2023)

[MCB22b] Marmoret, A., Cohen, J. E., & Bimbot, F. (2022). Barwise Compression Schemes for Audio-Based Music Structure Analysis. 19th Sound and Music Computing Conference (SMC 2022)

[ME14] McFee, B., & Ellis, D. P. W. (2014). Analyzing Song Structure with Spectral Clustering. ISMIR, pp. 405–410

[Nie+20] Nieto, O., Mysore, G. J., Wang, C., Smith, J. B. L., Schlüter, J., Grill, T., & McFee, B. (2020). Audio-Based Music Structure Analysis: Current Trends, Open Challenges, and Applications. Transactions of the International Society for Music Information Retrieval

[SB03] Smaragdis, P., & Brown, J. C. (2003). Non-Negative Matrix Factorization for Polyphonic Music Transcription. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 177–180

[Ser+14] Serrà, J., Müller, M., Grosche, P., & Arcos, J. Ll. (2014). Unsupervised Music Structure Annotation by Time Series Structure Features and Segment Similarity. IEEE Transactions on Multimedia, 16(5), pp. 1229–1240

[SG18] Smith, J. B. L., & Goto, M. (2018). Nonnegative Tensor Factorization for Source Separation of Loops in Audio. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 171–175

[Smi+11] Smith, J. B. L., Burgoyne, J. A., Fujinaga, I., De Roure, D., & Downie, J. S. (2011). Design and Creation of a Large-Scale Database of Structural Annotations. ISMIR, 11, pp. 555–560

[Wu+22] Wu, H., Marmoret, A., & Cohen, J. E. (2022). Semi-Supervised Convolutive NMF for Automatic Music Transcription. 19th Sound and Music Computing Conference (SMC 2022)