PupuM2D, Explained: An Open Music AI That Learns From Spectrograms the Way Producers Read Them

Music is a new section of the EXEPERT blog. It covers how AI and software understand, make and change music, and the tools producers can actually use. It starts with a model built on something every producer already knows: you can see a song.

On 17 September 2026 the Threads account @pupunaama announced PupuM2D, a self-supervised music model from researchers at Spellbrush, Aalto University and the Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen). The post calls it a state-of-the-art music model built on the Masked Modeling Duo (M2D) architecture. It says the code and checkpoints are open source, and that tokenizers and integration with the MARBLEv2 benchmark will follow (pupunaama, 2026). It shared five figures and four links, collected at the end of this post.

Here is what PupuM2D does, how it works, what its benchmark numbers show and how to use it today.

What PupuM2D is

PupuM2D comes from the preprint "Frequency-Aware Self-Supervised Music Representation Learning" by Yicheng Gu, Junan Zhang, Jerry Li, Zhizheng Wu and Lauri Juvela (Gu et al., 2026a). Three ideas explain it.

• Self-supervised learning. The model learns from unlabelled audio: nobody tells it a song's genre, key or tempo. It hides parts of its input and learns to predict what is missing. The result is a representation, a set of numbers for each moment of audio that other tools can reuse.

• A spectrogram is a picture of sound. Time runs left to right, pitch from bottom to top, and brightness shows how much energy sits at each frequency at each moment. A mel-spectrogram spaces frequencies roughly the way human hearing does. PupuM2D reads 128 mel bands of 24 kHz mono audio and trains on slices of 10.24 seconds.

• 2D, not 1D. Leading music models such as MERT, MusicFM and MuQ treat audio as a one-dimensional sequence: the raw waveform, or spectrogram frames flattened into a line. The authors argue that this throws away the spatial and harmonic structure of the spectrogram. PupuM2D treats the spectrogram as a 2D image and cuts it into a grid of small patches, the way vision models handle photos.

PupuM2D is an encoder, not a generator: it turns music into features that other models use to recognise keys, beats, genres, moods and instruments. The code is under the MIT licence, and six checkpoints, from 5 million to 1.1 billion parameters, are on Hugging Face (Gu et al., 2026c).

A producer's intuition

The project page sums up the idea in one line: "Music SSL models should be trained on 2D Spectrograms" (Gu et al., 2026b). The author writes as a former producer and DJ, and nobody they know produces, mixes or masters by looking at a waveform. Producers write notes in piano rolls, watch real-time spectra in FabFilter Pro-Q 4 while mixing, and check finished songs as mel-spectrograms in iZotope RX. All of these are time-frequency views.

The worked example is Pixel Galaxy by Snail's House, a well-known kawaii bass track, rebuilt by a fan in FL Studio. The page takes the first two bars of the first drop and splits them into six stems: chord, lead, pluck, bass, kick and snare.

Each stem leaves a recognisable mark: short, bright bursts at the very bottom for the kick, long low lines for the bass, a thin vertical streak across many frequencies for the snare, and stacks of parallel lines for the chords. The thought bubbles belong to Pupu, the paper's bunny producer (pupu is Finnish for bunny), and show why this matters:

• Beat tracking. The kick at the start of each bar gives a tempo of 150 BPM.

• Key detection. The root notes of the chords give the key, C major.

• Genre. A half-time kick and snare under huge synth chords says future bass.

• Instruments. The six stems answer the question themselves.

Each is a standard music information retrieval (MIR) task, and a producer can solve each one by eye. The bet behind PupuM2D is that a model trained on the same picture can learn the same shortcuts.

How Masked Modeling Duo works

M2D comes from Daisuke Niizumi and colleagues at NTT, first presented at ICASSP 2023 and extended in a 2024 journal paper (Niizumi et al., 2023, 2024). It learns from masked inputs with two networks.

1. Cut and hide. The spectrogram is split into patches, and most of them are hidden: 80% in PupuM2D.

2. Context encoder. One network, the context (or online) encoder, sees only the visible patches.

3. Target encoder. A second network sees only the hidden patches and turns them into embeddings, compact summaries rather than pixels.

4. Predictor. A small predictor takes the context encoder's output, plus placeholders marking where the hidden patches were, and tries to predict the target encoder's embeddings.

5. A slow teacher. Only the context side learns directly. The target encoder is a slowly updated copy of it, an exponential moving average, which keeps the two from settling on a trivial answer.

The key choice is to predict embeddings, not the spectrogram itself. The project page and paper explain why the authors set the alternatives aside (Gu et al., 2026a, 2026b):

• Masked autoencoders (MAE) rebuild the hidden pixels. On music they tend to overfit spectrogram detail, because neighbouring patches are highly correlated in time and frequency.

• DINO-style methods compare differently augmented views of the same input. Music has no natural equivalent of cropping, rotating or recolouring an image, and effects such as filtering and reverb suit isolated tracks better than full mixes.

• JEPA (joint-embedding predictive architecture) is close to M2D, but its target encoder sees every patch. In the authors' tests that let the model take shortcuts, and its representations collapsed. M2D, which came earlier in audio, also performed better.

The name changed along the way. The first version of the preprint, posted on 24 June 2026, called the model PupuJEPA. The third version, from 9 September 2026, renames it PupuM2D and credits M2D for a target encoder that sees only the masked patches (Gu et al., 2026a).

What the authors changed for music

• Thin, tall patches. Vision models usually cut 16×16 patches. PupuM2D uses 4 time frames by 16 mel bands, which gives 25 feature frames per second, fine enough for beat tracking.

• Masks shaped like music. Besides random patches, the model hides rectangular blocks and strips that run across time or frequency. Training starts with random masks only and phases in the others over the first 100,000 steps.

• A sturdier Transformer. SwiGLU feed-forward layers, query-key normalisation, layer normalisation instead of batch normalisation, and rotary position embeddings for audio of any length. DropPath and LayerScale, common in vision models, made training collapse and were removed.

• A smoothed L1 loss. With the standard L1 loss, training also collapsed.

• New read-outs. Tasks use a learned combination of the encoder's upper layers. For song-level tasks, patches are averaged within time, frequency or block partitions rather than in one global average, which keeps some of the time-frequency layout.

The ablations on PupuM2D-Large show what the changes are worth. Removing SwiGLU alone drops GiantSteps key accuracy from 66.1 to 62.5 and emotion valence from 62.5 to 58.1.

Training used about 100,000 hours of in-house music: 500,000 steps with a global batch size of 2,048 on 32 NVIDIA B200 GPUs. The paper reports five sizes, from Tiny (5M parameters) to Huge (632M) (Gu et al., 2026a).

How it scores on MARBLEv2

MARBLE is a community-driven benchmark for music audio representations, first presented at NeurIPS 2023 (Yuan et al., 2023). Its second version replaced the original code on 4 June 2025 (a43992899, 2025). The paper uses linear probing: the encoder is frozen, and only a single linear layer is trained on top for each task. That tests how much musical information the features already hold in an easy-to-use form.

The tasks cover emotion on EmoMusic (EMO), scored by the coefficient of determination R² for valence and arousal; key detection on GiantSteps (GS), a dance-music set, and on HookTheory, scored by refined accuracy; beat tracking on GTZAN, scored by F1 (F1 Beat); and section labelling on HookTheory, scored by accuracy. Genre, tagging, instrument and mood tasks use GTZAN (accuracy), MagnaTagATune (MTT) and MTG-Jamendo (MTG), scored by ROC-AUC and average precision (AP).

The two tables hold 17 metric columns. This summary uses average precision for the tagging tasks:

Source: Gu et al. (2026a), Tables III and V.

Four things stand out.

• It beats the 1D music models on most tasks. PupuM2D-Large tops the best of MERT, MuQ and MusicFM on 13 of the 17 metrics. The exceptions are the two HookTheory tasks, MTG genre ROC-AUC (86.1 against 86.2) and MTG Top 50 ROC-AUC (82.8 against 83.0).

• It is not best everywhere. A PupuM2D model holds the top score in 7 of the 17 columns: both emotion metrics, GiantSteps key, GTZAN beats, both MTG instrument metrics and MTG Top 50 ROC-AUC. General audio models take most of the rest, with MATPAC++ leading GTZAN genre and MTT tagging and M2D-X leading MTG mood and theme. MERT and MusicFM keep the HookTheory tasks.

• The fairest comparison favours it. The authors retrained AudioMAE++, MATPAC++ and M2D at 307M parameters on their own data, with PupuM2D's Transformer and read-out. PupuM2D-Large beats the best of them on 13 of 17 metrics, for example 86.9 against 83.8 on GTZAN genre. With data, size and read-out held equal, that points to the training recipe.

• Small can be enough. PupuM2D-Tiny, with 5 million parameters, scores 64.2 on GiantSteps key, ahead of every official checkpoint tested on that task, including the 330M-parameter MERT-Large at 64.1. Only the retrained M2D-Large scores higher, at 65.0. The paper's text says Tiny beats all baselines; the demo page says almost all, which matches the table.

Scores rise from Tiny to Large, though not evenly, and then level off or slip at Huge. The authors suggest that very large models learn features a single linear layer struggles to use.

Where the model looks

The last figure asks whether the model has learned anything a producer would recognise. The authors hide part of the Pixel Galaxy spectrogram and plot where the predictor's final layer looks while it fills the gap (Gu et al., 2026a).

When a kick is masked, the model does not just look at neighbouring patches. Its attention jumps to the other kicks across the whole clip, the ones with a similar rhythmic pattern. When the root notes of a lead phrase are masked, it looks at the higher harmonics of the same notes, which are still visible, and at the lead melody just before the gap.

The authors read this as the model learning to rebuild a multitrack project in its head, as a producer does. It is a striking picture, but it is one song and a qualitative check. Attention maps hint at what a model uses; they do not prove it.

How to use it today

The GitHub repository is mainly a training recipe (Gu et al., 2026c). You create a Python 3.12 environment, point the scripts at a folder of FLAC files, compute the dataset's mean and standard deviation with a statistics script, and launch training with Hugging Face Accelerate using one of six configurations, from Tiny to Giant. The README warns that the defaults target large GPUs; if memory runs out, lower the batch size, add gradient accumulation or pick a smaller model.

The Hugging Face repository holds a checkpoint for each size. The model files range from about 192 MB for Tiny and 363 MB for Small to 1.05 GB for Base, 3.36 GB for Large, 6.86 GB for Huge and 12.2 GB for Giant. Each folder also stores the optimizer state, bringing the repository to about 48.9 GB, so download only the files you need. The 1.1-billion-parameter Giant model is not in the paper's results tables.

As of 13 October 2026, the README documents training but not feature extraction, and the model card only says to place the checkpoints in the repository's experiments folder. Until the promised MARBLEv2 integration arrives, using PupuM2D as a feature extractor means loading a checkpoint with the repository's model code yourself.

For producers, PupuM2D is not a plug-in. It is a building block for the tools around production, such as sample-library search, auto-tagging, key and tempo detection, and DJ software. The promised tokenizers, which would turn audio into discrete units, could also let generative models build on it.

Caveats

• A preprint under review. The arXiv listing says the paper was submitted to TASLP, an IEEE journal on audio, speech and language processing. It has not yet been peer reviewed, and it has been revised twice and renamed once.

• "State of the art" is the authors' claim. The tables show clear gains over 1D music models, but general audio models still lead several tagging and genre tasks.

• Different read-outs. Most baselines are official checkpoints run in the authors' own setup. For the 2D models, including PupuM2D, each reported score is the best across a sweep of read-out strategies, while the 1D models use a standard read-out.

• Linear probing only. There are no fine-tuning results, and MARBLEv2 scores cannot be compared directly with MARBLE v1 scores.

• Unknown training data. The 100,000 hours of music are described only as an in-house dataset, with no detail on sources or licensing.

• Still arriving. Tokenizers and MARBLEv2 integration were announced as coming soon on 17 September 2026.

Links

References

a43992899. (2025). MARBLE (Version 2) [Computer software]. GitHub. Retrieved October 13, 2026, from https://github.com/a43992899/MARBLE

Gu, Y., Zhang, J., Li, J., Wu, Z., & Juvela, L. (2026a). Frequency-aware self-supervised music representation learning (arXiv:2606.25713). arXiv. https://doi.org/10.48550/arXiv.2606.25713

Gu, Y., Zhang, J., Li, J., Wu, Z., & Juvela, L. (2026b). Frequency-aware self-supervised music representation learning [Project page]. Retrieved October 13, 2026, from https://www.yichenggu.com/PupuM2D/

Gu, Y., Zhang, J., Li, J., Wu, Z., & Juvela, L. (2026c). PupuM2D [Computer software and model weights]. GitHub and Hugging Face. Retrieved October 13, 2026, from https://github.com/sizigi/PupuM2D

Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., & Kashino, K. (2023). Masked modeling duo: Learning representations by encouraging both networks to model the input. In ICASSP 2023: 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 1–5). IEEE. https://doi.org/10.1109/ICASSP49357.2023.10097236

Niizumi, D., Takeuchi, D., Ohishi, Y., Harada, N., & Kashino, K. (2024). Masked modeling duo: Towards a universal audio pre-training framework. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32, 2391–2406. https://doi.org/10.1109/TASLP.2024.3389636

pupunaama. (2026, September 17). We just released PupuM2D… [Threads post]. Threads. https://www.threads.com/@pupunaama/post/DdZ5thSCr52

Yuan, R., Ma, Y., Li, Y., Zhang, G., Chen, X., Yin, H., Zhuo, L., Liu, Y., Huang, J., Tian, Z., Deng, B., Wang, N., Lin, C., Benetos, E., Ragni, A., Gyenge, N., Dannenberg, R., Chen, W., Xia, G., . . . Fu, J. (2023). MARBLE: Music audio representation benchmark for universal evaluation. Advances in Neural Information Processing Systems, 36, 39626–39647. https://proceedings.neurips.cc/paper_files/paper/2023/hash/7cbeec46f979618beafb4f46d8f39f36-Abstract-Datasets_and_Benchmarks.html