I liked seeing those diagrams on how transformer attention looks at relevant words and grammar for text or translation, and how it looks at important components for vision. What does attention on audio sound like?
The attention mechanism in transformers at the the first layer literally points to sections of the input audio, so I vibe-designed a system to use a model like wav2vec to go over the entire file like normal music playback, and take the most active heads and see what other regions of the file they were attending to, and played these back. This looks like using a cassette player’s tape heads to run over discontinuous sections of the tape, an idea that Nam June Paik explored with beautiful tape dissection on a wall and a person that would hold and move a tape head connected to a speaker, randomly over the wall [1].
I hooked up a few popular transformer based encoders including wav2vec 2.0, MuQ, and HuBERT and ran it through this system. Each file has three sections, each with more layers being sonified. The first layer of the transformer operates directly on the input audio. The second layer operates on the activations from the first layer’s attention heads, so the process is recursive. The lines help you see which head is attending to which range of audio or other heads.
Test signals
Mixture of tones, chirps, noise in a loop, first the input audio, then the wav2vec attention:
Music
Chopin Nocturne
(Original: https://www.youtube.com/watch?v=QR10Od1cLaM)
Radiohead
Miles Davis
John Cage speaking
Rain
Thoughts
I expected this to have some insight into the way transformers work on audio, and I think it is useful. Sometimes it does what you expect, attention goes to similar frames from other parts of the audio. And other times, it seems a bit stuck on some parts that don’t seem obviously relevant. Some heads are just not active across multiple songs even, but we can’t rule out that they aren’t useful. For music, you can kind of hear the original song even though we disallow playing frames near the playback cursor. I’m still not sure why the models attend to longer sequences for certain audio. The models did sound quite different from each other. Some part of this might be related to kv dot product thresholds for how long of a duration to playback and when to switch away.
To be clear, since I’ve done various noise-related computer music before, I consider this kind of inspection sonification without explicit musical intent, although there is an aesthetic in that the sonification should provide a new perspective on the original process. There definitely are related sounding things in avant garde computer music, like James Tenney’s Collage #1 [2] or Plunderphonics, but those works were intentionally music, and this is similar mostly at surface value. There’s something logical about this that is not musical at all, yet the patterns contain rhythm and meaning. This pattern is literally the information which the model has decided it can use to reconstruct the original.
Blue Suede Shoes (processed on original, not Tenney’s)
This is a proof of concept that was purely vibecoded. I asked for visuals that were between minimal fintech and what Ryoji Ikeda did. The sonification used some ideas from an old project of mine called treemusic.
I would love to get feedback on how or if there is a way to use this for diagnosis, or if there’s other more interesting sonification schemes to look at. I’m sure there are bugs in it that need working out. Here’s the Github.
References
[1] Nam June Paik, Random Access Audio Tape https://njpart.ggcf.kr/collections/283
[2] James Tenney, Collage #1 (Blue Suede Shoes) https://www.youtube.com/watch?v=P9TGvSUUR7s