Looking for structure in "Red is a Slow Colour"

This is an experiment in signal processing, visual analysis (and even building a small tool) to explore the internal structure of a song.

How detailed can we measure the structure of a song?

I decided to take a closer look at one of the tracks on How to Measure a Planet?: Red Is a Slow Colour.

Some years ago, during the last decade, I was listening too closely to this album. I noticed a subtle echoing overlay in this song. When Anneke sings "the picture on the wall is chaotic", the word "chaotic" seems to repeat a second later, almost like an echo that wasn't meant to be heard.

So, if this song (too) contains patterns within patterns, can we visualize them as we did before? Could we actually see the repeated "chaotic"?

I wanted to explore its internal structure using signal processing, again.

The experiment would combine three things:

For the initial experiment, I averaged the stereo channels into a mono signal. This simplified the analysis by focusing on the combined audio rather than spatial differences between the channels.

But, as we've seen above, a waveform alone isn't particularly useful for identifying recurring musical textures.

We needed a representation that described how the sound changes over time.

The short-time Fourier transform (STFT) provides a time-frequency representation of the signal. Instead of looking only at the amplitude of the waveform, we examine the distribution of spectral energy across successive time windows.

Window sizes

The duration of the STFT window affects the representation.

Short windows provide better temporal resolution, while longer windows provide better frequency resolution.

I decided to experiment with five window sizes:

The intention was to compare the resulting representations rather than assume that one resolution would reveal everything.

Once the audio has been converted into feature vectors, we can compare every frame with every other frame, and the result is a self-similarity matrix (SSM).

Each axis represents time. Every point in the matrix indicates how similar two moments are according to the selected features and similarity measure as we used cosine similarity to compare the spectral feature vectors.

The diagonal represents each frame compared with itself (that's why the similarity = 1). Away from the diagonal, repeated or similar passages can produce additional patterns.

For example, if a musical passage repeats later, we might see a corresponding structure away from the main diagonal.

The similarity matrix tells us that certain moments are similar under a particular representation.

The first problem: the matrix was larger than the machine could handle

The first implementation attempted to calculate a dense self-similarity matrix for the entire track.

It failed miserably at the smallest window size.

With a hop length of 64 samples, the STFT produced approximately 266,000 frames. Comparing every frame against every other frame required a matrix with roughly 70.7 billion elements 😸

At eight bytes per element, the matrix alone would occupy approximately... 527 GiB of RAM.

So, the Linux kernel eventually killed the process. This was a useful engineering lesson.

Halving the number of frames reduces the matrix to one-quarter of its original size. Reducing the frame count to one-quarter reduces the matrix to one-sixteenth.

We could have increased the STFT window size to reduce the number of frames, but that would have changed the temporal resolution of the experiment.

I wanted to retain the ability to inspect finer details without requiring a ridiculous amount of memory.

Changing the approach: analyze the song in smaller pieces

Instead of processing the entire track as one matrix, I decided to divide it into four equal blocks, each approximately 96.5 seconds long.

This reduced the size of each matrix substantially. With the same frame density, each block's matrix would be approximately one-sixteenth the size of the original whole-track matrix.

I also changed the matrix data type from 64-bit to 32-bit floating-point values (float32), halving its storage requirements.

Finally, I adjusted the hop length according to the window size to balance temporal coverage against computational cost.

The resulting implementation would calculate the matrices independently and save each visualization separately.

Four blocks multiplied by five window sizes meant 20 plots.

facundo@HP-245-G8:~/Documents/htmap_analysis$ ls self_similarity_output/*.png
self_similarity_output/block_01_start_0000s_window_1024.png
self_similarity_output/block_01_start_0000s_window_16384.png
self_similarity_output/block_01_start_0000s_window_256.png
self_similarity_output/block_01_start_0000s_window_4096.png
self_similarity_output/block_01_start_0000s_window_8192.png
self_similarity_output/block_02_start_0096s_window_1024.png
self_similarity_output/block_02_start_0096s_window_16384.png
self_similarity_output/block_02_start_0096s_window_256.png
self_similarity_output/block_02_start_0096s_window_4096.png
self_similarity_output/block_02_start_0096s_window_8192.png
self_similarity_output/block_03_start_0192s_window_1024.png
self_similarity_output/block_03_start_0192s_window_16384.png
self_similarity_output/block_03_start_0192s_window_256.png
self_similarity_output/block_03_start_0192s_window_4096.png
self_similarity_output/block_03_start_0192s_window_8192.png
self_similarity_output/block_04_start_0289s_window_1024.png
self_similarity_output/block_04_start_0289s_window_16384.png
self_similarity_output/block_04_start_0289s_window_256.png
self_similarity_output/block_04_start_0289s_window_4096.png
self_similarity_output/block_04_start_0289s_window_8192.png

This will allow to retain several analysis resolutions while keeping the experiment manageable on my laptop's hardware.

It also offered a practical advantage. I could inspect individual parts of the song without committing to an automatic segmentation algorithm. At some point I was thinking about auto-detect chapters (as I did at some point with some audio transcriptions) but it would add an extra layer of complexity, more things that could break and I just wanted that the thing to run within my humble laptop constraints.

Something we can explore

I had the matrices but I wanted to be able to listen to the song while examining the corresponding matrix, moving between sections without opening and switching between files.

This led to a second part of the project: a small, local web applet for exploring the precomputed results.

Build an interface around the existing analysis.

The applet would reuse the data and visualizations already produced by the previous analysis.

Defining regions of the song

To make navigation easier, I manually (listening and writing the time in a notepad) divided the track into sections.

The initial segmentation was:

These labels were working hypotheses, not definitive musical classifications. In particular, the long passage between 03:12 and 04:20 was provisionally classified as a "bridge".

The purpose of segmentation was practical: to make it easier to inspect individual passages and compare repetitions.

Each region had its own PNG visualization and NumPy array. The PNG provided a convenient visual representation, while the .npy file preserved the numerical matrix for potential future analysis.

Designing the web applet

The applet was built around a small set of requirements:

The resulting architecture used Python's standard library for the HTTP server and plain HTML, CSS, and JavaScript for the frontend.

The main components

Audio player: The browser plays the original audio file using its native controls.

Timeline and regions: The manually defined sections provide an initial map of the song and a way to navigate to particular passages.

Self-similarity matrix viewer: The relevant regional visualization is displayed alongside the audio player.

Playback-synchronized cursor: A cursor moves through the matrix as playback advances, connecting the current audio position to its corresponding position in the visualization.

Image controls: Rotation, flipping, zooming, and panning make it easier to inspect patterns from different orientations and scales.

Annotations: Observations can be recorded with timestamps and saved as JSON. This provides a lightweight way to document interesting patterns without introducing a database.

This is where the experiment becomes more interesting than simply generating weird plots.

A self-similarity matrix shows how similar different moments are under the chosen signal representation.

A visible pattern might correspond to a repeated verse, a recurring instrumental texture, or another form of spectral similarity. Establishing which interpretation is appropriate requires listening and further investigation.

Questions worth exploring include:

Explore hierarchical analysis

Instead of calculating high-resolution matrices everywhere, a future version could analyze the song at several levels:

RMS energy, spectral flux, and frequency-band energy could provide initial features for identifying candidate changes.

However, a strong spectral change does not necessarily indicate a meaningful musical boundary.

Any automatic method would need to be evaluated against the actual recording and listening observations.

What we learned

The main outcome of this experiment was a small, reproducible workflow for exploring that structure.

The process evolved through a series of practical decisions:

Each decision addressed a specific problem while keeping the experiment manageable.

The resulting tool is modest, but it connects numerical analysis with direct listening in a way that makes further investigation easier.

The broader principle is straightforward:

Use computation to produce evidence, visualization to inspect it, and listening to interpret it.

and the "chaotic"?

Picking a signal from a lot of noise and interference could be quite challenging. After all, engineers are paid to make and secure communications making them hard to guess but easy to recover if we have the right keys.

I wanted to treat the audio as a cake made of many ingredients: the instrumental background is the spongy body and cream, while the vocal event we're looking for is an M&M or strawberry embedded in it. If a quiet vocal sample appears twice at different times, we might be able to detect it just by comparing slices of the recording. Too easy... The hope is that the background, being sufficiently similar between two slices, will partially cancel out, leaving a residual that reveals the vocal component. We can just brute-force sweep across different delays and window sizes to find where the similarity is strongest. This is related to common-mode cancellation, although the background won't cancel perfectly... because the instruments, timing, and mix are constantly changing. We also need to account for the possibility that the second vocal event is quieter (which is), filtered, reverberated, or otherwise different from the first.

To improve our chances, we chose to use Demucs to separate the vocals from the instrumental accompaniment before searching for the repetition. I remembered that older karaoke systems often used a much simpler trick: subtracting the left and right stereo channels to suppress centrally mixed sounds, including some vocals. That method is limited because it also removes other centred instruments and fails when vocals have stereo effects. Demucs instead uses a pretrained source-separation model to estimate a vocal stem from the mixture, giving us a cleaner signal to investigate. We can then cut the first, clearly audible “chaotic” and search for similar patterns elsewhere in the vocal track, using waveform comparison and spectral-envelope analysis at different timescales.