COURSE 01 · AUDIO SIGNAL PROCESSING
From “what is sound?” to MFCC
Written for readers with no background at all. The whole course builds toward one project: let a program listen to one second of music and decide whether it is classical, jazz or rock.
23 lessons247 figures8 interactive labs23 runnable scripts
The lesson text is only available in Chinese for now. Lesson titles, introductions, menus and buttons are shown in English.
00Before you start
01—05Sound and digital recording
- 01What does a program have to get past before it can tell classical, jazz and rock apart?Give a program a photo of a cat and it receives a neat grid of colours. Give it the sound of a passing car and it receives a long list of numbers arriving one after another over time. Both jobs are called “classification”, but sound needs one extra step: someone first has to turn that list of numbers into evidence that can be compared, or the program has nothing to compare. That step is called audio signal processing, and it is what these 23 lessons are about. This lesson covers no specific algorithm. It settles three things: what task we want the program to do, the order the 23 lessons follow, and what background you need.
- 02How is sound made, and how does it become the line you see in a recording?The last lesson set the task: get a program to tell classical, jazz and rock apart. But what the program receives is a long list of numbers. Where do those numbers come from? This lesson starts at the very beginning. An object vibrates and pushes the air around it; that push spreads outward, ring after ring, until it reaches a microphone, and the microphone records how hard the air is pushing at each instant as a number. Plot those numbers over time and you get the wobbly line in a recording app, called a waveform. The line hides three things at once: how fast the vibration is, how big it is, and what it sounds like (its timbre). This lesson covers the first two; timbre is left for Lesson 03.
- 03The volume meter hasn’t moved, so why does another song sound clearly louder?The last lesson covered frequency and amplitude in a waveform, but two things are still missing: how do we actually measure how strong a sound is, and what is timbre? This lesson starts at the sound source and works toward the ear, one layer at a time: how much energy the source gives off each second (sound power) → how much of it reaches each square metre where you are (sound intensity) → from the faintest sound you can hear to the point where it hurts, there is a factor of ten trillion → so we use decibels to squeeze that into a readable number → yet decibels are still not the loudness you hear. Finally we answer the “same volume, but different” question: once pitch and loudness are matched, the difference that remains is called timbre, and it comes down to three things.
- 04How does continuous sound become a list of numbers, and then turn back into sound?The previous two lessons dealt only with things that change continuously: air pressure, frequency, loudness. A computer can store none of them as they are: it cannot keep infinitely many moments, or infinitely many possible values. This lesson follows the whole conversion chain: a microphone produces a continuously changing voltage → a component called an ADC turns it into a list of numbers → you compute whatever you like → finally a component called a DAC turns the numbers back into a continuous voltage that drives a speaker. The ADC does only two things: how often it measures (sampling) and how finely it measures each time (quantisation). Each has its own cost, and the two costs are completely different in kind: one can be fixed afterwards, the other can never be recovered.
- 05What should we compute from a recording and hand to the program? Five dimensions of audio featuresBy the end of the last lesson, 30 seconds of music had become a list of more than 600,000 numbers (over 22,000 per second, for thirty seconds). Handing that list straight to a classifier doesn’t work: it is too long, too detailed, and no single number means anything on its own. So we first compute some summaries, called audio features. Search for “which audio features should I use” and you get a long list of names, but those names don’t answer the same kind of question. Instead of memorising the list, this lesson gives you a map: any audio feature can be placed along five dimensions — level of abstraction, time scope, musical aspect, signal domain, and the method used to compute it.
06—10Cutting a recording into pieces you can compute with
- 06From a whole recording to a list of comparable numbers: what steps lie in between?The last lesson split “what to compute” into five dimensions, and the signal-domain dimension has three levels: time domain, frequency domain and time–frequency. This lesson walks through the full pipeline for the first two — from the list of numbers coming out of the ADC to the few numbers finally handed to the program — explaining each step and why none can be skipped. Both pipelines share the same first half (both start by cutting the signal into pieces). They part ways on the frequency-domain side, which has one extra step: windowing. And windowing creates a new problem, which leads to the conclusion that frames must overlap.
- 07Without any frequency analysis, what can the curve alone tell us?The last lesson built the pipeline, but the “compute per frame” box is still empty. This lesson fills it with three time-domain answers: the amplitude envelope (the highest value in this frame), the root mean square, or RMS (how strong this frame is overall), and the zero-crossing rate (how many times this frame crosses the centre line). None of them needs any frequency analysis; they look only at the curve itself. They are not three interchangeable names but three different questions, each with its own formula, weaknesses and uses. This lesson works through all three formulas by hand on a small frame of eight numbers, so you can check them with a pen.
- 08Turning the amplitude-envelope formula into code that actually runsThe last lesson explained three formulas but didn’t implement a single line. This lesson builds the first one from scratch: the amplitude envelope. We load three music clips, check their basic information, plot the waveforms, write the envelope by hand, convert frame numbers to seconds, and finally overlay the envelope on the waveform to compare the three genres. Along the way we find an easy-to-miss abs (absolute value): it makes almost no visible difference to the maximum over a whole clip, but frame by frame it causes 18% to 36% of the frames in the three clips to be underestimated.
- 09Call the library first, write it by hand second, then make them agreeThis lesson implements the two remaining time-domain features, in a different way from last time: first compute them in one line with librosa, then write them from scratch, then compare the two results frame by frame. This isn’t a formality. If the numbers don’t match, you and the library disagree about which samples make up a frame, and every comparison after that falls apart. For RMS the two agree exactly (largest per-frame difference 8×10⁻⁹). For the zero-crossing rate they deliberately don’t: the ratio is exactly 1024/1023 — the “divide by K or by K−1” question left open in Lesson 07.
- 10How does a computer find which high and low components a single curve is made of?The previous four lessons asked everything you can ask by looking at the curve alone. But some things it can never answer: press three piano keys at once and the microphone records just one rising and falling curve — nothing in the waveform shows that there are three notes inside. This lesson changes the question: not how the signal rises and falls over time, but which high and low components it is made of. The method is surprisingly plain: take a wave of known frequency, multiply it with the signal point by point, and average. If they match, the average isn’t zero; if they don’t, the positives and negatives cancel out to zero. Try every frequency, and the curve you get is the spectrum.
11—15From waveforms to time and frequency
- 11How can one number hold both “how strong” and “where it starts”?The last lesson left a loose end: a single probe wave can be wiped out depending on where the signal starts, so we need two. That means each frequency gives us not one number but two — the strength, and where it starts. But “strength” is an ordinary number, and one ordinary number can’t hold two things. Complex numbers, introduced in this lesson, exist to solve exactly this. A complex number is a point on a plane, and a point naturally carries two pieces of information: how far it is from the origin, and in which direction. The first corresponds to strength, the second to the starting point. No abstract maths here: we simply combine the two readings measured in the last lesson into one complex number.
- 12Writing “how strong” and “where it starts” into a single coefficientThe last lesson got complex numbers ready: the magnitude records strength, the angle records the starting point. This lesson builds them into the Fourier transform properly. For each frequency we compute one complex coefficient; its magnitude gives the magnitude spectrum and its angle gives the phase spectrum. The scary-looking formula, once taken apart, still does what Lesson 10 did — multiply by a wave, then average — except that Euler’s formula merges the two probe waves into a single exponential of e. Finally we do a full round trip, transforming and transforming back: the largest point-by-point error is 2.5×10⁻¹³. Throw the phase away before transforming back, and the error becomes 2.05 — eight trillion times larger.
- 13A computer only has a finite number of values. How can it do a Fourier transform?The last lesson’s formula assumed that time is continuous and that infinitely many frequencies can be checked. A real computer only has a finite number of samples. To make the formula actually run, we limit time and frequency once each: keep N samples, and compute only N frequency bins. The result is called the discrete Fourier transform, or DFT.
- 14Reading the spectrum of real sounds in Python: why do four sounds look so different?Play the same note and a violin, a piano and a saxophone still sound different; noise looks different again. This lesson hands four recordings to Python, compares how their strength is spread across frequencies, and turns the horizontal axis into real Hz.
- 15Why a whole-clip spectrum can’t tell when a sound happened: the short-time Fourier transformTwo clips each play a low note and a high note one after the other, only in swapped order — and their whole-clip frequency results can look exactly the same. To let the program recover which sound came first, we split the recording into small overlapping pieces and recompute at every position.
16—20Shaping spectra into model input
- 16Same numbers, but you only see them when you draw them differently: plotting a spectrogram in PythonThe last lesson turned a recording into a big table of which high and low components are how strong at which moment. This lesson actually draws it. But the first picture comes out almost entirely black — not because the maths is wrong, but because of how the colours are assigned: 99.70% of the cells in the table are less than one hundredth of the strongest cell, so colouring in plain proportion squeezes them all into the same black. Colour in decibels and put the vertical axis on a ratio scale, and the same numbers all become visible.
- 17Re-marking the frequency ruler to match how we hear: the mel scaleThe last lesson put the vertical axis on a ratio scale because, in music, going up an octave means doubling the frequency. But the human ear doesn’t hear strictly in ratios. An old experiment shows this: going from C2 up to C4 and from G6 up to A6 changes the vibration rate by 196 and 192 — almost the same. Yet the first sounds like a jump of two octaves and the second like a single whole tone. The same difference, but a twelvefold difference in what you hear. This lesson swaps the frequency axis for a ruler marked out by listening experiments, then uses ten triangles to squeeze 1025 rows down to 10.
- 18The shapes match, but every number is different: extracting a mel spectrogram in PythonThe last lesson built the set of mel filters by hand in five steps — ten triangles, each covering one band of frequencies, most sensitive in the middle and fading out toward the edges. This lesson has a ready-made library build them too and puts the two results side by side: the shapes are identical, yet not a single number agrees. Following the differences uncovers three defaults: the library’s default top frequency isn’t 8000, it rescales every triangle to equal area by default, and it doesn’t even use the same mel formula. Fix those three and the largest difference drops from 0.9999 to 0.0147. Even the small difference that remains has an explanation.
- 19Separating who is making the sound from which sound they make: what MFCCs areA voice carries two things at once: how fast the vocal folds vibrate (which sets the pitch) and what shape the mouth makes (which decides whether you hear “ah” or “ee”). In the spectrum these two are multiplied together, tangled so that neither can be pulled out. This lesson takes just one step: take a logarithm, and multiplication becomes addition; two things that are added can then be moved to opposite ends of one axis with another transform. Each step along the way is measured: 98.3% of the envelope sits on the left and 99.0% of the excitation on the right, so a single cut separates them. The final DCT brings the correlation between neighbouring mel bands down from 0.95 to 0.15, and the first 13 coefficients hold 98.2% of the variation in the original spectrum.
- 20The four steps one line of code does for you: getting MFCCs in PythonThe last lesson wrote the whole MFCC chain by hand. This lesson hands it to the library: librosa.feature.mfcc returns 13 rows of coefficients in one line, and librosa.feature.delta adds the first- and second-order rates of change — together, the familiar 39 dimensions per frame. As usual, the first thing to do after getting a one-line result is to break it back into the four hand-written steps from the last lesson and line them up digit by digit: only when all four defaults are set correctly does the largest difference drop to 0.0000. Two surprises come up along the way: delta is not “next frame minus previous frame”, and no single one of the 13 coefficients can separate the three genres on its own.
21—23Summing up a spectrum in a few numbers
- 21Squeezing one column of a spectrum into three numbers: band energy ratio, spectral centroid and bandwidthOne column of a spectrogram has over a thousand frequency cells. It is complete, but not suited to answering three simple questions directly: how much low frequency is there compared with high? Where is the centre of gravity of the frequencies? How widely are they spread around that centre? This lesson gives each question its own number: the band energy ratio, the spectral centroid and the spectral bandwidth. We also build three spectrum columns with known answers to show that the three numbers look at different sides of the sound, and none can stand in for another.
- 22Implementing the band energy ratio in Python: which cell should 2000 Hz fall in?The formula for BER (band energy ratio) is a single division, but the code hides two traps that raise no error: mistaking the number of time frames for the number of frequency cells, and using an approximate cell spacing that puts 2000 Hz on the wrong side of the cut. This lesson computes BER frame by frame for three 30-second music clips. The correct boundary is cell 186; mistake the 1292 time frames for the number of frequency cells and the boundary moves to cell 234 — about 2519 Hz in reality. The median BER of the Debussy clip then comes out 4.37 dB higher.
- 23Spectral centroid and bandwidth: measuring where frequencies are centred and how far they spreadThe spectral centroid and bandwidth are often dismissed with two library calls. This lesson goes the other way: first get the time tracks of three music clips with the library’s default settings, then break the two calls down into formulas and recompute them frame by frame yourself. The largest difference is only 1.917e-05 Hz for the centroid and 1.156e-05 Hz for the bandwidth. Lining them up also exposes a definition that is easy to mix up: mean absolute distance is p = 1, but the library computes the root-mean-square distance, p = 2, by default — and on these three clips the two differ by a factor of 1.30 to 1.54.