Fundamental frequency (F0)¶

Introduction¶

The fundamental frequency of a speech signal, often denoted by F0 or $F_{0}$, refers to the approximate frequency of the (quasi-)periodic structure of voiced speech signals. The oscillation originates from the vocal folds, which oscillate in the airflow when appropriately tensed. The fundamental frequency is defined as the average number of oscillations per second and expressed in Hertz. Since the oscillation originates from an organic structure, it is not exactly periodic but contains significant fluctuations. In particular, amount of variation in period length and amplitude are known respectively as jitter and shimmer. Moreover, the F0 is typically not stationary, but changes constantly within a sentence. In fact, the F0 can be used for expressive purposes to signify, for example, emphasis and questions.

Typically fundamental frequencies lie roughly in the range 80 to 450 Hz, where males have lower voices than females and children. The F0 of an individual speaker depends primarily on the length of the vocal folds, which is in turn correlated with overall body size. Cultural and stylistic aspects of speech naturally have also a large impact.

The fundamental frequency is closely related to pitch, which is defined as our perception of fundamental frequency. That is, the F0 describes the actual physical phenomenon, whereas pitch describes how our ears and brains interpret the signal, in terms of periodicity. For example, a voice signal could have an F0 of 100 Hz. If we then apply a high-pass filter to remove all signal components below 450 Hz, then that would remove the actual fundamental frequency. The lowest remaining periodic component would be 500 Hz, which correspond to the fifth harmonic of the original F0. However, a human listener would then typically still perceive a pitch of 100 Hz, even if it does not exist anymore. The brain somehow reconstructs the fundamental from the upper harmonics. This well-known phenomenon is however still not completely understood.

A speech signal with a fundamental frequency of approximately F0=93Hz.

The spectrum of a speech signal with a fundamental frequency of approximately F0=93Hz (original) and a high-pass filtered version of it such that the fundamental frequency has been removed (high-pass filtered).

A speech signal with a fundamental frequency of approximately F0=93Hz

A high-pass filtered version of it such that the fundamental frequency has been removed

If $F_{0}$ is the fundamental frequency, then the length of a single period in seconds is

The speech waveform thus repeats itself after every $T$ seconds.

A simple way of modelling the fundamental frequency is to repeat the signal after a delay of $T$ seconds. If a signal is sampled with a sampling rate of $F_{s}$, then the signal repeats after a delay of $L$ samples where

A signal $x_{n}$ then approximately repeats itself such that

In the Z-domain this can be modelled by an IIR-filter as

where the scalar $0\leq\gamma_L\leq 1$ scales with the accuracy of the period. The Z-transform of the signal $x_{n}$ can then be written as $X(z)=B^{-1}(z) E(z),$ where $E(z)$ is the Z-transform of a single period.

Segment of a speech signal, with the period length $L$, and fundamental frequency $F_0=1/L$.

Spectrum of speech signal with the fundamental frequency $F_{0}$ and harmonics $kF_{0}$, as well as the formants F1, F2, F3... Notice how the harmonics form a regular comb-structure.

The magnitude spectrum of $B^{-1}(z)$, has then a periodic comb-structure. That is, the magnitude spectrum has peaks at $k\,F_0$ , for integer $k$. For a discussion about the fundamental frequency in the cepstral domain, see Cepstrum and MFCC.

Spectrum of fundamental frequency model $B^{-1}(z)$, showing the characteristic comb-structure with harmonic peaks appearing at integer multiples of $F0$.

Fundamental frequency estimation¶

The fundamental frequency (F0) is central in describing speech signals whereby we need methods for estimating the $F_0$ from speech signals. In speech analysis applications, it can be informative to study the absolute value of the fundamental frequency as such, but more commonly, extraction of the $F_0$ is usually a pre-processing step. For example, in recognition tasks, $F_0$ is often used as a feature for machine learning methods. A voice activity detector could, for instance, set a lower and higher threshold on the $F_0$, such that sounds with an $F_0$ outside the valid range would be classified as non-speech.

The fundamental frequency is visible in multiple different domains:

• In an acoustic time signal, the $F_0$ is visible as a repetition after every $T$ samples.

• In the autocovariance or -correlation, the $F_0$ is visible as a peak at lag $T$ as well as its integer multiples $kT$.

• In the magnitude, power or log-magnitude spectrum, the $F_0$ is visible as a peak at the frequency $F_0=F_{s}/T$, as well as its integer multiples, where $F_{s}$ is the sampling frequency.

• In the cepstrum, the $F_0$ is visible as a peak at quefrency $T$ as well as its integer multiples $kT$.

Consequently, we can use any of these domains to estimate the fundamental frequency $F_0$. A typical approach applicable in all domains except the time domain is peak-picking. The fundamental frequency corresponds to a peak in each domain, such that we can determine the $F_0$ by finding the highest peak. For better robustness to spurious peaks and computational efficiency, we naturally limit our search to the range of valid $F_0$’s, such as  $80\leq F_0\leq 450.$

The harmonic structure, however, poses a problem for peak-picking. Peaks appear at integer multiples of either $F_0$ or lag T, such that sometimes, by coincidence or due to estimation errors, the harmonic peaks can be higher than the primary peak. Such estimation errors are known as octave errors because the error in $F_0$ corresponds to the musical interval of an octave. A typical post-processing step is, therefore, to check for octave jumps. We can check whether $F_0/2$ or $F_0/3$ correspond to a sensible $F_0$. Alternatively, we can check whether the previous analysis frame had an $F_0$, an octave, or two octaves off. Depending on the application, we can then fix errors or label problematic zones for later use.

Another problem with peak picking is that peak locations might not align with the samples. For example, in the autocorrelation domain, the actual length of the period could be 100.3 samples. However, the peak in the autocorrelation would then appear at lag 100. One approach would then be to use quadratic interpolation between samples near a peak and use the location of the maximum of the interpolated peak to estimate the peak location. Interpolation makes the estimate less sensitive to noise. For example, background noise could peak at lag 102, so the desired maximum of 100 is lower than the peak at 102. By using data from more samples, as in the interpolation approach, we can, therefore, reduce the likelihood that a single corrupted data point would cause an error.

An alternative approach to peak-picking is to use all distinctive peaks to estimate the $F_0$ jointly. That is, if you find $N$ peaks at frequencies $p_{k}$, which approximately correspond to harmonic peaks $kF0$, then you can approximate $F_0 \approx \frac1N \sum_{k=1}^N p_k/k.$ Another alternative is to calculate the distance between consecutive peaks and estimate $F_0 \approx \frac1{N-1} \sum_{k=1}^{N-1} (p_{k+1}-p_k).$ We can also combine these methods as we like.

A further potential improvement is to compare subsequent frames. The pitch in speech is fairly continuous over time, and subsequent frames should, therefore, have similar F0s. This can be used to reduce the danger of octave jumps.

In many other applications, we use discrete Fourier transform (DFT) or cosine transforms (DCT) to resolve frequency components. It would, therefore, be tempting to apply the same approach here. In such a domain, we would also already have a joint estimate which does not rely on single data points. However, note that the spectrum is already the DFT of the time signal, and the cepstrum is the DCT or DFT of the log-spectrum. Additional transforms, therefore, usually do not resolve the ambiguity between harmonics.

A classic algorithm for fundamental frequency estimation is YIN De Cheveigné & Kawahara (2002), which has been later improved with a probabilistic version pYIN Mauch & Dixon (2014). YIN is based on peak picking in the autocorrelation domain while emphasizing continuity over time.

Examples¶

Excerpts of each tone (columns) with a square (upper row) and a Hann window (lower row).

The (loosly) periodic structure is visible for each tone as a repeated waveform. Left to right, the pitch increases and the period length becomes shorter such that more periods fit the 30ms analysis window. We also note that for the lowest tone, the periods are nicely visible with the square window (upper row), but less so with the Hann window.

To estimate the fundamental frequency, recall that for a zero-mean signal $x_k$ with a period length $L$, we should have $x_k \approx x_{k-L}$ or equivalently, zero expectation $E\left[\left(x_k - x_{k-L}\right)^2\right]=0.$ For a sample segment of length $N$, the expectation has

Here ${\mathbf x}_k$ is a vector containing the samples of $x_k$ to $x_{k+N}$.

To find the fundamental frequency, we can then find that $L$ where the above norm is closest to zero.

where constants that do not depend on $L$ were omitted, and the minimum was replaced by the maximum by changing the sign. We thus obtained the maximization of the correlation between the signal and its delayed version. Note that this makes this approach equivalent with estimating the maximum of the autocorrelation function.

To make comparisons between the estimators in this chapter concrete, we score each one against a reference pitch using the mean absolute error (MAE), expressed in cents -- a logarithmic, perceptually motivated unit of pitch distance in which 100 cents make a semitone and 1200 cents make an octave:

where $\hat F_{0,n}$ is the estimated and $F_{0,n}$ the reference pitch of frame $n$. Cents make errors comparable across octaves -- an octave error is always 1200 cents, whether it happens at 100 Hz or at 400 Hz -- which a plain difference in Hz would not. We use pYIN’s own estimate as the reference throughout this chapter, since it is consistently the most accurate of the methods discussed here, though, as the closing Discussion of this chapter points out, it remains an estimate rather than a true ground truth. To keep the many figures below comparable, we also fix one colour per estimator for the remainder of the chapter.

The upper row shows the whole correlation, while the lower row zooms near the valid range of lags (indicated by the red lines). Red crosses show the maximum peak.

With this easy sound example, the maxima are easy to discern. However, with both the lowest (leftmost) and highest (rightmost) pitches, other peaks are very close in magnitude to the true pitch. As this was an easy signal, it is thus not difficult to imagine more challenging sounds where peak picking would land at the incorrect peak.

We can repeat the experiment using the power spectrum (squared absolute spectrum).

Observe that the y-axis is now expressed in decibels. The upper row has the complete spectrum of each tone, while the lower row zooms in on the frequency range where the fundamental is visible.

We immediately observe that for the lowest (leftmost) tone, the pitch is incorrectly estimated as it picks up the second peak of the harmonic structure. In fact, for the two highest pitches, the second harmonic peak is higher than the first, but it is not incorrectly picked up only because it is outside the defined frequency analysis range. Furthermore, the accuracy of estimated frequencies is low because that the density of frequencies is not sufficient. The accuracy can however be improved by oversampling the spectrum, that is, by extending the analysis window by zeros before the Fourier transform.

In other words, let us extend the vector with zeros to a length equal to the sampling rate (44100).

It is now much easier to determine the exact location of each peak and thus the frequency estimates are also more accurate. However, this did not solve the octave jump in the lowest (leftmost) tone, where the second harmonic is still larger than the first, true peak.

The third main feature often used for F0 estimation is the cepstrum.

The peaks are much more pronounced here in the cepstrum in the sense that they are sharp and easily distinguishable from the noise. Still, the same danger of octave jumps persists in the lowest and highest tones, as there are other high peaks in the search range (red lines). However, since the peaks are now narrow, we cannot easily interpolate between points to find a more accurate location of the peak.

Pitch contours and -tracking¶

Pitch is used in speech for expressive purposes. In particular, relatively high pitch (and intensity) segments typically indicate an emphasis. It is therefore important to observe changes in pitch over time. Such pitch-over-time plots are known as pitch contours. The challenge of finding “good” pitch contours, that avoids intermittent octave-jumps, is known as pitch tracking.

Here is a sample sentence that can be used to demonstrate pitch contours and -tracking.

The figure illustrates four different estimates of the pitch contour; the above presented three methods as well as pYIN, which is an improvement of the classic YIN method De Cheveigné & Kawahara (2002)Mauch & Dixon (2014).

We can observe that near 35s, all four methods are nicely aligned, giving a continuous pitch contour. However, elsewhere, the three homebrew methods are very noisy. The first obvious reason is that the presented methods estimate a pitch in all frames, also when no voiced signal is present. We can use voice activity detection to limit analysis to only speech frames. This still leaves unvoiced frames for which we need some additional detection. We can, for example, check whether the detected peak is “prominent” or large enough in its surroundings such that it really is a peak and not noise. Also, if the detected maximum is at the border of the analysis range, that often indicates an anomaly, e.g. the maximum is likely outside the analysis range. Such heuristics should then be carefully tuned.

The pYIN library function used here (from Librosa) provides such a measure of voicing activity. By filtering the pitch contours to include only voiced frames, we obtain the following.

This seems vaguely better as a large portion of the noise has been omitted. A better visualization could be to compare the three first estimates with pYIN individually and with discrete points for each frame rather than a line.

The (auto)correlation method clearly yields estimates which are closest to the high-quality estimates of pYIN, with a mean error of only about 85 cents -- a small fraction of a semitone -- over the voiced frames above. Estimates extracted from the power spectrum are quantized to discrete levels, which makes them less accurate even where they are otherwise correct, and they also feature many large errors, where often the estimate has clearly an octave error, such as at the very first segment where the red line is near 100 Hz, and some points are at the double frequency (an octave higher) near 200 Hz. These octave mistakes, each worth almost exactly 1200 cents, are frequent enough here (roughly one in ten voiced frames) that they dominate the average, giving the spectral estimate the largest mean error of the three methods, at around 450 cents.

The cepstral method has a smaller mean error, at around 400 cents, but this hides a different kind of problem: rather than clean octave jumps, most of its large errors are (seemingly) random values that do not correspond to simple multiples of the true pitch. This distinction matters for post-processing: octave errors can be fixed by checking whether the result could be a multiple of the true value and correcting when one is found, but that trick does not help with the cepstral method’s more haphazard mistakes. So despite its lower average error here, the cepstral method’s mistakes are the hardest of the three to remedy after the fact.

Neural estimators¶

More accurate and robust results can potentially be achieved by deep neural networks such as Crepe or DeepF0 Kim et al. (2018)Singh et al. (2021). Rather than picking peaks in a hand-crafted correlation, spectral, or cepstral domain, such networks are trained to map short frames of audio (or a time-frequency representation of them) directly to an $F_0$ estimate, learning the relevant feature extraction from data.

An openly available, actively maintained example of this approach is PENN (Pitch Estimating Neural Networks), a PyTorch toolkit for training and running neural pitch and periodicity estimators Morrison et al. (2023). Like the classical methods presented in this chapter, PENN estimates pitch independently for each short analysis frame and, alongside it, a periodicity score that serves as a voicing confidence. It then offers several ways of turning these per-frame estimates into a pitch contour: a simple per-frame argmax, a pYIN-style probabilistic threshold, or Viterbi decoding of the most likely path through the per-frame pitch likelihoods, the latter enforcing the kind of temporal continuity that our correlation, spectral, and cepstral estimates lacked. Its default pretrained model, FCNF0++, is trained across multiple datasets (e.g. MDB-stem-synth and PTDB) so as to generalise across recording conditions better than earlier, single-domain neural pitch trackers. PENN is thus a convenient, ready-to-use illustration of how such deep-learning-based estimators are packaged for practical use.

Let’s put PENN to the test on the same speech excerpt used above (sounds/f0speechsample.wav), and compare it with the classical estimators and pYIN.

Technical implementation details: PENN’s default decoder, Viterbi, relies on an optional companion package (torbi) that ships precompiled binaries for specific torch/CUDA version combinations and has no fallback for unsupported ones; to keep this example reproducible regardless of the reader’s installed torch version, we instead use the 'argmax' decoder, which — like our own correlation, spectral, and cepstral estimators — decides the pitch of each frame independently, without enforcing continuity with neighbouring frames.

Despite FCNF0++ being, per frame, a considerably more accurate pitch model than our hand-crafted estimators, PENN’s argmax-decoded contour tracks pYIN worse than the simple autocorrelation estimator does (around 200 cents mean absolute error, versus about 85 cents for Correlation). The reason is decoding, not the network: pYIN’s own decoder (like PENN’s Viterbi and pYIN decoders) chooses the most likely path through consecutive frames, whereas 'argmax' commits to the best bin in each frame independently, in exactly the same frame-by-frame fashion as our Correlation, Spectrum, and Cepstrum estimators - and so it inherits the same vulnerability to octave errors and other isolated mistakes. Applying the light median filter above narrows the gap (down to roughly 160 cents at a 7-frame window) but does not close it, since a fixed-length filter is a crude substitute for the dynamic-programming search a real Viterbi decoder performs. This is a useful reminder that a neural network’s raw per-frame predictions are only part of the estimator; the decoding strategy that turns those predictions into a pitch contour matters just as much, for neural and classical estimators alike.

Discussion¶

Analysis of fundamental frequency often makes sense as it is a prominent feature of speech and has an expressive function. Classical DSP methods yield good results, though octave errors will always mar estimation. Here, we presented examples only with clean voices without background noises or reverberation. More realistic scenarios, that include background noise and competing speakers, obviously bring more problems.

Neural estimators such as those presented above can improve on these classical methods, provided their decoder makes proper use of the continuity between neighbouring frames, as our own PENN example illustrated. Beyond decoding, a data-driven approach brings its own challenge, namely to define the ground truth used for training. What is the true pitch? Who defines it and how is it determined? The ground truth is required by most data-driven approaches, yet both manual and automatic pitch estimation are susceptible to errors. The danger is that the model then approximates the method which was used to create the ground-truth, rather than the underlying phenomenon.

References¶