How to Implement Acoustic Echo Cancellation in Python

Acoustic echo cancellation (AEC) removes the sound your speaker plays from the signal your microphone records. Without it, the person on the other end of a call hears their own voice a moment later. In Python you can use bindings to an established echo canceller or write a simple adaptive filter yourself. Both need the same groundwork: a correct reference signal, matching sample rates and sensible input levels. This guide covers that groundwork, then how to apply and check the canceller.

Identify the symptom and environment

Start by confirming that the problem is echo. Echo is a delayed, recognisable copy of the far-end audio in your microphone signal. A rising howl is acoustic feedback. A hollow room sound is reverberation of your own voice. Hiss or hum is background noise. AEC targets the first case, so describe what you hear before changing code.

Record when the echo occurs:

• Only when audio plays through speakers, or also with headphones
• From the start of a session, or only after some time
• Only at high playback volume, or at any volume
• Only with one input device or output device, or with all of them
• Only while both sides are speaking, or also when only the far end speaks

Also write down the environment: operating system, Python version, the audio I/O library you use to open streams, the names of the input and output devices, the sample rate and frame size of each stream, and whether the echo canceller runs live or on recorded files. These details decide which of the causes below are possible.

Isolate where the fault occurs

Try the same session with headphones. If the echo disappears, the sound travels from speaker to microphone through the air, and the canceller is the right tool. If the echo stays with headphones, the far-end audio reaches the input by another route. Check whether the selected input is a loopback, monitor or mixed-output source, and whether your code accidentally adds playback samples to the captured buffer. No echo canceller can fix routing like that. Fix the device selection or the code path first.

Next, check the inputs to the canceller:

• Reference signal: it must be the exact audio sent to the output device, not a copy taken earlier in your pipeline before volume changes or resampling.
• Sample rate: the microphone and reference must use the same rate as the canceller. A mismatch makes the echo path look like it is constantly changing.
• Clock: separate input and output devices run on separate clocks that can drift apart slowly. A canceller that works at first and degrades over minutes often points to drift.
• Input level: if the microphone clips, the echo is no longer a linear copy of the reference and a linear filter cannot remove it fully.
• Other processing: the operating system, driver or calling app may already apply echo cancellation, noise suppression or automatic gain. Stacked processing can change the signal in ways your canceller does not expect.

To compare causes, run the canceller offline on a recorded pair of files: the reference and the microphone signal, captured in the same session. If it works offline but not live, look at buffering and timing in your stream code. If it fails offline too, look at alignment, levels and the algorithm settings.

Try reversible fixes in order

1) Open one full-duplex stream where possible. Many Python audio libraries can open a single stream with both input and output on the same device. That keeps the two directions on one clock and delivers matching blocks to your callback. If you must use separate devices, plan for drift compensation or resampling.

2) Use a fixed frame size and a single sample rate. Pick one rate for capture, playback and processing, and convert once at the edge if needed. Many echo cancellers process short frames, often around 10 ms. At 16 kHz that is 160 samples per frame. Check the frame size your chosen library expects instead of assuming.

3) Measure the delay between reference and microphone. Play a known test signal, record it, and cross-correlate the reference with the microphone signal. The lag of the correlation peak is the round-trip delay through buffers, converters and air. Either delay the reference by most of that amount or make sure the canceller's filter is long enough to cover it. A filter must also span the room's reverberation tail, so in large or hard rooms it needs to be longer.

4) Choose the canceller. For production use, Python bindings exist for established echo cancellers such as the Speex DSP library and the WebRTC audio processing module. Their package names, maintenance status and function signatures change between releases, so check the documentation of the version you install. The typical pattern is the same: create a canceller with the frame size, filter length and sample rate, then for each frame pass the microphone block and the matching reference block and receive a cleaned block.

5) If you want to understand or prototype the algorithm, implement a normalised least mean squares (NLMS) filter with NumPy. Keep a buffer of the last N reference samples as vector x and a weight vector w of the same length. For each new sample, estimate the echo as y = w·x, compute the error e = d − y where d is the microphone sample, then update w by adding mu · e · x / (x·x + eps). The error e is the cleaned output. The step size mu must stay between 0 and 2 for stability, and small values adapt more slowly but more steadily. The small constant eps prevents division by zero during silence. A sample-by-sample Python loop is too slow for live audio at typical rates with long filters, so for real-time use move to a block or frequency-domain variant, or a compiled library.

6) Handle double-talk. When the near-end person speaks at the same time as the far end, their voice appears in the error signal, and the filter tries to adapt to it. That pulls the filter away from the true echo path. Add a double-talk detector that slows or freezes adaptation when the microphone level is high relative to the reference. A simple approach compares the microphone magnitude with the recent peak of the reference. Library cancellers usually include their own handling, so do not add a second one on top without testing.

7) Order the processing chain. Apply echo cancellation before noise suppression and automatic gain control. Those stages change the microphone signal in non-linear, time-varying ways that break the relationship between reference and echo. Add residual echo suppression after the linear filter if faint echo remains.

8) Set levels conservatively. Lower the input level until speech peaks stay clearly below full scale, and avoid driving the speaker into distortion. Both changes are reversible and often remove the remaining non-linear echo that a linear filter cannot model.

Confirm the result

Test in stages with the same recordings or setup you used to describe the symptom:

• Far-end only: play speech through the speaker with nobody talking locally. After a short convergence period the output should be close to silence. You can quantify this with echo return loss enhancement (ERLE): 10 · log10 of the microphone power divided by the output power over the same segment. Higher is better, and you should compare values from your own runs rather than assume a target.
• Near-end only: speak with no playback. Your voice should pass through without obvious distortion.
• Double-talk: both sides speak at once. The near-end voice should remain intelligible, and the echo should not return strongly after the overlap ends.
• Path changes: move the microphone or change the speaker volume. The canceller should re-converge, and you should note roughly how long that takes.
• Long session: run for several minutes to reveal clock drift or slow divergence.

Finally, test a real call or recording with a second person listening on the far end. Confirm that they no longer hear their own voice and that your voice still sounds natural.

If the problem remains

Keep a small, reproducible evidence set: the time-aligned reference and microphone files, the canceller output, the sample rate and frame size, the measured delay, filter length, step size or library settings, the device names and the library versions. These recordings contain voices, so store and share them only according to the privacy rules that apply to the people recorded.

If the canceller works offline but not live, focus on buffering, callback timing and clock drift in your stream code. If it fails with one library but works with another on the same files, report the case to that library's maintainers with the minimal recordings. If the echo is present only with one specific device, check whether that device or its driver applies its own processing. Contact device support only when the device misbehaves without your code involved, for example when the microphone distorts or picks up playback even with headphones. An echo problem on its own does not prove that the hardware is faulty.