Logged sensor data rarely arrives clean. A pressure trace from a pump test, a temperature log from a greenhouse, an accelerometer recording from a conveyor — all of them pick up mains hum, electrical interference, quantisation noise and the occasional physically impossible spike. The question is not whether to filter, but where: in the time domain, sample by sample, or in the frequency domain, after transforming a block of data into its spectrum.
Both approaches are standard, both are a few lines of code in Python, MATLAB or C, and both can quietly ruin your measurements if applied carelessly. The choice comes down to three things: what the noise looks like, which features you cannot afford to blur, and how much computation and delay your application will tolerate.
Where the two approaches differ
Time-domain filtering works directly on the sequence of samples. Each output value is a function of a small neighbourhood of input values: an average, a median, a weighted sum. The filter runs as the data arrives, keeps a short history in memory, and produces an output almost immediately.
Frequency-domain filtering treats a block of samples as a sum of sinusoids. You transform the block with an FFT, multiply or zero parts of the spectrum, then transform back. This works best when the noise occupies narrow, steady bands and the signal behaves consistently across the block.
For linear filters, the two views are mathematically equivalent — a moving average has a frequency response, and a low-pass filter can be written as a convolution. The practical differences are what matter: how much data you need before producing output, how the ends of the record behave, and how the filter distorts features you care about.
Moving averages: cheap, smooth and slightly late
A moving average replaces each sample with the mean of the last N samples. With a ring buffer it costs one addition and one subtraction per sample, whatever N is. For uncorrelated noise it reduces the standard deviation by roughly the square root of N, which is a decent return for very little work.
The price is blur. A clean step edge becomes a ramp of length N; a short peak both shrinks and widens. A causal moving average also delays the signal by about (N−1)/2 samples, so a threshold alarm fires later than it should. A centred version removes the lag but needs future samples, which is fine for offline analysis and no use at all for live control.
Useful rule: choose the shortest window that brings noise down to the level your downstream logic needs, then check the step response. If the ramp is longer than the event you are trying to detect, the window is too long.
Two relatives are worth knowing. An exponential moving average uses less memory and gives more weight to recent samples, but still lags. A Savitzky–Golay filter fits a low-order polynomial across a window and preserves peak height and width far better than a plain average at similar noise reduction — often the better choice for logged spectra and chromatograms.
Median filters: built for spikes
A median filter outputs the middle value of its window. It is non-linear, so it has no simple frequency response, and that is exactly why it works. An isolated wild reading — a dropout, a bit error, a loose connector — is simply outvoted by its neighbours. Step edges survive almost untouched, and unlike a mean, a single large outlier cannot drag the result with it.
Typical windows are 3, 5 or 7 points, and the length should be odd. If the disturbance is wider than half the window, the median will follow it rather than remove it. The usual cost is sorting, though for small fixed windows a sorting network is quick and branch-free versions exist.
The risk is over-enthusiasm. A genuine narrow pulse lasting two samples disappears under a 5-point median, and long windows turn slowly changing signals into visible staircases. For logs with both spikes and hiss, a median followed by a short average is a sound pairing.
FFT filtering: powerful, but pick your moment
FFT filtering shines when interference is narrowband and steady. Mains hum at 50 Hz and its harmonics, a resonant vibration from a pump, a carrier leaking in from nearby equipment — all sit in a few bins, and attenuating those bins removes them without smearing the rest of the trace. A moving average would need to be very long to touch the same problem, and would blur everything along the way.
The mechanics matter. Take a block of N samples, apply a window function such as Hann to reduce leakage, transform, adjust the spectrum, transform back, then stitch blocks together with overlap-add or overlap-save to avoid wrap-around glitches at the joins.
Three constraints govern the results. Frequency resolution is the sample rate divided by N, so separating hum from a nearby signal demands a long block. The signal must be roughly stationary across that block, which drifting or intermittent data rarely is. And zeroing bins amounts to a brick-wall filter, which rings: transients acquire a pre-echo and a tail. A gentle roll-off in the spectrum causes far less trouble than hard zeros.
Computationally it costs N log N per block, which for long filters beats direct convolution comfortably. It also cannot recover anything above the Nyquist frequency — if noise has already aliased into your band, no amount of spectral surgery will remove it.
Comparing the three on real logs
- Noise removal: moving average for broadband hiss, median for impulsive spikes, FFT for narrowband interference such as mains hum.
- Edge preservation: median first, Savitzky–Golay close behind, moving average last.
- Cost: a moving average is a couple of operations per sample; a small median is also cheap; FFT filtering costs N log N per block plus buffer management.
- Latency: causal averages lag by half the window, medians lag less, block-based FFT filtering needs a whole block before it can output anything.
- Weak spots: averages on spikes, medians on sustained bursts, FFT on drifting signals and sharp transients.
A workflow that keeps you out of trouble
- Plot the raw trace before touching it. Note spikes, drift, steps and any visible ripple.
- Look at the spectrum of a representative chunk. A clear peak at 50 Hz tells you the interference is narrowband; a flat floor says it is broadband.
- Decide what must survive: step edges, peak amplitudes, the timing of events.
- Start with the simplest filter that reduces noise enough. A 5-point median for spikes, a short average for hiss.
- If interference is narrowband, notch it — in the frequency domain for offline logs, or with a time-domain notch for streaming data where latency matters.
- Compare filtered and raw output on a section containing a known event. Note how much the peak shrank and how far it moved.
- Record the filter settings alongside the data. Re-tuning a mystery filter six months later is nobody's idea of fun.
Keep the raw log
Whatever you choose, store the unfiltered measurements and filter a copy. Filters are decisions, and decisions get revisited: a window that suited a slow temperature survey will flatten a fast pressure transient. Keeping the raw record means you can re-run the whole analysis when the question changes. In most logging work, the honest answer is not one filter but two — a median to kill the spikes, then a short average or a spectral notch to settle the rest, with both settings written down next to the data.
Photo: Pexels / Pixabay
