Signal Processing: Filtering Out The Noise

In IT operations, the signal is the telemetry that indicates something requires attention. The noise is everything else, including normal variance, autoscaling events, holiday traffic patterns, and data collection gaps.

Separating signal from noise is what turns monitoring data into actionable insight. It separates alerts that drive action from those that waste an engineer's time.

In this article, we'll examine the most common sources of noise in production monitoring data and the filtering techniques used to extract actionable insights.

What Causes Noise in Production Monitoring Data?

Production environments introduce several data fidelity issues. Some of these include: 

• Data collection issues: gaps or errors in how metrics are gathered
• Missing data: incomplete time series that distort trend analysis
• External factors: autoscaling events or traffic shifts that affect metrics without indicating a problem
• Concept drift: changes in the relationship between input and output variables over time, even when inputs appear stable

The following plot shows an observed signal (in blue) with noise and the underlying signal without noise (in red).

Unlike this simplified example, production monitoring data rarely lends itself to visual inspection alone. Given the volume of telemetry generated in modern environments, algorithmic approaches are essential.

What Makes a Monitoring Insight Actionable?

Consider the time series plot below, showing Wait Time over 12 days for healthcare.gov. Performance is strong at night (low Wait Time) and degrades during the day (high Wait Time). That's informative, but it isn't actionable because a normal operating baseline hasn't been defined.

While there are spikes in Wait Time in this example, you first need to define the threshold at which a spike signals a capacity issue. If the upper bound on Wait Time were set at, say, 120 ms, then the data shows multiple instances above that threshold, pointing to potential capacity problems. For operations teams managing infrastructure at scale, this is exactly the kind of insight that drives action.

In the plot below, we can see a gradual increase across all three metrics.

Linear regression detects the consistent upward trend across these metrics, signaling the need to investigate capacity before performance degrades further.

How to Filter Noise from Monitoring Data

Noise reduction in monitoring data draws from the same filtering techniques used in image, audio, and video processing, adapted for time series. A wide variety of filters exist, but they fall into two broad categories:

Filter typeWhat it doesExampleLow passPasses signals below a frequency cutoff, attenuates above itSimple Moving Average (SMA)High passPasses signals above a frequency cutoff, attenuates below itUsed to isolate spikes or rapid changes

The red line in the plot above is the SMA of the original signal shown in blue. SMA filters out most of the noise and closely approximates the underlying signal shown earlier. Note that, by construction, there's a lag between SMA and the underlying signal.

Depending on the requirement, either linear filters (such as SMA) or non-linear filters (such as a median filter) can be used.

More advanced filtering techniques are available for environments where simple smoothing isn't sufficient:

• Kalman filters continuously estimate the true system state from noisy measurements, making them useful for dynamic environments.
• Recursive Least Squares (RLS) and Least Mean Squares (LMS) adapt to changing data over time
• Wiener-Kolmogorov filters estimate an underlying signal when the statistical properties of the noise are known

Noise reduction can also be performed in either the time domain or the frequency domain.

Frequency-domain approaches use transforms such as the Fourier Transform or Wavelet Transform to separate signal from noise before reconstructing a cleaner time series. The appropriate technique depends on the characteristics of the data and the operational problem being solved.

These techniques are effective, but applying them manually across millions of telemetry points isn't realistic. Modern observability platforms automate this analysis continuously, allowing teams to detect meaningful changes without tuning filters by hand.

Put Signal Processing to Work in Your Environment

LM Envision applies AI-powered anomaly detection, adaptive baselines, predictive forecasting, and event correlation across hybrid infrastructure monitoring.

By correlating metrics, logs, and events with operational context, LM Envision reduces alert noise, surfaces probable root causes, and helps operations teams focus on issues that require action.

When performance issues originate beyond your infrastructure, LM Internet Performance Monitoring extends visibility across internet paths, ISPs, cloud providers, CDNs, and other external dependencies, helping teams distinguish internet-related issues from problems within their own environment before users are affected.

Together, these capabilities help operations teams spend less time sorting through alert noise and more time resolving the issues that actually affect performance and reliability.