Consolidation and Modernization in Enterprise Observability

Organizations are seeing measurable benefits from investing in observability, including faster issue resolution, cost reduction, and improved business outcomes. However, challenges remain, including rising costs, tool fragmentation, and the need for more comprehensive monitoring of internet dependencies and user experience. Let's explore these challenges and the best practices organizations are adopting to address them.

Rising Concern Over Observability Costs

Observability is too expensive. In 2022 during an earnings call, Datadog revealed that a customer was spending $65 million on their observability tools. This news triggered increased awareness of the rising costs of observability.

Jeremy Burton wrote "Some observability tool vendors say organizations should allocate up to 30% of their total infrastructure cost to monitoring and understanding the state of their IT system. That's just nuts."

MELT Myopia Driving Higher Costs

One of the root causes of higher costs is the belief in the three pillars of observability (now MELT: infrastructure Metrics, Events, Logs, and code Traces) and the idea that everything needs to be monitored this way. Storage vendors, in particular, love logs as they drive significant storage needs.

Many IT teams are moving away from a tools-first approach to monitoring. Instead, teams are adopting an approach that begins by determining the best way to monitor specific applications or pieces of infrastructure based on the criticality of the application and the value of monitoring data.

The Challenge of Multiple APM Tools

Many IT leaders face the challenge of managing multiple APM tools, each with its own set of features, dashboards, and data sources. This fragmentation can lead to inefficiencies, as teams spend valuable time correlating data from different tools to identify and resolve issues.

The SRE Report shows a significant percentage of large enterprises have more than five observability tools due to different needs and silos. The variety is often driven by the preference of application teams. Many large enterprises use a mix of observability tools, including most of the top 5 APM platforms.

SRE Report, "How many monitoring or observability tools does your organization use?"

The Future of APM: Tool Consolidation, OpenTelemetry, and Cost Savings

According to research by Elastic, 79% of organizations are looking to streamline and consolidate their observability and monitoring tools, up from 74% the prior year.

One of the most compelling reasons for consolidating APM tools is the potential for cost savings. By reducing the number of tools and standardizing on a single platform, organizations can lower licensing costs, reduce training expenses, and streamline maintenance efforts.

OpenTelemetry (OTel) has emerged as a widely adopted standard for Observability. As an open-source project, it provides a unified set of APIs, libraries, agents, and instrumentation to capture and export telemetry data (metrics, logs, and traces) from applications. The adoption of OpenTelemetry is gaining momentum, with many organizations recognizing its potential to standardize data collection and reduce vendor lock-in.

This is Observability trend #1: organizations are making OTel a requirement for all new observability teams. It has the potential to break vendor lock-in, enable the consolidation of monitoring data, and give teams more flexibility.

Some teams are considering open-source technologies, but as the saying goes, open-source is free as a puppy. While it's effective for many use cases, in the case of observability, many teams realize that the effort to build, configure, integrate, and maintain a full open-source stack can easily add up to a higher cost than an optimized commercial platform option.

Centralized Observability Teams

The increased complexity of observability and the growing scrutiny of spend and platform adoption are resulting in the centralization of observability decisions. This is Observability trend #2: enterprises are either creating a central operations team that owns observability or an architecture team that defines standards, processes, vendors, and governance.

These central operations teams start by completing an inventory of applications, vendors, tools, reports, and critical needs. From here, they can define broad priorities, standards, and best practices. This trend is expected to accelerate in the coming months.

The Growing Need to Monitor Distributed, Internet-Centric Services and Dependencies

While APM tools are an established component of observability, they're increasingly recognized as systems-centric, with limited visibility into the many factors that influence application performance. Modern applications are cloud-centric, highly distributed, API-driven, and rely on a growing network of services that depend on the performance and availability of the global Internet.

This need is accentuated by the recognition that what really matters is not system uptime or code efficiency but the real-world digital user experience, regardless of where users are located. In the case of services exposed via APIs, what matters is the experience the other system encounters across the Internet, wherever it may happen to be.

Modern enterprises have recognized that customer experience is digital experience, and digital experience is customer experience. Even most offline processes are supported and dependent on digital systems that define the quality of the experience.

For instance, for a rental company, what matters is not that the cluster reservation system has an average CPU utilization of 72% but that the customer in line at the rental counter doesn't keep hearing, "Sorry, my computer is slow" after waiting 20 minutes in line.

As a result, we have observability trend #3: the requirement for Internet Performance Monitoring (IPM) as an essential component of an observability stack.

With the growing number of outages caused by elements of the Internet Stack and Internet-centric dependencies, it's no surprise GigaOM wrote, "IT decision-makers must choose between bearing the full weight of these outages or investing in an IPM solution to navigate them intelligently."

The Formula for Complete Observability

The combination of these trends has led to a common scenario in mature enterprises: consolidating to an APM platform (often among the top three) paired with an IPM platform. APM provides an inside-out view, while IPM delivers an outside-in perspective. APM provides a system-centric view, while IPM provides a global digital experience and internet health view.

This is why LogicMonitor integrates LM Internet Performance Monitoring (IPM) into its unified observability platform. Rather than requiring organizations to stitch together separate APM and IPM vendors, LogicMonitor combines LM Envision for hybrid infrastructure and application observability with LM Internet Performance Monitoring for Internet and digital experience monitoring, all connected through Edwin AI for intelligent prioritization and governed action. The result is a single platform that delivers both inside-out and outside-in visibility without the integration overhead.

In mature ITOps organizations, combining APM with IPM provides the operational insights these companies need to pursue resilience and optimal performance. These teams are characterized by being in a relatively advanced observability maturity stage where they aim to be not only proactive but also value-led and business-impact oriented.

Organizations adopting this approach are achieving operational efficiencies, dramatically increased uptime, and an improvement in user experience that's clearly having a positive impact on the business. This also results in better alignment between IT and business strategy and recognition of the value IT operations teams bring to the table. Companies like SAP, IKEA, Akamai, and one of the leading financial services institutions in North America are examples of those finding success with this model.

Conclusion

As IT leaders experience the complexities of IT operations, evolving monitoring and observability, consolidating APM tools, adopting OpenTelemetry, and implementing IPM with governance and best practices from a centralized team can provide significant advantages. By simplifying their observability strategies, organizations can achieve faster issue resolution, reduce costs, and drive better business outcomes.