Synthetic Monitoring Types
You can use synthetic monitoring to analyze multiple aspects of your web environment, including the paths that your end users take to reach your applications. When monitoring a web application, it is important to break down the overall application response time into dozens of incremental steps. This helps with discovering the reasons behind any performance issues. It is also important to account for all the different versions of browsers available, as well as how they behave when rendering your application’s user interface. We recommend simulating a full business transaction through synthetic monitoring to gauge the complete user experience and not stop at the first loaded page.
Use the following table to gain familiarity with all the types of monitoring options you should expect from an advanced synthetic monitoring tool.
- User Journey Transaction Testing: User journey transaction testing involves recording or scripting sequential steps to capture a complete web-based business transaction simulating real users, including login and SSO.
- Real Browser Testing: In order to truly emulate end user experience, web tests must be performed from various types and versions of browser.
- Mobile Simulation: This lets you simulate mobile users by platform (Android, iOS), browser (Safari, Chrome) or wireless speed (3G, 4G).
- SaaS Monitoring: Synthetic tests of mainstream SaaS applications. The monitoring solution should provide out-of-the-box templates or custom scripts for monitoring different workflows.
- API Monitor: Monitoring an Application Programming Interface (API) is a key requirement for both a single URI and a full multi-step transaction, and should include support for both local and global variables.
- Ping Test: A ping is used to test the basic reachability of a website or IP address. Your monitoring solution should support ICMP Ping, UDP Ping, and TCP Ping.
- TraceRoute Test: A traceroute tool shows you every hop sequentially and the total number of hops required. Segmented hop-by-hop network latency measurements should be supported via UDP, TCP and ICMP TraceRoute.
- SMTP Test: To know if your email server is functioning correctly, you need support for testing of SMTP, POP3 and IMAP protocols.
- DNS Monitoring: DNS testing needs to provide performance and error data for every step in the DNS resolution process, including IP address, SOA, MX and SRV records, and NS record and root servers. DNS monitoring should also involve tracking all individual servers involved in DNS resolution to help mitigate a DNS attack or outage.
- CDN Monitoring: Monitoring from as many geographic locations as possible is key to maintaining CDN performance. CDN monitoring will provide insight into server mapping and help identify whether incidents are isolated/regional or system-wide.
- FTP Monitoring: FTP monitoring entails testing file transfer services or delivery of data via the FTP protocol on TCP/IP networks. A test of an FTP server must exercise user login and issue requests to FTP simulating real users that are accessing your web applications.
- WebSocket Test: A WebSocket test checks the availability of your WebSocket service (typically used in chats, financial tickers, and gaming applications). WebSocket monitoring also offers the option to send a string message and verify its return data.
- Test NTP Server: This type of test should verify the NTP server and time that services are available, whether the public UDP port is reachable, and ensure that time offset is accurate.
- SSL Test: SSL testing involves checking SSL certificate validity, protocol support, key exchange, and cipher strength.
- MQTT Test: This type of testing allows you to monitor the performance and availability of IoT devices. MQQ testing requires publishing messages to the MQTT broker, listening for MQTT messages, and simulating connection issues.
Synthetic Monitoring Analytics
Once you have achieved full coverage of all the facets of your application environment with synthetic monitoring, it is time to analyze the collected data. To conduct effective analysis, it is important to be able to visualize data trends on a dashboard, set up alerts for latency and outage anomalies, and isolate any performance issues in the data path between your application and its end users. More advanced visualization options include charts such as a histogram or scatterplot, having access to a screenshot, or even a filmstrip, of the user interacting with the page. The sophistication of your tool’s analytics will allow you to shorten the mean time to detect an application performance problem before your customers have the chance to complain.
- Trends Data: Performance monitoring vendors should offer the ability to compare data over different time periods so that you can easily visualize changes day over day or year over year in order to determine trends.
- Waterfall of Granular KPIs: Waterfall data should capture standard HTTP metrics for all requests on a page, and include page-level metrics such as Render Start, DomComplete, and OnLoad.
- Data Aggregation: Data aggregation allows you to roll up raw data into minimum, maximum, average, median and 95 percentile groupings so that you can pinpoint any variations over time.
- Data Aggregation by Meta-Data: This feature allows users to group data by specific dimensions, such as node, ISP, user-defined tags, city, hour of day, error code and so on.
- Benchmark Comparison: This functionality allows users to compare performance across multiple websites to analyze how they stack up against competitors and industry benchmark standards.
- Scatterplot: A scatter plot shows each sample measurement as a single point on a graph according to its value and collection time which in turn helps identify outliers as a glance, especially when the points are color coded.
- Cumulative Distribution Function (CDF): CDF helps determine the probability of a poor end user experience by latency (e.g. 33% of users experience latency of 10 seconds or more) and answers the question, “Where does my performance long-tail start and end?
- Histogram: A histogram or frequency distribution shows how often an end user may experience a long latency (e.g. 20% between 1 or 2 minutes) and consequently a slow page load.
- Advanced Visualization: Data visualization features such as Bubble Chart, Grid Chart and Heat Maps allow you to better analyze and correlate data.
- Dashboards: Any analytics dashboard should be customizable, support AI-based smart-boards, offer heatmaps, and be shareable via public links so that any relevant employee can access it.
- Reports: A dashboard helps you in real-time while reports combine more than one test, helping you automate executive updates, perform trend analysis, or manage your SLAs.
- Alerting: Monitoring your website and application must include a comprehensive alerting system. Alerts allow you to monitor all alert events triggered in the system. Alerting on raw or aggregated data should be configurable by threshold, transaction step, percentage of failed checks, trend changes and/or transient incidents.
- False Positive Reduction: False positive reduction entails the suppression of false alerts using techniques such as generating alerts only after a specified percentage of concurrent or sequential checks fail.
- Fault Isolation: This feature allows you to isolate issues down to all elements of the service delivery chain, including the wireless provider, ISP, last mile provider, CDN provider and DNS provider...
- Cached Versus Uncached: This feature lets you measure the delay experienced by end users when the elements being loaded have not been cached (e.g. by a CDN).
- Service Level Agreement (SLA) Management: This critical feature analyzes your monthly service uptime and latency to determine whether your SLA was met outside of any pre-agreed maintenance windows so that you can hold your service providers to account.
Synthetic Monitoring Reachability
Guaranteeing the reachability of your digital services is one of the most important strategic aspects of synthetic monitoring. Your service may be perfectly reachable from AWS within the West Coast of the U.S., for instance, but not from a wireless network in Eastern Europe. The only way to ensure your service can be reached by your entire market is to choose a vendor that has heavily invested in deploying hundreds of nodes around the world as strategic vantage points. Nodes may be located in data centers owned by local Internet Service Providers (ISP), within the wireless coverage of major mobile service providers, in the backbone of the Internet, or even in the Local Area Network (LAN) where your application is deployed. Your ability to isolate a performance fault depends on the number and location of nodes you measure from.
- Public Cloud Nodes: Cloud is central to the application delivery chain and huge amounts of traffic flow to, from, and between cloud service providers. All synthetic monitoring providers implement vantage points, or nodes, on public clouds (e.g. AWS, Azure, Google Cloud). Some recent providers do so, however, mostly because it makes it easy for them to claim quick global coverage without investing in more node types.
- Internet Backbone Nodes: Backbone nodes allow you to monitor directly from ISPs, which carry the majority of the Internet’s global traffic, letting you measure performance excluding external network noise. They are critical in helping determine whether performance incidents stem from a problem with the edge or core of the Internet.
- Last Mile Nodes: To isolate reachability problems, nodes are required in every segment of the service delivery chain, including the “last mile” access network. Synthetic monitoring from the on and off-ramp of local ISPs to the backbone is essential for troubleshooting and localizing specific Internet problems.
- Wireless Nodes: To isolate mobile access issues, your provider must have nodes co-located with all the major wireless providers in your end user’s regions.
- International Nodes: For users in Asia, the Middle East, Africa, Australia and South America, you should ensure your provider has a robust set of international nodes with ISPs local to your market and your customer base.
- On-Premise Nodes: On-premise nodes allow you to use your own hardware and the provider’s software to implement your own node/s to monitor user experience behind the firewall and identify local issues specific to critical business applications.
- Detailed Network Monitoring: Troubleshooting of performance outages and slowdowns always requires a breakdown of network latency hop by hop and by Autonomous System Number (ASN). Synthetic monitoring providers should offer the capability for detailed network monitoring.
- Bandwidth Throttling: Bandwidth throttling allows the simulation of a mobile access point from a slow wireless network or an area with a weak wireless signal.