Modern monitoring is a big data problem

The rate of change and degree of diversity in the IT stack demands fine-grained and frequent monitoring, which turns monitoring into a real-time big data problem

Why did VMware acquire Wavefront? The start of the answer to this question comes with an understanding of what Wavefront is (or was). Wavefront was started by former Google engineers who set out to build a monitoring system for the commercial market that had the same features and benefits as the monitoring system that Google had built for itself.
Due to the massive scale of Google, such a system would have to have two key attributes:

1. The ability to consume and process massive amounts of data very quickly. In fact, the Wavefront website make the claim, "Enterprise-grade cloud monitoring and analytics at over 1 million data points per second."
2. The ability to quickly find what you want in this massive ocean of data

So, it is clear that the folks at Wavefront viewed modern monitoring to be a big data problem, and it is clear that some people at VMware were willing to pay a fair amount of money for a monitoring system that took a real-time and highly scalable approach to monitoring.


Why is modern monitoring a big data problem?Rather than just assume that VMware and Wavefront are right about the idea that modern monitoring is a big data problem, let's look at the underlying trends in the IT industry to determine what is changing monitoring in this manner. It is not enough that Google (and by extension other public clouds like Amazon Web Services and Microsoft Azure) have a big data monitoring problem. The question at hand is whether or not monitoring for the typical enterprise has become a big data problem.
The key insight is that the IT environment today could not be more different that it was as recently as five or 10 years ago. Ten years ago, there was one language (Java), and an application ran on one operating system, which ran on one physical server. Applications were updated as infrequently as possible, and changes in general were made as infrequently as possible.
 The image below depicts the typical "IT stack" at an enterprise today.

What is so different about the modern IT environment is the following:

1. In response to an unlimited demand for business functionality implemented in software, agile development and DevOps were invented to speed code to market.
2. In response to the same pressures, new languages and run times were created.
3. The above two changes lead to a very diverse application stack, with frequently arriving and changing applications.
4. At the infrastructure layer, everything (compute, networking and storage) is virtualized and is often subject to automated management.

In summary, the modern IT stack now consists of very diverse application stacks, with a rapid rate of change (many changes per hour) running on an abstracted and dynamic infrastructure.
How does monitoring have to respond?If you look at how monitoring is done today, it really has not changed in response to the changes in the IT stack listed above. Monitoring today consists of many different vendors, each collecting a slice of the total data, analyzing it, alerting on it, displaying it in dashboards and providing in reports.
Now, it might be tempting to try to find one vendor that can monitor that entire stack for you. Before you go down that road, remember what happened the last time that was tried. It was called Business Service Management with offerings from BMC, CA, HP and IBM, and it failed miserably (20 years ago) because even then the pace of innovation was so high that each vendor could not keep up. So they acquired companies to fill the gaps in their product lines, and they were never able to integrate them, which resulted in the mess, which in turn resulted in the failure of the BSM suites.
So, the first very important realization is that due to the accelerating pace of innovation in the industry, monitoring must remain a multi-vendor problem. This is so because various parts of the monitoring problem are "whole company" problems that require a significant investment of capital in intellectual property to solve.
Monitoring must also generally change to embrace the following principles:

1. If the stack is diverse (especially at the application layer), then each component and layer of the stack needs to be monitored. 
2. Transactions need to be monitored from their inception to their end in the application system (browser to database and back again).
3. The entire stack needs to be monitored from the top of the the stack to the infrastructure (browser to hard disk or storage device and back again).
4. So, the number of things that need to be monitored increases dramatically.
5. If the environment is dynamic due to frequent changes at the application layer and automation at the infrastructure layer, then monitoring needs to be much more frequent. Every five minutes is no longer frequent enough. Every minute is no longer frequent enough. 

The big data approach to monitoringIf we accept that monitoring is a multi-vendor problem due to the diversity of the problem, and we accept that the granularity of monitoring and the frequency of monitoring must increase due to the dynamic nature of the stack, then monitoring is a real-time multi-vendor problem. 
There are then two approaches to implementing real-time big data monitoring:

1. Have every vendor integrate with every other vendor and try to maintain a nightmare of a compatibility matrix.
2. Have every vendor integrate with a common high-performance, big data back end especially built for the real-time multi-vendor monitoring problem.

SummaryThe diversity of the modern application stack, and the rate of change at both the application and infrastructure layers, requires that monitoring become more granular and more frequent across the multiple vendors required to cover the IT estate. This turns monitoring into a multi-vendor big data problem.

​Source: http://www.networkworld.com/article/3191479/software/modern-monitoring-is-a-big-data-problem.html