Due to the massive scale of Google, such a system would have to have two key attributes:
- The ability to consume and process massive amounts of data very quickly. In fact, the Wavefront website make the claim, "Enterprise-grade cloud monitoring and analytics at over 1 million data points per second."
- The ability to quickly find what you want in this massive ocean of data
Why is modern monitoring a big data problem?Rather than just assume that VMware and Wavefront are right about the idea that modern monitoring is a big data problem, let's look at the underlying trends in the IT industry to determine what is changing monitoring in this manner. It is not enough that Google (and by extension other public clouds like Amazon Web Services and Microsoft Azure) have a big data monitoring problem. The question at hand is whether or not monitoring for the typical enterprise has become a big data problem.
The key insight is that the IT environment today could not be more different that it was as recently as five or 10 years ago. Ten years ago, there was one language (Java), and an application ran on one operating system, which ran on one physical server. Applications were updated as infrequently as possible, and changes in general were made as infrequently as possible.
The image below depicts the typical "IT stack" at an enterprise today.
- In response to an unlimited demand for business functionality implemented in software, agile development and DevOps were invented to speed code to market.
- In response to the same pressures, new languages and run times were created.
- The above two changes lead to a very diverse application stack, with frequently arriving and changing applications.
- At the infrastructure layer, everything (compute, networking and storage) is virtualized and is often subject to automated management.
How does monitoring have to respond?If you look at how monitoring is done today, it really has not changed in response to the changes in the IT stack listed above. Monitoring today consists of many different vendors, each collecting a slice of the total data, analyzing it, alerting on it, displaying it in dashboards and providing in reports.
Now, it might be tempting to try to find one vendor that can monitor that entire stack for you. Before you go down that road, remember what happened the last time that was tried. It was called Business Service Management with offerings from BMC, CA, HP and IBM, and it failed miserably (20 years ago) because even then the pace of innovation was so high that each vendor could not keep up. So they acquired companies to fill the gaps in their product lines, and they were never able to integrate them, which resulted in the mess, which in turn resulted in the failure of the BSM suites.
So, the first very important realization is that due to the accelerating pace of innovation in the industry, monitoring must remain a multi-vendor problem. This is so because various parts of the monitoring problem are "whole company" problems that require a significant investment of capital in intellectual property to solve.
Monitoring must also generally change to embrace the following principles:
- If the stack is diverse (especially at the application layer), then each component and layer of the stack needs to be monitored.
- Transactions need to be monitored from their inception to their end in the application system (browser to database and back again).
- The entire stack needs to be monitored from the top of the the stack to the infrastructure (browser to hard disk or storage device and back again).
- So, the number of things that need to be monitored increases dramatically.
- If the environment is dynamic due to frequent changes at the application layer and automation at the infrastructure layer, then monitoring needs to be much more frequent. Every five minutes is no longer frequent enough. Every minute is no longer frequent enough.
There are then two approaches to implementing real-time big data monitoring:
- Have every vendor integrate with every other vendor and try to maintain a nightmare of a compatibility matrix.
- Have every vendor integrate with a common high-performance, big data back end especially built for the real-time multi-vendor monitoring problem.
Source: http://www.networkworld.com/article/3191479/software/modern-monitoring-is-a-big-data-problem.html