Rutgers Big Data Certificate Program

Modern monitoring is a big data problem

4/26/2017

24 Comments

 
The rate of change and degree of diversity in the IT stack demands fine-grained and frequent monitoring, which turns monitoring into a real-time big data problem
Picture
Why did VMware acquire Wavefront? The start of the answer to this question comes with an understanding of what Wavefront is (or was). Wavefront was started by former Google engineers who set out to build a monitoring system for the commercial market that had the same features and benefits as the monitoring system that Google had built for itself.
Due to the massive scale of Google, such a system would have to have two key attributes:
  1. The ability to consume and process massive amounts of data very quickly. In fact, the Wavefront website make the claim, "Enterprise-grade cloud monitoring and analytics at over 1 million data points per second."
  2. The ability to quickly find what you want in this massive ocean of data
So, it is clear that the folks at Wavefront viewed modern monitoring to be a big data problem, and it is clear that some people at VMware were willing to pay a fair amount of money for a monitoring system that took a real-time and highly scalable approach to monitoring.


Why is modern monitoring a big data problem?Rather than just assume that VMware and Wavefront are right about the idea that modern monitoring is a big data problem, let's look at the underlying trends in the IT industry to determine what is changing monitoring in this manner. It is not enough that Google (and by extension other public clouds like Amazon Web Services and Microsoft Azure) have a big data monitoring problem. The question at hand is whether or not monitoring for the typical enterprise has become a big data problem.
The key insight is that the IT environment today could not be more different that it was as recently as five or 10 years ago. Ten years ago, there was one language (Java), and an application ran on one operating system, which ran on one physical server. Applications were updated as infrequently as possible, and changes in general were made as infrequently as possible.
 The image below depicts the typical "IT stack" at an enterprise today.
Picture
What is so different about the modern IT environment is the following:
  1. In response to an unlimited demand for business functionality implemented in software, agile development and DevOps were invented to speed code to market.
  2. In response to the same pressures, new languages and run times were created.
  3. The above two changes lead to a very diverse application stack, with frequently arriving and changing applications.
  4. At the infrastructure layer, everything (compute, networking and storage) is virtualized and is often subject to automated management.
In summary, the modern IT stack now consists of very diverse application stacks, with a rapid rate of change (many changes per hour) running on an abstracted and dynamic infrastructure.
How does monitoring have to respond?If you look at how monitoring is done today, it really has not changed in response to the changes in the IT stack listed above. Monitoring today consists of many different vendors, each collecting a slice of the total data, analyzing it, alerting on it, displaying it in dashboards and providing in reports.
Now, it might be tempting to try to find one vendor that can monitor that entire stack for you. Before you go down that road, remember what happened the last time that was tried. It was called Business Service Management with offerings from BMC, CA, HP and IBM, and it failed miserably (20 years ago) because even then the pace of innovation was so high that each vendor could not keep up. So they acquired companies to fill the gaps in their product lines, and they were never able to integrate them, which resulted in the mess, which in turn resulted in the failure of the BSM suites.
So, the first very important realization is that due to the accelerating pace of innovation in the industry, monitoring must remain a multi-vendor problem. This is so because various parts of the monitoring problem are "whole company" problems that require a significant investment of capital in intellectual property to solve.
Monitoring must also generally change to embrace the following principles:
  1. If the stack is diverse (especially at the application layer), then each component and layer of the stack needs to be monitored. 
  2. Transactions need to be monitored from their inception to their end in the application system (browser to database and back again).
  3. The entire stack needs to be monitored from the top of the the stack to the infrastructure (browser to hard disk or storage device and back again).
  4. So, the number of things that need to be monitored increases dramatically.
  5. If the environment is dynamic due to frequent changes at the application layer and automation at the infrastructure layer, then monitoring needs to be much more frequent. Every five minutes is no longer frequent enough. Every minute is no longer frequent enough. 
The big data approach to monitoringIf we accept that monitoring is a multi-vendor problem due to the diversity of the problem, and we accept that the granularity of monitoring and the frequency of monitoring must increase due to the dynamic nature of the stack, then monitoring is a real-time multi-vendor problem. 
There are then two approaches to implementing real-time big data monitoring:
  1. Have every vendor integrate with every other vendor and try to maintain a nightmare of a compatibility matrix.
  2. Have every vendor integrate with a common high-performance, big data back end especially built for the real-time multi-vendor monitoring problem.
SummaryThe diversity of the modern application stack, and the rate of change at both the application and infrastructure layers, requires that monitoring become more granular and more frequent across the multiple vendors required to cover the IT estate. This turns monitoring into a multi-vendor big data problem.

​Source: http://www.networkworld.com/article/3191479/software/modern-monitoring-is-a-big-data-problem.html

24 Comments
help with making a resume link
1/31/2018 01:21:05 am

Modem monitoring is really tiresome and time consuming. Other companies have adopted with the more renowned and advanced technology. There is a term called "cloud", which exists to monitor data from afar and from various places. It's quite different from modern monitoring, because you can access multiple databases from far away places. You can also gather data from various sources and use them for data analysis.

Reply
samira gupta link
10/29/2018 04:49:11 am

Good article with useful info

Reply
David Asarnow link
1/10/2019 09:36:49 pm

Agreed. Data monitoring is really tiresome and time consuming. Glad there are new technologies now to answer to that big data problem. Thanks for sharing an insightful article.


<a href="https://vimeo.com/158842489">David Asarnow</a>

Reply
playschools in ghaziabad link
6/23/2019 01:44:31 am

Top playschools in ghaziabad

https://veraxe.com/schools-in-ghaziabad

Reply
Low cost kvm vps in usa link
7/4/2019 06:38:05 am

Wow, nice content, please update more...

Reply
beautyworlduk link
8/24/2019 05:25:32 am

Wonderful article... thanks for sharing

Reply
Alpha status link
8/24/2019 05:27:17 am

thanks for sharing

Reply
double bed link
12/14/2019 02:25:28 am

double bed showroom in gurugram, krishnafurniture

Reply
auraa image management and consulting link
12/14/2019 02:26:18 am

auraa image management and consulting providing corporate training in gurgaon

Reply
mattress shop in gurgaon link
12/14/2019 04:34:21 am

mattress shop in gurgaon

Reply
Free Android Mod Apps link
2/19/2020 07:58:28 am

I am very impressive for this web portal, now i want to bookmark this website for future visit or get information about educational.

Reply
خرید پنل پیامکی link
8/5/2020 04:37:23 am

Very informative post indeed. Certainly looking forward to more boards like this one.

Thanks.

Reply
بهترین پنل اس ام اس link
9/15/2020 04:42:39 pm

Thanks for sharing this informative blog. This blog is very useful for everyone. keep it on.

Reply
MITRA BOILER link
11/25/2020 10:49:53 am

terimakasih, sangat bermanfaat buat kami

Reply
facebook link
12/19/2020 08:34:05 am

Hey! I simply wish to give a huge thumbs up for the good info you will have here on this post. I shall be coming back to your blog for more soon.

Reply
El7lwa.com link
12/21/2020 05:10:18 am

مجلة الحلوة موقع نسائي يهتم بالمرأة العربية وجمال المرأة احدث صيحات الموضة والازياء والديكور والأثاث والأجهزة المنزلية مجلة الحلوة مجلة عربية تقدم أحدث أنواع المحتوي في جميع المجالات

Reply
Turbo1.CO link
2/7/2021 06:43:43 pm

منصة رائدة في تقديم تقارير اسعار ومواصفات السيارات ، طريقك السهل للتعرف على أخبار ومراجعات السيارات ، قم بزيارة موقع موقع تيربو1 الالكتروني لتكون على اطلاع دائما بكل جديد.

Reply
جمس يوكن link
2/7/2021 06:46:16 pm

منصة رائدة في تقديم تقارير اسعار ومواصفات السيارات ، طريقك السهل للتعرف على أخبار ومراجعات السيارات ، قم بزيارة موقع موقع تيربو1 الالكتروني لتكون على اطلاع دائما بكل جديد

Reply
النترا 2021 link
2/7/2021 06:48:26 pm

A leading platform in providing price and specification reports for cars, your easy way to find out about car news and reviews, visit the Turbo 1 website to be always informed of all new developments.

Reply
نيسان كيكس 2021 link
2/7/2021 06:49:17 pm

thanks

Reply
توسان 2021 ان لاين link
2/7/2021 06:49:52 pm

thank you

Reply
توسان 2021 link
2/7/2021 06:50:47 pm

thanks a lot

Reply
مرسيدس EQA 2021 link
2/7/2021 06:51:43 pm

thanks for sharing.

Reply
distributor boiler link
2/20/2021 01:44:20 pm

thank you for sharing this information

Reply



Leave a Reply.

    Archives

    April 2017
    March 2017
    February 2017

    Categories

    All

    RSS Feed

LOCATIONS

New Brunswick, New Jersey
San Francisco Location, California
Online or on-site at your location

Contact us

Lisa Doehring
Program Manager
ldoering@cx.rutgers.edu
415 343 0264

BIGDAta@RUTGERS

The Big Data Certificate Program at Rutgers University is offered in connection with the Rutgers Center for Innovation Education and the Rutgers Professional Science Master's Degree.

SUBSCRIBE

Join our mailing list today!
Join Now