In my previous blog post, I gave my thoughts on the recent Strata Data Conference in New York and discussed the general movement I’ve observed toward a new phase in Big Data. That piece segues into another trend I’ve noticed: enterprises are becoming more focused on applications of Big Data rather than technologies.
In the past, people were focused on learning the various Big Data technologies: Hadoop, Spark, Kafka, Cassandra, etc. It took time for users to understand, differentiate and ultimately deploy them. There was a lot of debate and plenty of hype. Finally, organizations have cut through the noise and figured all that out. Now they’re concerned about actually putting their data to use.
You only start to get ROI from Big Data when you begin to employ applications. Simply deploying the technologies doesn’t achieve anything. As a result, applications are becoming mission critical for any organization that’s invested in Big Data.
What exactly do I mean by “applications”? Simply put, it’s the use of data to solve a problem. For instance, a common application in the financial sector is fraud detection. Banks use data to quickly identify suspicious transactions and freeze them as they occur. The banks don’t really care which technology was used, they care about solving the problem (fraud).
Another common application is the recommendation engine, which uses large quantities of data to forecast things. The recommendation engine is critical for most all web companies. Consider Netflix, for example. Their recommendation engine isn’t just a nice add-on that enhances the user experience, it’s absolutely fundamental to the experience and to Netflix’s bottom line. Their platform depends on the ability to accurately suggest relevant movies and TV shows to people – otherwise, it’d be almost impossible for viewers to dig through their enormous library.
An even better example of the power of Big Data applications is Climate Corporation. This company uses sophisticated weather, soil and field data to help farmers increase their yields. Using this data, farmers know when to best plant their seeds and when to harvest their crops. This data is also employed to anticipate any potential dangers – particularly weather related – so users can protect those crops. This doesn’t just save farmers money, it makes food cheaper and agriculture more sustainable. These are both critical as earth’s population continues to boom. The proper application of Big Data not only makes things more convenient, it can also save the world.
Again, in any of these examples, the enterprises don’t really care about the technology being used. It’s not important which distribution or database or analytics they’re using, what matters is the result. Enterprises have realized this and adopting an application-centric approach to Big Data.
That being said, applications like fraud detection and recommendation engines are only useful if they can run efficiently and finish on time, each time. An organization will not be able to depend on such applications if they finish slow one day or produce incorrect results the other. APM helps improve both correctness and performance, making big data applications reliable in the enterprise.