Over the last one year, we at Unravel Data have spoken to several enterprise customers (with cluster sizes ranging from 10’s to 1000’s of nodes) about their Big Data Application challenges. Most of them leverage Hadoop in a typical multi-tenant fashion wherein they have different applications (ETL, Business Intelligence, Machine Learning Apps etc.) accessing the common data substrate.
The conversations have typically been with both the Operations team, who manage and maintain these platforms and Developers/End Users, who are building these applications on the Big Data stack (Hive, Spark, MR, Oozie etc.).
Broadly speaking, the challenges can be grouped from the application perspective and cluster perspective. We have highlighted these challenges with actual examples that we have discussed with these customers.
Some of the most common Application challenges we have seen or heard of that are top of mind include:
Today, engineers end up going to five different sources (e.g CM/Ambari UI, Job History UI, Application Logs, Spark WebUI, AM/RM UI/Logs) to get an end to end understanding of application behavior and performance challenges. Further, these sources maybe insufficient to truly understand the bottlenecks associated with an applications (e.g detailed container execution profiles, visibility into the transformations that execute within a Spark stage etc.). See an example here on how to debug a Spark application. It’s cumbersome!
To add to the above challenges, many developers do not have access to the above sources and have to go via their Ops team, which adds to significant delays.
Some of the most common challenges we have seen or heard of that are top of mind for include:
The above application and operations challenges are real blockers preventing enterprise customers to make their Hadoop applications production ready. This in turn is slowing down the ROI enterprises are seeing from their Big Data investments.
Unravel’s vision is to address these broad challenges and more for Big Data Application and Operations. Our vision is to provide a full stack performance intelligence and self-serve platform that can improve your big data operations, make your applications reliable and improve overall cluster utilization. In our subsequent blog posts we will detail on how we go about doing this and the (data) science behind these solutions.
Learn more about how Unravel is the simplest way to resolve performance issues. Watch the webinar replay now!