Retailers are using big data to report on consumer demand, inventory availability, and supply chain performance in real time. Big data provides a convenient, easy way for retail organizations to quickly ingest petabytes of data and apply machine learning techniques for efficiently moving consumer goods. A top supermarket retailer has recently used Unravel to monitor its vast trove of customer data to stock the right product for the right customer, at the right time.
The supermarket retailer needed to bring point-of-sale, online sales, demographic and global economic data together in real-time and give the data team a single tool to analyze and take action on the data. The organization needed all the systems in their data pipelines to be monitored and managed end-to-end to ensure proper system and application performance and reliability. Existing methods were largely manual, error prone and lacked actionable insights.
Unravel Extensible Data Operations Platform
After failing to find alternative solutions to cluster performance management, the customer chose Unravel to help remove risks in their cloud journey. During implementation, Unravel worked closely with the ITOps team to support the customer, providing support and iterating in collaboration based on the insights and recommendations provided by Unravel. This enabled both companies to triage issues, and troubleshoot issues faster.
Get answers, not more charts and graphs
Bringing All The Data Into A Single Interface
The customer utilized a number of modern open source data projects in its data engineering workflow – Spark, MapReduce, HBase, YARN and Kafka. These components were needed to ingest and properly process millions of transactions a day. Hive query performance was a particular concern, as numerous downstream business intelligence reports depended on timely completion of these queries. Previously, the devops team spent several days to a week troubleshooting job failure issues, often blaming the operations team of improperly cluster configuration settings. The operations teams would in turn ask the devops team to re-check SQL query syntax for cartesian joins and other inefficient code. Unravel was able to shed light to these types of issues, providing usage based reporting which helped both teams pinpoint inefficiencies quickly.
Unravel was able to leverage its AI and automated recommendations engine to clean up hundreds of Hive tables, greatly enhancing performance. A feature that the company found particularly useful is the ability to generate custom failure reports using Unravel’s flexible API. In addition to custom reports, Unravel is able to deliver timely notifications through e-mail, serviceNOW, and PagerDuty.
Happy with the level of control Unravel was able to provide for Hive, the customer deployed Unravel for all other components – Spark, MapReduce, HBase, YARN and Kafka and made it a standard tool for DataOps management across the organization. Upon deploying Unravel, the team was presented with an end-to-end dashboard of insights and recommendations across the entire stack (Kafka, Spark, Hive/Tez, HBase) from a single interface, which allowed them to correlate thousands of pages of logs automatically. Previously, the team performed this analysis manually, with unmanaged spreadsheet tracking tools.
In addition to performance management, the organization was looking for an elegant means to isolate users who were consistently wasteful with the compute resources on the Hadoop clusters. Such reporting is difficult to put together, and requires cluster telemetry to not only be collected across multiple components, but also correlated to a specific job and user. Using Unravel’s chargeback feature, the customer was able to report not only the worst offenders who were over-utilizing resources, but the specific cost ramifications of inefficient Hive and Spark jobs. It’s a feature that enabled the company to recoup any procurement costs in a matter of months.
Examples of cluster utilization showing in the Unravel UI
Scalable modern data applications on the cloud are critical to the success of retail organizations. Using Unravel’s AI-driven DataOps platform, a top retail organization was able to confidently optimize its supply chain. By providing full visibility of their applications and operations, Unravel helped the retail organization to ensure their modern data apps are effectively architected and operational. This enabled the customer to minimize excess inventory and deliver high demand goods on time (such as water, bread, milk, eggs) and maintain long term growth.
FINDING OUT MORE
Download the Unravel Guide to DataOps. Contact us or create a free account.