Unsurprisingly, Modern data apps have become crucial across all areas of the financial industry, with apps for fraud detection, claims processing and compliance amongst others playing a business-critical role. Unravel has been deployed by several of the world’s largest financial services organizations to ensure these critical apps perform reliably at all times. One recent example is one of America’s ten largest banks, a corporation that encompasses over 3,000 retail branches, 5000 ATMs and over 70,000 employees. This is what happened.
This bank has been using big data for a variety of purposes, but its two most important apps are fraud detection and compliance. They deployed Informatica broadly in order to run ETL jobs. This was a massive focus for the bank’s DataOps team, which had many workflows running multiple Hive queries. They also made heavy use of Spark and Kafka Streaming in order to process tons of real-time streaming data for their fraud detection app.
Unravel Kafka Dashboard (main)
The bank suffered constant headaches before they deployed Unravel. First, their data apps tended to be slow and failed frequently. In order to figure out why, they had to dig through an avalanche of raw data logs, a process that could take weeks. Once they had identified the problem, they would have to do a long trial-and-error process to determine how to fix it. This again could take weeks, if they were fortunate enough to even find a fix for the issue.
There was another monitoring issue at the cluster usage level. They knew they weren’t optimally consuming their compute resources but had no visibility into how to improve utilization. The team only fully became aware of how critical their compute utilization problem was when it caused a critical data app to fail.
After deploying Unravel, the bank was able to quickly alleviate these problems. To begin, the platform’s reporting capabilities changed things dramatically. The team was able to monitor and understand its many different modern data stack technologies (Hive, Spark, Workflows, Kafka) from a single interface rather than relying on siloed views that didn’t enable correlation or many useful insights. The bank’s Kafka Streaming deployment had been particularly hard to monitor due to the massive input of streaming data. In addition, they previously had no way to track if Informatica and Hive queries for ETL jobs were hitting SLAs. Unravel changed all of that, delivering detailed insights that told the team how every workload was performing.
The insights were just as valuable at the cluster usage level, with Unravel providing cluster optimization opportunities to further boost performance and reduce wasteful resource consumption. This was the first time the bank really felt they understood what was happening in each and every cluster.
On top of the monitoring and visibility capabilities, Unravel yielded a significant boost in app performance. This is where the platform’s AI and automated recommendations were huge boon for the customer. After first automatically diagnosing several root cause issues, Unravel delivered cleanup recommendations for almost half million Hive tables, resulting in tremendous performance improvements. The platform also enabled the team to set notifications for specific failures and gave them the option to run automated fixes in these circumstances.
See Unravel AI and automation in action
Examples of “Stalled” and “Lagging” Consumer Groups (name = “demo”) showing in the Unravel UI
While the bank isn’t currently deploying data apps in the cloud, they do have plans to migrate soon. One of the hardest parts of any cloud migration is the planning phase. Unravel’s cloud assessment capabilities give the bank detailed insights to streamline this painstaking preliminary phase: The assessment mapped out the bank’s on-premises big data pipelines and then told them which apps are best fit for the cloud and how those apps should be configured using specific instance type recommendations and forecasting costs and consumption. This move saved the customer from having to hire an expensive consulting firm to evaluate and advise their move to the cloud, accelerated their decision timeline and critically provided data driven insights instead of relying on guesswork.
Modern data apps are the backbone of any major financial institution. Unravel’s AI-driven DataOps platform allowed this bank to leverage these critical data apps to their full potential for the first time. Unravel has been so transformative that the customer has been able to open their data lake to broader business users, democratizing data apps so they provide value to the team outside of the developer and IT operations staff. In the bank’s own words, Unravel is helping drive a cultural shift by ensuring big data delivers on its true potential and is future proofing architectural decisions as hybrid cloud deployments are evaluated.