As technology evolves at a rapid pace, the healthcare industry is transforming quickly along with it. Tech breakthroughs like IoT, advanced imaging, genomics mapping, artificial intelligence and machine learning are some of the key items re-shaping the space. The result is better patient care and health outcomes. To facilitate this shift to the next generation of healthcare services – and to deliver on the promise of improved patient care – organizations are adopting modern data technologies to support new use cases.
We are a large company operating healthcare facilities across the US and employing over 20,000 people. Despite our size, we understand that we must be nimble and adapt fast to keep delivering cutting edge healthcare services. We only began leveraging big data about three years ago, we’ve grown fast and built out a significant modern data stack, including Kafka, Impala and Spark Streaming deployments, among others. Our focus here was always on the applications, developer needs and, ultimately, business value.
We’ve built a number of innovative data apps on top of our growing data pipelines, providing great new services and insights for our customers and employees alike. During this process, though, we realized that it’s extremely difficult to manually troubleshoot and manage such data apps. We have a very developer-focused culture –the programmers are building the very apps that are ushering in the next generation of healthcare, and we put them front and center. We were concerned when we noticed these developers were sinking huge chunks of their time fixing and diagnosing failing apps, taking their focus away from creating new apps to drive core business value.
Impala and Spark Streaming are two modern data stack technologies that our developers increasingly employed to support next generation use cases. These two technologies are commonly used to build apps that leverage streaming data, which is prevalent in our industry. Unfortunately, both Impala and Spark Streaming are difficult to manage. Apps built with these two were experiencing frequent slowdowns and intermittent crashes. Spark Streaming, in particular, was very hard to even monitor.
Our key data apps were not performing the way we expected and our programmers were wasting tons of time trying to troubleshoot them. When we deployed Unravel Data, it changed things swiftly, providing new insights into aspects of our data apps we previously had no visibility of and drastically improving app performance.
Impala – improving performance by 12-18x !
Impala is a distributed analytic SQL engine running on the Hadoop platform. Unravel provided critical metrics that helped us to better understand how Impala was being used, including:
- Impala memory consumption
- Impala queries
- Detail for queries using drill-down functionality
- Recommendations on how to make queries run faster, use data across nodes more efficiently, and more
Unravel analyzed the query pattern (insert, select, data, data locality across the Hadoop cluster) against Impala and offered a few key insights. For one, Unravel saw most of the time was spent scanning Hadoop’s file system across nodes and combining the results. After computing stats on the underlying table – a simple operation – we were able to dramatically improve performance by 12-18x.
Unravel provides detailed insights for Impala
Spark Streaming – Reducing memory requirements by 80 percent!
Spark Streaming is a lightning quick processing and analytics engine that’s perfect for handling enormous quantities of streaming data. As with Impala, Unravel provided insights and recommendations that alleviated the headaches we were having with the technology. The platform told us we didn’t have the memory for many Spark Streaming jobs, which was ultimately causing the all the slowdowns and crashes. Unravel then provided specific recommendations on how to re-configure Spark Streaming, a process that’s typically complicated and replete with costly missteps. In addition, Unravel found that we could save significant CPU resources by sending parallel tasks to cores.
Overall, two critical Spark Streaming jobs saw memory reductions of 74 percent and 80 percent. Unravel’s parallelism recommendations saved us 8.63 hours of CPU per day.
Spark Streaming performance recommendation
The Bigger Picture
Unravel is straight forward to implement and immediately delivers results. The platform’s recommendations are all configuration changes and don’t require any changes to coding. We were stunned that we could improve app performance so considerably without making a single tweak to the code, yielding an immediate boost to critical business apps. Unravel’s full-stack platform delivers insights and recommendations for our entire modern data stack deployment, eliminating the need to manage any data pipeline with a siloed tool.
Modern data apps are fueling healthcare’s technical transformation. By improving data app performance, We have been able to continue delivering a pioneering healthcare experience, achieving better patient outcomes, new services and greater business value. Without a platform like Unravel, our developers and IT team would be bogged down troubleshooting these apps rather than creating new ones and revolutionizing our business. Unravel has helped create a deep cultural shift to do more with our data and evolve to a DataOps mindset.