CDO Battlescars Takeaways: Standardizing Business Metrics & Democratizing Experimentation at Intuit
CDO Battlescars is a podcast series hosted by Sandeep Uttamchandani, Unravel Data’s CDO. He talks to data leaders across data engineering, analytics, and data science about the challenges they encountered in their journey of transforming raw data into insights. The motivation of this podcast series is to give back to the data community the hard-learned lessons that Sandeep and his peer data leaders have learned over the years. (Thus, the reference to battlescars.)
In this episode, Sandeep talked to Anil Madan about battlescars in two areas: Standardization of Business Metrics and Democratization of Experimentation at Intuit.
At Intuit, Anil was the VP of Data and Analytics for Intuit’s Small Business and Self Employed group. He has over 25 years of experience in the data space across Intuit, PayPal, and eBay, and is now the VP of Data Platforms at Walmart. Anil is a pioneer in building data infrastructure and creating value from data across products, experimentation, digital marketing, payments, Fintech, and many other areas. Here are the key talking points from this insightful chat!
Standardization of Business Metrics
The Problem: Slow and Inconsistent Processing
- Anil led data and analytics for Intuit’s QuickBooks software. The goal of QuickBooks is to create a platform where they can power all the distinct needs of small businesses seamlessly.
- As they looked at their customer base, the key metric Anil’s team measured was signups and subscriptions. This metric needed to be consistent in order to have standards that could be relied on to drive business.
- This metric used to be inconsistent, and the time-to-value ranged from 24 to 48 hours because their foundational pipelines had several hubs.
The Solution: Simplify, Measure, Improve, and Move to Cloud
To solve this problem, Anil shared the following insights:
- Determine what the true metric definition was and establish the right counting rules and documentation.
- Define the right SLA for each data source.
- Invest deeply in programmatic tools, building a lineage called Superglue, which would start traversing right from the source all the way into reporting.
- Create micro-Spark pipelines, moving away from traditional MapReduce and monolithic ways of processing.
- Migrate to the cloud to help them auto-scale their compute capacity.
- Establish the right governance so that schema changes in any source would be detected through their programs.
The analytics teams, business leaders, and finance teams all looked at business definitions. As they launch new product offerings, or new monetization and consumption patterns, they review the metrics against these new patterns and ensure that business definitions are still holding true.
Democratization of Experimentation
Moving to democratizing experimentation, Anil was involved in significantly transforming the experimentation trajectory.
Anil breaks this transformation into people, processes, and technology:
- From a people perspective, what the challenges were, and where the handoffs were happening, determining if the right skills, and hiring data analysts with specialization in the experimentation space.
- On the processing side, ensuring a single place where they could measure what’s really happening at every step.
- On the technology side, looking at several examples from the industry to decide what to invest in the platform capabilities. They invested in things like metrics generation to establish overall evaluation criteria in each experiment. They also looked at techniques to move faster, like sequential testing, interleaving, and even quasi-experimentation.
Focusing on key metrics and ensuring that there are no mistakes in setting up and running the experiment became extremely important. They invested in a lot of programmatic ways to do that.
Anil’s team looked into where time is spent in the overall experimentation process and then focused on addressing those areas through a combination of building programs, leveraging programs that are open source, and even buying the needed software.
Focusing on the foundational pieces around how your pipelines work and how you analyze things, and investing in those foundational pieces, is key.
Though we highlighted the key talking points, Sandeep and Anil talked about so much more in the 29-minute podcast. Be sure to check out the full episode and the rest of the CDO Battlescars podcast series!
Listen to Podcast on Spotify