The reason the traditional approach cannot scale is because there are just too many potential problems, across too many different systems, for DevOps to troubleshoot issues through trial-and-error. Let’s look at an example: Many things can go wrong (at multiple levels) including the app, containers, resource management, network performance, and data storage. Additional factors contribute to the complexity of moving Spark from pilot to production. There are many patterns of Spark applications, submission methods, as well as infrastructure choices. All require a more simple, end-to-end way to manage performance and utilization.