MesosCon Asia

By using resource managers, like Mesos, Spark can improve resource utilization by allowing multiple Spark frameworks to share the same infrastructure. Resource requirements, such as memory and CPU, change quickly within the lifetime of a single Spark job and managing these dynamic resources is even more challenging when you have multiple users. Even with the newly introduced coarse-grained scheduler for Mesos in Spark 2.0, this fundamental problem is not resolved.
To address this, we have developed a Spark session scheduler on top of Mesos. The Spark session scheduler allocates resources with "fine-granularity" to a user's Spark application without introducing execution overhead. These tasks are controlled by overall process limits and support resource reclaims through controlled tasks preemption protocols. The source codes will be opened in near future.