[ANNOUNCE] Apache YuniKorn (Incubating) 0.8.0 Release
Apache YuniKorn (Incubating) is a standalone resource scheduler, designed for managing, and scheduling Big Data workloads on the container orchestration framework for on-prem and on-cloud use cases. This release ships a fully functional resource scheduler for Kubernetes with a number of useful features that empower to run Big Data workloads on K8s.
The 0.8.0 release contains a number of significant features and enhancements. The notable features are listed below.
- Resource Queues: YuniKorn divides a cluster into a hierarchy of resource queues. This enables fine-grained resource capacity management for a multi-tenant environment. Queues can be statically configured or dynamically managed by the scheduler. With the dynamic queue management feature, users can set up placement rules to delegate queue management. This enables use cases like mapping a K8 namespace to a YuniKorn queue. YuniKorn will dynamically create a queue for each namespace, see an example here. Using this feature, there is no other setup needed to run YuniKorn on a K8s cluster.
- Pluggable Application Management (Including native Spark support): A pluggable app management framework is being added to this release, with this feature, YuniKorn can be easily integrated with 3rd party K8s operators, such as Spark-k8s-operator, Flink-k8s-operator, etc. To learn more about this feature and the design, please see this doc.
- Resource Reservation: YuniKorn community has updated the core scheduling logic to include reservations. Reservation is the methodology used to avoid large/picky resource requests being starved in the batch scheduling scenario.
- Pluggable Node Sorting Policies: There are 2 builtin node sorting policies (FAIR and BinPacking) available in YuniKorn with regards to allocation distribution. Which are good for data-center/cloud use cases. The node sorting policies are pluggable, users can implement their own policies and plug that into the scheduler.
- Improved Scheduling Performance: When running Big Data batch workloads, e.g Spark, on K8s, scheduler throughput becomes one of the main concerns. In YuniKorn, there are lots of optimizations to improve the performance, such as a fully async event-driven system and low-latency sorting policies. Based on our test using Kubemark, YuniKorn is 2x+ faster than the default K8s scheduler.
We are planning to release the next version (0.9) by the third quarter of 2020, the main features planned for the upcoming release are priority scheduling, gang scheduling:
- YUNIKORN-1: Support app/task priority aware scheduling
- YUNIKORN-2: Support Gang Scheduling
- YUNIKORN-33: Performance benchmark with Kubemark
Since the birth of the YuniKorn project, the feedback has been well-received. We are thankful to have such a diverse community where we have members from Microsoft, LinkedIn, Alibaba, Apple, Tencent, Nvidia, and Cloudera, to help and guide the project’s growth.
Many thanks to everyone who contributed to the release, and everyone in the Apache YuniKorn (Incubating) community! The release is a result of direct and indirect efforts from many contributors, listed below are those who contributed directly by submitting patches: Adam Antal, Akhil Puthenveettil Balan, Bhuvan Mysore, Jetly Jaimin, Kinga Marton, Sunil Govindan, Tao Yang, Tarun Parimi, Wangda Tan, Wanqiang Ji, Weiwei Yang, Wilfred Spiegelenburg, Vinod Kumar Vavilapalli.
Thanks for the mentors Felix Cheung, Holden Karau, Jason Lowe, Junping Du for providing the guidance to the community, and coach us the Apache way.
Posted at 12:39AM May 05, 2020 by Weiwei Yang in Technology | |