文献类型：期刊文献

英文题名：A resource occupancy ratio-oriented load balancing task scheduling mechanism for Flink

作者：Dai, Qinglong[1];Qin, Guangjun[1];Li, Jianwu[2];Zhao, Jun[3];Cai, Jifan[1]

第一作者：Dai, Qinglong

通讯作者：Dai, QL[1]

机构：[1]Beijing Union Univ, Smart City Coll, Beijing 100101, Peoples R China;[2]Beijing Inst Technol, Adv Technol Reseach Inst, Beijing, Peoples R China;[3]Chinatelecom Res Inst, Inst Big Data & Artificial Intelligence, Beijing, Peoples R China

第一机构：北京联合大学继续教育学院

通讯机构：[1]corresponding author), Beijing Union Univ, Smart City Coll, Beijing 100101, Peoples R China.|[1141733]北京联合大学继续教育学院;[11417]北京联合大学;

年份：2023

卷号：44

期号：2

起止页码：2703-2713

外文期刊名：JOURNAL OF INTELLIGENT & FUZZY SYSTEMS

收录：;EI(收录号：20230813602438);Scopus(收录号：2-s2.0-85148068060);WOS:【SCI-EXPANDED(收录号:WOS:000925063400077)】；

基金：This work was supported by Science and Technology General Projects of Beijing Education Commission (Research on Optical and Wireless converged Access Network Networking Technology in Smart Traffic, No. KM202111417010), China Computer Federation (CCF) Opening Project of Information System (Research on Massive Event Flow oriented Stream Computing Framework, No. CCFIS2019-01-01). And thanks to Ms. Li.

语种：英文

外文关键词：Unbounded data; bounded data; integrated stream processing; Flink; load balancing

摘要：Flink is regarded as a promising distributed data processing engine for unifying bounded data and unbounded data. Unbalanced workloads upon multiple workers/task managers/servers in the Flink bring congestion, which will lead to the quality of service (QoS) decreasing. The balanced load distribution could efficiently improve QoS. Besides, existing works are lagging behind the current Flink version. To distribute workloads upon workers evenly, a resource-oriented load balancing task scheduling (RoLBTS) mechanism for Flink is proposed. The capacities of CPU, memory, and bandwidth are taken into consideration. Based on the barrel principle, the memory, and the bandwidth are respectively selected to model the resource occupancy ratio of the physical node and that of the physical link. On the based of modeled resource occupancy ratio, the data processing of load-balancing resource usage in Flink is formulated as a quadratic programming problem. Based on the self-recursive calling, a RoLBTS algorithm for scheduling task-needed resources is presented. Trough the numerical simulation, the superiority of our work is evaluated in terms of resource score, the number of possible scheduling solutions, and resource usage ratio.