文献类型：会议论文

英文题名：DDLB: A dynamic and distributed load balancing strategy

作者：Liu, Bingqi[1,2]; Chang, Jiulong[1,2]; Xiao, Limin[1]; Qin, Guangjun[3]; Wei, Bing[1,2]; Huo, Zhisheng[1,2]

第一作者：Liu, Bingqi

机构：[1] State Key Laboratory of Software Development Environment, Beihang University, Beijing, 100191, China; [2] School of Computer Science and Engineering, Beihang University, Beijing, 100191, China; [3] Smart City College, Beijing Union University, Beijing, 100101, China

第一机构：State Key Laboratory of Software Development Environment, Beihang University, Beijing, 100191, China

会议论文集：Proceedings - 21st IEEE International Conference on High Performance Computing and Communications, 17th IEEE International Conference on Smart City and 5th IEEE International Conference on Data Science and Systems, HPCC/SmartCity/DSS 2019

会议日期：August 10, 2019 - August 12, 2019

会议地点：Zhangjiajie, China

语种：英文

外文关键词：Bandwidth - Decision making - Dynamics - File organization - Predictive analytics - Smart city

摘要：In view of the problem of load imbalance for distributed file systems, the current solutions either adopt centralized algorithm and implement load balancing by increasing the number of files as the replica, or lack considerations for network transmission and network bandwidth. Therefore, the current solutions are relatively inefficient in distributed file system. To solve this problem, this paper proposes DDLB, a dynamic and distributed load balancing algorithm which is completely based on distributed load balancing architecture. DDLB can not only monitor the IO load on the data server dynamically and real time, but also use the load prediction model, which based on weighted mean, to reduce the impact of network delay and collection delay on load decision-making. In order to minimize the frequency of remote load collection and decrease network bandwidth occupation, DDLB adopts the remote distributed load collection method based on threshold and the cooling collection mechanism. Moreover, dynamic replica management mechanism of DDLB takes data blocks instead of files as the basic unit for replica replication, and reduces the IO load of the data server by adding replicas of hotspot data blocks. For proving the effectiveness of DDLB, this paper conducts a comprehensive evaluation in the HDFS cluster environment. The results show that compared with the traditional load balancing algorithm, DDLB can effectively balance the IO load between data servers in the distributed file system and improve the data access performance of IO-intensive applications. ? 2019 IEEE.