文献类型：期刊文献

中文题名：面向大数据处理应用的广域存算协同调度系统

英文题名：A wide-area collaborative scheduling system oriented to big data processing applications

作者：张晨浩[1,2];肖利民[1,2];秦广军[3];宋尧[1,2];蒋世轩[1,2];王继业[4]

第一作者：张晨浩

机构：[1]软件开发环境国家重点实验室,北京100191;[2]北京航空航天大学计算机学院,北京100191;[3]北京联合大学智慧城市学院,北京100101;[4]国家电网有限公司大数据中心,北京100031

第一机构：软件开发环境国家重点实验室,北京100191

年份：2021

卷号：7

期号：5

起止页码：82-97

中文期刊名：大数据

外文期刊名：Big Data Research

收录：CSTPCD;;国家哲学社会科学学术期刊数据库

基金：国家重点研发计划资助项目(No.2017YFB1010000)。

语种：中文

中文关键词：广域存算协同调度;大数据处理应用;虚拟数据空间;高性能计算环境

外文关键词：wide-area collaborative scheduling;big data processing application;global virtual data space;high-performance computing environment

摘要：以我国研发的高性能计算虚拟数据空间系统为基础,针对大数据处理应用如何统筹利用广域存储和计算资源的问题,设计并实现了一套面向大数据处理应用的广域存算协同调度系统。该系统可依据应用的计算特征和数据布局,通过存算协同、负载均衡、数据局部性感知等策略,在广域环境中协同调度应用数据和计算任务,统筹利用广域计算和存储资源,有效提升大数据处理应用的运行性能。在国家高性能计算环境中实际测试的结果表明,提出的调度方法可有效地支撑大数据处理应用,跨域目标协同识别、分子对接等典型应用的运行效率可提升3~4倍。
Based on the high-performance computing global virtual data space system,a wide-area collaborative scheduling system for big data processing applications was designed and implemented.This system can address the issue of how big data processing applications unified use wide-area storage and computing resources.And it can collaborative schedule of application data and computing tasks based on the computing characteristics of the application and data layout through collaborative scheduling,load balancing scheduling,data locality scheduling strategies.By unified scheduling of application data and computing tasks in the wide-area environment,it can coordinate the utilization of wide-area computing and storage resources,and effectively improve the running performance of big data processing applications.The actual test results in the national high-performance computing environment show that the scheduling method proposed can support big data processing applications effectively,and the running efficiency of typical applications such as wide-area target collaborative recognition and molecular docking can be increased by 3~4 times.