帮助 关于我们

返回检索结果

基于Spark Streaming的明安图射电频谱日像仪实时数据处理
Real-Time Data Processing in Mingantu Ultrawide Spectral Radio Heliograph Based on Spark Streaming

查看参考文献12篇

文摘 目前天文观测中对数据的实时处理需求越来越多,性能要求也越来越高,我国明安图射电频谱日像仪(MingantU SpEctral Radioheliograph, MUSER)是同时以高时间、高空间和高频率分辨率对太阳进行射电频谱成像的设备。在低频部分的日常观测中,包含了两方面的需求:(1)对历史数据的处理;(2)5秒钟抽样观测数据的处理。抽样观测数据需要实时处理,并在监控终端显示,数据处理过程包含了数据校验、修正、成图、洁化等多个步骤,传统的单机处理模式已无法满足大数据量下的实时性要求。因此,实时数据计算中,使用 Spark Streaming流式计算这一新兴的分布式计算方法,设计了自定义的接收器,并将多个图形处理器节点加入到分布式集群中。通过实验对性能进行评估,结果证明基于内存的高速执行引擎的特点能显著提高性能。期待能通过实验进一步优化算法和配置,获得更好的结果,并最终运用到实际环境中。
其他语种文摘 There is a growing demand for real-time processing in astronomical observations in recent years, meanwhile, the requirement for performance is also increasing dramatically. Mingantu Ultrawide Spectral Radio Heliograph ( MUSER) is a synthetic aperture radio interferometer with high temporal,spatial and spectral resolution. In daily observation of low frequency, MUSER contains two aspects of data processing, historical data processing and sampling observational data which is produced every 5 seconds and processed in real-time mode. The procedure of raw data processing contains validation, correction, clean and other processing steps, then the results need to be transmitted in real-time mode to monitoring end without user constantly refreshing or sending a request. The traditional stand-alone processing mode has been unable to meet the requirements of large amounts of data in real-time mode. In this paper, we explored the use of Spark Streaming in a new approach for MUSER real-time calculations across multiple machines and evaluated its effectiveness and efficiency. A customized receiver was created for real-time binary stream of MUSER. We also extended the Spark cluster by adding multiple GPU's nodes. The experiments have shown that Spark Streaming can significantly improve MUSER real-time processing performance for its memory-based execution engine. We might look forward to optimize the algorithm through experiments and configurations so as to obtain better results,and apply it to the actual environment of MUSER finally.
来源 天文研究与技术 ,2017,14(4):421-428 【核心库】
关键词 MUSER ; 射电天文 ; Spark ; 流式计算 ; 实时计算
地址

昆明理工大学, 云南省计算机技术应用重点实验室, 云南, 昆明, 650500

语种 中文
文献类型 研究性论文
ISSN 1672-7673
学科 自动化技术、计算机技术
基金 国家自然科学基金
文献收藏号 CSCD:6071970

参考文献 共 12 共1页

1.  颜毅华. 关于太阳厘米-分米波段频谱日像仪研究进展. 天文研究与技术--国家天文台台刊,2006,3(2):91-98 被引 22    
2.  张建勋. 云计算研究进展综述. 计算机应用研究,2010,27(2):429-433 被引 59    
3.  孙大为. 大数据流式计算:关键技术及系统实例. 软件学报,2014,25(4):839-862 被引 99    
4.  Dean J. MapReduce: simplified data processing on large clusters. Communications of the ACM,2008,51(1):107-113 被引 737    
5.  Koschel A. Efficiency experiments on hadoop and giraph with PageRank. 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing,2016:328-331 被引 1    
6.  Seo S. Hama: an efficient matrix computation with the mapreduce framework. IEEE Second International Conference on Cloud Computing Technology and Science,2010:721-726 被引 1    
7.  Kathleen E. On the performance of distributed clustering algorithms in file and streaming processing systems. IEEE International Conference on Utility and Cloud Computing,2011:33-40 被引 1    
8.  Chauhan J. Performance evaluation of Yahoo! S4: afirst look. 2012 Seventh International Conference on P2P,Parallel, Grid, Cloud and Internet Computing (3PGCIC),2012:58-65 被引 1    
9.  Samosir J. An evaluation of data stream processing systems for data driven applications. Procedia Computer Science,2016,80:439-449 被引 2    
10.  Stoa S. Online analysis of myocardial ischemia from medical sensor data streams with esper. 2008 ISABEL'08 First International Symposium on Applied Sciences on Biomedical and Communication Technologies,2008:1-5 被引 1    
11.  Ghesmoune M. Micro-Batching growing neural gas for clustering data streams using spark streaming. Procedia Computer Science,2015,53(1):158-166 被引 1    
12.  Wang F. Distributed data-processing pipeline for Mingantu Ultrawide Spectral Radioheliograph. Publications of the Astronomical Society of the Pacific,2015,127(950):383-396 被引 10    
引证文献 3

1 袁慧宇 基于SVM的食双星光变曲线自动分类算法 天文研究与技术,2019,16(2):187-193
被引 0 次

2 梅盈 一种基于多尺度带通滤波的洁化算法与GPU实现 天文研究与技术,2018,15(3):285-291
被引 0 次

显示所有3篇文献

论文科学数据集
PlumX Metrics
相关文献

 作者相关
 关键词相关
 参考文献相关

版权所有 ©2008 中国科学院文献情报中心 制作维护:中国科学院文献情报中心
地址:北京中关村北四环西路33号 邮政编码:100190 联系电话:(010)82627496 E-mail:cscd@mail.las.ac.cn 京ICP备05002861号-4 | 京公网安备11010802043238号