基于Spark Streaming的明安图射电频谱日像仪实时数据处理
Real-Time Data Processing in Mingantu Ultrawide Spectral Radio Heliograph Based on Spark Streaming
查看参考文献12篇
|
文摘
|
目前天文观测中对数据的实时处理需求越来越多,性能要求也越来越高,我国明安图射电频谱日像仪(MingantU SpEctral Radioheliograph, MUSER)是同时以高时间、高空间和高频率分辨率对太阳进行射电频谱成像的设备。在低频部分的日常观测中,包含了两方面的需求:(1)对历史数据的处理;(2)5秒钟抽样观测数据的处理。抽样观测数据需要实时处理,并在监控终端显示,数据处理过程包含了数据校验、修正、成图、洁化等多个步骤,传统的单机处理模式已无法满足大数据量下的实时性要求。因此,实时数据计算中,使用 Spark Streaming流式计算这一新兴的分布式计算方法,设计了自定义的接收器,并将多个图形处理器节点加入到分布式集群中。通过实验对性能进行评估,结果证明基于内存的高速执行引擎的特点能显著提高性能。期待能通过实验进一步优化算法和配置,获得更好的结果,并最终运用到实际环境中。 |
|
其他语种文摘
|
There is a growing demand for real-time processing in astronomical observations in recent years, meanwhile, the requirement for performance is also increasing dramatically. Mingantu Ultrawide Spectral Radio Heliograph ( MUSER) is a synthetic aperture radio interferometer with high temporal,spatial and spectral resolution. In daily observation of low frequency, MUSER contains two aspects of data processing, historical data processing and sampling observational data which is produced every 5 seconds and processed in real-time mode. The procedure of raw data processing contains validation, correction, clean and other processing steps, then the results need to be transmitted in real-time mode to monitoring end without user constantly refreshing or sending a request. The traditional stand-alone processing mode has been unable to meet the requirements of large amounts of data in real-time mode. In this paper, we explored the use of Spark Streaming in a new approach for MUSER real-time calculations across multiple machines and evaluated its effectiveness and efficiency. A customized receiver was created for real-time binary stream of MUSER. We also extended the Spark cluster by adding multiple GPU's nodes. The experiments have shown that Spark Streaming can significantly improve MUSER real-time processing performance for its memory-based execution engine. We might look forward to optimize the algorithm through experiments and configurations so as to obtain better results,and apply it to the actual environment of MUSER finally. |
|
来源
|
天文研究与技术
,2017,14(4):421-428 【核心库】
|
|
关键词
|
MUSER
;
射电天文
;
Spark
;
流式计算
;
实时计算
|
|
地址
|
昆明理工大学, 云南省计算机技术应用重点实验室, 云南, 昆明, 650500
|
|
语种
|
中文 |
|
文献类型
|
研究性论文 |
|
ISSN
|
1672-7673 |
|
学科
|
自动化技术、计算机技术 |
|
基金
|
国家自然科学基金
|
|
文献收藏号
|
CSCD:6071970
|
参考文献 共
12
共1页
|
|
1.
颜毅华. 关于太阳厘米-分米波段频谱日像仪研究进展.
天文研究与技术--国家天文台台刊,2006,3(2):91-98
|
CSCD被引
22
次
|
|
|
|
|
2.
张建勋. 云计算研究进展综述.
计算机应用研究,2010,27(2):429-433
|
CSCD被引
60
次
|
|
|
|
|
3.
孙大为. 大数据流式计算:关键技术及系统实例.
软件学报,2014,25(4):839-862
|
CSCD被引
99
次
|
|
|
|
|
4.
Dean J. MapReduce: simplified data processing on large clusters.
Communications of the ACM,2008,51(1):107-113
|
CSCD被引
746
次
|
|
|
|
|
5.
Koschel A. Efficiency experiments on hadoop and giraph with PageRank.
24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing,2016:328-331
|
CSCD被引
1
次
|
|
|
|
|
6.
Seo S. Hama: an efficient matrix computation with the mapreduce framework.
IEEE Second International Conference on Cloud Computing Technology and Science,2010:721-726
|
CSCD被引
1
次
|
|
|
|
|
7.
Kathleen E. On the performance of distributed clustering algorithms in file and streaming processing systems.
IEEE International Conference on Utility and Cloud Computing,2011:33-40
|
CSCD被引
1
次
|
|
|
|
|
8.
Chauhan J. Performance evaluation of Yahoo! S4: afirst look.
2012 Seventh International Conference on P2P,Parallel, Grid, Cloud and Internet Computing (3PGCIC),2012:58-65
|
CSCD被引
1
次
|
|
|
|
|
9.
Samosir J. An evaluation of data stream processing systems for data driven applications.
Procedia Computer Science,2016,80:439-449
|
CSCD被引
2
次
|
|
|
|
|
10.
Stoa S. Online analysis of myocardial ischemia from medical sensor data streams with esper.
2008 ISABEL'08 First International Symposium on Applied Sciences on Biomedical and Communication Technologies,2008:1-5
|
CSCD被引
1
次
|
|
|
|
|
11.
Ghesmoune M. Micro-Batching growing neural gas for clustering data streams using spark streaming.
Procedia Computer Science,2015,53(1):158-166
|
CSCD被引
2
次
|
|
|
|
|
12.
Wang F. Distributed data-processing pipeline for Mingantu Ultrawide Spectral Radioheliograph.
Publications of the Astronomical Society of the Pacific,2015,127(950):383-396
|
CSCD被引
10
次
|
|
|
|
|
了解气象卫星更多信息，请访问