Share Email Print

Proceedings Paper

RabbitQR: fast and flexible big data processing at LSST data rates using existing, shared-use hardware
Author(s): Ralf Kotulla; Arvind Gopu; Soichi Hayashi
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

Processing astronomical data to science readiness was and remains a challenge, in particular in the case of multi detector instruments such as wide-field imagers. One such instrument, the WIYN One Degree Imager, is available to the astronomical community at large, and, in order to be scientifically useful to its varied user community on a short timescale, provides its users fully calibrated data in addition to the underlying raw data. However, time-efficient re-processing of the often large datasets with improved calibration data and/or software requires more than just a large number of CPU-cores and disk space. This is particularly relevant if all computing resources are general purpose and shared with a large number of users in a typical university setup. Our approach to address this challenge is a flexible framework, combining the best of both high performance (large number of nodes, internal communication) and high throughput (flexible/variable number of nodes, no dedicated hardware) computing. Based on the Advanced Message Queuing Protocol, we a developed a Server-Manager- Worker framework. In addition to the server directing the work flow and the worker executing the actual work, the manager maintains a list of available worker, adds and/or removes individual workers from the worker pool, and re-assigns worker to different tasks. This provides the flexibility of optimizing the worker pool to the current task and workload, improves load balancing, and makes the most efficient use of the available resources. We present performance benchmarks and scaling tests, showing that, today and using existing, commodity shared- use hardware we can process data with data throughputs (including data reduction and calibration) approaching that expected in the early 2020s for future observatories such as the Large Synoptic Survey Telescope.

Paper Details

Date Published: 8 August 2016
PDF: 7 pages
Proc. SPIE 9913, Software and Cyberinfrastructure for Astronomy IV, 99131U (8 August 2016); doi: 10.1117/12.2233527
Show Author Affiliations
Ralf Kotulla, Univ. of Wisconsin-Madison (United States)
Arvind Gopu, Indiana Univ. (United States)
Soichi Hayashi, Indiana Univ. (United States)

Published in SPIE Proceedings Vol. 9913:
Software and Cyberinfrastructure for Astronomy IV
Gianluca Chiozzi; Juan C. Guzman, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?