| Issue |
EAS Publications Series
Volume 78-79, 2016
Mathematical Tools for Instrumentation & Signal Processing in Astronomy
|
|
|---|---|---|
| Page(s) | 179 - 208 | |
| Section | Mathematical Concepts, Methods and Tools | |
| DOI | https://doi.org/10.1051/eas/1678009 | |
| Published online | 02 September 2016 | |
D. Mary, R. Flamary, C. Theys and C. Aime (eds)
EAS Publications Series, 78–79 (2016) 179-208
Processing Solutions for Big Data in Astronomy
Université Côte d'Azur, CNRS, I3S, CS 40121, 06903 Sophia Antipolis Cedex, France
This paper gives a simple introduction to processing solutions applied to massive amounts of data. It proposes a general presentation of the Big Data paradigm. The Hadoop framework, which is considered as the pioneering processing solution for Big Data, is described together with YARN, the integrated Hadoop tool for resource allocation. This paper also presents the main tools for the management of both the storage (NoSQL solutions) and computing capacities (MapReduce parallel processing schema) of a cluster of machines. Finally, more recent processing solutions like Spark are discussed. Big Data frameworks are now able to run complex applications while keeping the programming simple and greatly improving the computing speed.
© EAS, EDP Sciences, 2016