We live in a world where data is generated from a multitude of sources, and it is really cheap to collect and store such data. However, the real benefit is not related to the data itself, but to the algorithms that are able to process that data in a tolerable time, and extract valuable insights from it. Therefore, the use of Big Data Analytics tools provides very significant advantages for both industry and academia.
Data pre-processing techniques are dedicated to correcting or mitigating errors in data. Dimensionality reduction, instance reduction, noise filtering and discretisation are examples of the most widespread data pre-processing techniques. Although we can find many theoretical proposals for data pre-processing and Big Data, there is little software development dedicated to the free provision of algorithms in Big Data environments.
BigDapTOOLS is a package of tools created with the aim of providing and unifying software developments related to data pre-processing and Big Data. This project started with funding from the BBVA Foundation. To date, we have several developments in three well-known platforms in Data Science, although the package will continue to grow in the coming years. The developments highlighted are:
- Software in R. These algorithms address problems such as data reduction with autoencoders, data preprocessing for unbalanced data sets, ordinal and noisy data, as well as a general purpose library for data preprocessing called ‘smartdata’, which collects the state of the art of algorithms for data preprocessing in R, being a container of algorithms that provides a uniform interface to other packages.(https://sci2s.ugr.es/BigDaPR).
- Software in Spark. Apache Spark is an open source engine developed specifically to handle large-scale data processing and analysis. The developed software is available in Spark Packages and contains a set of data pre-processing algorithms for feature selection, discretisation, noise filtering and missing value imputation.(https://sci2s.ugr.es/BigDaPSpark)
- Software in Flink. Apache Flink is a recent and novel Big Data framework, following the MapReduce paradigm, focused on distributed stream and batch data processing. This library contains six of the most popular data pre-processing algorithms for data streams, three for discretisation and the rest for feature selection. The associated work is in:(https://arxiv.org/abs/1810.06021)
Period
October 2016 – current
Researchers
Francisco Charte, Alberto Fernández, Francisco Herrera, Salvador García, Julián Luengo, Sergio Ramírez, Diego Jesús García, Ignacio Cordón.
