BIG DATA WITH MATLAB: HADOOP, MAPREDUCE, CLUSTERS, AND PARALLEL COMPUTING

Big data analytics examines large amounts of data to uncover hidden patterns, correlations and other insights. With today’s technology, it’s possible to analyze your data and get answers from it almost immediately – an effort that’s slower and less efficient with more traditional business intelligence solutions. Large data sets can be in the form of large files that do not fit into available memory or files that take a long time to process. A large data set also can be a collection of numerous small files. There is no single approach to working with large data sets, so MATLAB® includes a number of tools for accessing and processing large data. Begin by creating a datastore that can access small portions of the data at a time. You can use the datastore to manage incremental import of the data. To analyze the data using common MATLAB functions, such as mean and histogram, create a tall array on top of the datastore. For more complex problems, you can write a MapReduce algorithm that defines the chunking and reduction of the data. Tall arrays are used to work with out-of-memory data that is backed by a datastore. Datastores enable you to work with large data sets in small chunks that individually fit in memory, instead of loading the entire data set into memory at once. Tall arrays extend this capability to enable you to work with out-of-memory data using common functions. mapreduce is a programming technique which is suitable for analyzing large data sets that otherwise cannot fit in your computer's memory. Using a datastore to process the data in small chunks, the technique is composed of a Map phase, which formats the data or performs a precursory calculation, and a Reduce phase, which aggregates all of the results from the Map phase. Read more