INFRASTRUCTURE FOR BIG DATA ANALYTICS

In order to support the aforementioned challenges, there is a standard reference architecture for analyzing big data, which is characterized by a multilayered structure, as depicted in Figure 5.2.

FIGURE 5.2 Reference IT infrastructure for big data analytics.

Source: Adapted (significantly modified) from Shirley Coleman et al. [6].

The architecture exhibits seven main layers:

• • Hardware and Networking Infrastructure: Consisting of the physically connected devices providing the core computational power and memory requirements.
• • Data Storage: Representing the software components to store and manage large data repositories. We can further distinguish two different sublayers:

о File Systems: Distributed file systems providing storage, fault tolerance, scalability, reliability, and availability. Core technologies in this layer are, for example, the Google file system and the Hadoop system.

о Datastores: These are the evolution of traditional application databases, which guarantee high-performance distributed access and querying of data into heterogeneous formats. NoSQL databases feature flexible modes, support for simple and easy copy, simple application programming interfaces, eventual consistency, and support of large volumes of data.

• • Data Processing: Representing the core technologies and frameworks for streaming, hiteractive, real-time, batch, and iterative data processing. These frameworks work on top of the data storage infrastructure and provide support to more complex data analytics and integration components by enabling scalable primitives for the access and management of data.
• • Data Integration: Representing the supporting technologies for data ingestion, extraction, transformation, loading, and metadata management.
• • Data Analytics: Refers to the analytical tools, which help us to explore, predict, and analyze data and learning. Besides that, this layer includes libraries and systems supporting out-of-the-box implementations (such as R, MATLAB, Mahout, MLLib).
• • Visualization and Reporting: For outputting results in support of the interpretation phase.

Finally, the orthogonal layer of infrastructure management-the operational frameworks for security, benchmarking, and performance optimization to manage workloads, resource scheduling, and management and activity coordination.