Development of Big Data
The use of data to inform decision making in organizations is not new; business organizations have been storing and analyzing large volumes of data since the advent of data warehouse systems in the early 1990s. However, the nature of data available to most organizations today is rapidly changing. According to IBM, 80% of the data organizations currently generating data are unstructured. Data also come in a variety of formats, such as text, video, audio, diagrams, images, and combinations of any two or more of these formats. With traditional solutions becoming too expensive to scale or adapt to rapidly evolving conditions, business organizations are looking for affordable technologies that will help them store, process, and query all of their data. Organizations store most of their data that are meant to support decision making in data warehouses. A data warehouse refers to a central repository of data or a centralized database system used for analyzing and reporting data. Data warehousing also represents an ideal vision of maintaining a central repository of data that provides an organization with a living memory of data that can be leveraged for better decision making.
Recent developments in database technologies made it possible to collect and maintain large and complex amounts of data in many forms and store them in multiple sources within multiple points in time and space. In addition, there are analytical tools available that can turn this complex data into meaningful patterns and value, a phenomenon referred to as Big Data. Theoretically, Big Data describes data that are fundamentally too big and move too fast, exceeding the processing capacity of conventional database systems (Manyika et al. 2010).
The notion of big, the term itself, is misleading as contested by critics as it does not reflect only data size, but complexity. Yang (2013) points out the definition of Big Data has little to do with the data itself because the analysis of large quantities of data is not new, but rather Big Data includes an emergent suite of technologies that can process massive volumes of data of various types at faster speeds than ever before.