The Challenges & Benefits of Big Data In Real-Time, Part 2

By Derek Rodner, vice president of product strategy, Agilence

In this first part of this series we identified the different types of data (structured and unstructured) and we formalized a definition of Big Data from a retailer’s perspective. As a recap, the definition is as follows:

The collection of large amounts of (structured and unstructured) data from multiple, disparate, and often unrelated systems into a single repository to enable retailers to more efficiently and effectively understand their business and their customers in a timely manner.

Part 3 of this series will propose a reference architecture for deploying a real-time big data solution in retail. But first, let’s turn our attention to some of the challenges facing retailers and some of the benefits that await at the end of the journey.

In July, a 44-page “Big Data Insights and Opportunities” report released by CompTIA, the nonprofit association for the information technology industry, shows executives are aware of what big data means to getting their business right. Greater than two-thirds of respondents agreed with the statement, “If we could harness all of our data, we would be a much stronger business”. However, corralling all of that data is getting more difficult. The market research firm IDC says the total amount of world data is at 2.7 zettabytes (1,000,000,000,000,000,000,000 bytes) and the volume will double every 2 years. Closer to home, the Harvard Business Review estimates that Walmart is collecting 2.5 petabytes of data from customer transactions every hour.

Within the next two years, 30% of retailers will run out of storage capacity, according to The Edgell Knowledge Networks, and while nearly 80% are aware of the Retail Big Data concept, less than half understand the implications it has on their business. One immediate implication is clear, if retailers continue to ignore this issue they will be overrun by it.