What Is Big Data? Balancing Big Data And Real-Time, Part 1

By Derek Rodner, vice president of product strategy, Agilence

Big Data has been a term thrown around the IT space for several years now. However, depending on whom you talk to the meaning is different. As a baseline, let’s defer to Wikipedia that defines big data as “a collection of data sets so large and complex that it becomes awkward to work with using on-hand database management tools.” In other industries such as genomics, meteorology, and physics this is a major issue with data sets being measured in Exabytes of data. To put that in some context, consider that a single MP3 is about 3MB; a full length HD movie is 10GB. A single Exabyte of data is equal to 333 billion songs or 100 million full-length HD movies. As of April 2012 iTunes has only 28 million songs in its library.

In retail the data is not quite that large … yet. But, the time is coming where it’s not out of the realm of possibility to be crunching that much data in retail. Impossible? Not really. Retailers already collect a ton of information in the store and new technologies such as RFID are becoming pervasive. When you consider tagging every item with an RFID tag and following that item throughout its lifecycle, the amount of data becomes significant. One of the first users of RFID technology that I remember was a major chicken processor. Each and every chicken had an RFID tag attached to its leg at birth and it followed that chicken through its life and through the final processing facility. Their RFID database was generating 2TB of data per day!