Data Odyssey 2010: To Pluto and Back

The world is awash in digital data created by individuals, and by 2010 the amount of data added annually will swell from 161 exabytes to 988 exabytes, according to a report released Tuesday by technology research firm IDC. An exabyte is equal to one billion gigabytes.

In 2006 alone, the researchers said, there was enough data to stack books from the earth to the moon a dozen times. In 2010, there could be enough data to extend the book stack from the sun to Pluto and back again.

The report, sponsored by EMC (NYSE: EMC), an information infrastructure solutions firm, predicts that 70 percent of this ever-expanding glut of digital information will pass through an "organization along the way -- on a network, in a data center, at a hosting site, at a telephone or Internet switch, or in a backup system."

Various organizations -- governments, associations, agencies or businesses of any size -- will be required to ensure the security, privacy, reliability and compliance of some 85 percent of this data.

"This ever-growing mass of information is putting a considerable strain on the IT infrastructures we have in place today," said Mark Lewis, executive vice president and chief development officer at EMC. "This explosive growth will change the way organizations and IT professionals do their jobs, and the way we consumers use information."

"Given that 85 percent of the information created and copied will be the responsibility of organizations and businesses; we must take steps as an industry to ensure we develop flexible, reliable and secure information infrastructures to handle the deluge," Lewis noted.

Digital Black Hole

The IDC study followed in the footsteps of research previously conducted at the University of California, Berkeley. The university's research dealt with only original information, and estimated how much digital information all of that would translate into.

The IDC researchers expanded the scope to encompass information created and copied in the digital universe. The data was then segmented by region "and put it in context with the available storage capacity," according to the report.

"We believe ours is the first-ever study to size and forecast the rate of expansion of the digital universe," the IDC researchers said.

The digital universe -- all information that is either created or captured in digital form and then replicated -- will grow by some 57 percent annually, researchers predict.

The IDC found that roughly one quarter of the digital universe is comprised of original data, such as pictures recorded, keystrokes in an e-mail and phone calls. The remaining 75 percent is replicated data, including forwarded e-mails, backed up transaction records and movies on DVD.

Digital information has become an integral part of the modern world. From the airwaves to telephone circuits to computer cables, data streams are constantly abuzz. New and not-so-new technologies including HDTVs, the Internet and digital cameras generate enormous bits of information.

For instance, YouTube hosts 100 million video streams every day, billions of songs are shared online as MP3s, and London's traffic surveillance cameras send 64 trillion bits per day to its command center, the researchers found.

Images, both moving and still, accounted for the bulk of digitized bits, the study said. Each picture taken with a digital camera can easily generate one megabyte or more of digital data. Video or digital TV can produce dozens of megabytes per second. Voice signals and keystrokes are at the bottom of the list -- it would take a good typist more than 36 hours to produce a megabyte of keystrokes. Voice signals create less than one megabyte per second.

Digital information created and replicated within the workplace is growing at a faster pace than any other segment of the digital universe. In 2006, according to the IDC, about 25 percent of the data generated, both original and copied, came from the business world. By 2010, that portion will increase to 30 percent.

The study attributes the increase in organization information output to "the increased computerization of small businesses, regulations mandating new archiving and privacy standards, and industry-specific applications -- from security imaging and Internet commerce to medical imaging, sensor networks, and customer support applications that now include Web-based 'click-to-talk' service."

Information Overload?

The IDC forecasts that the media available on which to store all of this newly created and replicated digital data will grow at a rate of 35 percent each year between 2006 to 2010, from 185 exabytes to 601 exabytes. Rather than a shortfall in storage, however, it is the expanding responsibility to maintain the data that will put pressure on existing computing operations and drive organizations to develop more information-centric computing architectures. Companies, however, should not start panicking.

"It's important to understand that not all digital information created is stored today," Dave Farmer, a spokesperson for EMC, told TechNewsWorld. "A growing percentage of what's not stored today will need to be stored, but not all."

Hypothetically, Farmer explained, if we wanted to store all information created and replicated in 2007 (255 exabytes), we will fall short of the storage capacity available this year (246 exabytes), according to IDC's projections.

This hypothetical gap will increase in the future, as information will grow at an annual average of 57 percent over the next four years versus the 35 percent average annual growth in projected storage capacity.

"Considering the fact that organizations will be responsible for the security, reliability, privacy and compliance of at least 85 percent of this information, current infrastructures are inadequately equipped and do not have the right information management tools to handle the expected growth and provide the level of service required," he continued.

"From a technology perspective, organizations will need to employ ever-more sophisticated techniques to transport, store, secure and replicate the additional information that is being generated every day," he added.