Suddenly, we love data. It’s the hero in our TV shows, saves our baseball teams, generates our art, and is the topic of the moment at Davos. Our obsession borders on religion: we believe data is infallible, containing a single, emergent truth (the Guardian’s Datablog’s subtitle is “Facts are Sacred”). If our businesses or institutions are failing we say they need more data.
But when we embrace “Big Data” we neglect the operations and environments we hope the data will improve. Which is unfortunate because data’s value is determined by how well we capitalize on the intelligence it produces. If a film studio identifies a new trend they’re limited by how long a film takes to produce. If a retail outlet discovers that peanut butter buyers can also be sold bananas, they’re limited by how quickly employees can rearrange shelves.
Adding more data to the mix doesn’t help if infrastructures aren’t upgraded. If we ignore the systems we aim to improve, the data we crave is worth only a fraction of its potential value.
For those of us investing in or producing data, the bottleneck is our users, culture, and infrastructure. We’d be wise to learn from examples in other fields and scenarios. Take Dubai, for instance, where skyscrapers sprung up while sewage systems stagnated.
No One Wants their Name on a Sewer
Dubai’s record-setting skyline emerged over the last decades, fueled by rising fuel prices. Wikipedia lists 195 skyscrapers built or under construction in the once quiet city, whose population has more than quintupled in the last 30 years.
But all is not rosy: while the skyline was funded the sewage system was ignored. In 2008, The Wall Street Journal wrote, “By one estimate, some $300 billion in new projects are going up in Dubai in the next 10 years… But Dubai’s single, 30-year-old sewage-treatment plant isn’t keeping up. Sewage output here is rising by 25% a year.” The 160-floor Burj Khalifa alone was designed to house 25,000 people, nearly a 10th of Dubai’s total population when their sewage treatment plant was built.
Without adequate infrastructure, Dubai employed an army of tanker trucks to empty septic tanks throughout the city. The WSJ detailed the situation, though I’ve omitted the more squeamish bits:
Every day, hundreds of tanker trucks line up for almost two miles at the three approaches to the Dubai treatment plant to dump sewage. The wait can be as long as 10 hours, drivers say.
In order to avoid the wait and processing fees, some truckers have been discharging loads onto streets of desolate areas in the city, Dubai officials say, adding that they have fined several violators.
Only when these problems manifested were funds allocated for a new sewage plant.
Why did Dubai fail to update its sewage systems while it furiously updated everything else? On Fresh Air, Anatomy of a City author Kate Ascher suggested Dubai was dismayed by the complexity of creating a sewage system:
When you bring in the world’s, you know, most sophisticated architects and engineers, you can literally build anything, including a building of 140 or 150 stories. But designing a municipal network of sewage treatment is, in some ways, more complex - certainly requires more money and more time to make it happen. So one just seemed to jump ahead of the other.
But if Dubai were truly complexity-adverse they wouldn’t have doubled their coastline by dredging “a hundred million cubic yards of sand from the Persian Gulf, along with seven million tons of rock to form a man-made Island in the shape of a palm.” Complexity may have contributed to Dubai’s sewage neglect but it did not prohibit the construction of a new treatment plant.
Prestige must enter the equation. It’s easy to sign a check paying for the world’s tallest building but more difficult to allocate funds for an invisible system to handle waste. There are more television specials, movie roles, and parties for the former. The Burj Khalifa’s Wikipedia page lists 18 broken records and a section dedicated to its appearances in popular culture.
Meanwhile, no one wants their name on a sewage system. In San Francisco, citizens once attempted to rename a waste treatment plant out of spite. Contrast this with the renaming of the Burj Khalifa, formerly named Burj Dubai. During the financial crisis, Dubai found itself in dire financial straits. After their UAE sibling Abu Dhabi bailed out Dubai to the tune of $10 billion, the world’s tallest building was rechristened in honor of UAE President Khalifa bin Zayed Al Nahyan for his crucial support.
Data is our Skyscraper, Habits and Infrastructure are our Sewers
Ever since Steve Jobs unveiled a massive data center during a keynote and Jeff Bezos responded in kind, pictures of newly constructed data centers have become the must-have PowerPoint slide for the executive set. Data centers, analytics, and all the other totems of “Big Data” are in vogue.
Data has become our skyscrapers. Our sewer systems are our cultural and operational habits, which are nuanced, less visible, difficult to change, and limit our data from realizing its full potential. My personal favorite example is the business which insisted on near-real-time web traffic data to inform digital adverting buys, which were formally purchased via fax. The fax, I was told, felt more secure and comfortable.
Or there’s the digitization of the US health care system, which is still very much a work in progress. $18 billion was made available for the transition as part of a federal stimulus program, but only 10% of hospitals have qualified for additional allocations by meeting “meaningful use” requirements, which the CIO of NorthShore University Health System describes as, “showing that you did more than install [systems and software], but you are actually using it.”
Changing habits and creating new workflows isn’t as simple as acquiring data or installating software. What makes adoption even more difficult is when Big Data excitement adds more data to the equation before organizations are finished (or even start) adjusting. Sadly, The Economist claims this is likely the norm: “62% of [workers] say that the quality of what they do is hampered because they cannot make sense of the data they already have, according to Capgemini, a consultancy. This will only get worse: the data deluge is expected to grow more than 40 times by 2020.”
Data value is dependent on infrastructure as well as users. When data cannot be reliably delivered to users there’s zero chance of adoption. Consider the U.S Military, who’s struggling with a lack of bandwidth for their data-producing drones:
The cameras on drone aircraft in Afghanistan take such precise pictures that not all the data can quickly travel over the local Internet connections to analysts in the United States. The usual method is to store everything in a local cargo container full of receiving gear, computers and storage, then airlift it home when it’s full, swapping out another container to absorb more info from the drones.
True, the military is able to act on parts of their feeds in near realtime, but full analysis is dependent on flight times back to bases with more robust connections. Meanwhile, new drones sporting the Army’s new Autonomous Real-time Ground Ubiquitous Surveillance Imaging System (ARGUS) are set to arrive and pour fuel on the fire. Danger Room’s Spencer Ackerman describes the imminent challenge:
Just how powerful is ARGUS? You know, just a 1.8 gigapixel camera package, consisting of 92 five-megapixel imagers. One blink of ARGUS’ eye covers up to 36 square miles, depending on the quality of the resolution; it will give its remote pilots at least 65 independent, scaleable video windows within that blink… In a single day, ARGUS collects six petabytes of video — the equivalent of 79.8 years‘ worth of HD video.
The military, and everyone for that matter, is hindered by the fact that capture and storage technologies are advancing faster than communication technology. State of the art equipment scans the Afghan landscape, storing every bit on absurdly cheap hard drives before they’re loaded on a plane and flown home. Similarly, an iPhone 4S captures 1080P video to a 64GB SSD the size of thumbnail but chokes on a spotty 3G network.
Looking Forward to Filtering, Design, and (Eventually) Better Infrastructure
Culture and infrastructure are our new bottlenecks; we have more than enough of data. Based on the cases above, I’ve identified 3 tactics which mitigate or solve the “data deluge.” They are, from most to least difficult:
- Building better infrastructure
- Investing in design
- Getting rid of data
Building better infrastructure usually requires tough, big, unpopular decisions with high risks. Kicking off these retrofitting or rebuilding projects requires either a strong leader or a crafty plan.
A strong leader is behind the current restructuring JC Penney. CEO Ron Johnson plans to close 47 stores and cut 5,500 jobs as part of an effort to save the chain, end confusing discounts, and refactor inefficient operations. According to Johnson, only 0.2% of last year’s sales came from full price items and Penney’s bloated reporting structure is costs an extra $90 million per year. Johnson’s ability to push through difficult changes is due the bleakness of the situation and his past record leading the conception and creation of Apple’s retail efforts over the last decade.
Lacking a strong, willing leader, more creative methods may be employed. Back in Dubai, sewage problems are being alleviated by the construction of a massive wastewater pipeline. The plan is largely overlooked because its construction is bundled in with the construction of the Dubai Metro system. As workers lay the foundation for the world’s largest autonomous train network, they’re also laying sewage pipes in the same ditches. Like a parent hiding vegetables in dessert, Dubai’s planners have snuck in a sewage system with yet another record-breaking project.
In a way, the disguise of the sewage system was part of it’s design, which brings us to tactic number 2, investing in design. If you’re unable to reboot, designing interfaces to connect existing cultures and infrastructure to new data and its output might lead to success. Once a small linkage succeeds, it can act as a beachhead to evolve user habits towards the data’s natural state.
A good example of this tactic can be seen in the rollout of Apple’s music ecosystem. Jobs and company had a vision of a digital music future where a wealth of songs would be available at any moment, delivered to listeners through a single store to a ubiquitous device. But at the time, music habits and music players were dependent on physical media. Rather than dropping the full ecosystem into the marketplace, Apple slowly evolved towards their vision from a single, simple interface compatible with existing culture and infrastructure: iTunes. Apple initially positioned iTunes as a way to import existing CDs and burn new ones (though only the Power Mac came with a burner). 10 months later they launched the iPod. After 2 years of gaining traction were they able to convince the labels to join them and the iTunes Store was launched. With the iPhone’s 2007 launch the device and the market were finally merged, largely completing the original blue-sky vision after 6 years1.
Speaking of the iPhone, the rise of smartphones heightens Big Data’s need to invest in design. Fitting petabytes of information through a 5-by-7 centimeter screen at 3G speeds requires smart choices about what you show and how you show it. Interfaces must map new data to existing behaviors with less room to maneuver. These choices are part and parcel of tactic 3, getting rid of data.
The easiest way to assuage data overload is to reduce the amount of data at hand. Businesses probably don’t need all the data they want. In fact, they almost certainly don’t use all the data they already own. Determining just how big your “Big Data” needs to be will save you many infrastructure and cultural headaches.
In addition to restructuring, Ron Johnson’s changes at JC Penney significantly reduce the amount of data aggregated and published by the chain. Going forward, JC Penney’s will reduce the frequency of its sales reports and profit forecasts, from monthly to quarterly and quarterly to annually, respectively. Further, they will conduct 12 pricing promotions per year instead of the 590 which ran last year, reducing the signage, pricing, and product changes required by retail staff. Johnson and his team realized the bottleneck in Penney’s process was the speed at which employees could manage merchandise and customers could follow store and pricing changes. Insights arriving faster than either of these processes are at best wasted and at worst distracting.
As more and more people excitedly embrace ‘Big Data’, we’d be wise to remember that such solutions are not turn-key. Without updating the infrastructure and educating the people we aim to improve data’s full value can never be realized.
Icons: Iconfactory’s Smooth Icons