We’ve all heard the promises of big data—that it will revolutionize organizations and industries alike, providing real-time insights gleaned from massive amounts of data—and not just from traditional sources, but from just about everywhere, including social media and more.
But what, exactly is big data, and more importantly, what can it do for midsized companies like yours?
What it is
Big data is an amalgam of software and hardware solutions that permits analysis of data with greater volume, variety, and velocity than traditional methods. It doesn’t supplant traditional data services, such as enterprise resource planning (ERP) suites (at least for now), but rather combines them with other data sources to provide richer and more nuanced insights.
“Big data is something of a misnomer,” says Joe Caserta, president of Caserta Concepts, a New York-based innovation technology and consulting firm. “The data doesn’t necessarily have to be big, although it most often is. It also could be thought of as low-cost data.” That’s because while traditional data generally requires large investments in proprietary software and computing hardware, so-called big data works with low-cost, open-source tools that can be run on commodity hardware or in the cloud.
How it works
Because big data treats all data—whether structured (traditional databases), semi-structured (such as email files) or unstructured (essentially everything else)—the same, once a system is set up, it is relatively easy to direct it toward different solutions. “One of the exciting things about big data is that it’s much more nimble than a big, structured environment,” Caserta says. “It is not uncommon for an IT department to build a traditional database solution only to have a business change slightly or decide to capture data a little bit differently. Suddenly, that can become a six-month project. The business just can’t tolerate that. But big data allows you to change things on the fly, almost immediately.” What’s more, mining big data is fast—fast enough to permit real-time analysis at a highly granular level.
What it’s good for
The most familiar use for big data is in marketing, something Web companies like Amazon, Google, and Facebook have been doing for years. In that scenario, customer behavior is tracked not only on an e-commerce website itself, but across the Web, allowing companies to create highly refined customer profiles that help predict behavior and target marketing. Newer techniques let “clicks and mortar” retailers correlate online activity with offline customers, essentially creating a 360-degree view of a customer cohort’s online and offline behavior. But marketing is hardly the only application of big data, nor is it limited to retail. “Anyone can find a use for it,” Caserta contends. Indeed, the technique has been applied to new-product development, strategic business planning, predictive supply chain management and fraud detection and prevention.
Where to begin
Caserta suggests starting with a question you want answered—one that can’t be satisfactorily addressed with the systems you have on hand. “We work with our clients to try to identify critical business use cases, and then we figure out which one would return the most value for the investment,” he says. “It has to be driven by the business.” Implementation of an initial big data solution can take up to three months, Caserta says, but once operational, it can be directed toward multiple queries.
Big data is not a panacea, of course, and like any analytical process, it is only as good as the information it is based on. In fact, because of its “everything and the kitchen sink” nature, big data may be particularly susceptible to error based on misused or irrelevant input. But it is already opening new avenues for mining the massive amounts of data that are being produced—inside and outside your company—every day.
