Business Intelligence: Meet the Data Wranglers

Regardless of the business sector in which your company operates, chances are you’re struggling with data overload. In fact, the wrangling of data has become a huge area of focus for most IT organizations.

“Everybody is drowning in data, and we haven’t even seen a massive amount of data yet,” says Hossein Eslambolchi, CEO of 2020 Venture Partners, which provides IT consulting service to venture capital firms. Eslambolchi is author of 2020 Vision (Silicon Press, 2006), an exploration of how IT will transform business over the next decade. He expects data to become a growing challenge over the next 10 years, as wireless carriers beef up networks to support data consumption via super-fast broadband connections.

Some companies are doing what they can today to squeeze the most value out of the massive amounts of data they’re shepherding. What follows are tales of how four very different enterprises have made headway in wrangling data. For more on this topic, read the article “Business Intelligence: Five Tips for Managing Data.”

ComScore: Analyzing Data Where It Sits

Few companies can claim that data are more important to their business than ComScore, which monitors the online behavior of some 2 million consumers around the world. The company collects information ranging from general online activity–such as sites visited–to granular details, such as mouse movements and clicks. It then uses that information to provide projections that companies use to plan every part of their digital strategies, including marketing, sales and product development.

The amount of data ComScore accumulates has been growing rapidly, forcing the company to constantly revamp how it crunches that data to assemble projections. ComScore captures 120 million pages a day in analyzing online transactions, and its database stores 2 trillion records totaling 147 terabytes. The company had been pulling data into individual applications for analysis, an approach that had become increasingly inefficient and time-consuming, says Scott Smith, vice president of data warehousing.

“At the end of the day, our whole business model is predictive analytics,” says Smith. “We needed to get faster in coming up with projections.”

To speed its data analysis, ComScore recently went live with a platform that combines Sybase’s IQ analytics server with software from Fuzzy Logix, enabling the company to analyze information in the database. Doing so, Smith says, is paying off for ComScore on a number of fronts:

• It no longer has to move massive data files–sometimes as large as 500 GB–from the database to any number of applications.
• Programmers no longer have to write complex analyses in SQL, saving countless hours.
• Vast amounts of bandwidth that had been devoted to the movement of data between the database and the apps have been freed up for other uses.
• Because the new setup prevents data from being used repeatedly after it’s been removed from the database, ComScore’s projections are based on fresh, accurate and accessible data.

“It gives me a lot more confidence in terms of how the analysis is done because we know what the state of the data is,” Smith says. “Previously, we would have had to extract the data and prep it to do all that analysis. Now, we’re doing it all directly in the data warehouse.”