The growing variety of health care data presents many challenges for IT leaders as the velocity and volume of this data continues to explode.
Data promises to revolutionize health care in the years ahead. However, the path to success is paved with more than a few bumps and bruises. Not only must health care firms amass large and varied data sets, it's essential to manage them and combine them in ways that produce real-world results. "Designing and building the data models of the future is an extremely challenging task," said Dan Blake, CTO at Valence Health, a provider of value-based care solutions for hospitals and health systems.
Valence Health, which specializes in data services and technology platforms designed to drive better health outcomes, supports more than 85,000 physicians and 135 hospitals that represent care solutions for upwards of 20 million patients in the U.S. "We design and build the models that deliver data and results," Blake said. "Although we don't have the volume of data that other industries encounter, we have an extreme variety of data—and we are facing a growing velocity of data." Consequently, the company is moving away from an environment where reports and files "come in once a month to near real-time systems."
It's no small task. Altogether, the firm manages a staggering 2,000 inbound data feeds with 45 different types of data, including unstructured formats. On a busy day, more than 20 million records stream in. However, the challenges don't stop there. Rapid client growth and the associated increase in data volumes had pushed the company's existing technology infrastructure to the breaking point. "In some cases, the user interfaces and architectures in place wouldn't work adequately. We would sometimes wind up receiving some data elements but have to wait on others before we could process everything. We stepped back and realized that in today's connected world, we needed a better data platform," Blake explained.
As a result, Valence Health has migrated to a sophisticated data platform driven by big data solutions provider MapR. It relies on a Hadoop distribution model to build a data lake that now serves as the main repository for the firm's most valuable data assets. "When we took a close look at the environment, how data and digital business is evolving and where we needed to be, we recognized that the Hadoop world was where we needed to be. It beat everything else hands down in terms of speed and flexibility," Blake said. After surveying Hadoop vendors, Valence Health opted to deploy MapR, which, among other things, delivers a robust and highly configurable data ingestion mechanism that allows the firm to process its data faster and more efficiently. After launching the big data platform in December 2014, it went live with the technology in September.
The results have been nothing less than stellar. In addition to generating higher quality data and making broader data sets and analysis available to analysts and clients, Valence Health can now on-board new data sets in minutes rather than 18 to 22 hours. It can also view point-in-time recovery snapshots in granular detail. This has helped free staff time and resources so that the firm can address client needs faster and more efficiently. In addition, the initiative has helped the firm venture deeper into data science. Today, Valence Health is able to examine new and more varied types of data, including socioeconomic and demographic information, immunization records or mobile health device data.