Unlocking Ever-Increasing Value from NSW Health's Petabytes of Data

Lachlan Rudd, Director, Data & Analytics, eHealth NSW

Lachlan Rudd, Director, Data & Analytics, eHealth NSW

I've yet to meet a person who pursued a health career because they’re passionate about data entry. The digital age, however, presents a vision where the accurate recording of information leads to the efficient and safe operation of hospital systems, and ultimately better patient care. This vision implies all recorded data will be accessible, but in reality, this is difficult to realise. With the volumes of data generated within NSW Health, accessing and exploring it with existing tools can feel like panning for gold with a thimble. Such a limitation can disincentivise quality data entry.

Electronic medical record (EMR) systems do an exceptional job of filing and retrieving individual patient records. Viewing an electronic record concerning critical patient information at 3 am, when the paper-based records office is closed, can be the difference between life and death. However, there is no out-of-the-box functionality to assemble a multi-patient view, across complex patient journeys, and threaded through multiple clinical and corporate systems.

Traditional solutions to this problem revolve around the construction of a single monolithic enterprise data warehouse. However, the evolving nature and complexity of health information make this prohibitively challenging to achieve. So much so, that not a single successful case study exists within the health domain globally.

That said, targeted data warehouses do have their place. Outcomes data collected into data warehouses across NSW Health are the fuel that drives population health, value-based care and activity-based administration and funding, returning significant value to the system. However, even data warehouses comprised of only this standardised outcomes information can be difficult to expand as needs evolve.

There is also growing demand for the process data (which leads to outcomes) that has historically been buried within clinical and corporate systems. As NSW Health moves towards Horizon Three of the eHealth Strategy for NSW Health: 2016-2026, there is need to deliver personalised care, and better tailor the hospital journey to the needs of individual patients. This can’t be achieved without understanding the paths that patients take towards outcomes.

NSW Health is faced with the challenge that new data cannot be readily extracted from operational systems. NSW Health comprises over 220 hospitals and health facilities and more than 150,000 employees. Data extraction timelines explode when dealing with multiple data owners, application administrators, vendor systems, software versions and local data models.

NSW health is faced with the challenge that new data cannot be readily extracted from operational systems

A seemingly simple request, such as understanding the scope of childhood obesity by calculating a body mass index measure from hospital data, can take significant effort to deliver state-wide. This makes sense when considering several types of system hold this information, with multiple vendor solutions, deployed across 17 Local Health Districts and Specialty Health Networks operating across seven different electronic medical records (EMR) databases. Each solution has its own application owner, bottlenecking analyst access as they rightly guard against performance and stability impacts to operational systems.

To build a new data extract today, the data must first be fully understood. Paradoxically, this understanding must come before analysts ever see the data. To develop this understanding, analysts must find the right mix of local subject matter experts with the right capacity, who have the right understanding of local clinical processes and data models in each health district. Even when data extract specifications are formulated, the application owners tasked with safely delivering the extracts must find capacity to write and deploy them.

For NSW Health, the time from hypothesis to data-driven conclusion can be prohibitively long if new data needs to be extracted. By the time data arrives, the organisational focus has often shifted. This curtails our ability to truly unlock the value of the petabytes of data we collect. For these reasons, NSW Health has delivered a proof-of-concept platform for the tactical acquisition of data.

The past 18 months have focused on resolving the complexities of near real-time replication of operational system data, to a central data hub. By centralising data to a single system, teams collating new data assets no longer need to deal with myriad application owners. Widely known query languages can be used to interrogate the most granular of state-wide data, removing the need for specialist technical skills to access data in each vendor system. The time from idea to data-driven value can be slashed from months to weeks.

Frontline LHD analysts engaged in the proof of concept are ecstatic about the data unlocked. The list of new possible projects grows daily. A small sample of these includes understanding optimal treatment pathways for stroke patients; the risks to childhood development resulting from premature birth; and evidence-based personalisation of care for unique populations of patients.

There are many additional benefits arising from a central source for data. Existing data extractions can be moved from operational systems, substantially improving system performance for end-users. Redundant activity across similar data extracts can be dropped, drastically reducing extract support costs for NSW Health.

In modern vernacular, NSW Health is delivering a data lake. However, it’s essential to realise that this naive replication of operational data leads to more of a data swamp. A single EMR application may have over 6,000 tables requiring navigation. This is of no use to the average analyst.

The platform proven in this proof of concept is only the first piece in a larger puzzle. The technology developed will become the foundational enabler for a broader data hub, accompanied by a catalogue of value-added tools, infrastructure and services.

For an organisation as large and as complex as NSW Health, analytics can never be efficiently centralised. There is, however, significant value in delivering a platform that meets the analytics needs of the state, democratises data, uplifts capability and consolidates efforts into standardised tools and techniques.