Unlocking Ever-Increasing Value from NSW Health's Petabytes of Data

A seemingly simple request, such as understanding the scope of childhood obesity by calculating a body mass index measure from hospital data, can take significant effort to deliver state-wide. This makes sense when considering several types of system hold this information, with multiple vendor solutions, deployed across 17 Local Health Districts and Specialty Health Networks operating across seven different electronic medical records (EMR) databases. Each solution has its own application owner, bottlenecking analyst access as they rightly guard against performance and stability impacts to operational systems.

To build a new data extract today, the data must first be fully understood. Paradoxically, this understanding must come before analysts ever see the data. To develop this understanding, analysts must find the right mix of local subject matter experts with the right capacity, who have the right understanding of local clinical processes and data models in each health district. Even when data extract specifications are formulated, the application owners tasked with safely delivering the extracts must find capacity to write and deploy them.

For NSW Health, the time from hypothesis to data-driven conclusion can be prohibitively long if new data needs to be extracted. By the time data arrives, the organisational focus has often shifted. This curtails our ability to truly unlock the value of the petabytes of data we collect. For these reasons, NSW Health has delivered a proof-of-concept platform for the tactical acquisition of data.

The past 18 months have focused on resolving the complexities of near real-time replication of operational system data, to a central data hub. By centralising data to a single system, teams collating new data assets no longer need to deal with myriad application owners. Widely known query languages can be used to interrogate the most granular of state-wide data, removing the need for specialist technical skills to access data in each vendor system. The time from idea to data-driven value can be slashed from months to weeks.

Frontline LHD analysts engaged in the proof of concept are ecstatic about the data unlocked. The list of new possible projects grows daily. A small sample of these includes understanding optimal treatment pathways for stroke patients; the risks to childhood development resulting from premature birth; and evidence-based personalisation of care for unique populations of patients.

There are many additional benefits arising from a central source for data. Existing data extractions can be moved from operational systems, substantially improving system performance for end-users. Redundant activity across similar data extracts can be dropped, drastically reducing extract support costs for NSW Health.

In modern vernacular, NSW Health is delivering a data lake. However, it’s essential to realise that this naive replication of operational data leads to more of a data swamp. A single EMR application may have over 6,000 tables requiring navigation. This is of no use to the average analyst.

The platform proven in this proof of concept is only the first piece in a larger puzzle. The technology developed will become the foundational enabler for a broader data hub, accompanied by a catalogue of value-added tools, infrastructure and services.

For an organisation as large and as complex as NSW Health, analytics can never be efficiently centralised. There is, however, significant value in delivering a platform that meets the analytics needs of the state, democratises data, uplifts capability and consolidates efforts into standardised tools and techniques.