How Have Recent Advancements in Big Data Been Impacting Businesses?

Marc Solomon, CIO, Bvn Architecture

Marc Solomon, CIO, Bvn Architecture

Data, big, large, or small, is only useful once transformed into information. Furthermore, data is only useful when it delivers actionable intelligence. You must be able to change course, take corrective actions, be that manually or programmatically, as a result of that intelligence for it to be truly advantageous.

Therefore, "Big Data" has to incorporate the ecosystem surrounding the data. The computing power required to process it, the myriad of algorithms, machine learning (ML) tools, and necessary languages are all now considered to be part of the universal term - Big Data.

So what are recent advancements?

Literacy

The term "Big Data" has been around since the '90s, and while slow to take hold initially, big business has now fully embraced it.

Increasing numbers of people are conscious and fundamentally have an understanding of the terminology, the skills required to lead, develop, or manage Big Data solutions. All the mega-vendors, Microsoft, Amazon, and Google, have created education programs (available for free) that aim to build solid foundations in software concepts, and an understanding of the services required to build software, and integrate ML.

Young students, mature students, career changers are all looking to upskill in a critical ingredient of the 4th industrial revolution — the rise of the knowledge worker.

Educational institutions are listening and getting in on the act too, cashing in on the massive volume of people wanting to start a career in the Data Science field.

Literacy has reached a point where organizations can unilaterally understand the benefits of having big data programs in place and see measurable returns for their investment.

The proliferation of APIs

Without traditional software programming, Big Data initiatives can become costly projects whereby model predictions and insights languish unused in a slide deck, without reaching their true potential.

Any significant Big Data initiative is useless without software to deliver the suggestion/score/prediction to the right place at the right time, to alter the course.

The proliferation of clearly documented and defined APIs across modern software architecture has resulted in the adoption and delivery of Big Data insights where it matters most - in the customer or business workflows.

With big data comes big responsibility and big questions regarding security and more fundamentally, ethics

Building on this simple and effective connectivity, we have the rise of automation. The bot's making use of Big Data intelligence to service customer requests or to spot trends and make suggestions are commonplace now in retail platforms or videos services.

IoT Devices & the internet

Internet of Things (IoT) devices fuel the good stuff, the data!

IoT devices have provided vastly varied data sets (Surely I should have mentioned the 4 V's at some point!). Temperature, CO2, heat, heart (ECG), light, sound, and any number of sensors are cropping up all over. Every smartphone, every video camera is an IoT device producing vast volumes of data to be processed and interpreted.

All this data is so readily collectible now thanks to the rapid spread of network connectivity and the internet.

Developed economies continue to invest heavily in high-quality networks. In Australia, the NBN (National Broadband Network) has brought faster, more reliable connectivity to new homes and businesses. While less developed countries are leapfrogging the developed world by launching mobile (5G) networks capable of ultra-fast bandwidth speeds. They are avoiding maintenance and reliance on legacy equipment and Telecomms platforms by going to straight to mobile solutions.

So broad is the spread of the internet it's getting hard to find places outside of network range, areas to switch off, and be human again - but that is perhaps another article altogether.

The Impact

With Big Data comes big responsibility and big questions regarding security and more fundamentally, ethics.

From a security perspective, handling data, managing encryption at rest, in transit, while enabling people in your organization the freedom and autonomy to use the data can cause weighty burdens on security teams and Chief Information Security Officers (CISOs). Governments and regulators are often struggling to keep up; the reaction, Privacy laws design to prevent the evil actions and misuse of PII (Personally Identifiable Information) developed without the consideration for the consequence of implementation.

Even after we meet the security requirements, we must understand the biases wired into training data, potentially tainted with a lack of diversity and polluted with discrimination. We need to pose ethical questions; what should I understand about a person if I have harvested location data from their phone? Their heart rate data from their smartwatch? With whom they have met or connected with via social media. What have they purchased recently? Differential pricing or suggestions based on this type of data, or a combination, delivers a fine line between ethical and legal dilemma.

Closing statement

Literacy, IoT, and APIs have all generated phenomenal advances in the adoption of Big Data solutions, mostly for the increased bottom line, and some the betterment of society.

It is no longer the technological constraints that are slowing or preventing Big Data solutions seeing the light of day. It is arguably the security and ethics conversations surrounding big data that is having the most significant impact on businesses today.

My advice, consider the impact Big Data and technology has on the people in your team, organization, or personal life. Have the conversation and set some boundaries, draw your own ethical and moral lines before it is too late.