Big Data Analytics in Cyber Security

Vahid Behbood, Chief Data Scientist, BT Security

Vahid Behbood, Chief Data Scientist, BT Security

Introduction

The current cyber security ecosystem is vastly different from how it was a couple of decades ago. Nowadays companies enjoy more sophisticated, diverse and versatile digital environments which comprise of technologies such as cloud computation, software as a service, IoT devices, automated robots, etc. Although these new technologies increase efficiency and provide significant value for companies, the accompanying increase in the complexity of the digital environment provides a fertile landscape for adversarial activities. Organisations operating in these environments are therefore faced with the challenging task of maintaining an efficient and holistic security practice, while realising the benefits of these new technologies.

Modern advancements in computational science could provide the solution. Organisations can now efficiently digest and analyse huge volumes of data from a multitude of sources such as desktop computers, servers, mobile phones, applications and IoT devices. Effective use of this data could provide the key to tackling the sophisticated and ever-changing cyber threat landscape. Unlike conventional static signature(rule)-based security solutions, big data analytics enables security teams to build data-driven and dynamic solutions that learn to recognise malicious activity patterns. These solutions can then increase the efficacy of the protection, detection, investigation and response stages of the security monitoring process.

Protection

Protection is the first security layer of organisation where big data analytics plays acritical role. The protection layer includes security devices and solutions such as firewall, web proxy, IAM, CASB,EDR, IDPS, and DLP that aim to protect organisations against cyber threat. These devices conventionally rely on static signatures and configuration provided by experts and threat intelligence in confronting known threats and malware. However, the current threat landscape is so advanced and dynamic that these security devices can struggle to protect an organisation’s vital assets efficiently.

One effective solution is to learn and recognise complex threat patterns, while handling the high velocities and varieties of network traffic that pass through a modern organisation’s digital environment.. Big data plays an essential role in both identifying malicious behaviours and providing a framework to handle network data. An analytics solution built upon massive data from various sources can be embedded into security solutions on the protection layer as a complementary capability to signature-based methods.

Detection

Since cyber threats are immense in their number and diversity, it would be naïve to think that the protection security layer guarantees the organisation immunity against adversarial events. Hence, it is realistic to assume that the organisation has already been or will be compromised, and that some form of malware may exist in its digital environment. Therefore, the detection layer becomes crucial. Solutions like security information and event management (SIEM), user and entity behaviour analytics (UEBA) and network traffic analytics (NTA) focus on detecting malicious events and activities that are present and active in a digital environment.

Considering the high number of devices and entities that normally operate in a modern organisation and, more importantly, the massive amount of data that they produce in the form of log events, it is apparent that big data is hugely helpful in this scenario. The data volume is in the order of terabytes per day, originating from many sources generating millions of events per second. Finding the malicious events in this digital environment is analogous to finding a needle in haystack, which makes the big data platform and framework invaluable for generating actionable insights into malicious behaviour patterns.

Investigation

Another layer of security for an organisation is the investigation layer which is sometimes referred to as threat hunting. In the investigation stage, security experts dig into data lakes of past events produced by digital entities to explore and find historical malicious events and incidents. In typical practice, the analyst performs retrospective investigation spanning from 6 months’ worth of data up to a couple of years. Considering that timeline, a cyber analyst could deal with unimaginable amount of data, and it would be near impossible to carry out an investigation without acknowledgement of the big data ecosystem and architecture.

The investigative analytics on this massive amount of data span from basic descriptive analytics to advanced analytics like Artificial Intelligence. Descriptive analytics can include simply understanding general behaviour of traffic – such as the amount of web traffic during the normal business hours and the geographical destination of web connections. Advanced analytics such as Deep Learning leverage massive amounts of data to mine the historical data and explore interesting patterns of adversaries that have been unnoticed and undetected by cyber experts(e.g. due to the novelty and complexity of attack or just simply too much data to be monitored). These advanced analytics are a great example of human experts being augmented by big data, rather than being replaced.

Response

The last security layer, but definitely not the least, is the response layer. It focuses on triage, analysis and response to detected malicious events and incidents. The response layer is the most critical, intensive and resource-consuming stage o the security monitoring process. It requires numerous cyber experts with different levels of security expertise to examine every single alert supplied by multiple security tools within an organisation. For example, Security Operation Centres (SOCs) monitoring environments for large, frequently-targeted organisations such as banks could receive in the order of thousands of alerts per day. These events need to be analysed by cyber analysts to find malicious incidents and respond as quickly as possible in order to contain and/or remediate damage done by the threat.

The Response layer is the most resource intensive part of security monitoring process and it also represents the key performance indicator for cyber security departments. Therefore, many companies are attracted by the idea of introducing smart automation, which should be able to learn and adapt the response process when receiving different types of alerts. This allows it to act promptly and with minimum human intervention when action needs to be taken.

To be smart and adaptive, it requires access to the big data environment that contains the historical data of case incidents including alerts and responses. Utilising this data, smart automation forms robust learning of the relation between alerts and the relevant response actions. Smart automation unlocks the potential of security experts by reducing the time spent on menial tasks and increasing the time spent on advanced tasks. Moreover, it considerably reduces the time to detection and response and improves the productivity of security departments.

Cyber Security Platform

Big data technology has successfully been imbedded into analytics products for various business sectors such as finance, retail, health and telecom. Its now also plating a crucial role in today’s cyber security industry, due to the volume, velocity and variety of the data in this domain. In fact, it’s now almost impossible to unravel the actual view of security information without using big data technologies and frameworks in the core of security architecture. Big data analytics empower companies to offer holistic, accurate and efficient security monitoring mechanism and continually maintain the standard security posture.BT is pioneering in this space with its Cyber Security Platform; combining big data engineering and advanced analytics. Leveraging huge volumes of data and innovative analytics capabilities is actively helping to protect BT as well as our partners and customers, from today's advanced threat landscape.