Unleash the power of the dark side of data

When there is light, there is also darkness, and where there is data, there is lots of darkness. It sounds ominous, perhaps even sinister, and as far as dark data goes, it’s all that information that has been collected through regular business processes, such as server log files, machine data, and countless interactions that your employees have daily with various systems and devices, it’s all this that you don’t know what to do with, or in many cases, don’t know that it even exists. These data multiples, it attracts more, and it is something that shouldn’t be ignored. Welcome to the dark side of data.

What really is Dark Data?

Firstly, let’s look at how Gartner defines it. “Dark Data is information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes)”. They then go on to give some examples such as analytics, business relationships, and direct monetising. You can read the full inclusion in their technology glossary here.

Let’s break it down more. Dark Data is estimated to be more than half of an organisation's data, 55% if we are going to be more specific. A Splunk report seems to suggest that this number could even be as high as 75%!

This data can be potentially used to drive new revenue streams, eliminate waste, and reduce costs. Some of it, however, can be discarded as it could contain old data that has become obsolete over time.

How to Find the Dark Side of Data

Don’t look where the light is already shining, as you will end up missing most of the dark data. You need to look where the light doesn’t shine. Stored away in some dark place, away from plain sight.

• Ok, so I’m getting a little carried away with making dark data out to be sinister, you should look in the following places:
• Look at your storage costs year on year, if your storage costs are not increasing in a suspected linear fashion, there is a good chance that you have a good source of dark data
• Create data orchestration pipelines that help pool and aggregate dark data silos
• Use people, hire a human that’s job is specifically to handle data management, like a Chief Data Officer

The Tools you Need

The whole reason for finding dark data is to generate insight from it. To this end you need tools to support various capabilities:

Snorkel - Developed by Stanford University and released as an open-source tool for accelerating dark data extraction.

Dark Vision - A technology demonstrator using IBM Watson service to extract dark data from video.

Microsoft Purview eDiscovery - Microsoft’s auto-classification tools help to manage dark data, and organise it in a central, secured cloud.

Microfocus ControlPoint - Tools for increasing visibility of unstructured data, helping to improve security and compliance.

There are an enormous amount of data hiding in the dark, this unleveraged information can help businesses to form a better, more predictive strategy, over traditional business intelligence solutions. This provides additional data points loaded with future-looking insights that are yet to be uncovered.

The key to unlocking this information is starting with a clear intention, and to contact our team of in-house expert data consultants at Nexus Technology.