3 ways to prevent dark data from casting a shadow on the planet

Vinod Bijlani July 07, 2023 09:26

The aftershocks from the global pandemic and multiple geopolitical tensions over the last three years are still reverberating through much of the global business community. The experience has taught leaders two clear lessons. First, that it is critical to build resilience through digital transformation and second, that every enterprise must take proactive measures to mitigate against future risks, whether it be pandemic preparedness, climate change adaptation, or supply chain interruptions.

While technology holds the promise to help organizations with both resilience and risk mitigation, our digital-first world is not without its challenges. The proliferation of devices has led to unprecedented amounts of data being produced every second across sensors, data centres, and high-tech infrastructure, resulting in an enormous pile-up of ‘dark data’.

Dark data is data that has been collected and stored, but underutilized, unused, or ignored. This dark data can cost organizations dearly—financially, because of it costs money to store this data and because of the missed opportunity to extract business value from the ignored data, and environmentally, because storing vast amounts of data in servers is energy intensive. Experts estimate that more than 80% of data is unstructured dark data. A Harvard Business Review webinar revealed that only 50% of structured data is used in making decisions, and that less than 1% of unstructured data is ever analysed or used at all. This unused data is estimated to be pumping millions of tonnes of carbon emissions into the atmosphere each year, and is on track to rise annually as the world creates even more data. Faced with this reality and considering the urgent global imperative to contain greenhouse gas emissions, it is crucial that organizations adopt a robust strategy to combat enterprise dark data.

The Many Costs of Disregarding Dark Data

Unstructured dark data has the potential to cost organizations not only financially and environmentally, but also creates additional administrative and security burdens for the IT teams because, as data gets more fragmented and siloed, it becomes progressively challenging to organize and manage. This creates vulnerabilities in the infrastructure because dark data can contain sensitive information that could be at risk of security and privacy breaches or regulatory non-compliance. Additionally, unorganized dark data could thwart organizational efforts to accurately gauge market trends, customer needs, or business risks. So, addressing dark data can help enterprises be more competitive, operate more efficiently, and defend against potential regulatory and security risks.

How to Address the Dark Data Conundrum

Businesses that capture data with purpose make better operational and investment decisions, yielding superior social, financial, and sustainable results. Here are three ways in which organizations can take a strategic and sustainable data-first approach to efficient dark data management:

1. Foster a Data Culture

To instil a data culture across their businesses, organizations must build capabilities around data and treat data as a valuable company asset. Establishing a data culture forms the basis of a robust data culture and helps ensure purposeful data collection and helps avoid the creation of dark data dumps.
As with any organizational culture shift, a culture of data requires buy in from the top. Leadership must not only consider data to be an organizational asset rather than a siloed departmental asset, it must also promote data accountability by inculcating a data-first value system where everyone is on board. An inclusive mindset can give the much-needed boost to decentralized, democratized, frontline decisioning based on data insights. Industry experts have pointed out that organizations with a strong data culture demonstrate a healthy and balanced decision culture as well. This helps frame an organization’s behaviours, beliefs, and practices around data, facilitating quick and informed decisions rooted in data-driven insights. So, organizations must align all gathered data with organizational and business objectives to reduce or eliminate large reservoirs of dark data.

2. Plan for Effective Data Governance

Data democratization, with open and easy access to the data gateway, is a pivotal step towards a strong data culture, dissolving the need for data gatekeeping. With a free data gateway, everyone in the organization will have access to data, enabling greater data discovery. Data discovery, a key constituent of data governance, can help combat dark data by acknowledging the presence of data, understanding its quality, and identifying its issues. Regular data quality checks can reveal the source and purpose of the collected data. Organizations must insist on traceability—knowing where and how the data originated—to gainfully leverage data as needed.

For effective governance, organizations must enforce stringent guidelines on data labelling and the right use of metadata. They must design a sustainable data cataloguing strategy to classify, structure, and label data to manage it at scale. Data tagging and labelling can go a long way in preventing dark data accumulation. Tier-based data categorization and value-based data storage can help enterprises leverage actionable insights from the stored data.
Organizations must study data movement and evaluate usage patterns to ensure compliance, assess business intelligence relevance, and prevent unnecessary hoarding of dark data. This helps them avert the risk of extracting insights from inconsistent and conflicting data. They must carry out regular data audits to ensure security and regulatory compliance. They should invest in advanced computing tools and artificial intelligence (AI) powered automated data discovery solutions to analyse data relevancy rapidly and at scale.

3. Decarbonize Data Processes

To design an effective data management strategy, organizations must focus on decarbonizing their data processes by connecting enterprise-wide data over compatible systems to harness actionable insights. They must also maximize data utilization. To effectively reuse existing data, organizations should consciously cultivate data as a renewable resource, avoiding dark data and reducing the carbon footprint of their data.
Businesses can work toward eliminating waste from data storage using lean principles, bringing about data stability. They must purge irrelevant, duplicated, and outdated data to reduce data clutter. AI tools with machine learning (ML) algorithms trained to analyse and classify data based on predefined rules and policies can help with cautious purging to avoid accidental data loss.
Beyond data optimization, storage, and pruning clutter, organizations can leverage low-cost AI-powered sensors and the cloud to streamlines their data processes.

Start Building a Unified Data Management Strategy Now

Organization must build sustainable and unified data analysis strategy to optimize operational performance, business success flow, and overall enterprise resilience. Unless companies shift to enterprise-wide good data hygiene, carbon emissions from data-led digital transformations will continue to be overlooked. They need to carefully consider ‘data efficiency’ as a key component of any IT efficiency assessment. To be sure, this is no simple task and requires thoughtful data regulations in addition to strong leadership, clear vision, and a willingness to invest in technology for good.

-- Contact us at [email protected]

Vinod Bijlani

AI & Data Practice Leader, APAC at HPE