Archiving and data handling is one of those things that happens on the back end for most organizations; no one really thinks about it too hard. But that’s also one of the places your performance can start flagging and your costs start rising, all entirely out of view. Let’s look at archive optimization – and specifically, at dark data.
Why do we care about archiving in the first place?
Archiving meets several key needs for any organization. In particular, it hits the following points:
- Compliance: Meets legal and industry retention requirements
- Cost efficiency: Moves less‑used data to cheaper storage
- Performance: Improves system speed by offloading old data
- Risk mitigation: Protects against accidental deletion or corruption
It is the middle two – cost efficiency and performance – that we’re going to be taking a look at this week.
What is dark data?
Per IBM, Dark data is the information that organizations accumulate but often never use for analytics or decision-making. This is a large and growing category – especially now that AI is entering the picture, it is a commonly held wisdom that as much data as can be retained should be in case it becomes valuable at some future point.
Therefore, dark data is a growing pool that starts absorbing all information the organization has lost track of, or perhaps never really realized they had in the first place. Forbes’ research found that 94% of enterprises struggle with unstructured data, having some amount of data outside regular storage structures.
The issue is less one of the data being retained – rather, the problem begins when the data falls outside standard formats or categories and is left behind. Even for the AI use case outlined above, it can cause issues as AI begins drawing on learning materials that it was not intended to access or use to inform its answers and behavior.
Furthermore, dark data accumulates at a rapid rate at a time when data storage costs are and increasing concern for enterprises worldwide. Organizations whose data volume begins to spiral out of control would benefit from knowing what information they can safely delete, or at least offload into as cheap a storage solution as is possible.
How can archives help?
Archived data by definition ceases to be dark data because it suddenly gains structure. Archives can help reduce issues by immediately solving several of the more common issues:
- Keeping data too long: Compliance requirements demand that some data is maintained in your systems for a given period. A well-structured archive will also let you track when it is safe to no longer retain this data
- ROT data: Redundant, obsolete or trivial data can be minimized – specifically, the redundant aspect can be mitigated by modern archives offering an intelligent deduplication layer
- Data governance: Archive structures prevent data from easily becoming lost and unusable
- Data awareness: The structure of an archive, ideally combined with search tools, means that you can keep awareness of what information is being stored, where and why
Your Data in Your hands – With TECH-ARROW
by Matúš Koronthály
Image generated by Canva
