It’s not news that effective metadata management is essential for organizations that want to be nimble and agile in today’s fast-moving business climate. However, what may not be as well known is the role that automated active metadata plays in effective metadata management.
It’s one thing to have metadata; it’s another thing to use it to its full potential and derive true value from it. Existing data management solutions contain an abundance of unstructured data that most organizations find difficult to identify, let alone analyze.
According to Gartner:
“Through 2024, active metadata capabilities will expand to include monitoring, evaluating, recommending design changes, and orchestrating processes in third-party data management solutions.”
“Through 2024, organizations that adopt aggressive metadata analysis across their complete data management environment will decrease time-to-delivery of new data assets to users by as much as 70%.”
Both active and passive metadata add value to your data pipeline. But active metadata provides insights that passive metadata alone cannot.
Passive metadata contains basic information about data such as data profiles (business qualification, quality score, etc.) or data operational characteristics (who accesses the data, how often, popular data sets, etc.). It provides a generic overview of the data landscape, but it is static, can’t be acted upon, and won’t be of much help with providing complete visibility into complex data pipelines, unlike properly activated metadata.
Active metadata can tell you the story behind the static profile of your data. It shows how and where the data flows in a data pipeline, including all changes, data transformations, and calculations. Knowing this, you can find any blind spots in the data landscape and fix them before they become a problem for your organization.
The impact of metadata management by itself is very small. But when it’s combined with automated data lineage, that’s when you see a huge, positive effect on the business. The only way to properly activate metadata is a complete, panoramic overview of all direct and indirect dependencies among data elements in the environment. After all, you can’t activate what you don’t know you have.
That’s what makes proper, true data lineage the key component of active metadata management. It’s time we moved on from over-simplistic data lineage tools to advanced automated solutions that provide much deeper information about the data landscape than just tracking data sources and data journeys from one table to another. What should you be looking for in a data lineage solution to make it a vital part of a successful active metadata management system?
The volume of data your organization processes requires constant changes to the technology stack. The potpourri of mature, established technologies, cloud solutions, and open-source tools becomes too much for basic solutions. Such tools deliver fragmented lineage information extracted only from the technologies they support. Failing to connect to other parts of the environment leaves a vast segment of the metadata undiscovered, not to mention unactivated.
To ensure the most accurate and up-to-date information about metadata, deploy a solution that scans various databases, data modeling, data integration, and ETL tools; reporting and analysis software; and programming languages. And one that even lets you build your own metadata ingestion process for technologies that there’s no formal scanner for.
Regular and automated discovery is essential to keep up with the pace of the data that is moving across the organization’s systems. With dependencies hiding in places you might never think of, the manual approach (either to metadata discovery or scan scheduling) fails to provide a complete picture of the data landscape and takes you back to the drawing board.
Combining automated metadata discovery with the ability to run automatic scans of the entire environment provides you with the most comprehensive overview of your data pipelines and lays a solid foundation for metadata activation.
Metadata activation serves as an aid for users who want to discover what they can do with their organization’s lineage diagram without going through each and every node manually. To facilitate this, it’s important to make sure that every possible data user (business and technical) can actually understand and filter out lineage information that is important to them. To do so, equip your employees with the solution that allows them to switch between different lineage views (based on their needs and technical understanding), highlight important information in the context of the data pipeline directly in the lineage diagram (for example, data quality issues), and organize lineage metadata into their own folders and hierarchies.
Activating metadata with data lineage will move your metadata management maturity to the next level and allow you to benefit from the organization’s data in new ways. Turning harvested metadata into proactive alerts and recommendations will help you:
Once you have laid the groundwork, your journey to fully understanding the metadata begins. Activating metadata allows you to constantly validate it and stop relying on outdated, passive metadata descriptors that are misleading in dynamic environments. This is the gateway to making metadata available for operational and analytical purposes, closing the knowledge gap between how much metadata your organization stores and how little has surfaced, and eventually planning the orchestration services and designing the data fabric for the enterprise.
Learn more about active metadata management, levels of metadata management technology maturity, the current state of the market, and more from the 2021 Gartner® Market Guide for Active Metadata Management. Download your free copy of the report. Want to know more about how to activate metadata with data lineage? Reach out to us at manta@getmanta.com.