I often hear when implementing data lineage, or metadata in general, that we only can use it when it is complete – unless there is 100% support, 100% of metadata collected, it is useless. I respectfully disagree. 100% or “complete” is an illusion or holy grail that we can aspire for, but is a moving target (and in a way, it does not exist). Instead, we must ask “Where is the money?” and adjust our strategy based on ROI and true value for our organization. We must use incremental improvements as soon as they are available to help maximize the ROI. In this way, we must be more agile.
Fairly often, we drive to visit my parents who live in a different part of the country. Over the years, the whole journey has gotten better and better. It is a traditional last mile problem - most of the journey is on highway, but getting from my home to highway and then getting from the highway to my parents house is through small rural townships, and similar smaller roads.
Over the past 10 years, it has been great to see incremental improvements and progress made along the way. First, there was a really painful segment where you drive comfortably at 90km/h but just before I enter the highway, there’s a small town. It takes only 5 minutes to go through it, but it goes directly through the center, meaning road crossings, crosswalks, people, dogs, etc. - but when you get through it, there is a highway entrance. About 5 years ago, they opened a bypass of that town to go from the rural road directly on the highway. Not a huge time saver, but it definitely makes me much more relaxed. I am sure that it made life of all the town citizens better as well since all the through-traffic is gone now and only people who live there or need something specific drive in.
About two years ago, something similar happened closer to my parents - an extension of the highway bypassing two small villages where 90% of the traffic was through-traffic. I admit, it used to be a convenient shortcut to the highway, but was quite an inconvenience to those living in the villages. This year, they just opened a continuation of this road introducing an off-grade junction not to mix local and through-traffic, eliminating another dangerous point on the road for cars but even more dangerous for cyclists.
What is the lesson learned? Well, simply said:
The question you may be asking is why don’t we use the same approach to building our metadata environment? I keep hearing from customers and prospects that if only a part of the environment is covered by metadata/lineage, it cannot be consumed by the end users because it is not complete or it does not provide enough value. I understand that it may not yet cover a significant part of the environment for a particular use case in question, but I am sure that it can already provide enough value for other use cases. Remember my “road” story above? The people in the village who finally get a much quieter neighborhood by eliminating through traffic? Or the cyclist at a dangerous at-grade crossing?
Of course, using analogies can be dangerous as they do not 100% match the situation but we can still learn from them. My observations and takeaways are:
What do you think? Does it make sense to start using agile in metadata as well? Share your thoughts in the comments.
Manta’s automated data lineage can help you unlock and harness the insights provided by metadata management, creating trust in your data so that you can operationalize it well. In fact, we’ve been recognized by Gartner in the Market Guide for Metadata Management. Get your copy here!
You can also chat with a member of our team to learn more about how Manta can get you on the road to better metadata management. Click here to get started.
P.S. This post was written by a human.