Agile Methodology: What Roads and Metadata Management Should Have in Common
I often hear when implementing data lineage, or metadata in general, that we only can use it when it is complete – unless there is 100% support, 100% of metadata collected, it is useless. I respectfully disagree. 100% or “complete” is an illusion or holy grail that we can aspire for, but is a moving target (and in a way, it does not exist). Instead, we must ask “Where is the money?” and adjust our strategy based on ROI and true value for our organization. We must use incremental improvements as soon as they are available to help maximize the ROI. In this way, we must be more agile.
What Can We Learn About Metadata Management from Building Roads?
Fairly often, we drive to visit my parents who live in a different part of the country. Over the years, the whole journey has gotten better and better. It is a traditional last mile problem - most of the journey is on highway, but getting from my home to highway and then getting from the highway to my parents house is through small rural townships, and similar smaller roads.
Over the past 10 years, it has been great to see incremental improvements and progress made along the way. First, there was a really painful segment where you drive comfortably at 90km/h but just before I enter the highway, there’s a small town. It takes only 5 minutes to go through it, but it goes directly through the center, meaning road crossings, crosswalks, people, dogs, etc. - but when you get through it, there is a highway entrance. About 5 years ago, they opened a bypass of that town to go from the rural road directly on the highway. Not a huge time saver, but it definitely makes me much more relaxed. I am sure that it made life of all the town citizens better as well since all the through-traffic is gone now and only people who live there or need something specific drive in.
About two years ago, something similar happened closer to my parents - an extension of the highway bypassing two small villages where 90% of the traffic was through-traffic. I admit, it used to be a convenient shortcut to the highway, but was quite an inconvenience to those living in the villages. This year, they just opened a continuation of this road introducing an off-grade junction not to mix local and through-traffic, eliminating another dangerous point on the road for cars but even more dangerous for cyclists.
What is the lesson learned? Well, simply said:
- We are using an agile (incremental) approach to build our roads and more importantly
- We built a segment of a road and we immediately started using it.
Applying the Agile Approach to Metadata Management
The question you may be asking is why don’t we use the same approach to building our metadata environment? I keep hearing from customers and prospects that if only a part of the environment is covered by metadata/lineage, it cannot be consumed by the end users because it is not complete or it does not provide enough value. I understand that it may not yet cover a significant part of the environment for a particular use case in question, but I am sure that it can already provide enough value for other use cases. Remember my “road” story above? The people in the village who finally get a much quieter neighborhood by eliminating through traffic? Or the cyclist at a dangerous at-grade crossing?
Of course, using analogies can be dangerous as they do not 100% match the situation but we can still learn from them. My observations and takeaways are:
- People do not understand the value that a solution (“a road” from my analogy) may have for others until we start talking about it with them. Way too often is the “builder” focused on a particular problem and misses the other uses completely.
- Building something but not letting people use it just because it does not seem to have any immediate value is not smart for the reasons described in #1. Unfortunately for metadata, the completed segment of metadata/lineage is often not as visible and obvious to others as the completed segment of a road from my road analogy is visible/obvious to the public.
- Some topics like metadata are so vast that "doing it all" strategy is like boiling the ocean. In real life, it would be similar to forbidding people from using already existing roads until all roads are built (and not knowing what "all roads" even mean). If this seems too harsh, I have an alternative analogy - forbidding people from using a map of part of your road network until the complete map is available (while new roads are still being built … so it will never be complete).
What do you think? Does it make sense to start using agile in metadata as well? Share your thoughts in the comments.
Get Started with Manta
Manta’s automated data lineage can help you unlock and harness the insights provided by metadata management, creating trust in your data so that you can operationalize it well. In fact, we’ve been recognized by Gartner in the Market Guide for Metadata Management. Get your copy here!
You can also chat with a member of our team to learn more about how Manta can get you on the road to better metadata management. Click here to get started.
P.S. This post was written by a human.