Database Documentation

Data documentation isn’t sexy. But it matters — big time.

Data documentation is paramount for any data team. Without accurate and up-to-date documentation, how will your team understand data to make accurate and reliable decisions? Imagine you are a 5-star Chef at a Michelin Star restaurant, but all of your ingredients look the same, and you have zero labels. Do you think you'd be able to cook a world-class meal? This is what it is like working without documentation. Even the most talented data scientists and analysts can't execute to their full potential without understanding the full context behind the data.

Too often, data is created and left without documentation leaving others to use it at their own will. This free-reign can result in guesswork and a complete misinterpretation of variables and key data assets. Even worse, what happens when someone on your team leaves and the documentation lives in their head?  It doesn’t matter if you have the best, most sophisticated talent in the world if the data context used is based on inaccurate or outdated documentation - how can anyone thrive as a data worker? Data documentation prevents loss of knowledge, incorrect data use and gives analysts the confidence to self-serve data.

Data documentation can seem like a tedious, daunting task enforced as a corporate governance effort. In reality, data documentation enables faster and better decisions by giving your team access to the right context behind your data. To truly reap the benefits of having a data-driven decision-making process in place, you have to ensure that accurate and up-to-date documentation is in use.

Quality documentation starts with the naming convention. Field names should represent what they mean and have a shared style across an organization. Never name your variable “X” or “test”; instead, fill it with a level of detail that is understandable to anyone in your area. In fact, we recall one organization whose analyst used the acronym “NA” for “North America.” For years, the entire company thought that referred to a null value in their sales data.

When creating a new table, include descriptions in your data warehouse. This pays dividends when data is transferred or moved.  Always have an additional document that provides an outline for the schema. Data lineage is useful for an ERD diagram to show how the data is created and how it connects to other tables. Needless to say, any additional acronyms or abbreviations should be called out in this additional documentation. ‍

Helpful tips to keep accurate and up to date documentation:

• Assign clear ownership: Data should have a technical and business owner.
• Centralize the documentation: Documentation should live in one place that is accessible to anyone working with the data. It’s like the instruction manual for building a piece of IKEA furniture. The process will be a whole lot easier with findable documentation.
• Establish a process for documentation: Things at most companies, especially in the data environment, are dynamic. As data changes, the documentation should evolve with it. Make sure to have a defined process for updating documentation as things change.

While it doesn’t have a glamourous name like “machine learning”, documenting data is crucial to deliver the most accurate insights. Datalogz offers a great way to simplify the data documentation process, among other great features!