Data observability:what is it?

The data management landscape has dramatically changed over the last decade with the evolution and massive adoption of BigData, ML/AI, and an ever-increasing number of data sources and volumes. Ensuring high data quality and completeness is critical for driving valuable business decisions for enterprises. Yet statistics show that 87% of machine learning projects don’t make it into production, and data engineers spend 80% of their time cleaning the data.

Around 5-8 years ago, enterprises started seeing similar patterns around cloud infrastructure. Organizations were investing heavily in cloud architecture with low returns on their investments. There was a lack of monitoring and minimal predictability regarding anomalies and service failures. This need gave birth to the evolution of Cloud Infrastructure Observability.

Running SaaS operations at an ever-growing scale and the need for efficiency and reliability put the focus on observability products, and today, there are several major players in this space like Splunk, DataDog, NewRelic, Dynatrace. I have noticed various interpretations of what Observability is, but recently it has converged into three key concept pillars. In the world of Cloud Infrastructure Observability, these pillars are metrics, traces, and logs and they try to answer the following questions:

1. Do I have a problem, how bad is it?
2. Where is my problem and what is the impact?
3. What went wrong?

Initially, metrics monitoring tools addressed the first pillar, tracing tools addressed the second, and logs tools addressed the third. However, the observability area is very dynamic and experiencing explosive growth, so we see many new tools emerging that address the needs of each pillar.

However, data problems are hidden from these tools, and even when all the metrics, traces, and logs look normal for a data pipeline, it still can produce garbage data. This is a significant problem for businesses, which leads not only to constant escalation mode and burnout of data engineers who are troubleshooting discrepancies in the reports but also hurts business big time.

Hence, a similar concept of observability has now emerged in data management for data quality use cases. More and more companies realize they need to focus on addressing data issues, or what is sometimes referred to as “data downtime.” It is not surprising to see the growing interest in Data Observability.

Just like Cloud Observability, Data Observability Suites are trying to get answers to the same three questions. However, there is no established consensus on the naming. Let me offer my take on the Data Observability pillars:

• Monitoring of the Data Quality detects a variety of problems or anomalies in the data. There are numerous things that could go wrong, but we can break it down into three high-level categories: missing/incomplete data, incorrect data, and stale data
• Data Lineage helps understand the impact and source of a data anomaly. You need to know how various sources of data relate to each other and how they contribute to downstream systems and reports
• Data Troubleshooting helps to find the root cause of an issue. In the App Observability, it is the Logs. In the case of Data, Observability logs are of minimal help since the pipeline or an application is still operating as expected, but they are processing the wrong data and ultimately causing wrong decision-making

Given the complexity of the domain, I anticipate a wide range of tools being introduced for each pillar, far greater than what we saw for Cloud Observability.

So, what is Data Observability? In short, it is a new discipline that is trying to fill the gaps where traditional data management systems like data quality, data profiling, and lineage are failing to help data engineers achieve operational excellence and deliver business results.

#dataquality #dataobservability #dataops

‍

Passionate about data quality? Get expert insights and guides delivered straight to your inbox – click here to subscribe to our newsletter now.