Use Cases

Data teams, and data engineers in particular, use data observability to optimize data quality and data pipelines across a lifecycle of three stages: validate and detect, assess and predict, and resolve and prevent.

• Validate and detect. During this stage, data teams identify patterns in all the signals, as well as breaks in the patterns that might include anomalous trends, outlier values, job failures, or system errors. These pattern breaks indicate issues that need attention.
• Assess and predict. During this stage, data teams triage issues such as null values or pipeline failures, and measure their impact on the business in terms of affected users, revenue, etc. They trace workflows and correlate events across various elements—data sources, clusters, targets, etc.—to isolate the root cause.
• Resolve and prevent. Once data teams find the culprit—perhaps a server failure or misfiring Kaka producer—they work to get things back under control. They might debug pipeline code, optimize hardware utilization, or cleanse data to remove null value, all in an effort to prevent or at least minimize downstream damage to the business.