Market Trends

Five market trends shape the market for data pipeline tools: adopters, priorities, users, use cases, and tools.

• Adopters. Data pipeline adopters divide into two camps, each with distinct characteristics and requirements. Cloud-native companies born since 2010 embrace the latest pipeline tools with little if any need to worry about technical debt. Large and midsized enterprises, on the other hand, use pipelines to synchronize data across a mix of old and new elements in their hybrid/multi-cloud environments.
• Priorities. Data teams seek to keep migrating analytical and operational workloads to the cloud; democratize business access to data; and build competitive advantage with AI/ML projects.
• Users. Data pipeline users and managers include data engineers that ingest and transform data; analytics engineers that prepare and document data for analytics; analysts that serve themselves with basic discovery and preparation; and data scientists that define features and train AI/ML models.
• Use cases. The use cases for data pipelines break out into three loose categories: scaling out BI projects in terms of users, tools, datasets, etc.; testing and bringing AI/ML projects into production; and merging analytical outputs with operational applications to drive smarter action.
• Tools. Eckerson Group research shows that most practitioners expect to increase the number of pipeline tools they use in 2023. It also shows that more than 40% of practitioners already use ChatGPT to assist their data engineering work.