{"id":2160,"date":"2023-05-31T16:37:11","date_gmt":"2023-05-31T16:37:11","guid":{"rendered":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/?page_id=2160"},"modified":"2023-05-31T16:37:11","modified_gmt":"2023-05-31T16:37:11","slug":"overview","status":"publish","type":"page","link":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/overview\/","title":{"rendered":"Challenges & Product Evaluation Criteria"},"content":{"rendered":"\n

<\/h2>\n\n\n
\n\t
\n\t\t\t\t

Challenges & Product Evaluation Criteria<\/h1>\n\t\t\n\t\t
\n\n\t\t\t\t\t\t
\n\t\t\t

The challenges facing modern data pipelines are five-fold. As described earlier, rising data volumes, driven by data transformation and proliferating data sources, put significant strains on traditional pipelines. An accelerating business environment, meanwhile, drives the need for real-time data delivery and access. In addition, data engineers, analytics engineers, data scientists, and analysts\u2014especially the more business-oriented ones\u2014can struggle to build the right pipeline scripts in SQL, Python, or other popular languages. All the while, data environments grow more diverse, adding elements to accommodate new business demands and projects. A final challenge is compliance with stringent regulations, for example to ensure privacy and avoid bias.\u00a0<\/span><\/p>\n

With these challenges in mind, Eckerson Group recommends evaluating data pipeline products according to five criteria.<\/span><\/p>\n

    \n
  • Breadth of functionality. <\/b>Your<\/span> data pipeline tool should support\u2014either natively or via easy integration with third-party tools\u2014multiple data integration patterns, such as ETL, ELT, ELTL, and reverse ETL.\u00a0<\/span><\/li>\n
  • Performance and scale<\/b>. Your tool should put minimal processing burden on your current architecture, avoid source agents, use log-based change data capture, and meet your business\u2019 most rigorous service level agreements (SLAs).<\/span><\/li>\n
  • Ease of use<\/b>. Your tool should require minimal training, automate basic tasks, provide an intuitive graphical interface, easily change pipeline elements, and improve the productivity of your data team.\u00a0<\/span><\/li>\n
  • Open architecture<\/b>. Your tool should provide an open architecture that works with various sources, targets, processors, and programming languages. Open data formats and APIs can help ensure your tools interoperate with one another and give you the flexibility you need to migrate or integrate data as needed.<\/span><\/li>\n
  • Governance<\/b>. Look for a tool that centralizes pipeline metadata, provides granular role-based access controls, masks sensitive data, tracks lineage, and audits user actions to assist compliance efforts.<\/span><\/li>\n<\/ul>\n\t\t\t<\/div>\n\t\t\t\t\t\t
    \n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t

    Eckerson Group recommends evaluating data pipeline products according to their breadth of functionality, performance and scale, ease of use, open architecture, and governance<\/p>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\n\n\n\n\n\t\t\t\t\t\t\t\t

    \n\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t
    youtu.be\/SNNPZzuSulU<\/div>\n\t\t\t\t\t\t