{"id":14,"date":"2022-06-09T12:49:23","date_gmt":"2022-06-09T12:49:23","guid":{"rendered":"https:\/\/eck.uidev.site\/?page_id=14"},"modified":"2023-05-31T13:39:37","modified_gmt":"2023-05-31T13:39:37","slug":"challenges","status":"publish","type":"page","link":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/challenges\/","title":{"rendered":"Evolution of Data Pipelines"},"content":{"rendered":"<div class=\"entry-content\">\n\t<div style=\" z-index: 2\" class=\"simplesub wow zoomIn bggrey b100w notoppover footover ceven withtitle singleno\">\n\t\t\t\t<h1>Evolution of Data Pipelines<\/h1>\n\t\t\n\t\t<div class=\"textblockwrap\">\n\n\t\t\t\t\t\t<div class=\"txt\">\n\t\t\t<p><span style=\"font-weight: 400\">The role of data pipelines has evolved over three decades to support increasingly complex, cloud-driven data environments. This evolution comprises three phases:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><b>Phase 1: Load (1990-2010)<\/b><span style=\"font-weight: 400\">. During this phase companies used basic ETL pipelines that loaded periodic batches of database records, perhaps hourly, daily, or weekly, into a central data warehouse for business intelligence projects such as operational reporting and dashboards.<\/span><\/li>\n<li style=\"font-weight: 400\"><b>Phase 2: Consolidate (2010s)<\/b><span style=\"font-weight: 400\">. In the second phase, companies modernized their environments by migrating analytics workloads to new data warehouses in the cloud. They sought to consolidate data from a rising number of sources, including IoT sensors, log files, and SaaS applications as well as traditional databases, into a data warehouse for BI or data lake for data science.<\/span><\/li>\n<li style=\"font-weight: 400\"><b>Phase 3: Synchronize (2020s)<\/b><span>. Despite efforts to consolidate, data environments grow more diverse than ever. Companies maintain some data on premises due to regulatory concerns, data gravity, and the sheer cost of moving it all. While data warehouses and lakes start to merge into lakehouses, companies often have multiple such platforms across two or even three cloud providers. Data pipelines must synchronize data across these distributed elements in real time to support BI and AI\/ML projects, as well as merged workflows in which analytical outputs trigger operational action.<\/span><\/li>\n<\/ul>\n\t\t\t<\/div>\n\t\t\t\t\t\t<div class=\"sub\">\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t<div class=\"quote\">\n\t\t\t\t\t<p>Data pipeline management has evolved over three phases: from loading (1990-2010) to consolidation (2010s) to synchronization (2020s)<\/p>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\n\n\n\n\n\t\t\t\t\t\t\t\t<div class=\"vidimg\">\n\t\t\t\t\t<div class=\"img\">\n\t\t\t\t\t\t\t\t\t\t\t<div class=\"iframelink\">youtu.be\/d8xUDmA0RsI<\/div>\n\t\t\t\t\t\t<iframe class=\"videoem\" src=\"https:\/\/www.youtube.com\/embed\/d8xUDmA0RsI?rel=0\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen=\"\"><\/iframe>\n\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<div class=\"txtlnk\">\n\t\t\t\t\t\t<p><\/p>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\t\n\t<\/div>\n<\/div>\t\n\n<div class=\"entry-content\">\n\t<div style=\" z-index: 2\" class=\"simplesub wow zoomIn bgblue b100w notoppover footover cignore singleyes\">\n\t\t\n\t\t<div class=\"textblockwrap\">\n\n\t\t\t\t\t\t<div class=\"sub\">\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\t<div class=\"cfigure bgwhite\">\n\t\t\t\t\t<h2>Evolution of Data Pipelines<\/h2>\n\t\t\t\t\t\n\n\n\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" src=\"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-content\/uploads\/sites\/18\/2023\/05\/Screenshot-2023-05-31-at-9.35.52-AM.png\" alt=\"\" \/>\n\t\t\t\t\t\n\n\n\n\n\t\t\t\t\t\t\t\t\t\t<a class=\"close\" href=\"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-content\/uploads\/sites\/18\/2023\/05\/Screenshot-2023-05-31-at-9.35.52-AM.png\">Close<\/a>\n\t\t\t\t<\/div>\n\t\t\t\t\n\n\n\n\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\t\n\t<\/div>\n<\/div>\t\n\n<div class=\"entry-content\">\n\t<div style=\" z-index: 1\" class=\"wow fadeInDown resources bgblue topunder colsingle b60w\">\n\t<h2>Additional Resources<\/h2>\n\t<ul>\n\t\t\t\t<li><a class=\"icon sml iwebsite\" target=\"_blank\" href=\"https:\/\/www.eckerson.com\/register?content=the-rise-of-streaming-etl-transforming-data-in-flight-for-real-time-insight-and-action\">The Rise of Streaming ETL<\/a><\/li>\n\t\t\t<\/ul>\n<\/div>\n\t<\/div>","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":1343,"parent":0,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"class_list":["post-14","page","type-page","status-publish","has-post-thumbnail","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/pages\/14","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/comments?post=14"}],"version-history":[{"count":58,"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/pages\/14\/revisions"}],"predecessor-version":[{"id":2036,"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/pages\/14\/revisions\/2036"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/media\/1343"}],"wp:attachment":[{"href":"https:\/\/markets.eckerson.com\/modern-data-pipelines\/wp-json\/wp\/v2\/media?parent=14"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}