انتقل إلى وضع عدم الاتصال باستخدام تطبيق Player FM !
المدونة الصوتية تستحق الاستماع
برعاية


1 72: If You Want to Grow—Stop Fixing the Wrong Problem 16:32
Building an End-to-End Data Observability System at Netflix with Joseph Machado
Manage episode 482858005 series 2948506
Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.
Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.
Key Takeaways:
.
(03:14) Supporting data privacy and engineering efficiency within data systems.
(10:41) Validating outputs with reconciliation checks to catch transformation issues.
(16:06) Applying standardized patterns for auditing, validating and publishing data.
(19:28) Capturing historical check results to monitor system health and improvements.
(21:29) Treating data quality and availability as separate monitoring concerns.
(26:26) Using containerization strategies to streamline pipeline executions.
(29:47) Leveraging orchestration platforms for better visibility and retry capability.
(31:59) Managing business pressure without sacrificing data quality practices.
(35:46) Starting simple with quality checks and evolving toward more complex frameworks.
Resources Mentioned:
https://www.linkedin.com/in/josephmachado1991/
Netflix | LinkedIn
https://www.linkedin.com/company/netflix/
Netflix | Website
https://www.netflix.com/browse
https://www.startdataengineering.com/
https://airflow.apache.org/
https://www.getdbt.com/
https://greatexpectations.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 حلقات
Building an End-to-End Data Observability System at Netflix with Joseph Machado
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 482858005 series 2948506
Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.
Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.
Key Takeaways:
.
(03:14) Supporting data privacy and engineering efficiency within data systems.
(10:41) Validating outputs with reconciliation checks to catch transformation issues.
(16:06) Applying standardized patterns for auditing, validating and publishing data.
(19:28) Capturing historical check results to monitor system health and improvements.
(21:29) Treating data quality and availability as separate monitoring concerns.
(26:26) Using containerization strategies to streamline pipeline executions.
(29:47) Leveraging orchestration platforms for better visibility and retry capability.
(31:59) Managing business pressure without sacrificing data quality practices.
(35:46) Starting simple with quality checks and evolving toward more complex frameworks.
Resources Mentioned:
https://www.linkedin.com/in/josephmachado1991/
Netflix | LinkedIn
https://www.linkedin.com/company/netflix/
Netflix | Website
https://www.netflix.com/browse
https://www.startdataengineering.com/
https://airflow.apache.org/
https://www.getdbt.com/
https://greatexpectations.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 حلقات
كل الحلقات
×
1 The Future of Airflow Telemetry at Metyis with Bolke de Bruin 21:55

1 Transforming the Airflow UI for Cloudera’s Users with Shubham Raj 22:28

1 Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing 19:34

1 Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19

1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21

1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26

1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27

1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46

1 Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy 27:40

1 Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot 17:54

1 Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling 33:45

1 Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer 43:39

1 Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar 26:00

1 The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant 21:19

1 Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall 38:54

1 Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado 24:35

1 Exploring the Power of Airflow 3 at Astronomer with Amogh Desai 30:24

1 Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta 24:11

1 Maximizing Business Impact Through Data at GlossGenius with Katie Bauer 25:49

1 Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade 27:47

1 How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari 28:44

1 Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino 28:29

1 Inside Airflow 3: Redefining Data Engineering with Vikram Koka 30:08

1 Building a Data-Driven HR Platform at 15Five with Guy Dassa 20:25

1 The Intersection of AI and Data Management at Dosu with Devin Stein 20:18

1 AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma 26:11

1 From Task Failures to Operational Excellence at GumGum with Brendan Frick 24:06

1 From Sensors to Datasets: Enhancing Airflow at Astronomer with Maggie Stark and Marion Azoulai 22:25

1 Mastering Data Orchestration with Airflow at M Science with Ben Tallman 24:36


1 Enhancing Business Metrics With Airflow at Artlist with Hannan Kravitz 23:51

1 Cutting-Edge Data Engineering at Teya with Alexandre Magno Lima Martins 23:46

1 Airflow Strategies for Business Efficiency at Campbell with Larry Komenda 26:10

1 How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard 23:58

1 How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing 23:51
مرحبًا بك في مشغل أف ام!
يقوم برنامج مشغل أف أم بمسح الويب للحصول على بودكاست عالية الجودة لتستمتع بها الآن. إنه أفضل تطبيق بودكاست ويعمل على أجهزة اندرويد والأيفون والويب. قم بالتسجيل لمزامنة الاشتراكات عبر الأجهزة.