Building an End-to-End Data Observability System at Netflix with Joseph Machado

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Player FM - Internet Radio Done Right

Added four years ago

المحتوى المقدم من The Data Flowcast. يتم تحميل جميع محتويات البودكاست بما في ذلك الحلقات والرسومات وأوصاف البودكاست وتقديمها مباشرة بواسطة The Data Flowcast أو شريك منصة البودكاست الخاص بهم. إذا كنت تعتقد أن شخصًا ما يستخدم عملك المحمي بحقوق الطبع والنشر دون إذنك، فيمكنك اتباع العملية الموضحة هنا https://ar.player.fm/legal.

The Wednesday Season 2 Official Woecast

1
Hide and Woe Seek: Georgie Farmer, Joy Sunday, Tom Turnbull & Angela Robinson 36:10

منذ 14 يومًا36:10

التشغيل لاحقا

قوائم

إعجاب

احب

36:10

WARNING! This podcast contains spoilers for episodes 1 – 5 of Wednesday Season 2. Your favorite gorgon, Georgie Farmer, joins Caitlin Reilly to talk all things Ajax: his relationship with Enid, his friendship with Bianca, and his true feelings for Bruno. And then Joy Sunday talks about how the adults in Bianca’s life are really NOT making it easy for her this season. And who she’s turning to for support. Plus, visual effects supervisor Tom Turnbull breaks down how they shoot those Hyde scenes. And the crows! Finally, we hear from director Angela Robinson on filming the action packed scenes with Slurp and on turning Gwendoline Christie into a ghost! 00:00 - intro 1:02 - Georgie Farmer 1:31 - What’s changed between Ajax and Enid? 2:32 - How does Ajax feel about Bruno? 4:14 - How does Ajax feel about Bruno and Enid together? 4:53 - Ajax and Bianca’s bond 6:32 - Ajax is the unsung hero of Season 2 7:03 - Georgie’s Gorgon makeup 8:07 - Working with Noah Taylor 9:08 - Georgie’s favourite memories from the set of Wednesday 11:11 - Joy Sunday 11:33 - Bianca’s story arc in season 2 13:08 - Joy’s take on Ajax and Bianca 14:19 - Working with Georgie 14:51 - Joy’s relationship with her mother 16:30 - Bianca’s siren powers 19:00 - Working with Catherine Zeta Jones 21:17 - Tom Turnbull 21:34 - Tom’s favourite piece of VFX work from Season 2 22:42 - Working on the Hyde 25:10 - Christopher Lloyd as Professor Orloff 27:36 - Angela Robinson 27:50 - Becoming part of the Wednesday crew 28:25 - Collaborating with Al and Miles 29:49 - Pugsley and Gomez 32:14 - Pugsley and Slurp share a moment 33:30 - Weems returns! 34:59 - Outro…

منذ عام واحد 38:54

MP3•منزل الحلقة

Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.

Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.

Key Takeaways:

(03:14) Supporting data privacy and engineering efficiency within data systems.

(10:41) Validating outputs with reconciliation checks to catch transformation issues.

(16:06) Applying standardized patterns for auditing, validating and publishing data.

(19:28) Capturing historical check results to monitor system health and improvements.

(21:29) Treating data quality and availability as separate monitoring concerns.

(26:26) Using containerization strategies to streamline pipeline executions.

(29:47) Leveraging orchestration platforms for better visibility and retry capability.

(31:59) Managing business pressure without sacrificing data quality practices.

(35:46) Starting simple with quality checks and evolving toward more complex frameworks.

Resources Mentioned:

Joseph Machado

https://www.linkedin.com/in/josephmachado1991/

Netflix | LinkedIn

https://www.linkedin.com/company/netflix/

Netflix | Website

https://www.netflix.com/browse

Start Data Engineering

https://www.startdataengineering.com/

Apache Airflow

https://airflow.apache.org/

dbt Labs

https://www.getdbt.com/

Great Expectations

https://greatexpectations.io/

https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/

https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

70 حلقات

#Tech #Airflow

Building an End-to-End Data Observability System at Netflix with Joseph Machado

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

published منذ عام واحد