انتقل إلى وضع عدم الاتصال باستخدام تطبيق Player FM !
Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich
Manage episode 485580771 series 2948506
Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.
In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code.
Key Takeaways:
(03:23) Code duplication creates long-term problems.
(08:16) Frameworks bring order to complex pipelines.
(09:41) Shared functions cut down repetitive code.
(17:18) Auto-generated docs stay accurate by design.
(22:40) On-demand DAGs support real-time workflows.
(25:08) Task-level sensors improve run efficiency.
(27:40) Combine local runs with automated tests.
(30:09) Clean code helps teams scale faster.
Resources Mentioned:
https://www.linkedin.com/in/gilreich/
Wix | LinkedIn
https://www.linkedin.com/company/wix-com/
Wix | Website
https://www.wix.com/
https://airflowsummit.org/slides/2024/92-refactoring-dags.pdf
https://airflow.apache.org/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 حلقات
Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 485580771 series 2948506
Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.
In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code.
Key Takeaways:
(03:23) Code duplication creates long-term problems.
(08:16) Frameworks bring order to complex pipelines.
(09:41) Shared functions cut down repetitive code.
(17:18) Auto-generated docs stay accurate by design.
(22:40) On-demand DAGs support real-time workflows.
(25:08) Task-level sensors improve run efficiency.
(27:40) Combine local runs with automated tests.
(30:09) Clean code helps teams scale faster.
Resources Mentioned:
https://www.linkedin.com/in/gilreich/
Wix | LinkedIn
https://www.linkedin.com/company/wix-com/
Wix | Website
https://www.wix.com/
https://airflowsummit.org/slides/2024/92-refactoring-dags.pdf
https://airflow.apache.org/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 حلقات
Minden epizód
×
1 The Future of Airflow Telemetry at Metyis with Bolke de Bruin 21:55

1 Transforming the Airflow UI for Cloudera’s Users with Shubham Raj 22:28

1 Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing 19:34

1 Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19

1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21

1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42
مرحبًا بك في مشغل أف ام!
يقوم برنامج مشغل أف أم بمسح الويب للحصول على بودكاست عالية الجودة لتستمتع بها الآن. إنه أفضل تطبيق بودكاست ويعمل على أجهزة اندرويد والأيفون والويب. قم بالتسجيل لمزامنة الاشتراكات عبر الأجهزة.