The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines.In this episode, we’re joined by <a href="https://www.linkedin.com/in/ian-buss/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Ian Buss</a>, Principal Software Engineer at Astronomer, and <a href="https://www.linkedin.com/in/piotr-chomiak-b1955624/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Piotr Chomiak</a>, Principal Product Manager at <a href="https://www.linkedin.com/company/astronomer/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Astronomer</a>, who share insights into the Astro Executor and remote execution.Key Takeaways:00:00 Introduction.04:13 How product leadership drives scalability for enterprise needs.08:23 Architectural changes that improve reliability and remove bottlenecks.10:15 Metrics that enhance visibility into system performance.12:54 The role of remote execution in addressing security requirements.15:56 Differences between open-source solutions and managed offerings.19:04 Broad industry adoption and applicability of remote execution.20:39 Future advancements in language support and multi-tenancy.Resources Mentioned:<a href="https://www.linkedin.com/in/ian-buss/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Ian Buss</a>https://www.linkedin.com/in/ian-buss/<a href="https://www.linkedin.com/in/piotr-chomiak-b1955624/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Piotr Chomiak</a>https://www.linkedin.com/in/piotr-chomiak-b1955624/<a href="https://www.astronomer.io" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Astronomer</a> | Websitehttps://www.astronomer.io<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://airflow.apache.org/community/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Airflow Slack Community</a>https://airflow.apache.org/community/<a href="https://astronomer.io/beyond/dataflowcast" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Beyond Analytics conference</a>https://astronomer.io/beyond/dataflowcast﻿Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews