The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments.In this episode, <a href="https://www.linkedin.com/in/jens-scheffler/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Jens Scheffler</a>, Test Execution Cluster Technical Architect at <a href="https://www.linkedin.com/company/bosch/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Bosch</a>, shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release.Key Takeaways:(02:39) The role of remote execution in supporting large-scale testing needs.(04:44) How community support contributed to the Edge Executor’s development.(08:41) Navigating network and infrastructure limitations within secure environments.(13:25) Transitioning from database-heavy processes to an API-driven model.(14:16) How the new task SDK in Airflow 3 improves distributed task execution.(16:54) What is required to set up and configure the Edge Executor.(19:36) Managing multiple queues to optimize tasks across different environments.(23:30) Examples of extreme distance use cases for edge execution.Resources Mentioned:<a href="https://www.linkedin.com/in/jens-scheffler/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Jens Scheffler</a>https://www.linkedin.com/in/jens-scheffler/<a href="https://www.linkedin.com/company/bosch/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Bosch</a> | LinkedInhttps://www.linkedin.com/company/bosch/<a href="https://www.bosch.com/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Bosch</a> | Websitehttps://www.bosch.com/<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Edge Executor (Edge3 Provider Package)</a>https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html<a href="https://www.astronomer.io/docs/astro/astro-executor/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Astronomer’s Astro Executor</a>https://www.astronomer.io/docs/astro/astro-executor/<a href="http://astronomer.io/beyond/dataflowcast" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Beyond Analytics Conference</a>http://astronomer.io/beyond/dataflowcast﻿Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews