The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.In this episode, we are joined by <a href="https://www.linkedin.com/in/willgraham2/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">William Graham</a>, Senior Data Engineer at <a href="https://www.linkedin.com/company/pattern-hq/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Pattern</a>, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.Key Takeaways:00:00 Introduction.02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.04:27 How Airflow became the central scheduler for batch jobs.08:57 Credential management challenges that led to decoupling scheduling and orchestration.12:21 Heimdall simplifies multi-application access through a unified interface.13:15 Standardized operators in Airflow using Heimdall integration.17:13 Open-source contributions and early adoption of Heimdall within Pattern.21:01 Community support for Airflow and satisfaction with scheduling flexibility.Resources Mentioned:<a href="https://www.linkedin.com/in/willgraham2/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">William Graham</a>https://www.linkedin.com/in/willgraham2/<a href="https://www.linkedin.com/company/pattern-hq/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Pattern</a> | LinkedInhttps://www.linkedin.com/company/pattern-hq/<a href="https://pattern.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Pattern</a> | Websitehttps://pattern.com<a href="https://airflow.apache.org" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org<a href="https://github.com/Rev4N1/Heimdall" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Heimdall on GitHub</a>https://github.com/Rev4N1/Heimdall<a href="https://netflix.github.io/genie/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Netflix Genie</a>https://netflix.github.io/genie/Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

Building a Unified Data Platform at Pattern with William Graham

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews