The integration of data orchestration and machine learning is critical to operational efficiency in healthcare tech. Vivian Health leverages Airflow to power both its ETL pipelines and ML workflows while maintaining strict compliance standards.<a href="https://www.linkedin.com/in/maxwell-calehuff/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Max Calehuff</a>, Lead Data Engineer at <a href="https://www.linkedin.com/company/vivianhealth/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Vivian Health</a>, joins us to discuss how his team uses Airflow for ML ops, regulatory compliance and large-scale data orchestration. He also shares insights into upgrading to Airflow 3 and the importance of balancing flexibility with security in a healthcare environment.Key Takeaways:00:00 Introduction.04:21 The role of Airflow in managing ETL pipelines and ML retraining.06:23 Using AWS SageMaker for ML training and deployment.07:47 Why Airflow’s versatility makes it ideal for MLOps.10:50 The importance of documentation and best practices for engineering teams.13:44 Automating anonymization of user data for compliance.15:30 The benefits of remote execution in Airflow 3 for regulated industries.18:16 Quality-of-life improvements and desired features in future Airflow versions.Resources Mentioned:<a href="https://www.linkedin.com/in/maxwell-calehuff/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Max Calehuff</a>https://www.linkedin.com/in/maxwell-calehuff/<a href="https://www.linkedin.com/company/vivianhealth/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Vivian Health</a> | LinkedInhttps://www.linkedin.com/company/vivianhealth/<a href="https://www.vivian.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Vivian Health</a> | Websitehttps://www.vivian.com<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://www.astronomer.io/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Astronomer</a>https://www.astronomer.io/<a href="https://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAE" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">AWS SageMaker</a>https://www.google.com/aclk?sa=L&ai=DChsSEwj3-fbz1tiQAxWXlKYDHXUBBVoYACICCAEQABoCdGI&ae=2&aspm=1&co=1&ase=2&gclid=Cj0KCQiA5abIBhCaARIsAM3-zFWbfj2olUvX4dqoiYNaE3q2fMf_ZifRjmbKNQCVX7D6ZMClaUXUkFkaAuwmEALw_wcB&cid=CAASQuRoMccxWhBvMq-1Uez3XOZti1ul7mTDotKvSMoDHv0q2xCsyS2FzMptO5dJf3tmfkLRu22TtD8ChTmdjvs6YetTjQ&cce=2&category=acrcp_v1_35&sig=AOD64_2xE2xolEEVbpDb56qXQluxTzs-Aw&q&nis=4&adurl&ved=2ahUKEwj7le3z1tiQAxWXcvUHHfZePbAQ0Qx6BAgUEAE<a href="https://www.getdbt.com/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">dbtLabs</a>https://www.getdbt.com/<a href="https://github.com/astronomer/astronomer-cosmos" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Cosmos</a>https://github.com/astronomer/astronomer-cosmos<a href="https://www.split.io/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Split</a>https://www.split.io/<a href="https://www.snowflake.com/en/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Snowflake</a>https://www.snowflake.com/en/﻿Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

The Role of Airflow in Building Smarter ML Pipelines at Vivian Health with Max Calehuff

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews