Building scalable, reproducible workflows for scientific computing often requires bridging the gap between research flexibility and enterprise reliability.In this episode, <a href="https://www.linkedin.com/in/anja-mackenzie/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Anja MacKenzie</a>, Expert for Cheminformatics at <a href="https://www.linkedin.com/company/covestro/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Covestro</a>, explains how her team uses Airflow and Kubernetes to create a shared, self-service platform for computational chemistry.Key Takeaways:00:00 Introduction.06:19 Custom scripts made sharing and reuse difficult.09:29 Workflows are manually triggered with user traceability.10:38 Customization supports varied compute requirements.12:48 Persistent volumes allow tasks to share large amounts of data.14:25 Custom operators separate logic from infrastructure.16:43 Modified triggers connect dependent workflows.18:36 UI plugins enable file uploads and secure access.Resources Mentioned:<a href="https://www.linkedin.com/in/anja-mackenzie/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Anja MacKenzie</a>https://www.linkedin.com/in/anja-mackenzie/<a href="https://www.linkedin.com/company/covestro/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Covestro</a> | LinkedInhttps://www.linkedin.com/company/covestro/<a href="https://www.covestro.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Covestro</a> | Websitehttps://www.covestro.com<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://kubernetes.io/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Kubernetes</a>https://kubernetes.io/<a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Airflow KubernetesPodOperator</a>https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html<a href="https://www.astronomer.io/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Astronomer</a>https://www.astronomer.io/<a href="https://www.udemy.com/user/lockgfg/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_GammaCatchall_NonP_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Gamma&utm_content=deal4584&utm_term=_._ag_169801645584_._ad_700876640602_._kw__._de_c_._dm__._pl__._ti_dsa-1456167871416_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21341313808&gbraid=0AAAAADROdO1_-I2TMcVyU8F3i1jRXJ24K&gclid=Cj0KCQjwvJHIBhCgARIsAEQnWlC1uYHIRm3y9Q8rPNSuVPNivsxogqfczpKHwhmNho2uKZYC-y0taNQaApU2EALw_wcB" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Airflow Academy by Marc Lamberti</a>https://www.udemy.com/user/lockgfg/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_GammaCatchall_NonP_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Gamma&utm_content=deal4584&utm_term=_._ag_169801645584_._ad_700876640602_._kw__._de_c_._dm__._pl__._ti_dsa-1456167871416_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21341313808&gbraid=0AAAAADROdO1_-I2TMcVyU8F3i1jRXJ24K&gclid=Cj0KCQjwvJHIBhCgARIsAEQnWlC1uYHIRm3y9Q8rPNSuVPNivsxogqfczpKHwhmNho2uKZYC-y0taNQaApU2EALw_wcB<a href="https://airflow.apache.org/docs/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Airflow Documentation</a>https://airflow.apache.org/docs/<a href="https://airflow.apache.org/docs/apache-airflow/1.10.9/plugins.html" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Airflow Plugins</a>https://airflow.apache.org/docs/apache-airflow/1.10.9/plugins.html﻿Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

How Covestro Turns Airflow Into a Simulation Toolbox with Anja Mackenzie

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews