PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible.In this episode, <a href="https://www.linkedin.com/in/kunaljubce/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Kunal Bhattacharya</a>, Senior Manager of Data Platform Engineering at <a href="https://www.linkedin.com/company/pepsico/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">PepsiCo</a>, shares how his team manages Airflow at scale while ensuring security, performance and cost efficiency.Key Takeaways:00:00 Introduction.02:31 Enabling developer delight by extending platform capabilities.03:56 Role of Snowflake, dbt and Airflow in PepsiCo’s data stack.06:10 Local developer environments built using official Airflow Helm charts.07:13 Pre-staging and PR environments as testing playgrounds.08:08 Automating labeling and resource allocation via DAG factories.12:16 Cost optimization through pod labeling and Datadog insights.14:01 Isolating dbt engines to improve performance across teams.16:12 Wishlist for Airflow 3: Improved role-based grants and database modeling.Resources Mentioned:<a href="https://www.linkedin.com/in/kunaljubce/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Kunal Bhattacharya</a>https://www.linkedin.com/in/kunaljubce/<a href="https://www.linkedin.com/company/pepsico/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">PepsiCo</a> | LinkedInhttps://www.linkedin.com/company/pepsico/<a href="https://www.pepsico.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">PepsiCo</a> | Websitehttps://www.pepsico.com<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://www.snowflake.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Snowflake</a>https://www.snowflake.com<a href="https://www.getdbt.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">dbt</a>https://www.getdbt.com<a href="https://kubernetes.io" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Kubernetes</a>https://kubernetes.io<a href="https://greatexpectations.io" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Great Expectations</a>https://greatexpectations.io<a href="https://www.montecarlodata.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Monte Carlo</a>https://www.montecarlodata.comThanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews