Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable.In this episode, <a href="https://www.linkedin.com/in/scroc/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Sébastien Crocquevieille</a>, Data Engineer at <a href="https://www.linkedin.com/company/numberly/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Numberly</a>, unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features.Key Takeaways:00:00 Introduction.02:13 Overview of the company’s operations and global presence.04:00 The tech stack and structure of the data engineering team.04:24 Running nearly 2,000 DAGs in production using Airflow.05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot.07:05 Details on the Kubernetes-based Airflow setup using Helm charts.09:31 Transition from GitSync to NFS for DAG syncing due to performance issues.14:11 Making every team member Airflow-literate through local installation.17:56 Using custom libraries and plugins to extend Airflow functionality.Resources Mentioned:<a href="https://www.linkedin.com/in/scroc/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Sébastien Crocquevieille</a>https://www.linkedin.com/in/scroc/<a href="https://www.linkedin.com/company/numberly/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Numberly</a> | LinkedInhttps://www.linkedin.com/company/numberly/<a href="https://numberly.com/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Numberly</a> | Websitehttps://numberly.com/<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://grafana.com/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Grafana</a>https://grafana.com/<a href="https://kafka.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Kafka</a>https://kafka.apache.org/<a href="https://airflow.apache.org/docs/helm-chart/stable/index.html" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Helm Chart for Apache Airflow</a>https://airflow.apache.org/docs/helm-chart/stable/index.html<a href="https://kubernetes.io/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Kubernetes</a>https://kubernetes.io/<a href="https://about.gitlab.com/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">GitLab</a>https://about.gitlab.com/<a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">KubernetesPodOperator – Airflow</a>https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html<a href="https://astronomer.io/beyond/dataflowcast" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Beyond Analytics Conference</a>https://astronomer.io/beyond/dataflowcast﻿Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews