Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed.In this episode, <a href="https://www.linkedin.com/in/atiannicelli/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Alex Iannicelli</a>, Staff Software Engineer at <a href="https://www.linkedin.com/company/overture-maps-foundation/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Overture Maps Foundation</a>, and <a href="https://www.linkedin.com/in/daniel-smith-analyst/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Daniel Smith</a>, Senior Solutions Architect at <a href="https://www.linkedin.com/company/wherobots" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Wherobots</a>, join us to discuss leveraging Apache Airflow and Apache Sedona to process massive geospatial datasets, build reproducible pipelines and orchestrate complex workflows across platforms.Key Takeaways:00:00 Introduction.03:22 How merging multiple data sources supports comprehensive datasets.04:20 The value of flexible configurations for running pipelines on different platforms.06:35 Why orchestration tools are essential for handling continuous data streams.09:45 The importance of observability for monitoring progress and troubleshooting issues.11:30 Strategies for processing large, complex datasets efficiently.13:27 Expanding orchestration beyond core pipelines to automate frequent tasks.17:02 Advantages of using open-source operators to simplify integration and deployment.20:32 Desired improvements in orchestration tools for usability and workflow management.Resources Mentioned:<a href="https://www.linkedin.com/in/atiannicelli/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Alex Iannicelli</a>https://www.linkedin.com/in/atiannicelli/<a href="https://www.linkedin.com/company/overture-maps-foundation/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Overture Maps Foundation</a> | LinkedInhttps://www.linkedin.com/company/overture-maps-foundation/<a href="https://overturemaps.org" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Overture Maps Foundation</a> | Websitehttps://overturemaps.org<a href="https://www.linkedin.com/in/daniel-smith-analyst/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Daniel Smith</a>https://www.linkedin.com/in/daniel-smith-analyst/<a href="https://www.linkedin.com/company/wherobots" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Wherobots</a> | LinkedInhttps://www.linkedin.com/company/wherobots<a href="https://www.wherobots.com" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Wherobots</a> | Websitehttps://www.wherobots.com<a href="https://airflow.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Airflow</a>https://airflow.apache.org/<a href="https://sedona.apache.org/" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Apache Sedona</a>https://sedona.apache.org/<a href="https://github.com/wherobots/airflow-providers-wherobots" target="_blank" style="background-color: transparent; color: rgb(17, 85, 204);">Github repo</a>https://github.com/wherobots/airflow-providers-wherobots﻿Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow #MachineLearning

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

AI Chapters

AI Insights

Reviews