Pandas is at a the core of virtually all data science done in Python, that is virtually all data science. Since it's beginning, Pandas has been based upon numpy. But changes are afoot to update those internals and you can now optionally use PyArrow. PyArrow comes with a ton of benefits including it's columnar format which makes answering analytical questions faster, support for a range of high performance file formats, inter-machine data streaming, faster file IO and more. Reuven Lerner is here to give us the low-down on the PyArrow revolution. 
 
Episode sponsors 
 
<a href='https://talkpython.fm/nordlayer'>NordLayer</a>
<a href='https://talkpython.fm/auth0'>Auth0</a>
<a href='https://talkpython.fm/training'>Talk Python Courses</a> 
 
<h2 class="links-heading">Links from the show</h2>
<div>Reuven: <a href="https://github.com/reuven?featured_on=talkpython" target="_blank" >github.com/reuven</a> 
Apache Arrow: <a href="https://github.com/apache/arrow?featured_on=talkpython" target="_blank" >github.com</a> 
Parquet: <a href="https://parquet.apache.org/?featured_on=talkpython" target="_blank" >parquet.apache.org</a> 
Feather format: <a href="https://arrow.apache.org/docs/python/feather.html?featured_on=talkpython" target="_blank" >arrow.apache.org</a> 
Python Workout Book: <a href="https://www.manning.com/books/python-workout?featured_on=talkpython" target="_blank" >manning.com</a> 
Pandas Workout Book: <a href="https://www.manning.com/books/pandas-workout?featured_on=talkpython" target="_blank" >manning.com</a> 
Pandas: <a href="https://pandas.pydata.org/?featured_on=talkpython" target="_blank" >pandas.pydata.org</a> 
PyArrow CSV docs: <a href="https://arrow.apache.org/docs/python/csv.html?featured_on=talkpython" target="_blank" >arrow.apache.org</a> 
Future string inference in Pandas: <a href="https://pandas.pydata.org/docs?featured_on=talkpython" target="_blank" >pandas.pydata.org</a> 
Pandas NA/nullable dtypes: <a href="https://pandas.pydata.org/docs/user_guide/integer_na.html?featured_on=talkpython" target="_blank" >pandas.pydata.org</a> 
Pandas `.iloc` indexing: <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.iloc.html?featured_on=talkpython" target="_blank" >pandas.pydata.org</a> 
DuckDB: <a href="https://duckdb.org?featured_on=talkpython" target="_blank" >duckdb.org</a> 
Pandas user guide: <a href="https://pandas.pydata.org/docs/user_guide/?featured_on=talkpython" target="_blank" >pandas.pydata.org</a> 
Pandas GitHub issues: <a href="https://github.com/pandas-dev/pandas/issues?featured_on=talkpython" target="_blank" >github.com</a> 
Watch this episode on YouTube: <a href="https://www.youtube.com/watch?v=IHd-bgeHrv0" target="_blank" >youtube.com</a> 
Episode transcripts: <a href="https://talkpython.fm/episodes/transcript/503/the-pyarrow-revolution" target="_blank" >talkpython.fm</a> 
 
--- Stay in touch with us --- 
Subscribe to Talk Python on YouTube: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a> 
Talk Python on Bluesky: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm at bsky.app</a> 
Talk Python on Mastodon: <a href="https://fosstodon.org/web/@talkpython" target="_blank" >talkpython</a> 
Michael on Bluesky: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes at bsky.app</a> 
Michael on Mastodon: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" >mkennedy</a> </div>

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

#503: The PyArrow Revolution

Talk Python To Me

AI Chapters

AI Insights

Reviews