We discuss common use cases and challenges for copying data between S3 buckets and S3-compatible object storage services. We share our experience building an open source Node.js CLI tool called S3-Migrate to efficiently migrate data with separate source and destination credentials. We cover performance considerations like streaming, chunk sizes, concurrency and parallelism.AWS Bites is brought to you in association with fourTheorem. If you need a friendly partner to support you and work with you to de-risk any AWS migration or development project, check them out at <a href="https://fourtheorem.com" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠fourtheorem.com⁠⁠⁠</a>In this episode, we mentioned the following resources:<ul><li>s3-migrate CLI tool: <a href="https://github.com/lmammino/s3-migrate" target="_blank" rel="noopener noreferer">https://github.com/lmammino/s3-migrate</a></li><li>DigitalOcean Spaces Object Storage: <a href="https://docs.digitalocean.com/products/spaces/" target="_blank" rel="noopener noreferer">https://docs.digitalocean.com/products/spaces/</a></li><li>Cloudflare R2: <a href="https://www.cloudflare.com/en-gb/developer-platform/products/r2/" target="_blank" rel="noopener noreferer">https://www.cloudflare.com/en-gb/developer-platform/products/r2/</a></li><li>Backblaze B2: <a href="https://www.backblaze.com/cloud-storage" target="_blank" rel="noopener noreferer">https://www.backblaze.com/cloud-storage</a></li><li>Wasabi Cloud Storage: <a href="https://wasabi.com/" target="_blank" rel="noopener noreferer">https://wasabi.com/</a></li><li>Linode / Akamai Object Storage: <a href="https://www.linode.com/products/object-storage/" target="_blank" rel="noopener noreferer">https://www.linode.com/products/object-storage/</a></li><li>MinIO (Self-hosted S3-compatible storage): <a href="https://min.io/" target="_blank" rel="noopener noreferer">https://min.io/</a></li><li>Basecamp / Hey’s move away from S3: <a href="https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563" target="_blank" rel="noopener noreferer">https://world.hey.com/dhh/it-s-five-grand-a-day-to-miss-our-s3-exit-b8293563</a></li><li>AWS re:Post - How to move objects between S3 buckets: <a href="https://repost.aws/knowledge-center/move-objects-s3-bucket" target="_blank" rel="noopener noreferer">https://repost.aws/knowledge-center/move-objects-s3-bucket</a></li><li>AWS Labs - Utility for S3 Migration: <a href="https://github.com/awslabs/utility-for-s3-migration" target="_blank" rel="noopener noreferer">https://github.com/awslabs/utility-for-s3-migration</a></li><li>s3s3mirror (Java-based tool): <a href="https://github.com/cobbzilla/s3s3mirror" target="_blank" rel="noopener noreferer">https://github.com/cobbzilla/s3s3mirror</a></li><li>rclone S3 Support: <a href="https://rclone.org/s3/" target="_blank" rel="noopener noreferer">https://rclone.org/s3/</a></li><li>knox-copy (Ruby-based, deprecated): <a href="https://github.com/goodeggs/knox-copy" target="_blank" rel="noopener noreferer">https://github.com/goodeggs/knox-copy</a></li><li>Flexify.io (paid cloud migration service): <a href="https://flexify.io/" target="_blank" rel="noopener noreferer">https://flexify.io/</a></li></ul>Do you have any AWS questions you would like us to address?Leave a comment here or connect with us on X/Twitter, BlueSky or LinkedIn:- <a href="https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbWlYNEdBeDBCSU56N3U1M0o1NDF0dS1sbkdRd3xBQ3Jtc0tra2RBajhnX0lBQ0dRVjZWMFZtNUdiX0JKVnZiVEdHZFBIcmtiSWpIU0Nvelc1UnliZDNZZ1hrS19tUUdtOHVwUTFzeTVrSkRuN2dwZDBKUXJlcmc5enY5YkJoUndoNEhIYVlsLUJKMmZRdUh2bVR4bw&q=https%3A%2F%2Ftwitter.com%2Feoins&v=lyMHJW-3l9k" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/eoins⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠</a> | <a href="https://bsky.app/profile/eoin.sh" target="_blank" rel="ugc noopener noreferrer">⁠⁠https://bsky.app/profile/eoin.sh⁠⁠</a> | <a href="https://www.linkedin.com/in/eoins/" target="_blank" rel="ugc noopener noreferrer">⁠⁠https://www.linkedin.com/in/eoins/⁠⁠</a>- <a href="https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqa3d0X2t5cUtreE02S2xuV1ZMTWNNMGdkamdqQXxBQ3Jtc0ttbDhQN2JicGhXOGxVX2Zqa1FLVVFKNERkY2VzUThBU3JONmdaM1dwWkVxRG5NQjdYeHU5TUxCWmxKSmZVclZLUXBSb2JGdkZUTnQzSEdpTUd1RDhGT3Y1LTR0TU5YUVZQbUNYOFJxRi1ZZTkxcGhkdw&q=https%3A%2F%2Ftwitter.com%2Floige&v=lyMHJW-3l9k" target="_blank" rel="ugc noopener noreferrer">⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://twitter.com/loige⁠⁠⁠⁠⁠⁠</a> | <a href="https://bsky.app/profile/loige.co" target="_blank" rel="ugc noopener noreferrer">⁠⁠https://bsky.app/profile/loige.co⁠⁠</a> | <a href="https://www.linkedin.com/in/lucianomammino/" target="_blank" rel="ugc noopener noreferrer">⁠⁠https://www.linkedin.com/in/lucianomammino/</a>

Podurama

Home

Trending

Personal

Your Podcasts

New Episodes

My Files

Favourite Episodes

Tags (Folders)

Recently Played

Bookmarks

Followed Playlists

142. Escape from S3

AWS Bites

AI Chapters

AI Insights

Reviews