Logo Cinquin Andy Signature

How to synchronize a MinIO bucket with a Cloudflare R2 bucket using Rclone

Développeur Freelance - Logo

The sites and the associated resources.

MinIO
Cloudflare R2
Rclone

How to synchronize a MinIO bucket with a Cloudflare R2 bucket using Rclone

Posted on May 27, 2024 -  by Andy Cinquin

MinIOCloudflare R2RcloneSynchronizationObject storageAmazon S3 APIBackupsData migration

Introduction

In this tutorial, we'll learn how to synchronize a MinIO bucket with a Cloudflare R2 bucket using the Rclone command-line tool. MinIO is an open source object storage server compatible with the Amazon S3 API, while Cloudflare R2 is an object storage service offered by Cloudflare, also compatible with the S3 API. By synchronizing buckets between these two services, you can create backups, migrate data or maintain synchronized copies of your files across multiple clouds.

Installing Rclone

Before you can synchronize your MinIO and Cloudflare R2 buckets, you need to install Rclone on your local machine. Here's how to do it:

On Linux :

  1. Open a terminal.
  2. Run the following command to automatically download and install the latest version of Rclone:
curl https://rclone.org/install.sh | sudo bash

On macOS (using Homebrew):

  1. Open a terminal.
  2. If you don't have Homebrew installed, install it first by running :
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  1. Install Rclone with Homebrew:
brew install rclone

On Windows:

  1. Visit the Rclone download page: https: //rclone.org/downloads/.
  2. Download the installation file for Windows (for example, rclone-current-windows-amd64.msi).
  3. Double-click on the downloaded file and follow the installation wizard instructions.
After installation, you can check that Rclone is correctly installed by running the following command in your terminal:
rclone --version
This will display the version of Rclone installed on your system.

Prerequisites

  • A MinIO server with a bucket created. See the MinIO documentation to find out how to configure a server and create buckets: https: //docs.min.io/.
  • A Cloudflare account with R2 enabled and a bucket created. Follow Cloudflare's guide to enable R2 and create buckets: https: //developers.cloudflare.com/r2/.
  • Rclone installed on your local machine.

Step 1: Configure remote MinIO

  1. Run rclone config to open the interactive Rclone configuration menu.
  2. Choose n to create a new remote.
  3. Give your remote a name, e.g. "minio-andy".
  4. Choose "s3" as storage type.
  5. Leave the provider field blank.
  6. Enter the URL of your MinIO server, your access_key_id and your secret_access_key. This information can be obtained from your MinIO administration interface. Make sure you keep these credentials secure.
  7. Leave the other options as default, or adjust them as required.
  8. Choose y to confirm remote creation.

Step 2: Configure the Cloudflare R2 remote

  1. Run rclone config again.
  2. Choose n to create a new remote.
  3. Give your remote a name, e.g. "r2-andy".
  4. Choose "s3" as storage type.
  5. Choose "Cloudflare" as provider.
  6. Enter your R2 access_key_id, your R2 secret_access_key and the URL of your R2 bucket. This information can be obtained from your Cloudflare R2 account. Make sure you keep these credentials secure.
  7. Leave the other options as default, or adjust them as required.
  8. Choose y to confirm remote creation.

Step 3: Synchronize buckets

  1. To synchronize your MinIO bucket with your R2 bucket, run the following command:
rclone sync "minio-andy:name-of-bucket-source" "r2-andy:name-of-bucket-destination"
Replace name-du-bucket-source with the name of your source MinIO bucket and name-du-bucket-destination with the name of your destination R2 bucket.
  1. Rclone will start synchronizing files from your MinIO bucket to your R2 bucket. It will display progress and a summary at the end.

Step 4: Verify synchronization

  1. To check that synchronization has been successful, you can list the contents of your R2 bucket:
rclone ls "r2-andy:bucket-name-destination"
Replace name-of-bucket-destination with the name of your destination R2 bucket.
  1. You should see the files that have been synchronized from your MinIO bucket.

Explanation

Rclone is a powerful tool for managing files on different cloud storage services. In this tutorial, we used Rclone to synchronize a MinIO bucket with a Cloudflare R2 bucket.
By configuring Rclone remotes for MinIO and R2, we can use the rclone sync command to easily synchronize files between the two buckets. Rclone compares source and destination files and transfers only those files that have been modified, making it effective for keeping multiple buckets in sync.
There are many advantages to using Rclone, including
  • Ease of use: Rclone offers a simple, intuitive command-line interface for managing your files on different cloud storage services.
  • Efficiency: Rclone optimizes file transfers by transferring only modified files, saving time and bandwidth.
  • Compatibility: Rclone supports a wide range of cloud storage services, enabling you to synchronize your files between different providers.

Conclusion

In this tutorial, we've learned how to synchronize MinIO and Cloudflare R2 buckets using Rclone. By following the remotes configuration steps and using the rclone sync command, you can easily keep your files synchronized between these two object storage services.


Thank you for your visit, feel free to contact me for any information, quote or collaboration proposal. I will be happy to answer you as soon as possible.
Did you like this article? Feel free to share it!

DEVELOP YOUR PROJECTS TOGETHER

An idea, a project? I'm here to answer your questions and help you.
I'd be delighted to discuss your project with you!
Contact me