Using Object Storage
Object storage is one of the most commonly used storage paradigms in cloud computing. It helps organizations and individuals manage and distribute large volumes of data efficiently. In Virtual Kubernetes Services (VKS) workflows, object storage is frequently used for staging, persistence, and transfer. We recommend using rclone to manage your object storage resources.
What is Rclone
Rclone is a powerful command-line program for managing cloud storage. It can synchronize, transfer, cache, and manage files across many providers (for example: Google Drive, Dropbox, OneDrive and S3-compatible services). This guide shows how to use rclone to manage your object storage on Alaya NeW.
Prerequisite
Object Storage has been provisioned for your account.
Procedure
Download and install
- Linux
- Windows
- macOS
1. Manual install Rclone**
#Download and install Rclone
curl https://rclone.org/install.sh | sudo bash
#Verify installation
rclone --version
2. Install Using a Package Manager
2.1 Using apt（Debian/Ubuntu）
# Update package list
sudo apt update
# Install Rclone
sudo apt install rclone
# Verify installation
rclone --version
2.2 Using yum（CentOS/RHEL）
# Install the EPEL repository (if not already installed)
sudo yum install epel-release
#Install Rclone
sudo yum install rclone
#Verify installation
rclone --version
2.3 Using dnf（Fedora）
# Install Rclone
sudo dnf install rclone
# Verify installation
rclone --version
-
Download the Installer：
- Visit the Rclone Official Website to download the latest version of the installation package.
- Once the download is complete, double-click the installation package to begin the installation.
-
Verify Installation：
- Open a command prompt and enter the following command：
rclone --version
Install Using Homebrew：
# Install Rclone
brew install rclone
# Verify Installation
rclone --version
Configuration
You can configure rclone either via the command-line interface or by creating a configuration file (using a config file is recommended).
- Configuration File
- config command
Example for Linux: create the config file at /root/.config/rclone/rclone.conf，with the following content： Note： access_key_id and secret_access_key are provided in your object storage provisioning SMS notification. The endpoint can be found on the Object Storage details page under Resource Center-Storage Management in the web console.
[s3_store] #Custom name for this remote
type = s3
provider = Ceph
access_key_id = ******* #Replace with your actual access_key_id
secret_access_key = ******* #Replace with your actual secret_access_key
endpoint = https://s3.hd-01.alayanew.com:8082
acl = public-read-write
no_check_bucket = true
When editing your configuration file, replace the following variables:
| Variable | Description | Source | Example |
|---|---|---|---|
| s3_store | Custom remote name | User-defined | s3_store |
| access_key_id | access_key_id | Provisioning SMS | Asjnf****233s |
| secret_access_key | secret_access_key | Provisioning SMS | sder*******nfksu9j5lw |
| endpoint | Service endpoint URL | Object Storage details page | http://s3.hd-01.alayanew.com:8082 |
Run the configuration wizard：
# Run the configuration wizard
rclone config
Follow the interactive prompts：
- Choose n to create a new remote.
- Enter a name for the remote (for example: s3_store).
- Select the storage provider (for example: s3).
- Follow the prompts to provide authentication details (access key, secret key, endpoint, etc.) or to complete any provider-specific auth flow.
- When configuration is complete, enter q to quit the wizard.
Usage
Common Commands
| Command | Description |
|---|---|
| rclone lsd | List all directories/buckets at a path. |
| rclone ls | List objects with size and path. |
| rclone copy | Copy files from source to destination. |
| rclone move | Move files from source to destination. |
| rclone delete | Delete specified files or directories. |
| rclone rmdir | Remove empty directories. |
| rclone sync | Sync files, updating only target files. |
| rclone help | Show rclone help documentation. |
| rclone help [command_name] | Show detailed help for a specific command. |
| rclone listremotes | List configured remotes. |
| rclone config show [s3_store] | Display configuration for a remote. |
Advanced Commands
| Command | Description |
|---|---|
| rclone find s3_store:/remote/path --name *.txt | Search files by name pattern. |
| rclone copy /local/path s3_store:/remote/path --bwlimit 1M | Limit transfer speed to 1 MB/s. |
| rclone copy /local/path s3_store:/remote/path --retries 3 | Retry failed transfers up to 3 times. |
| rclone copy /local/path s3_store:/remote/path --log-file transfer.log | Log transfers to a file. |
| rclone copy /local/path s3_store:/remote/path --progress | Show transfer progress. |
| rclone copy /local/path s3_store:/remote/path --include ".txt" --exclude ".log" | Include/exclude files during transfer. |
| rclone copy /local/path s3_store:/remote/path --transfers 4 | Set number of concurrent transfers. |
Advanced Features
Rclone also supports advanced capabilities such as caching and encryption. Please visit the Rclone official website for more details and usage instructions.
Important Notes
- Security：Keep your Access Key ID and Secret Access Key confidential to prevent unauthorized access.
- Performance：For large-scale data operations, use rclone in a stable, high-bandwidth network environment to ensure optimal performance.