Runs on CEPH

AutoMQ uses EBS and S3 storage, while CEPH supports POSIX and S3 access protocols, making it a suitable backend storage for AutoMQ. The following is a guide for deploying AutoMQ based on CEPH.

Prerequisites

• To install CEPH, please refer to: https://docs.ceph.com/en/latest/install/

• To install the S3-compatible component RGW for CEPH, please refer to: https://docs.ceph.com/en/latest/cephadm/services/rgw/

Configure WAL

1. For guidance on mounting raw devices on a Linux host from the Ceph official documentation, please refer to: https://docs.ceph.com/en/latest/rbd/

2. Configure the raw device path as /dev/vdb.

3. AutoMQ uses the raw device to store WAL data at the specified path. You can configure this via the startup parameter --override s3.wal.path=/dev/vdb.

Configure S3 URL

Create a User for CEPH

radosgw-admin user create --uid="automq" --display-name="automq"

The created user will have full permissions required by AutoMQ by default. For configuring minimal permissions, please refer to the official CEPH documentation for custom settings. The result after executing the above commands is as follows:

{
    "user_id": "automq",
    "display_name": "automq",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "automq",
            "access_key": "X1J0E1EC3KZMQUZCVHED",
            "secret_key": "Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

Create a Bucket for CEPH

1. Set environment variables to configure the Access Key and Secret Key required by the AWS CLI.

export AWS_ACCESS_KEY_ID=X1J0E1EC3KZMQUZCVHED
export AWS_SECRET_ACCESS_KEY=Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD

2. Use the AWS CLI to create an S3 bucket.

aws s3api create-bucket --bucket automq-data --endpoint=http://127.0.0.1:80

Deploy AutoMQ

The necessary parameters for generating an S3 URL are as follows:

Parameter Name	Default Value in This Example	Description
--s3-access-key	X1J0E1EC3KZMQUZCVHED	Replace this value with the actual key after creating the Ceph user
--s3-secret-key	Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD	Replace this value with the actual secret after creating the Ceph user
--s3-region	us-west-2	This parameter is not valid in Ceph, you can set it to any value, such as us-west-2
--s3-endpoint	http://127.0.0.1:80	This parameter specifies the address of the S3-compatible RGW component in Ceph for external services. If multiple machines are used, it is recommended to use a load balancer (SLB) to aggregate them into a single IP address.
--s3-data-bucket	automq-data	-
--s3-ops-bucket	automq-ops	-

After completing the WAL and S3URL configuration, you can now deploy AutoMQ. Please follow the Cluster Deployment on Linux▸ instructions.