Use S3 on Seqpipe Ceph filesystem

Log into Ceph dashboard at https://piglet.seqpipe.org:8443 with user seqpipe.

Note

Other available endpoints for the Ceph dashboard are:

• https://dory.seqpipe.org:8443

• https://pooh.seqpipe.org:8443

From the dashboard, you can explore available buckets and users and get the necessary Access Key and Secret Key.

Use aws command line tool to configure your access:

> aws configure
AWS Access Key ID [None]: <your access key>
AWS Secret Access Key [None]: <your secret key>
Default region name [None]:
Default output format [None]:

After that, you can try the following command to list the available buckets:

aws --endpoint-url=https://s31.seqpipe.org:7480 s3api list-buckets

{
    "Buckets": [
        {
            "Name": "grr-bucket",
            "CreationDate": "2022-04-19T06:25:10.649Z"
        },
        {
            "Name": "grr-bucket-test",
            "CreationDate": "2022-02-23T15:50:06.594Z"
        }
    ],
    "Owner": {
        "DisplayName": "seqpipe",
        "ID": "seqpipe"
    }
}

To copy a directory into an S3 bucket, you can use:

aws --endpoint-url=https://s31.seqpipe.org:7480 s3 cp --recursive \
    ./grr s3://grr-bucket-test/grr  \
    --exclude ".dvc/*" --exclude ".git/*" --exclude "*/.grr/*"