Working with Storage¶

You can use any software that supports the S3 protocol to work with S3 storage. In most cases, you need to specify the Access Key, Secret Key, the storage server address, and select the connection protocol (HTTP or HTTPS).

For complete documentation on S3 protocol capabilities in the Storage service, refer to the official documentation. The S3 protocol was originally developed for Amazon’s AWS S3 service.

Endpoints¶

Currently, the Storage S3 service supports the following types of endpoints in each location:

Public access point (HTTP and HTTPS): s3.loc.icdc.io.
Internal access point (HTTP and HTTPS): s3.local.loc.icdc.io.
A DirectConnect access point can be configured upon request to the support team.

The internal access point can be used to access the S3 service only from within the account’s networks (for example, from Compute virtual machines) and does not use internet traffic, which may have bandwidth limitations.
Once a bucket is created, it becomes accessible through all available endpoints. Using the internal endpoint does not restrict access to the bucket via the public endpoint.

path-style access

Currently, Storage supports only path-style access, which means buckets are accessed using the following format:
s3.loc.domain/bucket-name
The DNS-style access format such as <bucket>.s3.loc.domain is not supported at this time.

S3 Browser (GUI)¶

Below is an example of configuring S3 Browser, a client for working with S3-compatible object storage. It allows you to create and delete buckets, browse stored objects, and upload new files to S3 storage. Download the S3 Browser distribution package from the developer’s website.

When launching S3 Browser for the first time, a connection configuration dialog will appear.
Fill in the fields using the information from the S3 user page (Overview tab):

Account type: select S3 Compatible Storage.
REST Endpoint: specify the storage address, for example: s3.loc.icdc.io, where loc is the name of the location.
Access Key ID: Access Key ID that you received when creating the cart (bucket).
Secret Access Key: the Secret Key that you received when creating the cart (bucket).
Use secure transfer (SSL/TLS): required option to use an encrypted SSL/TLS channel to interact with the storage.
Advanced Settings > Addressing model: select option Path-style

After pressing Add new account, the configuration of s3browser for working with S3 will be completed. Instructions for further work with S3 Browser can be found on developer’s website.

s3cmd (CLI)¶

s3cmd is a command line tool and console client for managing S3 object stores. To install S3cmd, you need to download the distribution from developer's website.

To configure S3cmd, use the s3cmd --configure command and fill in the values for the following parameters:

Access Key: Access Key identifier that you received when creating the bucket.
Secret Key: the Secret Key that you received when creating the bucket.
Default Region: leave blank.
S3 Endpoint: you can specify the public storage address s3.loc.icdc.io or the internal address s3.local.loc.icdc.io, where loc is the name of the location.
DNS-style bucket+hostname:port template for accessing a bucket: specify s3.loc.icdc.io (but DNS-style %(bucket)s.s3.loc.icdc.io is not supported at the moment).

The s3cmd --configure command will save the settings in the ~/.s3cfg file in the format:

[default]
access_key = ACCESS_KEY
secret_key = SECRET_KEY
host_base = https://s3.loc.icdc.io
host_bucket = s3.loc.icdc.io
use_https = True

Public access¶

An example of outputting a file to public access using the s3cmd command:

Example

# 1. Configure keys of S3 user created in Storage into file ~/.s3cfg
# 2. Make Bucket with public ACL
$ s3cmd mb --acl-public s3://test-anon

$ echo "hello world" > index.html

# 3. Save object with public ACL
$ s3cmd put --acl-public index.html s3://test-anon
upload: 'index.html' -> 's3://test-anon/index.html'  [1 of 1]
12 of 12   100% in    0s    18.92 B/s  done
Public URL of the object is: http://s3.loc.icdc.io/test-anon/index.html

$ s3cmd info s3://test-anon/index.html
s3://test-anon/index.html (object):
  File size: 12
  Last mod:  Wed, 25 Oct 2023 08:51:52 GMT
  MIME type: text/plain
  Storage:   STANDARD
  MD5 sum:   6f5902ac237024bdd0c176cb93063dc4
  SSE:       none
  Policy:    none
  CORS:      none
  ACL:       *anon*: READ
  ACL:       test123___icdc___user@example.com: FULL_CONTROL
  URL:       http://s3.loc.icdc.io/test-anon/index.html
  x-amz-meta-s3cmd-attrs: atime:1698223852/ctime:1698223852/gid:1000/gname:danix/md5:6f5902ac237024bdd0c176cb93063dc4/mode:33188/mtime:1698223852/uid:1000/uname:dan

The file is available at the following address: https://s3.loc.icdc.io/test-anon/index.html

Public access to objects in Storage V2

When using curl to access public objects in buckets created in Storage V2, you must use the account:bucket format in the URL, like this:

curl https://s3.loc.icdc.io/<account>:<bucket>/<object>

This does not affect access through s3cmd, which continue to use the standard bucket/object format.

S3 bucket policies¶

The ability to restrict access is configured within the S3 protocol using S3 policies.

$ cat > examplepol
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": ["arn:aws:iam::usfolks:user/fred"]},
    "Action": "s3:PutObjectAcl",
    "Resource": [
      "arn:aws:s3:::happybucket/*"
    ]
  }]
}

$ s3cmd setpolicy examplepol s3://happybucket
$ s3cmd delpolicy s3://happybucket