From 68a4ec9313c9bbce325f071db64a21857ecd3939 Mon Sep 17 00:00:00 2001 From: "Michael V. DePalatis" Date: Sun, 13 Jun 2021 08:58:45 -0600 Subject: [PATCH 1/3] Add additional commentary on limitations More specifically: - Point out that opening a file can file when empty "directory" objects don't exist - Provide a hint that passing `strict=False` can resolve some issues when writing empty objects isn't an option --- docs/index.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 9899ac8..6aedc77 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -62,12 +62,15 @@ directory exists. If you create all your files and directories with S3FS, then you can forget about how things are stored under the hood. Everything will work as you expect. You *may* run in to problems if your data has been -uploaded without the use of S3FS. For instance, if you create a +uploaded without the use of S3FS. For instance, if you create or open a `"foo/bar"` object without a `"foo/"` object. If this occurs, then S3FS may give errors about directories not existing, where you would expect -them to be. The solution is to create an empty object for all +them to be. One solution is to create an empty object for all directories and subdirectories. Fortunately most tools will do this for you, and it is probably only required of you upload your files manually. +Alternatively you may be able to get away with creating a `S3FS` object +directly and passing ``strict=False`` to bypass some consistency checks +which will fail if empty objects in an object's prefix exist. Authentication From c14cbba964e9574033791829835fb4e0ffed1319 Mon Sep 17 00:00:00 2001 From: "Michael V. DePalatis" Date: Sun, 13 Jun 2021 09:02:50 -0600 Subject: [PATCH 2/3] Update README to link to limitations docs --- README.md | 4 +++- README.rst | 45 ++++++++++++++++++++++++--------------------- 2 files changed, 27 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 5e704f5..c999b95 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,9 @@ S3FS is a [PyFilesystem](https://www.pyfilesystem.org/) interface to Amazon S3 cloud storage. As a PyFilesystem concrete class, [S3FS](http://fs-s3fs.readthedocs.io/en/latest/) allows you to work with S3 in the -same way as any other supported filesystem. +same way as any other supported filesystem. Note that as S3 is not strictly +speaking a filesystem there are some limitations which are discussed in detail +in the [documentation](https://fs-s3fs.readthedocs.io/en/latest/#limitations). ## Installing diff --git a/README.rst b/README.rst index 0624209..fcba978 100644 --- a/README.rst +++ b/README.rst @@ -6,7 +6,10 @@ Amazon S3 cloud storage. As a PyFilesystem concrete class, `S3FS `__ allows you to work -with S3 in the same way as any other supported filesystem. +with S3 in the same way as any other supported filesystem. Note that as +S3 is not strictly speaking a filesystem there are some limitations +which are discussed in detail in the +`documentation `__. Installing ---------- @@ -15,7 +18,7 @@ You can install S3FS from pip as follows: :: - pip install fs-s3fs + pip install fs-s3fs Opening a S3FS -------------- @@ -24,37 +27,37 @@ Open an S3FS by explicitly using the constructor: .. code:: python - from fs_s3fs import S3FS - s3fs = S3FS('mybucket') + from fs_s3fs import S3FS + s3fs = S3FS('mybucket') Or with a FS URL: .. code:: python - from fs import open_fs - s3fs = open_fs('s3://mybucket') + from fs import open_fs + s3fs = open_fs('s3://mybucket') Downloading Files ----------------- To *download* files from an S3 bucket, open a file on the S3 filesystem for reading, then write the data to a file on the local filesystem. -Here's an example that copies a file ``example.mov`` from S3 to your HD: +Here’s an example that copies a file ``example.mov`` from S3 to your HD: .. code:: python - from fs.tools import copy_file_data - with s3fs.open('example.mov', 'rb') as remote_file: - with open('example.mov', 'wb') as local_file: - copy_file_data(remote_file, local_file) + from fs.tools import copy_file_data + with s3fs.open('example.mov', 'rb') as remote_file: + with open('example.mov', 'wb') as local_file: + copy_file_data(remote_file, local_file) Although it is preferable to use the higher-level functionality in the -``fs.copy`` module. Here's an example: +``fs.copy`` module. Here’s an example: .. code:: python - from fs.copy import copy_file - copy_file(s3fs, 'example.mov', './', 'example.mov') + from fs.copy import copy_file + copy_file(s3fs, 'example.mov', './', 'example.mov') Uploading Files --------------- @@ -77,9 +80,9 @@ to a bucket: .. code:: python - import fs, fs.mirror - s3fs = S3FS('example', upload_args={"CacheControl": "max-age=2592000", "ACL": "public-read"}) - fs.mirror.mirror('/path/to/mirror', s3fs) + import fs, fs.mirror + s3fs = S3FS('example', upload_args={"CacheControl": "max-age=2592000", "ACL": "public-read"}) + fs.mirror.mirror('/path/to/mirror', s3fs) see `the Boto3 docs `__ @@ -91,9 +94,9 @@ and can be used in URLs. It is important to URL-Escape the .. code:: python - import fs, fs.mirror - with open fs.open_fs('s3://example?acl=public-read&cache_control=max-age%3D2592000%2Cpublic') as s3fs - fs.mirror.mirror('/path/to/mirror', s3fs) + import fs, fs.mirror + with open fs.open_fs('s3://example?acl=public-read&cache_control=max-age%3D2592000%2Cpublic') as s3fs + fs.mirror.mirror('/path/to/mirror', s3fs) S3 URLs ------- @@ -102,7 +105,7 @@ You can get a public URL to a file on a S3 bucket as follows: .. code:: python - movie_url = s3fs.geturl('example.mov') + movie_url = s3fs.geturl('example.mov') Documentation ------------- From 4a69cd77eab743ad68f81925856ad31869be60fa Mon Sep 17 00:00:00 2001 From: "Michael V. DePalatis" Date: Sun, 13 Jun 2021 09:07:42 -0600 Subject: [PATCH 3/3] Rephrase sentence about strict=False workaround --- docs/index.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 6aedc77..bfef353 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -68,9 +68,9 @@ may give errors about directories not existing, where you would expect them to be. One solution is to create an empty object for all directories and subdirectories. Fortunately most tools will do this for you, and it is probably only required of you upload your files manually. -Alternatively you may be able to get away with creating a `S3FS` object -directly and passing ``strict=False`` to bypass some consistency checks -which will fail if empty objects in an object's prefix exist. +Alternatively you may be able to get away with creating the `S3FS` object +directly with ``strict=False`` to bypass some consistency checks +which could fail when empty objects are missing. Authentication