The file was not readable while being written

[whitejdae]

Mountpoint for Amazon S3 version

v1.12

AWS Region

No response

Describe the running environment

• The local kubernetes cluster uses the version: 1.31
• Use Kustomize to install
• The operating system that the pod runs is:

cat /etc/os-release

NAME="Rocky Linux"
VERSION="9.3 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

Mountpoint options

Default loading configuration：https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/install.md#deploy-driver

What happened?

• The java program configures the GC log parameters to mount to s3, but s3 must wait until the GC log is written before uploading to s3, and an error message is reported

Feb 17 22:10:39 k8s-node04 mount-s3[1683809]: [WARN] mountpoint_s3::cli: failed to detect network throughput. Using 10 gbps as throughput. Use --maximum-throughput-gbps CLI flag to configure a target throughput appropriate for the instance. Detection failed due to: failed to get instance type: IMDS query failed: Unknown CRT error
Feb 17 22:11:08 k8s-node04 mount-s3[1683809]: [WARN] getxattr{req=76 ino=8 name="security.capability"}: mountpoint_s3::fuse: getxattr failed: operation not supported by Mountpoint
Feb 17 22:13:29 k8s-node04 mount-s3[1683809]: [WARN] open{req=308 ino=8 pid=1686052 name=aliex-grantcenter-ts-778d8bfb46-t8r9x-2025-02-17_22-10-47.log.0.current}: mountpoint_s3::fuse: open failed with errno 1: inode error: inode 8 (full key "dev/GCLog/aliex-grantcenter-ts/aliex-grantcenter-ts-778d8bfb46-t8r9x/aliex-grantcenter-ts-778d8bfb46-t8r9x-2025-02-17_22-10-47.log.0.current") is not readable while being written

• Before mounting, enter the pod and use tail to access normally, but an error is reported after mounting the s3 driver

[root@aliex-grantcenter-ts-778d8bfb46-t8r9x aliex-grantcenter-ts-778d8bfb46-t8r9x]# tail -f aliex-grantcenter-ts-778d8bfb46-t8r9x-2025-02-17_22-10-47.log.0.current
tail: cannot open 'aliex-grantcenter-ts-778d8bfb46-t8r9x-2025-02-17_22-10-47.log.0.current' for reading: Operation not permitted
tail: no files remaining

• If you use the s3-app program to test and write in a loop, this problem will not exist.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: s3-app-1
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: s3-app-1
  template:
    metadata:
      labels:
        app: s3-app-1
    spec:
      containers:
        - name: app
          image: centos
          command: ["/bin/sh"]
          args:
            - "-c"
            - |
              mkdir -p /data/s3-app-1;
              while true; do
                echo "Hello from the container! $(date -u)" >> /data/s3-app-1/log.txt;
              done
          volumeMounts:
            - name: persistent-storage
              mountPath: /data
      volumes:
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: dev-deploy-log-s3-novax

Relevant log output

[whitejdae]

pv and pvc configuration：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: dev-deploy-log-s3-novax
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1200Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: dev-deploy-log-s3-novax
    namespace: dev
  csi:
    driver: s3.csi.aws.com
    volumeAttributes:
      bucketName: dev-deploy-log-s3-novax
    volumeHandle: dev-deploy-log-s3-novax-storage
  mountOptions:
  - region ap-southeast-1
  - --allow-other
  - prefix dev/
  - --allow-overwrite
  - --incremental-upload
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dev-deploy-log-s3-novax
  namespace: dev
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1200Gi
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: dev-deploy-log-s3-novax

[muddyfish]

Hi, thanks for your interest in Mountpoint & the CSI Driver.

This is expected behaviour - in Mountpoint's semantics documentation, it says that files cannot be read from whilst they're being written to.

Unrelated to your problem, I noticed you're using the --incremental-upload flag, which is only supported in S3 Directory buckets. I'd recommend either removing the flag or using a directory bucket for your log files

[whitejdae]

Why can I write to S3 in real time by looping through the mount point using the s3-app service? Is it because of the file system problem?

[vladem]

I don't think that subsequent writes are actually completed in your example. The following command ignores the error reported by Mountpoint:

echo "Hello from the container! $(date -u)" >> /data/s3-app-1/log.txt;

You'll get an error if you'll try to append to an existing object in S3 Standard via python script:

$ target/debug/mount-s3 <bucket> <path> --incremental-upload --log-directory logs
bucket <bucket> is mounted at <path>

(25-02-21 15:25:26) <1> [~/local/mountpoint-s3]
$ python -c "import datetime; open('<path>/file4.txt', 'a').write(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))"

(25-02-21 15:25:37) <0> [~/local/mountpoint-s3]
$ cat <path>/file4.txt
2025-02-21 15:25:37%

(25-02-21 15:25:45) <0> [~/local/mountpoint-s3]
$ python -c "import datetime; open('<path>/file4.txt', 'a').write(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))"
close failed in file object destructor:
IOError: [Errno 5] Input/output error

(25-02-21 15:25:50) <0> [~/local/mountpoint-s3]
$ cat <path>/file4.txt
2025-02-21 15:25:37%

In any case file is not overwritten and following error is reported in Mountpoint logs:

2025-02-21T15:25:50.713169Z  WARN flush{req=58 ino=2 fh=4 pid=20946 name="file4.txt"}: mountpoint_s3::fuse: flush failed with errno 5: put failed: put request failed: Service error: The server does not support the functionality required to fulfill the request

This behaviour may be misleading, so we'll consider making the error in this case reported earlier.