Blobfuse2 Streaming: Write fails when using ADLS, works with Blob?

[rmodig]

Hi,

I'm encountering some problems using Blobfuse2, would very much appreciate any suggestions.

Use Case

Restore and host multiple SQL Anywhere databases backups (.db-files) on a Linux VM in Azure. The files are fairly large (40+gb) so I would like to stream directly from blob storage.

Configuration

• Ubuntu 20.04.6 LTS
• Blobfuse2 2.0.3
• VM service principal has Blob Storage Data Owner role on the storage account.
• The file I'm experimenting with is ~3gb and the VM is oversized, so should not be an issue.
• Executing as root.

Mount command:

blobfuse2 mount /root/mount/ --config-file=/root/config/blobfuse.yaml --use-adls=true

Configuration file:

allow-other: true

logging:
type: base
level: log_debug
file-path: ~/blobfuse.log

components:
- libfuse
- stream
- azstorage

stream:
block-size-mb: 0
max-buffers: 0
buffer-size-mb: 0
file-caching: true

libfuse:
attribute-expiration-sec: 0
entry-expiration-sec: 0
negative-entry-expiration-sec: 0
disable-writeback-cache: true

azstorage:
type: adls
account-name: <redacted>
endpoint: https://<redacted>.dfs.core.windows.net
mode: msi
container: <redacted>

Problem

The database backups need to be recovered, which is done automatically when starting a database server. This requires modifying the database file, but the write fails with the following error:

A write failed with error code: (-5), Unknown error (-5)
Fatal error:  A disk write to file "/root/mount/<redacted>/<redacted>.db" failed with error code: (-5)

Exploring the log, I discover the following:

Wed May  3 07:57:42 UTC 2023 : blobfuse2[6841] : LOG_ERR [libfuse_handler.go (744)]: Libfuse::libfuse_write : error writing file <redacted>/<redacted>.db, hand
le: 12 [-> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/cloudtest/go/pkg/mod/github.com/!azure/[email protected]
2ad61/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=BlobOperationNotSupported) =====
Description=Blob operation is not supported.
RequestId:1fe72c44-c01e-0050-2694-7d0429000000
Time:2023-05-03T07:57:42.2587931Z, Details: 
   Code: BlobOperationNotSupported
   PUT https://<redacted>.blob.core.windows.net/<redacted>/<redacted>/<redacted>.db?blockid=I5OFx4RATAVmVZZezJRV1A%3D%3D&comp=block&timeout=901
   Authorization: REDACTED
   Content-Length: [4194304]
   User-Agent: [Azure-Storage-Fuse/2.0.3 (Ubuntu 20.04.6 LTS) Azure-Storage/0.14 (go1.16.2; linux)]
   X-Ms-Client-Request-Id: [f532ac7f-1d48-4f30-6703-a012405ad73c]
   X-Ms-Version: [2020-08-04]
   --------------------------------------------------------------------------------
   RESPONSE Status: 409 Blob operation is not supported.
   Content-Length: [226]
   Content-Type: [application/xml]
   Date: [Wed, 03 May 2023 07:57:42 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [f532ac7f-1d48-4f30-6703-a012405ad73c]
   X-Ms-Error-Code: [BlobOperationNotSupported]
   X-Ms-Request-Id: [1fe72c44-c01e-0050-2694-7d0429000000]
   X-Ms-Version: [2020-08-04]
]

Partial solution

I observed in the log that the PUT-operation specifies a Blob-endpoint, not DFS. By putting my files in the container root folder and changing azstorage-type to "block" and the endpoint url to "blob" instead of "adls", and removing the "--use-adls"-parameter I can successfully restore the database. Once the database is restored, I can interact with the databases as expected.

This is only a partial solution, because each database instance generates its own log-file, and unfortunately these have the same name, so they can't co-exist in the same directory. An ad hoc solution would be to use multiple containers, but it's far from ideal.

As an aside, I'm unsure if the "--use-adls"-parameter is necessary when the azstorage-type is already set to "adls". Any clarification would be helpful!

Attempted solutions

• Attempted authentication with Access Key to rule out permission issues.
• Investigated file permissions. Current permissions are "-rw-r-----" on the file and "drwxr-x---" on the directory. My understanding is that read/write permissions should be sufficient.
• Experimented with setting "disable-writeback-cache" and "file-caching" to false.

Thanks very much for taking the time, any suggestions would be most appreciated!

[vibhansa-msft]

• If your account is HNS enabled then you shall provide "type: adls" or "--use-adls=true" options.
• In case of HNS account, file/directory permissions will be based on the ACLs configured on that path in storage.
• As you have streaming configured and trying with ~3GB file, can you give file-cache a shot and see if that works well for the same test case or not.
• When you attempt to use streaming can you configure the block size to a higher value and max-buffers also to a higher value and give it a try.

[rmodig]

• If your account is HNS enabled then you shall provide "type: adls" or "--use-adls=true" options.
• In case of HNS account, file/directory permissions will be based on the ACLs configured on that path in storage.
• As you have streaming configured and trying with ~3GB file, can you give file-cache a shot and see if that works well for the same test case or not.
• When you attempt to use streaming can you configure the block size to a higher value and max-buffers also to a higher value and give it a try.

Thank you.

I made some progress, successfully restoring the database after manually copying it (using Azure Storage Explorer) from a child-directory into the root-folder (the mounted path), without changing azstorage-type to block or changing the endpoint.

I think it may be a permission issue, but I've tried changing the ACLs on the storage account, even changing the owner to the VM managed identity. Considering the restoration works after manually copying the file to the root folder, do you think I'm on the right track?

Apologies that I'm not too knowledgable concerning ACLs and Linux File System permissions, I might be missing something obvious.

I have experimented with other configurations. I will experiment some more to see if it turns out to be an authentication issue and if it's not I will try your other suggestions next.

Attaching the log file where I successfully restored the database from the root folder, then failed to restore it in a child-directory.

blobfuse.log

[rmodig]

Update: Was able to restore database after copying manually to a new sub-directory. I conclude that the issue is either due to misconfigured permissions or due to the directory name containing hyphens. Renamed the directory and it did not resolve the issue.

I will look into the documentation to see if there is any guidance for configuring permissions. Thanks again for taking the time.

[rmodig]

I've done some more testing and discovered a few things.

• Most importantly; the database that seems to deny write access has been copied using Azure Data Factory copy activity.
• All other files that I've uploaded from my computer or even copies made in Azure Storage Explorer of the database that won't work can be operated on successfully.
• I can set POSIX permissions using chmod or in Storage Explorer
• These permissions are being altered by Blobfuse. Curiously also for the files that are restored successfully, their ACLs seem to be changed to read-only for all users. This may be causing other issues, but does not seem to prevent me from reading from the database.

I've browsed through the source code and found that Blobfuse does indeed change the ACLs, probably due to misconfiguration on my part.

I will keep investigating, but if I might be on the wrong track or if I'm missing anything obvious, I'd be grateful if you or anyone else would point it out. I'm still learning so it's possible I've misunderstood something very obvious.

[vibhansa-msft]

" Blobfuse does indeed change the ACLs" : what do you mean by this. If user executes 'chmod' command, then only ACLs will be modified.
Storage explorer and Blobfuse are cross compatible but I am not too sure about ADF.

[rmodig]

I mean that after changing permissions, either from the VM or using storage explorer, and then attempting to access the file through Blobfuse, the file permissions are reset to "-r--------", as far as I can tell.

I'm not certain if this is the issue, but since the only difference I can discover between the files copied from ADF and those uploaded manually is the owner/owning group and the permissions. To be clear, the only interaction with ADF is copying the file into blob storage to later be read by Blobfuse.

Looking at the source code, it seems as if there's a problem fetching ACLs, they will be set to default values: reference to component/azstorage/utils.go

I have enabled API logging in my configuration and will try to dig deeper into the logs and experiment with the configuration to see what is going on.

[vibhansa-msft]

Kindly share the complete log file as well to better understand what all was going on.

[rmodig]

I've made some very modest progress in figuring out the problem.

Part of it appears to be that the server utility I'm using (SQL Anywhere dbsrv17) alters file permissions to -r--r--r--. So I appear to have been mistaken above when I suspected that Blobfuse changes these. There are also many telling outputs in the log ("unexpected permissions"), possibly indicating an issue where the ACLs are modified by the server utility while Blobfuse is interacting with the file.

Currently some database files cannot be recovered, while others can. The curious thing is that after cloning the file to the same directory, recovery succeeds. Copying the file from the mounted storage account to the local file system also allows for successful recovery.

Other database files that have been copied using ADF can be successfully recovered.

This may be more of an SQL Anywhere issue than an issue with Blobfuse. Something feels off about the fact that I'm able to recover the database successfully through blobfuse after cloning it though.

Here is my current config file:

allow-other: true

logging:
  type: base
  level: log_debug
  file-path: ~/blobfuse.log

components:
  - libfuse
  - stream
  - azstorage

stream:
  block-size-mb: 0
  max-buffers: 0
  buffer-size-mb: 0
  file-caching: false

libfuse:
  default-permissions: 0777
  attribute-expiration-sec: 0
  entry-expiration-sec: 0
  negative-entry-expiration-sec: 0
  disable-writeback-cache: false
  ignore-open-flags: true
  fuse-trace: true

azstorage:
  type: adls
  account-name: <redacted>
  mode: msi
  container: <redacted>
  sdk-trace: true

And here is a recent log file: blobfuse.log

The log is from performing the following steps:

• Successfully restored one database.
• Successfully restored a cloned copy of another database (cloned in storage explorer).
• Failure to restore the original of the cloned database.

I have tried to increase block size and max buffers seemingly without any effect. File-caching also has made no difference.

I think I will have to find a workaround, possibly copying all files to local storage before attempting recovery. Very frustrating though, as streaming directly from blobfuse appears very close to working.

Thanks for taking the time to help resolve this issue. It may not be an issue with Blobfuse at all, but if you see anything strange either in the logs or my configuration I'd appreciate any input.

[vibhansa-msft]

"unexpected permissions" : this is expected if you have assigned some Identity/Service principals' special permissions for your blob. Other then rwx triplets if we receive anything extra in the permissions section, we will report it using this error message so this can be ignored for now.

[rmodig]

Thanks for all your assistance. I've decided to change my workflow to instead copy my database backups to the VM using AzCopy, which is disappointing but still workable given the volume of data I'm dealing with.

I've settled on the problem possibly being caused by the fact that the database recovery tool alters permissions to read-only and the behavior can't be overridden. I suspect that ACLs are being cached and Blobfuse tries to write to the file after it has been changed to read-only. I could also be completely off track as I'm still learning as I go.

I tried the same workflow with NFS v3 instead and seemed to encounter the same problem.

Anyway, hopefully these comments may be of some use for anyone else wanting to serve SAP SQL Anywhere / Sybase databases on Blobfuse .