[Bug]: File uploads intermittently fail with "Object not found after writing" due to object storage race condition #55814
Description
⚠️ This issue respects the following points: ⚠️
- This is a bug, not a question or a configuration/webserver/proxy issue.
- This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
- Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- I agree to follow Nextcloud's Code of Conduct.
Bug description
I have encountered a bug where our Nextcloud server cannot manage to upload files, in a normal or team folder:
- File uploads fail with "Object not found after writing" error due to a race condition in object storage validation.
Steps to reproduce
- Use Nextcloud with object storage backend (S3/Swift/etc.)
- Upload files
- Error occurs intermittently due to timing
- The timing-dependent nature makes it difficult to reproduce consistently.
Expected behavior
Retry mechanism & File uploads should succeed
Nextcloud Server version
32
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.3
Web server
Apache (supported)
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Upgraded to a MAJOR version (ex. 31 to 32)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
- Default user-backend (database)
- LDAP/ Active Directory
- SSO - SAML
- Other
Configuration report
{
"system": {
"htaccess.RewriteBase": "\/",
"memcache.local": "\\OC\\Memcache\\APCu",
"apps_paths": [
{
"path": "\/var\/www\/html\/apps",
"url": "\/apps",
"writable": false
},
{
"path": "\/var\/www\/html\/custom_apps",
"url": "\/custom_apps",
"writable": true
}
],
"memcache.distributed": "\\OC\\Memcache\\Redis",
"memcache.locking": "\\OC\\Memcache\\Redis",
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"password": "***REMOVED SENSITIVE VALUE***",
"port": 6379
},
"overwriteprotocol": "https",
"trusted_proxies": "***REMOVED SENSITIVE VALUE***",
"objectstore": {
"class": "\\OC\\Files\\ObjectStore\\S3",
"arguments": {
"bucket": "fneaurabuckets3",
"region": "sbg",
"hostname": "s3.sbg.io.cloud.ovh.net",
"port": "443",
"storageClass": "",
"objectPrefix": "urn:oid:",
"autocreate": true,
"use_ssl": true,
"use_path_style": true,
"legacy_auth": false,
"key": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***"
}
},
"upgrade.disable-web": true,
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"intranet.next.fne-aura.org"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "mysql",
"version": "32.0.0.13",
"overwrite.cli.url": "https:\/\/intranet.next.fne-aura.org",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"default_phone_region": "FR",
"enable_previews": false,
"maintenance_window_start": 1,
"default_language": "fr",
"default_locale": "fr_FR",
"default_timezone": "Europe\/Paris",
"auto_logout": true,
"knowledgebaseenabled": false,
"skeletondirectory": "",
"mail_smtpmode": "smtp",
"mail_smtpsecure": "ssl",
"mail_sendmailmode": "smtp",
"mail_smtpauth": 1,
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "465",
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"loglevel": 0,
"force_language": "fr",
"force_locale": "fr_FR",
"trashbin_retention_obligation": "auto, 90",
"versions_retention_obligation": "disabled",
"memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
"memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
"memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
"memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
"app_install_overwrite": [
"libresign"
],
"maintenance": false,
"defaultapp": "dashboard,files,activity"
}
}List of activated Apps
Enabled:
- activity: 5.0.0-dev.0
- app_api: 32.0.0
- bruteforcesettings: 5.0.0-dev.0
- calendar: 6.0.1
- circles: 32.0.0
- cloud_federation_api: 1.16.0
- comments: 1.22.0
- contacts: 8.0.3
- contactsinteraction: 1.13.1
- dashboard: 7.12.0
- dav: 1.34.2
- deck: 1.16.0
- federatedfilesharing: 1.22.0
- federation: 1.22.0
- files: 2.4.0
- files_downloadlimit: 5.0.0-dev.0
- files_pdfviewer: 5.0.0-dev.0
- files_reminders: 1.5.0
- files_sharing: 1.24.0
- files_trashbin: 1.22.0
- files_versions_s3: 1.3.0
- firstrunwizard: 5.0.0-dev.0
- forms: 5.2.1
- impersonate: 3.0.0
- libresign: 11.4.1
- logreader: 5.0.0-dev.0
- lookup_server_connector: 1.20.0
- mail: 5.5.7
- nextcloud_announcements: 4.0.0-dev.0
- notes: 4.12.3
- notifications: 5.0.0-dev.0
- notify_push: 1.2.0
- oauth2: 1.20.0
- password_policy: 4.0.0-dev.0
- polls: 8.5.0
- privacy: 4.0.0-dev.0
- profile: 1.1.0
- provisioning_api: 1.22.0
- recommendations: 5.0.0-dev.0
- related_resources: 3.0.0-dev.0
- richdocuments: 9.0.0
- serverinfo: 4.0.0-dev.0
- settings: 1.15.1
- sharebymail: 1.22.0
- spreed: 22.0.0
- survey_client: 4.0.0-dev.0
- systemtags: 1.22.0
- text: 6.0.0-dev.0
- theming: 2.7.0
- twofactor_backupcodes: 1.21.0
- updatenotification: 1.22.0
- user_status: 1.12.0
- viewer: 5.0.0-dev.0
- weather_status: 1.12.0
- webhook_listeners: 1.3.0
- workflowengine: 2.14.0
Disabled:
- admin_audit: 1.22.0 (installed 1.21.0)
- collectives: 3.2.0 (installed 3.2.0)
- encryption: 2.20.0 (installed 2.19.0)
- files_external: 1.24.0 (installed 1.24.0)
- files_versions: 1.25.0 (installed 1.24.0)
- photos: 5.0.0-dev.1 (installed 2.5.0)
- support: 4.0.0-dev.0 (installed 3.0.0)
- suspicious_login: 10.0.0-dev.0
- twofactor_nextcloud_notification: 6.0.0-dev.0
- twofactor_totp: 14.0.0
- user_ldap: 1.23.0 (installed 1.22.0)Nextcloud Signing status
No errors have been found.Nextcloud Logs
Additional info
We use S3 bucket provided by OVH. Maybe not fast as aws s3.
In the upload code, objectExists() verification happens immediately after writing, but object storage has latency.
Maybe add retry mechanism or make validation optional for object storage backends.
I've temporarily resolved this issue by removing/disabling the objectExists() validation check: