Dynamic Concurrency "snapshotPersistenceEnabled" not working

[H-Finch-404]

We want to use dynamic concurrency with snapshotPersistenceEnabled on our ServiceBus triggers, but it does not seems to be storing the concurrency configuration snapshots to the connected storage account.

Repro steps

1. Enable dynamic concurrency including snapshot persistence as described in https://learn.microsoft.com/en-us/azure/azure-functions/functions-concurrency#dynamic-concurrency-configuration. Configured on root level of host.json like below:

"concurrency": {
        "dynamicConcurrencyEnabled": true,
        "snapshotPersistenceEnabled": true
    }

2. Check the connected storage account's azure-webjobs-hosts container
3. No directory named 'concurrency' with a concurrencyStatus.json file is created as described in https://techcommunity.microsoft.com/t5/apps-on-azure-blog/see-how-dynamic-concurrency-works-in-azure-function-app-with-a/ba-p/3502772
4. Check Application Insights logs with "Host.Concurrency": "Trace" enabled. New function instances (or when restarted) start from concurrency value 1 again.

Expected behavior

When dynamic concurrency is enabled together with snapshotPersistenceEnabled setting to true, concurrency configuration should be read/written to/from connected storage account so that a function instance does not have to relearn its concurrency settings from start.

Actual behavior

When dynamic concurrency is enabled together with snapshotPersistenceEnabled setting to true, concurrency settings should be saved and read from connected storage account.

Known workarounds

None.

Related information

I traced back the repository code that handles the concurrency snapshot persistence -> https://github.com/Azure/azure-webjobs-sdk/blob/7b8916156d27e6661d3e287a5902cdba98f953cd/src/Microsoft.Azure.WebJobs.Host.Storage/BlobStorageConcurrencyStatusRepository.cs. Initial log search debugging with following query :

traces   
| where cloud_RoleName has "<function-name>"
| where message contains "azure-webjobs-hosts/concurrency"

shows that attempts to read the concurrencyStatus.json file are made:
GET https://storage account URL/azure-webjobs-hosts/concurrency/function name/concurrencyStatus.json

However, I do not see any PUT's/POSTS to create the file in the first place.

• Package version
  3.0.39 - WebJobs
  5.13.4 - WebJobs.Extensions.ServiceBus
• Links to source
  https://github.com/Azure/azure-webjobs-sdk/blob/7b8916156d27e6661d3e287a5902cdba98f953cd/src/Microsoft.Azure.WebJobs.Host.Storage/BlobStorageConcurrencyStatusRepository.cs

[H-Finch-404]

Turns out restarting the Function App manually from Portal created the file. However we did multiple deployments in the timespan of enabling the dynamic concurrency (with snapshot persistence), which should trigger a restart if I remember correctly? This is of course also not expected behavior since we want to avoid downtime by manually restarting the Function App.

[H-Finch-404]

Turns out restarting the Function App manually from Portal created the file. However we did multiple deployments in the timespan of enabling the dynamic concurrency (with snapshot persistence), which should trigger a restart if I remember correctly? This is of course also not expected behavior since we want to avoid downtime by manually restarting the Function App.

The manual restart only seemed to persist the concurrency config to storage once with the following contents: {"Timestamp":"2023-11-13T15:16:52.1757275Z","NumberOfCores":1,"FunctionSnapshots":{}}
As you can see the FunctionSnapshots are completely empty. We also do not see any subsequent attempts (e.g. PUT requests) being made to store an updated concurrency configuration (even though we can see the dynamic concurrency working by verifying logs that incidicate concurrency increases)