Error: Too many pending tasks in queue ./database/config.json in notificationproxy_v1.1.0.d_impl

[Rajitharaj1234]

Seeing this error in latest NP with latest tag v1.1.0.d_impl.
[root@mwappserver2 _data]# docker logs -f notificationproxy_v1.1.0.d_impl | head -100
Tue, 02 Sep 2025 13:17:10 GMT body-parser deprecated undefined extended: provide extended option at node_modules/oas3-tools/dist/middleware/express.app.config.js:22:33
{"level":"WARN","timestamp":"2025-09-02T13:17:11.450Z","logger":"kafkajs","message":"KafkaJS v2.0.0 switched default partitioner. To retain the same partitioning behavior as in previous versions, create the producer with the option "createPartitioner: Partitioners.LegacyPartitioner". See the migration guide at https://kafka.js.org/docs/migration-guide-v2.0.0#producer-new-default-partitioner for details. Silence this warning by setting the environment variable "KAFKAJS_NO_PARTITIONER_WARNING=1""}
Mock mode: disabled
[13:17:10.722] INFO (1): Your server is listening on port 4025 (http://localhost:4025)
[13:17:10.723] INFO (1): Swagger-ui is available on http://localhost:4025/docs
Producer successfully connected to kafka client
POST /v1/register-yourself 204 656.736 ms - -
Initiated application registration to HTTP://X.X.X.X:3007/v1/register-yourself expected 204 is 204

callback : POST http://X.X.X.X:XXXX/v1/record-service-request header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"860CfA31-3f54-01bc-B174-D7a91DC18A7E","trace-indicator":"1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"x-correlator":"8ba65833-A80e-887d-AeF0-0211af78A3C6","trace-indicator":"1","user":"NotificationProxy","originator":"NotificationProxy","application-name":"NotificationProxy","release-number":"1.1.0","operation-name":"/v1/register-yourself","response-code":204,"timestamp":"2025-09-02T13:17:12+00:00","exec-time":373}response code:204

callback : POST http://X.X.X.X:3031/v2/register-application header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"8ba65833-A80e-887d-AeF0-0211af78A3C6","trace-indicator":"1.1","customer-journey":"unknown","operation-key":"n.a.","Content-Type":"application/json"}body :{"application-name":"NotificationProxy","release-number":"1.1.0","embedding-operation":"/v1/embed-yourself","client-update-operation":"/v1/update-client","operation-client-update-operation":"/v1/update-operation-client","dispose-remainders-operation":"/v1/dispose-remainders-of-deregistered-application","preceding-release-operation":"/v1/inform-about-preceding-release","subsequent-release-operation":"/v1/update-client-of-subsequent-release","tcp-server":{"protocol":"HTTP","port":3007,"address":{"ip-address":{"ipv-4-address":"X.X.X.X"}}}}response code:204
POST /v1/redirect-topology-change-information 200 656.512 ms - 154410
POST /v1/embed-yourself 204 1141.124 ms - -

callback : POST http://X.X.X.X:XXXX/v1/record-service-request header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"ffadC88e-8f34-F493-F571-1FCeCd595d9A","trace-indicator":"1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"x-correlator":"8ba65833-A80e-887d-AeF0-0211af78A3C6","trace-indicator":"1.1.1.1.1.1","user":"NotificationProxy","originator":"ApplicationLayerTopology","application-name":"NotificationProxy","release-number":"1.1.0","operation-name":"/v1/redirect-topology-change-information","response-code":200,"timestamp":"2025-09-02T13:17:14+00:00","exec-time":508}response code:204

callback : POST http://X.X.X.X:XXXX/v1/record-service-request header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"dCa857Bb-5ce4-31c0-b961-b76Bf44Be7e9","trace-indicator":"1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"x-correlator":"8ba65833-A80e-887d-AeF0-0211af78A3C6","trace-indicator":"1.1.1.1.2","user":"NotificationProxy","originator":"RegistryOffice","application-name":"NotificationProxy","release-number":"1.1.0","operation-name":"/v1/embed-yourself","response-code":204,"timestamp":"2025-09-02T13:17:14+00:00","exec-time":966}response code:204
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications

callback : POST http://X.X.X.X:3036/v1/update-ltp header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"8ba65833-A80e-887d-AeF0-0211af78A3C6","trace-indicator":"1.1.1.1.2.1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"uuid":"np-1-1-0-tcp-c-or-1-0-0-000","ltp-direction":"core-model-1-4:TERMINATION_DIRECTION_SINK","client-ltp":["np-1-1-0-http-c-or-1-0-0-000"],"server-ltp":[],"layer-protocol":[{"local-id":"0","layer-protocol-name":"tcp-client-interface-1-0:LAYER_PROTOCOL_NAME_TYPE_TCP_LAYER","tcp-client-interface-1-0:tcp-client-interface-pac":{"tcp-client-interface-configuration":{"remote-protocol":"tcp-client-interface-1-0:PROTOCOL_TYPE_HTTP","remote-address":{"ip-address":{"ipv-4-address":"X.X.X.X"}},"remote-port":3007}}}]}response code:204
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications

callback : POST http://X.X.X.X:XXXX/v1/record-service-request header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"5cBffF0a-cE60-bb8B-B28A-f7Db38Cca094","trace-indicator":"1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"x-correlator":"dEECB931-Fd9f-cBdD-8CA3-Ac544BcaAD5e","trace-indicator":"1","user":"NotificationProxy","originator":"NotificationProxy","application-name":"NotificationProxy","release-number":"1.1.0","operation-name":"SubscribeToControllerNotificationsStep1","response-code":200,"timestamp":"2025-09-02T13:17:14+00:00"}response code:204
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
[13:17:14.732] WARN (1): count not parse notification - not json: 'Device connected: 407251016'
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications
message [object Object] successfully sent to all_notifications

callback : POST http://X.X.X.X:XXXX/v1/record-service-request header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"ebF63a23-eFC0-B0B2-732a-Fe4cbcdcF83A","trace-indicator":"1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"x-correlator":"7ddC56b3-4733-07fe-eA7d-01697000C434","trace-indicator":"1","user":"NotificationProxy","originator":"NotificationProxy","application-name":"NotificationProxy","release-number":"1.1.0","operation-name":"SubscribeToControllerNotificationsStep2","response-code":200,"timestamp":"2025-09-02T13:17:14+00:00"}response code:204
message [object Object] successfully sent to all_notifications
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
callback : POST http://X.X.X.X:XXXX/v1/record-service-request header :{"user":"NotificationProxy","originator":"NotificationProxy","x-correlator":"2BeEe15e-bBb3-d06E-d8aB-a861717188B2","trace-indicator":"1","customer-journey":"unknown","operation-key":"-------------------------------------------","Content-Type":"application/json"}body :{"x-correlator":"B29CE099-B6eB-2e7c-91Fa-bC7d6D9dDBa8","trace-indicator":"1","user":"NotificationProxy","originator":"NotificationProxy","application-name":"NotificationProxy","release-number":"1.1.0","operation-name":"/v1/notify-device-alarms","response-code":200,"timestamp":"2025-09-02T13:18:00+00:00"}response code:204
Error: Too many pending tasks in queue ./database/config.json
at AsyncLock.acquire (/home/openbackhaul/NotificationProxy/node_modules/async-lock/lib/index.js:195:15)
at Object.exports.readFromDatabaseAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/databaseDriver/JSONDriver.js:30:23)
at Function.getLogicalTerminationPointListAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/onfModel/models/ControlConstruct.js:32:59)
at Object.exports.getKafkaClient (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:44:60)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:782:49)
at notifyAllDeviceSubscribers (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:736:13)
at exports.handleDeviceNotification (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:713:17)
at EventSource.eventSource.onmessage (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationStreamManagement.js:153:9)
at EventSource.emit (node:events:513:28)
at _emit (/home/openbackhaul/NotificationProxy/node_modules/eventsource/lib/eventsource.js:287:17)
Error: Too many pending tasks in queue ./database/config.json
at AsyncLock.acquire (/home/openbackhaul/NotificationProxy/node_modules/async-lock/lib/index.js:195:15)
at Object.exports.readFromDatabaseAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/databaseDriver/JSONDriver.js:30:23)
at Function.getLogicalTerminationPointListAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/onfModel/models/ControlConstruct.js:32:59)
at Object.exports.getKafkaClient (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:44:60)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:782:49)
at notifyAllDeviceSubscribers (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:736:13)
at exports.handleDeviceNotification (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:713:17)
at EventSource.eventSource.onmessage (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationStreamManagement.js:153:9)
at EventSource.emit (node:events:513:28)
at _emit (/home/openbackhaul/NotificationProxy/node_modules/eventsource/lib/eventsource.js:287:17)
TypeError: Cannot read properties of undefined (reading 'layer-protocol')
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:66:41)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
TypeError: Cannot read properties of undefined (reading 'layer-protocol')
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:66:41)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
ReferenceError: y is not defined
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:70:28)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
ReferenceError: y is not defined
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:70:28)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error: Too many pending tasks in queue ./database/config.json
at AsyncLock.acquire (/home/openbackhaul/NotificationProxy/node_modules/async-lock/lib/index.js:195:15)
at Object.exports.readFromDatabaseAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/databaseDriver/JSONDriver.js:30:23)
at Function.getLogicalTerminationPointListAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/onfModel/models/ControlConstruct.js:32:59)
at Object.exports.getKafkaClient (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:44:60)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:782:49)
at notifyAllDeviceSubscribers (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:736:13)
at exports.handleDeviceNotification (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:713:17)
at EventSource.eventSource.onmessage (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationStreamManagement.js:153:9)
at EventSource.emit (node:events:513:28)
at _emit (/home/openbackhaul/NotificationProxy/node_modules/eventsource/lib/eventsource.js:287:17)
Error: Too many pending tasks in queue ./database/config.json
at AsyncLock.acquire (/home/openbackhaul/NotificationProxy/node_modules/async-lock/lib/index.js:195:15)
at Object.exports.readFromDatabaseAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/databaseDriver/JSONDriver.js:30:23)
at Function.getLogicalTerminationPointListAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/onfModel/models/ControlConstruct.js:32:59)
at Object.exports.getKafkaClient (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:44:60)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:782:49)
at notifyAllDeviceSubscribers (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:736:13)
at exports.handleDeviceNotification (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:713:17)
at EventSource.eventSource.onmessage (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationStreamManagement.js:153:9)
at EventSource.emit (node:events:513:28)
at _emit (/home/openbackhaul/NotificationProxy/node_modules/eventsource/lib/eventsource.js:287:17)
Error: Too many pending tasks in queue ./database/config.json
at AsyncLock.acquire (/home/openbackhaul/NotificationProxy/node_modules/async-lock/lib/index.js:195:15)
at Object.exports.readFromDatabaseAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/databaseDriver/JSONDriver.js:30:23)
at Function.getLogicalTerminationPointListAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/onfModel/models/ControlConstruct.js:32:59)
at Object.exports.getKafkaClient (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:44:60)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:782:49)
at notifyAllDeviceSubscribers (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:736:13)
at exports.handleDeviceNotification (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:713:17)
at EventSource.eventSource.onmessage (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationStreamManagement.js:153:9)
at EventSource.emit (node:events:513:28)
at _emit (/home/openbackhaul/NotificationProxy/node_modules/eventsource/lib/eventsource.js:287:17)
Error: Too many pending tasks in queue ./database/config.json
at AsyncLock.acquire (/home/openbackhaul/NotificationProxy/node_modules/async-lock/lib/index.js:195:15)
at Object.exports.readFromDatabaseAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/databaseDriver/JSONDriver.js:30:23)
at Function.getLogicalTerminationPointListAsync (/home/openbackhaul/NotificationProxy/node_modules/onf-core-model-ap/applicationPattern/onfModel/models/ControlConstruct.js:32:59)
at Object.exports.getKafkaClient (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:44:60)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:782:49)
at notifyAllDeviceSubscribers (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:736:13)
at exports.handleDeviceNotification (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:713:17)
at EventSource.eventSource.onmessage (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationStreamManagement.js:153:9)
at EventSource.emit (node:events:513:28)
at _emit (/home/openbackhaul/NotificationProxy/node_modules/eventsource/lib/eventsource.js:287:17)
TypeError: Cannot read properties of undefined (reading 'layer-protocol')
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:66:41)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
TypeError: Cannot read properties of undefined (reading 'layer-protocol')
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:66:41)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
TypeError: Cannot read properties of undefined (reading 'layer-protocol')
at Object.exports.getKafkaTopicName (/home/openbackhaul/NotificationProxy/service/individualServices/KafkaHandler.js:66:41)
at sendMessageToKafka (/home/openbackhaul/NotificationProxy/service/individualServices/NotificationManagement.js:783:44)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)

[IswaryaaS]

Fix provided in NP v1.1.0.e_impl

Since it is a performance related issue, it is currently under observation in pre-prod environment with notifications enabled against primary controller with full load of devices.

[Ziabatcha]

This issue needs to be tested after MWDI Issue 1545 delivered.