Asterisk crashes after running for a period of time

[hexuanyi]

Asterisk crashes after running for a period of time

Excessive refcount 100000 reached on ao2 object 0x563b7888d528

[RoEdAl]

Could you tell me more? Is two-month period enough? What version of Asterisk and chan_quectel you're using? What platform does your Asterisk service run on?

[hexuanyi]

Ubuntu22.01 asterisk18.10.1 Compile the latest code used , used is i5,16GB memory ,In 30 concurrent situations,
FRACK may occur irregularly!, Failed assertion Excessive refcount 100000 reached on ao2 object
Then I modified the Asterisk source code

file main/astobj2. c

EXCESSIVE_REF_COUNT 900000
#File include/ostrisk/taskprocessor. h
AST-TASKPROCESSOR-HIGHT-WATER-LEVEL 20000
still reports FRACK error!,
Failed assertion Excessive refcount 900000 reached on ao2 object

[RoEdAl]

You have two options:

1. Build chan_quectel from experimental ao2 branch.
2. Set refdebug = yes in /etc/asterisk/asterisk.conf, restart service and investigate /var/log/asterisk/refs file.

[hexuanyi]

Connected to Asterisk 18.10.0 currently running on asbt (pid = 11502)
-- [quectel0] Opening data port: /dev/ttyUSB2
-- [quectel0] Opening audio port: /dev/ttyUSB1
-- [quectel0] Ports opened, initializing...
core dump

[hexuanyi]

use Build chan_quectel from experimental ao2 branch. core dump

[RoEdAl]

• Change initstate to stop in quectel.conf.
• Reload Asterisk.
• Increase debug level to 4.
• Increase verbose level to 3.
• Run quectel0 manually: quectel start quectel0.

[hexuanyi]

Can you give me a contact information?

[hexuanyi]

Asterisk Ready.
*CLI> core set debug 4
Core debug was OFF and is now 4.
*CLI> core set verbose 3
Console verbose was 8 and is now 3.
*CLI> quectel start quectel0
[quectel0] Start scheduled
*CLI> -- [quectel0] Opening data port: /dev/ttyUSB2
-- [quectel0] Opening audio port: /dev/ttyUSB1
-- [quectel0] Ports opened, initializing...
Segmentation fault (core dumped)

[RoEdAl]

You have to configure Asterisk to dump debug messages to console or check out log file for debug messages.

[hexuanyi]

Can you wait for me for a few minutes?

[hexuanyi]

#0 0x0000555d52c2f28a in __ao2_unlock (user_data=0x7fcca4b5a7b2, file=0x7fcb534e5168 "src/chan_quectel.c", func=0x7fcb534e5db0 <PRETTY_FUNCTION.68> "pvt_taskproc_lock_and_execute", line=686, var=0x7fcb534e553e "ptd->pvt") at astobj2.c:290
#1 0x00007fcb534ce939 in __ao2_unlock_and_unref () from /usr/lib/asterisk/modules/chan_quectel.so
#2 0x00007fcb534bee73 in pvt_taskproc_lock_and_execute () from /usr/lib/asterisk/modules/chan_quectel.so
#3 0x00007fcb534ba6b0 in at_response_taskproc () from /usr/lib/asterisk/modules/chan_quectel.so
#4 0x0000555d52dcc6ee in ast_taskprocessor_execute (tps=0x7fcb58002220) at taskprocessor.c:1302
#5 0x0000555d52dd74a6 in execute_tasks (data=0x7fcb58002220) at threadpool.c:1350
#6 0x0000555d52dcc6ee in ast_taskprocessor_execute (tps=0x555d55818770) at taskprocessor.c:1302
#7 0x0000555d52dd4c42 in threadpool_execute (pool=0x555d55819c00) at threadpool.c:367
#8 0x0000555d52dd6bca in worker_active (worker=0x7fcb3c001160) at threadpool.c:1137
#9 0x0000555d52dd6903 in worker_start (arg=0x7fcb3c001160) at threadpool.c:1056
#10 0x0000555d52de2866 in dummy_start (data=0x7fcb58002760) at utils.c:1607
#11 0x00007fcb9ac64ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
#12 0x00007fcb9acf6850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
this core dump gdb info

[hexuanyi]

[Jun 21 10:34:22] DEBUG[51191]: src/at_queue.c:102 at_queue_add: [quectel0][AT] ↵ [OK] cmds:19 at tail
[Jun 21 10:34:22] DEBUG[51191]: src/at_queue.c:214 at_queue_run: [quectel0][AT] → [AT\r]
[Jun 21 10:34:22] DEBUG[51185]: threadpool.c:536 grow: Increasing threadpool chan-quectel/pool's size by 1
[Jun 21 10:34:22] DEBUG[51185]: threadpool.c:536 grow: Increasing threadpool chan-quectel/pool's size by 1
[Jun 21 10:34:22] DEBUG[51192]: src/at_response.c:2737 show_response: [quectel0][AT] ← [UNK][AT\r]
'Jun 21 10:34:22] DEBUG[51192]: src/at_response.c:3018 at_response: [quectel0] Ignoring unknown result: 'AT

[hexuanyi]

[Jun 21 10:34:13] DEBUG[51117] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:13] DEBUG[51118] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:13] DEBUG[51119] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:13] DEBUG[51113] threadpool.c: Destroying worker thread 2
[Jun 21 10:34:13] DEBUG[51113] threadpool.c: Destroying worker thread 3
[Jun 21 10:34:13] DEBUG[51113] threadpool.c: Destroying worker thread 4
[Jun 21 10:34:14] DEBUG[51114] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:14] DEBUG[51116] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:14] DEBUG[51113] threadpool.c: Destroying worker thread 0
[Jun 21 10:34:14] DEBUG[51113] threadpool.c: Destroying worker thread 1
[Jun 21 10:34:22] DEBUG[51186] src/chan_quectel.c: [dev-manager] Got scan event
[Jun 21 10:34:22] DEBUG[51186] src/chan_quectel.c: [dev-manager][quectel0] Starting device
[Jun 21 10:34:22] DEBUG[51191] src/at_queue.c: [quectel0][AT] ↵ [OK] cmds:19 at tail
[Jun 21 10:34:22] DEBUG[51191] src/at_queue.c: [quectel0][AT] → [AT\r]
[Jun 21 10:34:22] DEBUG[51185] threadpool.c: Increasing threadpool chan-quectel/pool's size by 1
[Jun 21 10:34:22] DEBUG[51185] threadpool.c: Increasing threadpool chan-quectel/pool's size by 1
[Jun 21 10:34:22] DEBUG[51192] src/at_response.c: [quectel0][AT] ← [UNK][AT\r]

[hexuanyi]

[Jun 21 10:33:54] VERBOSE[51111] asterisk.c: Asterisk Ready.
[Jun 21 10:34:13] DEBUG[51117] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:13] DEBUG[51118] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:13] DEBUG[51119] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:13] DEBUG[51113] threadpool.c: Destroying worker thread 2
[Jun 21 10:34:13] DEBUG[51113] threadpool.c: Destroying worker thread 3
[Jun 21 10:34:13] DEBUG[51113] threadpool.c: Destroying worker thread 4
[Jun 21 10:34:14] DEBUG[51114] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:14] DEBUG[51116] threadpool.c: Worker thread idle timeout reached. Dying.
[Jun 21 10:34:14] DEBUG[51113] threadpool.c: Destroying worker thread 0
[Jun 21 10:34:14] DEBUG[51113] threadpool.c: Destroying worker thread 1
[Jun 21 10:34:22] DEBUG[51186] src/chan_quectel.c: [dev-manager] Got scan event
[Jun 21 10:34:22] DEBUG[51186] src/chan_quectel.c: [dev-manager][quectel0] Starting device
[Jun 21 10:34:22] VERBOSE[51186] src/chan_quectel.c: [quectel0] Opening data port: /dev/ttyUSB2
[Jun 21 10:34:22] VERBOSE[51186] src/chan_quectel.c: [quectel0] Opening audio port: /dev/ttyUSB1
[Jun 21 10:34:22] VERBOSE[51186] src/chan_quectel.c: [quectel0] Ports opened, initializing...
[Jun 21 10:34:22] DEBUG[51191] src/at_queue.c: [quectel0][AT] ↵ [OK] cmds:19 at tail
[Jun 21 10:34:22] DEBUG[51191] src/at_queue.c: [quectel0][AT] → [AT\r]
[Jun 21 10:34:22] DEBUG[51185] threadpool.c: Increasing threadpool chan-quectel/pool's size by 1
[Jun 21 10:34:22] DEBUG[51185] threadpool.c: Increasing threadpool chan-quectel/pool's size by 1
[Jun 21 10:34:22] DEBUG[51192] src/at_response.c: [quectel0][AT] ← [UNK][AT\r]
[Jun 21 10:34:22] DEBUG[51192] src/at_response.c: [quectel0] Ignoring unknown result: 'AT

[RoEdAl]

Sync your repo, rebuild and run again.