Crash Sphinx 2.4.1 (Manticore)

[iivanov]

Have crash and strange log from manticore. I din't find CRASH DUMP END in log.

------- FATAL: CRASH DUMP -------
[Sun Nov 12 09:31:16.536 2017] [27499]

--- crashed SphinxQL request dump ---
SELECT * FROM section_2020 WHERE id in (15893587) LIMIT 20000 OPTION max_matches=20000
--- request dump end ---
Sphinx 2.4.1 3b31a97@171017 id64-release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DMYSQL_CONFIG_EXECUTABLE=/bin/mysql_config -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_ODBC=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON
Host OS is Linux 01ea43a41e6c 4.8.0-45-generic #48~16.04.1-Ubuntu SMP Fri Mar 24 12:46:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f23e9ffaeff, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x4)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x4, stack=0x7f23ea000000, stacksize=0x100000)
Trying system backtrace:
[Sun Nov 12 09:31:17.337 2017] [27498] watchdog: main process 27499 killed dirtily with signal 11, will be restarted
[Sun Nov 12 09:31:17.338 2017] [27498] watchdog: main process 24605 forked ok
[Sun Nov 12 09:31:17.387 2017] [24605] listening on all interfaces, port=9312
[Sun Nov 12 09:31:17.387 2017] [24605] listening on all interfaces, port=9306
[Sun Nov 12 09:31:27.539 2017] [24605] WARNING: wordlist size mismatch (size=18, checkpoints=0)
[Sun Nov 12 09:31:48.719 2017] [24605] last message repeated 5 times
[Sun Nov 12 09:31:48.719 2017] [24605] binlog: replaying log /var/lib/sphinx//binlog.043

Find error in /var/log/messages:
kernel: : [423829.349058] searchd[27523]: segfault at 54 ip 00007f24f8f034d3 sp 00007f23e97d5fb8 error 4 in libstdc++.so.6.0.19[7f24f8e8e000+e9000]

I tried to execute the query once more, it was executed correctly.

[tomatolog]

Could you start your daemon with --coredump option and enable core file generation at box via command
ulimit -c 128000000000 for user that starts daemon?

This way next time daemon crashed it saves core file that might help of crash investigation.

Could you also check your section_2020 index with following command

indextool -c your.conf --check section_2020

to make sure that index is valid.

[iivanov]

Thanks. I add coredump and up ulimit.

I didn't find any erros in index:

indextool -c /etc/sphinx/sphinx.conf --check section_2020
Manticore 2.4.1 3b31a97@171017 id64-release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)

using config file '/etc/sphinx/sphinx.conf'...
checking index 'section_2020'...
checking RT segment 0(26)...
checking RT segment 1(26)...
checking RT segment 2(26)...
checking RT segment 3(26)...
checking RT segment 4(26)...
checking RT segment 5(26)...
checking RT segment 6(26)...
checking RT segment 7(26)...
checking RT segment 8(26)...
checking RT segment 9(26)...
checking RT segment 10(26)...
checking RT segment 11(26)...
checking RT segment 12(26)...
checking RT segment 13(26)...
checking RT segment 14(26)...
checking RT segment 15(26)...
checking RT segment 16(26)...
checking RT segment 17(26)...
checking RT segment 18(26)...
checking RT segment 19(26)...
checking RT segment 20(26)...
checking RT segment 21(26)...
checking RT segment 22(26)...
checking RT segment 23(26)...
checking RT segment 24(26)...
checking RT segment 25(26)...
checking disk chunk 0(1)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.7 sec elapsed
check passed, 0.7 sec elapsed

[airolg]

So let's close the issue than?

[tomatolog]

we might wait for crash with core to get more data or close in case no crashes happen

[iivanov]

I got few crashes at night (crash.log, last was:

------- FATAL: CRASH DUMP -------
[Thu Nov 16 08:38:18.261 2017] [ 3948]

--- crashed SphinxQL request dump ---
SELECT `id`, `parent_id`
                FROM section_651
                WHERE
`id` in (16041574,16041749,16041798,16041861,16041945,16042085,16042204,16042253,16042281,
16042421,16042533,16042547,16042617,16042715,16042750,16042813,16042820,16042981,16043149,
16043198,16043219,16043254,16043520,16043653,16043723,16043730,16043793,16043807,16043856,
16043870,16077704,16077710,16077716,16077770,16077782,16077854,16077860,16077878,16077884,
16078052,16078058,16078064,16099706,16099712,16099742,16099748,16099754,16099760,16099874,
16099952,16100276,16100282,16100288,16100300,16100324,16102712,16102718,16102724,16102742,
16102748,16102760,16102766,16102772,16102778,16102784,16102790,16179800,16179836,16180664,
16180772,16181660,16182464,16182524,16182542,16182566,16183796,16771604,16771610,16902140,
16902182,16902206,16902212,16902218,16902242,16902272,16902296,16902314,16902380,16902386,
16902392,16950824,16951268,16951280,17010806,17010902,17011928,17011946,17011976,17012024,
17012078,17012084,17012180,17012228,17012336,17012342,17012402,17012408,17012516,17012540,
17012552,17012564,17118272,17120379,17128142,17128177,17128408,17128415,17128422,17128674,
17129612,17129626,17129647,17129703,17129787,17129815,17129997,17130326,17130501,17134946,
17134960,17134981,17135142,17135149,17135163,17135184,17135429,17135443,17135590,17135604,
17135639,17135779,17135793,17135800,17136017,17136024,17136101,17136213,17136325,17136339,
17136486,17136493,17136892,17136976,17137053,17137228,17137249,17141659,17141666,17141834,
17142394,17146671,17147266,17147952,17148729,17148953,17151928,17152341,17152411,17153440,
17155554,17155694,17157248,17157318,17157542,17157570,17157941,17157962,17158032,17158207,
17158991,17159586,17159656,17160195,17161161,17161504,17220493,17220521,17225435,17225449,
17225456,17225463,17225470,17225477,17225484,17225491,17225498,17225505,17225512,17225519,
17225526,17225533,17225547,17225554,17225561,17225568,17225582,17225589,17225596,17225603,
17225610,17225617,17225624,17225631,17280154,17392014,17392616,17392665,17392679,17393491,
17393582,17394338,17394345,17394394,17394443,17402738,17402745,17402801,17403403,17403431,
17403459,17523551,17523600,17717274,17717298,17717304,17717310,17717364,17717370,17717376,
17718522,17718528,17718534,17718540,17718546,17718552,17718558,17721816,17721822,17721828,
17721834,17721840,17721846) AND `parent_id` != 1162030
                LIMIT 20000
OPTION max_matches=20000
--- request dump end ---
Sphinx 2.4.1 3b31a97@171017 id64-release
Handling signal 7
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DMYSQL_CONFIG_EXECUTABLE=/bin/mysql_config -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_ODBC=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON
Host OS is Linux 01ea43a41e6c 4.8.0-45-generic #48~16.04.1-Ubuntu SMP Fri Mar 24 12:46:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f75351f7eff, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x4)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x4, stack=0x7f7535200000, stacksize=0x100000)
Trying system backtrace:
begin of manual symbols:
647dea
4cd0f0
7f764a4ca5e0
6f72f0
6f7603
580d2b
5a4cfc
758ae4
7384db
50868b
50dee9
50f3f9
4f1886
53df3b
519238
519e34
6501e5
64f0ff
7f764a4c2e25
7f76495b634d
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (http://sphinxsearch.com/bugs) and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(/usr/share/doc/sphinx/sphinx.txt or http://sphinxsearch.com/docs/current.html#reporting-bugs)
--- BT to source lines (depth 20): ---
??:?
??:?
??:0
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:0
??:0
--- BT to source lines finished ---
--- 1 active threads ---
thd 0, proto sphinxql, state query, command select
------- CRASH DUMP END -------
[Thu Nov 16 08:38:19.179 2017] [15134] watchdog: main process 3948 killed dirtily with signal 7, will be restarted
[Thu Nov 16 08:38:19.180 2017] [15134] watchdog: main process 22887 forked ok
[Thu Nov 16 08:38:19.197 2017] [22887] listening on all interfaces, port=9312
[Thu Nov 16 08:38:19.198 2017] [22887] listening on all interfaces, port=9306
[Thu Nov 16 08:38:20.800 2017] [22887] FATAL: disk chunk /var/lib/sphinx/rtsection_651.6335: prealloc failed: failed to open /var/lib/sphinx/rtsection_651.6335.sph: No such file or directory
[Thu Nov 16 08:38:21.055 2017] [15134] watchdog: main process 22887 exited cleanly (exit code 1), shutting down

index 651 got broken after the last crash. After the search end the following is written
[Thu Nov 16 08:38:20.800 2017] [22887] FATAL: disk chunk /var/lib/sphinx/rtsection_651.6335: prealloc failed: failed to open /var/lib/sphinx/rtsection_651.6335.sph: No such file or directory
I found lost file in old files

ls /var/lib/sphinx/ |grep section_651.*.old
rtsection_651.3198.old.spa
rtsection_651.3198.old.spd
rtsection_651.3198.old.spe
rtsection_651.3198.old.sph
rtsection_651.3198.old.spi
rtsection_651.3198.old.spk
rtsection_651.3198.old.spm
rtsection_651.3198.old.spp
rtsection_651.3198.old.sps
rtsection_651.6335.old.spa
rtsection_651.6335.old.spd
rtsection_651.6335.old.spe
rtsection_651.6335.old.sph
rtsection_651.6335.old.spi
rtsection_651.6335.old.spk
rtsection_651.6335.old.spm
rtsection_651.6335.old.spp
rtsection_651.6335.old.sps 

and coppied it to file without old. After this manipulation searchd started correctly and now works without mistakes.

I check index after hot fix, indextool say that all ok:

indextool -c /etc/sphinx/sphinx_check.conf --check section_651
Manticore 2.4.1 3b31a97@171017 id64-release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)

using config file '/etc/sphinx/sphinx_check.conf'...
checking index 'section_651'...
checking disk chunk 0(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 3.1 sec elapsed
checking disk chunk 1(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 2(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 3(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 4(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 5(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 6(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 7(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 8(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 9(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 10(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
checking disk chunk 11(12)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.1 sec elapsed
check passed, 4.6 sec elapsed

[tomatolog]

could you load core files into gdb like this

$> gdb /path_to_searchd
gdb> core-file /path_to_core_file

then provide output of following command?

gdb> thread apply all backtrace

as source of crash is not clear from crash logs you provided.

I also wandering on RT index with old extension do you issue optimize command for this index? Could you provide excerpts from log file prior to crash I'd like check that might be messages related to crash.

[iivanov]

I catch the similar problem in second server (crash dump below). After crash we get mistakes like this:

[Thu Nov 16 08:39:17.533 2017] [26545] WARNING: rt optimize: index section_651: failed to merge /var/lib/sphinx/rtsection_651.6333 to /var/lib/sphinx/rtsection_651.6337 (error failed to open /var/lib/sphinx/rtsection_651.6333.spi: No such file or directory)

We have 4 servers. 2 of them in sphinx (Sphinx 2.2.11-id64-release) and 2 in (Manticore 2.4.1). These servers have the same data in db and the same sent queries (logical replication). At night 2 manticore servers caught mistakes and crashed rtsection_651 index. Sphinx servers were ok.
We didn't get similar problem before. But in last build we increased count of select queries to rtsection_651. And at night we usually get a lot of updates. I think this a reason why we have started getting the mistake.

------- FATAL: CRASH DUMP -------
[Thu Nov 16 05:47:10.990 2017] [ 4164]

--- crashed SphinxQL request dump ---
SELECT parent_id FROM section_651 WHERE status in (1) AND status_inherited in (1)
AND sell_status in (0, 1, 2, 3, 4, 7) AND  MATCH ('FNAMproducerVVAL11917ENVAL FNAMproducerVVAL11917ENVAL
')  GROUP BY parent_id LIMIT 3000 OPTION  max_matches=3000
--- request dump end ---
Sphinx 2.4.1 3b31a97@171017 id64-release
Handling signal 7
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DMYSQL_CONFIG_EXECUTABLE=/bin/mysql_config -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_ODBC=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON
Host OS is Linux 01ea43a41e6c 4.8.0-45-generic #48~16.04.1-Ubuntu SMP Fri Mar 24 12:46:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7fa985c92eff, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x4)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x4, stack=0x7fa985c90000, stacksize=0x100000)
Trying system backtrace:
begin of manual symbols:
647dea
4cd0f0
7faaa7a9f5e0
569c32
58613a
6fe750
7051b6
70b13d
70a95d
70cb91
72700e
70dc1e
586d94
5a4abf
758ae4
7384db
50868b
50dee9
50f3f9
4f1886
53df3b
519238
519e34
6501e5
64f0ff
7faaa7a97e25
7faaa6b8b34d
-------------- backtrace ends here ---------------
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(/usr/share/doc/sphinx/sphinx.txt or http://sphinxsearch.com/docs/current.html#reporting-bugs)
--- BT to source lines (depth 27): ---
??:?
??:?
??:0
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:0
??:0
--- BT to source lines finished ---
--- 2 active threads ---
thd 0, proto sphinxql, state query, command select
thd 1, proto sphinxql, state query, command select
------- CRASH DUMP END -------
------- FATAL: CRASH DUMP -------
[Thu Nov 16 05:47:10.990 2017] [ 4164]

--- crashed SphinxQL request dump ---
SELECT * FROM section_651 WHERE `id` in (10996864,10996871,11729953,11729995) LIMIT
20000 OPTION max_matches=20000
--- request dump end ---
Sphinx 2.4.1 3b31a97@171017 id64-release
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.8.5
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.18 -DMYSQL_CONFIG_EXECUTABLE=/bin/mysql_config -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_ODBC=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON
Host OS is Linux 01ea43a41e6c 4.8.0-45-generic #48~16.04.1-Ubuntu SMP Fri Mar 24 12:46:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7fa984c52eff, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x4)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x4, stack=0x7fa984c50000, stacksize=0x100000)
Trying system backtrace:
begin of manual symbols:
647dea
4cd0f0
7faaa7a9f5e0
7faaa7a9b05b
64fcb9
50dc8f
50f3f9
4f1886
53df3b
519238
519e34
6501e5
64f0ff
7faaa7a97e25
7faaa6b8b34d
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (http://sphinxsearch.com/bugs) and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(/usr/share/doc/sphinx/sphinx.txt or http://sphinxsearch.com/docs/current.html#reporting-bugs)
--- BT to source lines (depth 15): ---
[Thu Nov 16 05:47:11.581 2017] [2933] watchdog: main process 4164 killed dirtily with signal 11, will be restarted
[Thu Nov 16 05:47:11.582 2017] [2933] watchdog: main process 26496 forked ok

[tomatolog]

that might be issue with Sphinx RT indexes along with optimize that breaks RT index.

Here is PR sphinxsearch/sphinx#39 and bug report http://sphinxsearch.com/bugs/view.php?id=2590

I think copy RT index from Sphinx might cause issue or however from crash report I still see no root of issue - could you provide output of gdb? or provide symbols from searchd binary to decode addresses at crash log?

[iivanov]

We install manticore from this doc: http://docs.manticoresearch.com/latest/html/installation.html#installing-manticore-packages-on-redhat-and-centos
using rmp:
manticore-2.4.1_171017.3b31a97-1.x86_64

I coudn't get gdb. I catch mistake

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/searchd...(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install manticore-2.4.1_171017.3b31a97-1.x86_64

[iivanov]

sory. I am installing debuginfo-install manticore-2.4.1_171017.3b31a97-1.x86_64 now

[tomatolog]

oh I see in case you installed our package I will look for for debug symbols at our server.
I was sure that you built daemon at your box.

[tomatolog]

I decoded BT from crash log and see that daemon crashes on searching disk chunk of RT index.
Could you restart your daemons with --logdebug option then in case of crash provide excerpt from search log prior to crash I want to see information on file operations by daemon prior to crash.

Could you also provide these entries from daemon log there crash happens

cat searchd.log | grep "optimized chunk"

Do you able to install symbols? I'd also want to see output of thread apply all backtrace

In case it crashes alot and that related to optimize operation you also might try another optimize method by setting progressive_merge option at common section of config, like

common
{
   progressive_merge = 1
}

it disabled by default

[iivanov]

I add --logdebug

I added a log of the whole day searchdcrash.log

progressive_merge was disabled. I will try to enable it.

I will add output of thread apply all backtrace tomorrow.

[iivanov]

I caught the problem again.
Here is interesting part of log on my opinion

Sat Nov 18 06:36:10.129 2017] [29495] DEBUG: Hashing docinfo
[Sat Nov 18 06:36:10.132 2017] [29495] DEBUG: Preread successfully finished, hash=89
[Sat Nov 18 06:36:10.132 2017] [29495] DEBUG: Base set to /var/lib/sphinx/rtsection_651.6391.old
[Sat Nov 18 06:36:10.132 2017] [29495] DEBUG: Base set to /var/lib/sphinx/rtsection_651.6391
[Sat Nov 18 06:36:10.134 2017] [29495] DEBUG: Unlocking the index (lock /var/lib/sphinx/rtsection_651.6391.old.spl)
[Sat Nov 18 06:36:10.136 2017] [29495] DEBUG: Unlocking the index (lock /var/lib/sphinx/rtsection_651.6387.spl)
[Sat Nov 18 06:36:10.152 2017] [29495] DEBUG: Progressive merge - merging 2 (30225 kb) with 4 (30148 kb)
[Sat Nov 18 06:36:10.337 2017] [29495] DEBUG: CSphIndex_VLN::Preread invoked '/var/lib/sphinx/rtsection_651.6386.tmp'
[Sat Nov 18 06:36:10.339 2017] [29495] DEBUG: Hashing docinfo
[Sat Nov 18 06:36:10.341 2017] [29495] DEBUG: Preread successfully finished, hash=180
[Sat Nov 18 06:36:10.341 2017] [29495] DEBUG: Base set to /var/lib/sphinx/rtsection_651.6385.old
[Sat Nov 18 06:36:10.341 2017] [29495] DEBUG: Base set to /var/lib/sphinx/rtsection_651.6385
[Sat Nov 18 06:36:10.344 2017] [29495] DEBUG: Unlocking the index (lock /var/lib/sphinx/rtsection_651.6385.old.spl)
[Sat Nov 18 06:36:10.346 2017] [29495] DEBUG: Unlocking the index (lock /var/lib/sphinx/rtsection_651.6386.spl)
[Sat Nov 18 06:36:10.356 2017] [29495] DEBUG: Progressive merge - merging 2 (28168 kb) with 4 (0 kb)
[Sat Nov 18 06:36:10.524 2017] [29495] WARNING: rt optimize: index section_651: failed to merge /var/lib/sphinx/rtsection_651.6386 to /var/lib/sphinx/rtsection_651.6385 (error failed to open /var/lib/sphinx/rtsection_651.6386.spi: No such file or directory)

The full day logs
problem.log.tar.gz