Manticoresearch crashed after executing vector search while inserting data

[GreyPlane]

Bug Description:

-- Crash dump

[2026-05-09 10:06:16] fatal runtime error: failed to initiate panic, error 5, aborting
[2026-05-09 10:06:16] ------- FATAL: CRASH DUMP -------
[2026-05-09 10:06:16] [Sat May 9 02:06:16.637 2026] [ 1]
[2026-05-09 10:06:16] --- crashed SphinxQL request dump ---
[2026-05-09 10:06:16]
[2026-05-09 10:06:16] select reservation_uuid from stey_crs_reservation_note
[2026-05-09 10:06:16] where knn(embedding_vector,
[2026-05-09 10:06:16] '124124123')
[2026-05-09 10:06:16] group by reservation_uuid
[2026-05-09 10:06:16] limit 10
[2026-05-09 10:06:16]
[2026-05-09 10:06:16] --- request dump end ---
[2026-05-09 10:06:16] --- local index:stey_crs_reservation_note
[2026-05-09 10:06:16] Manticore 17.5.1 d4cc0969e@26020616 (columnar 10.2.0 9a5dd2f@26020506) (secondary 10.2.0 9a5dd2f@26020506) (knn 10.2.0 9a5dd2f@26020506) (embeddings 1.1.1)
[2026-05-09 10:06:16] Handling signal 6
[2026-05-09 10:06:16] -------------- backtrace begins here ---------------
[2026-05-09 10:06:16] Program compiled with Clang 16.0.6
[2026-05-09 10:06:16] Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_JIEBA=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
[2026-05-09 10:06:16] Built on Linux x86_64 (jammy) (cross-compiled)
[2026-05-09 10:06:16] Stack bottom = 0x7f67980a3fe0, thread stack size = 0x20000
[2026-05-09 10:06:16] Trying manual backtrace:
[2026-05-09 10:06:16] Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
[2026-05-09 10:06:16] Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f67980a0000, stacksize=0x20000)
[2026-05-09 10:06:16] Trying system backtrace:
[2026-05-09 10:06:16] begin of system symbols:
[2026-05-09 10:06:16] Trying boost backtrace:
[2026-05-09 10:06:16] 0# sphBacktrace(int, bool) in searchd
[2026-05-09 10:06:16] 1# HandleCrash(int) in searchd
[2026-05-09 10:06:16] 2# 0x00007F67B5A9C330 in /lib/x86_64-linux-gnu/libc.so.6
[2026-05-09 10:06:16] 3# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
[2026-05-09 10:06:16] 4# gsignal in /lib/x86_64-linux-gnu/libc.so.6
[2026-05-09 10:06:16] 5# abort in /lib/x86_64-linux-gnu/libc.so.6
[2026-05-09 10:06:16] 6# 0x00007F67B4E9FE0A in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 7# 0x00007F67B4E79BE9 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 8# 0x00007F67B4E79C8D in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 9# 0x00007F67B4E79150 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 10# 0x00007F67B4EA02C8 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 11# 0x00007F67B4EA0229 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 12# 0x00007F67B4EA021C in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 13# 0x00007F67B4CF318F in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 14# 0x00007F67B4CFC3B2 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 15# 0x00007F67B4CFC359 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 16# 0x00007F67B4BEB1E3 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 17# 0x00007F67B4BF8B79 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
[2026-05-09 10:06:16] 18# knn::TextToEmbeddings_c::Convert(std::vector<std::basic_string_view<char, std::char_traits >, std::allocator<std::basic_string_view<char, std::char_traits > > > const&, std::vector<std::vector<float, std::allocator >, std::allocator<std::vector<float, std::allocator > > >&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const in /usr/share/manticore/modules/lib_manticore_knn_avx2.so
[2026-05-09 10:06:16] 19# RtIndex_c::SetupAutoEmbeddings(CSphQuery const&, CSphQuery&, ISphSchema const&, CSphString&) const in searchd
[2026-05-09 10:06:16] 20# RtIndex_c::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in searchd
[2026-05-09 10:06:16] 21# CSphIndexStub::MultiQueryEx(int, CSphQuery const*, CSphQueryResult*, ISphMatchSorter**, CSphMultiQueryArgs const&) const in searchd
[2026-05-09 10:06:16] 22# 0x0000561260027B70 in searchd
[2026-05-09 10:06:16] 23# 0x0000561260533E7D in searchd
[2026-05-09 10:06:16] 24# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in searchd
[2026-05-09 10:06:16] 25# SearchHandler_c::RunLocalSearches() in searchd
[2026-05-09 10:06:16] 26# SearchHandler_c::RunSubset(int, int) in searchd
[2026-05-09 10:06:16] 27# SearchHandler_c::RunQueries() in searchd
[2026-05-09 10:06:16] 28# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in searchd
[2026-05-09 10:06:16] 29# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in searchd
[2026-05-09 10:06:16] 30# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in searchd
[2026-05-09 10:06:16] 31# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in searchd
[2026-05-09 10:06:16] 32# 0x000056126009F28F in searchd
[2026-05-09 10:06:16] 33# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in searchd
[2026-05-09 10:06:16] 34# make_fcontext in searchd
[2026-05-09 10:06:16]
[2026-05-09 10:06:16] -------------- backtrace ends here ---------------
[2026-05-09 10:06:16] Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
[2026-05-09 10:06:16] and attach there:
[2026-05-09 10:06:16] a) searchd log, b) searchd binary, c) searchd symbols.
[2026-05-09 10:06:16] Look into the chapter 'Reporting bugs' in the manual
[2026-05-09 10:06:16] (https://manual.manticoresearch.com/Reporting_bugs)

-- Detail

We have an event driven system which inserts record row by row, continuously into manticoresearch, and during this procedure, if executing vector search query to the same table, it will trigger this crash almost constantly.

We could provide more detail if needed.

Manticore Search Version:

starting daemon version '17.5.1 d4cc0969e@26020616 (columnar 10.2.0 9a5dd2f@26020506) (secondary 10.2.0 9a5dd2f@26020506) (knn 10.2.0 9a5dd2f@26020506) (embeddings 1.1.1)'

Operating System Version:

eclipse-temurin:11.0

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[sanikolaev]

Thanks for opening this issue. Is there any easy way to reproduce this issue?

While inserting data

Can you reproduce it if you're not inserting data? If you're inserting data with concurrency > 1?

[GreyPlane]

Thanks for opening this issue. Is there any easy way to reproduce this issue?

While inserting data

Can you reproduce it if you're not inserting data? If you're inserting data with concurrency > 1?

I think I was inserting data with multiple sources concurrently, I will try to create an isolate project to reproduce this problem.

[GreyPlane]

repo

Yes I was able to reproduce this crash by

1. Query while continuously inserting data with concurrency = 1.
2. Just inserting data with concurrency > 1.
3. Just query data with concurrency > 1.

[sanikolaev]

Thanks, I tried to reproduce this locally and so far I could not reproduce it.

What I tested

• exact reported image/version:
  manticoresearch/manticore:17.5.1
  Daemon 17.5.1 d4cc0969e@26020616
  Columnar 10.2.0 9a5dd2f@26020506
  Knn 10.2.0 9a5dd2f@26020506
  Embeddings 1.1.1
• auto-embeddings table using:
  MODEL_NAME='sentence-transformers/all-MiniLM-L6-v2' FROM='title'
• the same 2-arg text KNN form as in the issue, including:

  SELECT reservation_uuid
  FROM ...
  WHERE knn(embedding_vector,'124124123')
  GROUP BY reservation_uuid
  LIMIT 10
  

Stress scenarios I ran

1. query while continuously inserting
2. query-only concurrency
3. insert-only concurrency

Observed results

• mixed inserts + grouped KNN queries on 17.5.1 for 120s:
  ~2500 inserts, ~6000 queries, 0 failures, no crash
• heavier mixed run for 120s:
  ~6200 inserts, ~8700 queries, 0 failures, no crash
• query-only concurrency for 60s:
  ~61000 successful queries, 0 failures
• insert-only concurrency for 60s:
  ~13700 successful inserts, 0 failures

I also checked docker logs after each run and did not get a crash / lost connection / container exit.

So it looks like the current public repro is still missing an important ingredient.

Could you please share one of the following?

• exact CREATE TABLE for embedding_test / stey_crs_reservation_note
• whether the vector column uses only FROM='content' or multiple source fields
• whether the table is rowwise or columnar
• approximate text size per inserted row
• actual concurrency pattern:
  • number of insert workers
  • number of query workers
  • whether each query uses a fresh connection
• if possible, a minimal dataset + exact reproduction script that crashes 17.5.1 consistently

The crash stack points into auto-embeddings generation (TextToEmbeddings_c::Convert / SetupAutoEmbeddings), so the missing factor is likely schema/workload-specific rather than just "concurrent query + insert" by itself.

[GreyPlane]

Interesting, the report was so brief is only bcs it was so easy to reproduce on my side, sry for that, I would make some time to do a comprehensive one.

[GreyPlane]

Firstly the DDL is

CREATE TABLE embedding_test
(
    content text,
    embedding_vector FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2'
        MODEL_NAME='Qwen/Qwen3-Embedding-0.6B' FROM ='content'
); 

Then text size is 10 characters each row I generated every time randomly, I'm not sure how to determine the type of table, could you provide some instruction?

Also I tried different concurrency pattern for query:

1. Pumping 3 different threads firing query at the same time, crashed.
2. Start each one in different timing and continually querying, after all 3 threads fired, it crashed.
3. Just single thread querying once 2 seconds, didn't crash.

Though I'm not 100% sure, but I think most library we've using include a connection pool therefore it should involve some sort of connection reusing.

And there's might be a tricky thing that is since I couldn't get it work in my local environment, I use what have been deployed in our k8s cluster, and in that cluster, the model file was actually downloaded by me and manually put into the model cache directory, I wonder if that could be the issue?

I do agree it looks like other issue than just concurrency problem, could it relates to something about the base image?

[GreyPlane]

Also would you recommend us to try to upgrade to the newest version and see what will happen? I just realized we're far behind the updates.