Crash on query with GROUP BY on a JSON field

[rudyzeinoun]

Bug Description:

This problem has been happening repeatedly on production servers and also on the latest dev build (25.13.5). It starts okay, but after a few weeks of filling up with data, this happens. I copied over the crashed table folder and can reproduce this on my local machine with it.

CREATE TABLE table_name (
id bigint,
email_id string attribute,
sender string attribute,
recipient string attribute,
event_name string attribute,
event_time timestamp,
event_timeofday integer,
event_date integer,
type_id integer,
source string attribute,
properties json secondary_index='1'
)

manticore-1     | ------- FATAL: CRASH DUMP -------
manticore-1     | [Wed May 27 13:55:11.642 2026] [    1]
manticore-1     | 
manticore-1     | --- crashed SphinxQL request dump ---
manticore-1     | SELECT COUNT(*) as `cnt`, COUNT(DISTINCT email_id) AS `unique_count`, properties.url 
manticore-1     | 
manticore-1     | as `url` FROM cms:table_name WHERE source = 'Channel_Post:1939' 
manticore-1     | 
manticore-1     | and event_name = 'click' GROUP BY properties.url LIMIT 1000
manticore-1     | --- request dump end ---
manticore-1     | --- local index:table_name
manticore-1     | Manticore 25.13.5 0796982d0@26052509 dev (columnar 13.3.0 ff802ca@26052410) (secondary 13.3.0 ff802ca@26052410) (knn 13.3.0 ff802ca@26052410) (embeddings 1.1.1 ff802ca@26052410)
manticore-1     | Handling signal 11
manticore-1     | -------------- backtrace begins here ---------------
manticore-1     | Program compiled with Clang 16.0.6
manticore-1     | Configured with flags: Configured with these definitions: -DDISTR_BUILD=jammy -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_JIEBA=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
manticore-1     | Built on Linux x86_64 (jammy) (cross-compiled)
manticore-1     | Stack bottom = 0x794ec0063fc0, thread stack size = 0x20000
manticore-1     | Trying manual backtrace:
manticore-1     | Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
manticore-1     | Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x794ec0060000, stacksize=0x20000)
manticore-1     | Trying system backtrace:
manticore-1     | begin of system symbols:
manticore-1     | Trying boost backtrace:
manticore-1     |  0# sphBacktrace(int, bool) in searchd
manticore-1     |  1# HandleCrash(int) in searchd
manticore-1     |  2# 0x0000795090E59330 in /lib/x86_64-linux-gnu/libc.so.6
manticore-1     |  3# FastPForLib::FastPForImpl<8u, unsigned long>::decodeArray(unsigned int const*, unsigned long, unsigned long*, unsigned long&) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     |  4# unsigned int const* FastPForLib::CompositeCodec<FastPForLib::FastPFor<8u>, FastPForLib::VariableByte>::_decodeArray<unsigned long>(unsigned int const*, unsigned long, unsigned long*, unsigned long&) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     |  5# util::IntCodec_T<util::Int32SVBCodec_c, util::Int64FastPFORCodec_c>::DecodeDelta(util::Span_T<unsigned int> const&, util::SpanResizeable_T<unsigned long>&) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     |  6# SI::BlockValues_T<unsigned long>::Load(unsigned int, util::SpanResizeable_T<unsigned int>&, util::FileReader_c&, util::IntCodec_i*) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     |  7# int SI::BlockReader_T<unsigned long, unsigned long>::LoadBlockAndCreateIterator<SI::BlockReader_T<unsigned long, unsigned long>::CreateBlocksIterator(std::vector<SI::BlockIter_t, std::allocator<SI::BlockIter_t> > const&, common::Filter_t const&, std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&)::{lambda(int, int)#1}&>(int, int, unsigned long, SI::BlockReader_T<unsigned long, unsigned long>::CreateBlocksIterator(std::vector<SI::BlockIter_t, std::allocator<SI::BlockIter_t> > const&, common::Filter_t const&, std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&)::{lambda(int, int)#1}&) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     |  8# void SI::BlockReader_T<unsigned long, unsigned long>::CreateBlocksIterator<SI::BlockReader_T<unsigned long, unsigned long>::CreateBlocksIterator(std::vector<SI::BlockIter_t, std::allocator<SI::BlockIter_t> > const&, common::Filter_t const&, std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&)::{lambda(int, int)#1}>(SI::BlockIter_t const&, SI::BlockReader_T<unsigned long, unsigned long>::CreateBlocksIterator(std::vector<SI::BlockIter_t, std::allocator<SI::BlockIter_t> > const&, common::Filter_t const&, std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&)::{lambda(int, int)#1}&&) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     |  9# SI::BlockReader_T<unsigned long, unsigned long>::CreateBlocksIterator(std::vector<SI::BlockIter_t, std::allocator<SI::BlockIter_t> > const&, common::Filter_t const&, std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&) in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     | 10# SI::SecondaryIndex_c::GetValsRows(std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >*, common::Filter_t const&, SI::IteratorSettings_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     | 11# SI::SecondaryIndex_c::CreateIterators(std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&, common::Filter_t const&, SI::IteratorSettings_t const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const in /usr/share/manticore/modules/lib_manticore_secondary_avx512.so
manticore-1     | 12# SIContainer_c::CreateIterators(std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&, common::Filter_t const&, common::RowidRange_t const*, unsigned int, long, int, bool, CSphString&, CSphString&) const in searchd
manticore-1     | 13# SIIteratorCreator_c::CreateSIIterators(std::vector<common::BlockIterator_i*, std::allocator<common::BlockIterator_i*> >&, CSphFilterSettings const&, long, CSphString&) in searchd
manticore-1     | 14# SIIteratorCreator_c::Create(CSphString&) in searchd
manticore-1     | 15# SIContainer_c::CreateSecondaryIndexIterator(sph::Vector_T<SecondaryIndexInfo_t, sph::DefaultCopy_T<SecondaryIndexInfo_t>, sph::DefaultRelimit, sph::DefaultStorage_T<SecondaryIndexInfo_t> >&, sph::Vector_T<CSphFilterSettings, sph::DefaultCopy_T<CSphFilterSettings>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphFilterSettings> > const&, ESphCollation, ISphSchema const&, unsigned int, int, bool, CSphString&) const in searchd
manticore-1     | 16# CSphIndex_VLN::SpawnIterators(CSphQuery const&, sph::Vector_T<CSphFilterSettings, sph::DefaultCopy_T<CSphFilterSettings>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphFilterSettings> > const&, sph::Vector_T<JsonSIFilterTransform_t, sph::DefaultCopy_T<JsonSIFilterTransform_t>, sph::DefaultRelimit, sph::DefaultStorage_T<JsonSIFilterTransform_t> > const&, CSphQueryContext&, CreateFilterContext_t&, ISphSchema const&, sph::Vector_T<ISphSchema const*, sph::DefaultCopy_T<ISphSchema const*>, sph::DefaultRelimit, sph::DefaultStorage_T<ISphSchema const*> > const&, std::unique_ptr<ISphSchema, std::default_delete<ISphSchema> >&, CSphQueryResultMeta&, int, int, sph::Vector_T<CSphFilterSettings, sph::DefaultCopy_T<CSphFilterSettings>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphFilterSettings> >&, bool, ISphRanker*) const in searchd
manticore-1     | 17# CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const in searchd
manticore-1     | 18# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in searchd
manticore-1     | 19# 0x00005C0DFA5AE25A in searchd
manticore-1     | 20# 0x00005C0DFA6D943D in searchd
manticore-1     | 21# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in searchd
manticore-1     | 22# RtIndex_c::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in searchd
manticore-1     | 23# CSphIndexStub::MultiQueryEx(int, CSphQuery const*, CSphQueryResult*, ISphMatchSorter**, CSphMultiQueryArgs const&) const in searchd
manticore-1     | 24# 0x00005C0DFA198E9A in searchd
manticore-1     | 25# 0x00005C0DFA6D943D in searchd
manticore-1     | 26# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in searchd
manticore-1     | 27# SearchHandler_c::RunLocalSearches() in searchd
manticore-1     | 28# SearchHandler_c::RunSubset(int, int) in searchd
manticore-1     | 29# SearchHandler_c::RunQueries() in searchd
manticore-1     | 30# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in searchd
manticore-1     | 31# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in searchd
manticore-1     | 32# 0x00005C0DFA21D49D in searchd
manticore-1     | 33# LoopClientMySQL(unsigned char&, int, QueryProfile_c*, AsyncNetBuffer_c*) in searchd
manticore-1     | 34# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in searchd
manticore-1     | 35# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in searchd
manticore-1     | 36# 0x00005C0DFA217162 in searchd
manticore-1     | 37# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in searchd
manticore-1     | 38# make_fcontext in searchd
manticore-1     | 
manticore-1     | -------------- backtrace ends here ---------------
manticore-1     | Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
manticore-1     | and attach there:
manticore-1     | a) searchd log, b) searchd binary, c) searchd symbols.
manticore-1     | Look into the chapter 'Reporting bugs' in the manual
manticore-1     | (https://manual.manticoresearch.com/Reporting_bugs)
manticore-1 exited with code 0

Manticore Search Version:

25.13.5 0796982d0@26052509 dev (columnar 13.3.0 ff802ca@26052410) (secondary 13.3.0 ff802ca@26052410) (knn 13.3.0 ff802ca@26052410) (embeddings 1.1.1 ff802ca@26052410)

Operating System Version:

Docker

Have you tried the latest development version?

Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[tomatolog]

could you provide you table_name files as crash related to the data content of that table?

as the crash happens on reading secondary index from the disk chunk of that RT table and it is not clear:

• it could be invalid data
• some runtime error based on that data

you could upload your data as it described in our manual at https://manual.manticoresearch.com/dev/Reporting_bugs#Uploading-your-data

[rudyzeinoun]

I uploaded the table data (~7.5 GB).
I needed to anonymize it first, so I first replaced all recipient values with email@domain.com. The crash still happens after that. For safety, I also dropped the recipient column from the table before uploading. searchd still crashes on any basic SELECT ... WHERE.
Thanks.

[sanikolaev]

Hi @rudyzeinoun

Analysis of the provided .spidx file revealed that:

• it was built by an older version of the secondary library
• it's corrupted which causes the crash. Perhaps the reason is the same as here Crash on SELECT query with GROUP BY #4285

ALTER TABLE ... REBUILD SECONDARY should help.

[rudyzeinoun]

Indeed, I tried to upgrade Manticore to the latest dev build to see if the queries on the table still crash. So the index file was made with Manticore 10. I will upgrade the production server to the latest version, and see if this happens again. If it does on the latest release, should I try to rebuild the index or submit the table files to you?

[sanikolaev]

If it does on the latest release, should I try to rebuild the index or submit the table files to you?

Yes, if it keeps crashing after ALTER ... REBUILD SECONDARY, please let us know.