SIGSEGV in RtIndex_c::GetIndexFiles when querying table.@files under write load

[kino505]

Bug Description:

Summary
searchd crashes with SIGSEGV (signal 11) when an internal SELECT file, size FROM

.@files query is executed against an actively-written real-time table. The query appears to be issued by Manticore Buddy (it is not generated by our application). Crash happens reliably under sustained write load and has been observed multiple times on different nodes of the cluster within a single 24-hour test window.
Environment

Manticore version: 25.13.1 e605e80c5@26051415 dev (columnar 13.2.6 0b37507@26042817, secondary 13.2.6, knn 13.2.6, embeddings 1.1.1)
Buddy version: v3.46.2+26051413-42296708-dev
Built: Linux x86_64 (jammy), cross-compiled with Clang 16.0.6
Cluster: 3-node Galera-replicated cluster, all nodes Primary/Synced
Platform: Kubernetes (EKS)
Workload: sustained ~270 RPS SELECT + ~9 RPS write (REPLACE / UPDATE / DELETE) per node on multiple real-time tables, largest is product_variant (~5M docs)

What happened
Across multiple nodes in a 24-hour window, searchd segfaulted while serving a query to the @files virtual table. The crash dump consistently shows the same SQL and the same stack trace.
Crashed query
sqlSELECT file, size FROM product_variant.@files;SHOW META
This query is not generated by our application — we replay only the application's query.log and there are no @files references in it. The query is presumably issued internally by Manticore Buddy (e.g. for the backup or metrics plugin).
Backtrace
------- FATAL: CRASH DUMP -------
--- crashed SphinxQL request dump ---
SELECT file, size FROM product_variant.@files;SHOW META
--- request dump end ---
--- local index:product_variant.@files
Manticore 25.13.1 e605e80c5@26051415 dev ...
Handling signal 11

Trying system backtrace:
searchd(_Z12sphBacktraceib+0x227)[0x560a13ff4167]
searchd(_Z11HandleCrashi+0x2e3)[0x560a13cde8c3]
/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7f81ee436330]
searchd(+0x129987d)[0x560a13ef187d]
searchd(_ZNK13CSphIndex_VLN13GetIndexFilesERN3sph8Vector_TI10CSphStringNS0_13DefaultCopy_TIS2_EENS0_14DefaultRelimitENS0_16DefaultStorage_TIS2_EEEES9_PK17FilenameBuilder_i+0x87)[0x560a13ef10a7]
searchd(_ZNK9RtIndex_c13GetIndexFilesERN3sph8Vector_TI10CSphStringNS0_13DefaultCopy_TIS2_EENS0_14DefaultRelimitENS0_16DefaultStorage_TIS2_EEEES9_PK17FilenameBuilder_i+0x1ba)[0x560a14108c7a]
searchd(Z17HandleSelectFilesR11RowBuffer_iRK10CSphStringS3+0x116)[0x560a13e69166]
Demangled, the relevant frames are:

CSphIndex_VLN::GetIndexFiles(...)
RtIndex_c::GetIndexFiles(...)
HandleSelectFiles(RowBuffer_i&, CSphString const&, CSphString const&)

Frequency / reproducibility
Within a single 24-hour test window we observed:

Worker-1: 2 crashes from this exact @files query
Worker-2: 4 crashes from this exact @files query
Worker-0: 0 crashes from @files (but crashed once from a different bug, filed as #4577)

Pattern is reproducible — wait long enough under sustained write load and one of the nodes crashes from this query.
What was running at the time
The crashes happen while real-time disk chunks are being actively flushed. Lines like the following appear in the log very close to each crash:
[time] [tid] rt: table product_variant: diskchunk N(M), segments K saved in
0.932771 (0.938771) sec [attrs=836ms deadmap=0ms docs=94ms ...]
RAM saved/new 45507875/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)

[time] [tid] rt: table product_variant: merged chunks .../product_variant.169,
.../product_variant.184 to .../product_variant.185 in 5s ...
Suspicion: GetIndexFiles() enumerates .spa, .spi, .sps, .spb, etc. files from the RT index, but when disk chunks are being merged / renamed concurrently, the file list it captures references files that have already been deleted / superseded — leading to a null-deref or use-after-free.
Reproducer
Sustained write load against a real-time table for several hours, while the Buddy plugin that polls @files is enabled. We don't yet have a minimal reproducer because we don't know which Buddy plugin issues the query — knowing that would let us either trigger it on demand or temporarily disable it as a workaround.
Workaround for users hitting this
None known — the query is internal, applications don't trigger it directly. If Manticore team can confirm which Buddy plugin issues SELECT FROM table.@files periodically, users could disable that plugin via plugins config while waiting for a fix.
Questions for the team

Which Buddy plugin issues SELECT file, size FROM

.@files — backup, metrics, or something else?
Is there a way to disable that single plugin (we see plugins list: core: empty-string, backup, emulate-elastic, fuzzy, create-table, create-cluster, drop, insert, alias, select, replace-select, show, plugin, test, alter-column, alter-distributed-table, alter-rename-table, modify-table, knn, replace, queue, sharding, update, autocomplete, cli-table, distributed-insert, truncate, metrics, conversational-search)?
Is the @files virtual table expected to be safe under concurrent writes, or is this a known limitation?

Additional context
We have a shadow-replay rig that reproduces this reliably and full searchd log captures from each crash event. Happy to share privately if helpful.
Related crash (separate issue): SIGSEGV in Threads::Coro::RWLock_c::Unlock during UPDATE on attribute columns — filed as #4577.

Manticore Search Version:

dev-25.13.1

Operating System Version:

Ubuntu 22.04

Have you tried the latest development version?

Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[kino505]

"Cascade crash on 3-node cluster within seconds (worker-2 → worker-1 → worker-0)"
"Stack corruption: Wrong stack limit or frame pointer (fp=0x20000, stacksize=0x20000) — backtrace not recoverable"
"Triggered by UPDATE on attribute field that has secondary index"
"Resulting in 'Detected wrong count' on restart — real data loss on the recovering node"

[kino505]

After cascade crash, recovery may hang: applier-thread on rejoining node stays in working state for 15+ minutes on a single transaction. SHOW STATUS shows cluster_*_received=2, replicated=0 while recv_queue=21000+ and growing. The recovering node never reaches Synced state and requires manual pod recreation to recover. Suggests deadlock in applier when processing the transactions that originally caused the crash.

[kino505]

After cascade crash recovery, the rejoining node may enter a state where applier-thread hangs indefinitely on a single transaction. Observable as: node_state=joining, local_state=1, recv_queue growing linearly, received=2 (or other very low number) while replicated=0. Repl-thread working for 15+ minutes on single job. All other worker-threads idling. Pod recreation forces fresh SST which completes successfully in 5-7 minutes.

[tomatolog]

could be better to upload searchd.log files from all nodes to line up the crash events

If you could reproduce the crash - please start all nodes with --logreplication cli - this way daemon will post more replication events in the daemon log

[kino505]

Logs from 3 cluster nodes covering 2+ hours of activity including
two cascade crash episodes:

T+0 = 2026-05-18 11:21 UTC (cluster startup)
13:02:19 worker-2 SIGSEGV
13:02:30 worker-1 SIGSEGV (11 seconds after w2)
13:02:42 worker-0 SIGSEGV (12 seconds after w1)
13:03:42 worker-0 restart → "Detected wrong count" in 5 tables:
product_admin_v1 6087 docs missing
product_variant 3143 docs missing
order_package_admin 501 docs missing
favorite_variant_v1 2859 docs missing
favorite_customer_v1 303 docs missing

13:25-13:30 second cascade — same pattern repeated

Common trigger preceding each crash:

• WARNING attribute updated (origin_name, gender_code, categories)
• WARNING secondary index disabled for that attribute
• WARNING "Action message in non-primary configuration" (Galera)
• WARNING "sending install message failed: Resource temporarily unavailable"
• SIGSEGV with stack corruption (fp=0x20000, stacksize=0x20000)
  -> backtrace not recoverable

Recovery: applier-thread on rejoining node may hang for 15+ minutes
with received=2 and recv_queue growing. Requires kubectl delete pod
to force fresh SST, which then completes in 5-7 minutes.

Version: 25.13.1 e605e80c5@26051415 dev

2.gz
1.gz
0.gz

PS:I will try to run with --logreplication cli ASAP