Bound retry loops and detect reorgs to prevent sync stalls

[pragmaxim]

Refactored the sync coordinator and workers to handle persistent backend failures and chain reorgs more robustly. Previously, the sync process could get stuck in infinite retry loops if the chain tip moved or a block became temporarily unavailable.

• Abstract sync logic into helpers (getBlockHashForSync, waitForBlockWorkers) to reduce duplication between parallel and bulk sync modes.
• Implement deadlock prevention by using non-blocking selects on channel sends to workers.
• Add reorg detection that triggers a sync restart (errResync) when the requested height is found to be past the current chain tip.
• Enhance signal handling to ensure clean exits during worker synchronization.

This solution won over #1490

[Copilot]

Pull request overview

This PR hardens the DB sync coordinator/worker logic to avoid sync stalls caused by unbounded retries when the chain tip moves (reorg/rollback) or the backend temporarily can’t serve a block/hash.

Changes:

• Added bounded retry + “tip fell below requested height” detection in GetBlockHash coordination to trigger errResync instead of retrying indefinitely.
• Refactored worker-coordination error handling via recordBlockWorkerAbort and waitForBlockWorkers to prevent coordinator deadlocks and improve shutdown behavior.
• Added/expanded regression tests for reorg detection, worker abort propagation, and EthereumRPC.GetBlockHash error mapping to bchain.ErrBlockNotFound.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
db/sync.go	Adds helper functions for bounded hash retries, reorg detection, and abort-aware worker waiting; updates parallel/bulk connect loops to use them.
db/sync_test.go	Adds unit regression tests covering resync signaling, abort handling, and deadlock prevention paths.
bchain/coins/eth/ethrpc_blockhash_test.go	Verifies EthereumRPC.GetBlockHash maps “not found” errors to bchain.ErrBlockNotFound for sync retry logic.

Comments suppressed due to low confidence (1)

db/sync.go:695

• getBlockWorker unconditionally increments w.metrics.IndexResyncErrors on retryable GetBlock errors. Since SyncWorker.metrics is a pointer and constructors don’t enforce it being non-nil, this can panic in callers/tests that create a worker without metrics when a retryable error occurs. Either guard this increment with if w.metrics != nil (similar to getBlockHashForSync) or ensure metrics is always initialized before starting workers.

				w.metrics.IndexResyncErrors.With(common.Labels{"error": "failure"}).Inc()
				select {
				case <-terminating:
					return
				case <-w.chanOsSignal:
					return
				case <-time.After(cfg.RetryDelay):

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.