Feat/block proposal spec

[aimendj]

This PR implements the Execution Market — Block Construction Specification and adds an in-memory transaction mempool.

It also integrates the mempool with the node and chain, including proposal reconstruction, validation, transaction pruning, and restoration after reorgs.

Main Changes

Mempool

• Added an in-memory Mempool for Mantle transactions.
• Transactions are currently stored in FIFO order.
• Added a bounded GraceCache for proposal reconstruction.
• Added transaction pruning for committed and expired transactions.
• Proposal selection checks:
  • minimum transaction age
  • execution gas limit
  • current fee market prices
  • transaction balance coverage

Proposal Reconstruction

• Added a dedicated proposal.nim module.
• Compact proposals can be reconstructed using transactions from the mempool/grace cache.
• Reconstructed proposals are validated against the current ledger state.

Chain Integration

• Transactions are removed from the mempool when included in a block.
• Transactions from disconnected blocks are restored after a reorg.
• Transactions from the new chain branch are pruned.

Design Decisions

Dynamic Fee Market

We don't use a priority queue because execution and storage prices are dynamic and can change independently between transaction insertion and proposal construction.

A transaction's effective cost is:

execGas × execPrice + storGas × storPrice

Therefore, there is no stable priority value we can store when the transaction enters the mempool.

For now, transactions are kept in FIFO order and evaluated against the current ledger state during proposal construction.

Note Dependencies

Transactions can depend on notes created by other unconfirmed transactions.

For now, we process them in FIFO order using a temporary working ledger. Transactions that cannot be applied are skipped.

A dependency DAG could be added in a follow-up PR to allow better out-of-order selection.

Limitations

• Proposal reconstruction can fail if a required transaction is no longer in the bounded GraceCache.
• MempoolMinAgeSlots = 3 is currently a placeholder and should be adjusted based on propagation benchmarks.

[github-actions]

Unit Test Results

       5 files  ±  0     670 suites  +5   1m 5s ⏱️ -1s
   612 tests +16     599 ✔️ +16  13 💤 ±0  0 ❌ ±0 
3 220 runs  +80  3 155 ✔️ +80  65 💤 ±0  0 ❌ ±0 

Results for commit df79e03. ± Comparison against base commit 4340747.

This pull request removes 3 and adds 19 tests. Note that renamed tests count towards both.

isFutureDescendantOfImmutable accepts child extending past immutable tip
isFutureDescendantOfImmutable rejects block at or before immutable height
lcaBlockIdAndHeight with tip branch and sibling branch meets at genesis

add enforces monotonic slot order via doAssert
addBlockToTree prunes side-branch nodes below finality height
advanceEpochAndMarket rotates storage market and advances epoch without proof
canExtend accepts child extending past immutable tip
canExtend rejects candidate blocks that do not descend from latestImmutableId
capacity limit evicts oldest tx to graceCache
mandatory_fees for SignedMantleTx combines execution gas and storage gas
mempool lifecycle (add, contains, get, len)
multi-fork reorg correctly handles multiple competing branches
pruneBlockTxs removes committed block transactions
…

♻️ This comment has been updated with latest results.

[tersec]

This doesn't really handle reorgs well -- if some tx's get used by a block, then that block is reorged, then they're back in play. But they're gone from the FIFO.

[tersec]

There's no early exit for these, so every block proposal scans the entire tx pool. While there are other avoidable inefficiencies here which can make that less problematic, there should be some mechanism by which it detects it's sufficiently close to full, that no other transactions in the pool will likely fit, or won't be likely enough to be worth searching, and stop. This could be because cumulativeExecutionGas is already close enough to MAX_EXECUTION_GAS_PER_BLOCK, for example, that it couldn't be improved much.

Because of the shared/diluted/anonymous leader reward it optimizing for the absolute greatest reward doesn't matter anyway. The ROI for doing a truly thorough search is low.

Other block building performance points:

• casual benchmarking suggests that a single Ed25519 signature verification involved in e.g., tryApplyTx (without, e.g., batch verification) on the Zen 5 machine I tried it's 120μs with the current version (and incidentally, BoringSSL provides one which benchmarks at around 90μs on the same machine). This is for ChannelInscribe;
• zkSig/Groth verification looks to be around 1.7ms (~15x slower) on the same machine;
• it's wasteful to actually compute encodeSignedMantleTx(item[].tx) to compute encodeSignedMantleTx(item[].tx).len;
• another case to consider is a tx pool full of invalid signatures which don't apply and don't consume execution or storage tx, which becomes fairly pathological with the current design;
• it's especially wasteful (though, it's not nearly as CPU-intense as these signature verifications) to do so twice, i.e. mandatory_fees immediately does so again;
• and it's even more wasteful to verify signatures of any sort more than once at all across blocks -- once the signature is verified once, it should never have to be verified again. it does not become spontaneously invalid (and I'm not sure if there's any reason to keep invalid tx's in the pool, except to have some memory that they are invalid, so not to waste time verifying them again); and
• the var candidate = workingLedger and similar copies (sink doesn't actually work particularly well in refc, I've tried; with luck it works with move(), but otherwise apparently not at all that I've managed) aren't huge but they do end up copying slightly nontrivial things beyond the HAMT. This looks to be perf noise compared to the rest though, and changes with ORC anyway, where sink does do something useful, unlike refc where it's ... not very functional.

As is, this block building effectively DoSes itself/the node with any reasonable number of pool tx's.

[aimendj]

Yeah, agreed. About workingLedger, the remaining candidate = workingLedger copy is real, but it's needed in case applying the candidate transaction fails. Switching tryApplyTx to var already removes the extra copy inside the call, so I think it's fine to leave this as is unless profiling shows the remaining copy is actually significant.

[aimendj]

perf issues will be managed by PR183 and gossipsub to validate txs at ingress