feat(fill): enable shared pre-allocation groups and add BlockchainEngineReorgFixture (ethereum#1706)

* feat(fill): enable interface for two-phase shared alloc filling

* feat(fixtures,types): add shared pre-allocation data models

- Add `SharedPreStateGroup` and `SharedPreState` models.
- Support grouping tests by (fork, environment) hash.
- Enable serialization for phase 1/2 coordination.

* feat(fixtures): add `BlockchainEngineReorgFixture` and common base class

- Add `BlockchainEngineFixtureCommon` base class for shared functionality.
- Add `BlockchainEngineReorgFixture` with `post_state_diff` field.
- Remove unnecessary backwards compatibility validator from common class.
- Update `test_collect_only` framework test for new fixture.

* feat(fixtures): export new fixture classes and shared models

- Export `BlockchainEngineReorgFixture` and `BlockchainEngineFixtureCommon`.
- Export `SharedPreState` and `SharedPreStateGroup` models.
- Enable imports for shared pre-allocation functionality.

* feat(specs): add shared pre-allocation support to `BaseTest`

- Add `update_shared_pre_state()` and `compute_shared_pre_alloc_hash()`.
- Add `get_genesis_environment()` for polymorphic environment access.
- Support both `StateTest` and `BlockchainTest` formats.

* feat(fill): implement shared pre-allocation (`fill` pytest hooks)

- Add `pytest_sessionstart` and `pytest_sessionfinish` for phase coordination.
- Add `calculate_post_state_diff()` for memory-efficient storage.
- Add custom terminal summary with group statistics.
- Load/save shared pre-allocation state.

* feat(fixtures): enable global address allocation and shared pre-alloc path

- Add `shared_prealloc_path` property to `FixtureOutput`.
- Extend global address iterators to all test formats.
- Make the fixture scope of address iterators dynamic: session-scoped for shared pre-alloc generation, function-scoped for regular pre-alloc.
- Update and improve `pre_alloc` tests.

* feat(specs): add `BlockchainEngineReorgFixture` generation support

- Support generating reorg fixtures from blockchain tests.
- Enable shared pre-allocation integration with blockchain specs.

* feat(fixtures): add `pre_hash` field to test case index

- Support pre-allocation hash tracking in index files.
- Enable fixture consumption with shared pre-allocation metadata.

* chore(fill): only generate engine_reorg fixtures w/shared-alloc flags

* feat(fill): add a new pytest mark for custom prealloc group control

* chore(tests): apply `prealloc_group` to system contract & requests tests

* fix(tests): prevent iterator state persistence in beacon root tests during two-phase execution

The beacon root tests were failing when run with two-phase shared pre-allocation because they used `itertools.count()` objects directly in pytest parametrization. These iterator objects maintain internal state that persists across test phases when run in the same Python process.

Changes:
- Replace direct `count()` objects with factory functions that return fresh iterators.
- Update `beacon_roots` fixture to create new iterator instances on each invocation.
- Modify test parametrization to use `timestamps_factory` instead of `timestamps`.

This bug only manifested in two-phase execution because:
1. In regular single-phase filling, each test gets its own fresh Python process.
2. In two-phase execution within one process, the same parametrized iterator.
   objects are reused, causing timestamps to continue from where phase 1 left off.
3. The second phase would see timestamps like 21000 instead of starting at 1000.

The fix ensures each test invocation gets fresh iterators with reset state, preventing cross-phase contamination while maintaining the same test behavior.

* chore(cli): add `groupstats` script to analyze shared pre-allocation groups

Add a new CLI tool that provides comprehensive analysis of shared pre-allocation groups generated by the test framework. The tool helps identify optimization opportunities for client teams by analyzing group distributions, test coverage, and identifying problematic test functions that create multiple size-1 groups.

Key features:
- Display overall statistics (groups, tests, accounts)
- Show distribution by fork with average tests per group
- Analyze group size frequency distribution
- Calculate test coverage impact by group size
- List test modules by execution complexity (group count)
- Identify split test functions with Groups/Fork ratio analysis
- Progressive disclosure with -v and -vv verbosity levels

The Groups/Fork ratio metric distinguishes between necessary multi-fork coverage and problematic parameter-heavy tests, enabling targeted optimization strategies for CI workflows.

* docs(consume): add `BlockchainEngineReorgFixture` docs

* refactor(all): standardize shared pre-allocation naming conventions

- CLI flags: `--generate-shared-alloc` → `--generate-shared-pre`
- CLI flags: `--use-shared-alloc` → `--use-shared-pre`
- Property: `shared_prealloc_path` → `shared_pre_alloc_path`
- Pytest marker: `prealloc_group` → `pre_alloc_group`
- Entry point: `show_pre_alloc_groups` → `groupstats`

* docs: update changelog

* fix(fixtures/filler): Pydantic fixes

* feat(filler): pre_alloc.json is now a folder

* fix(cli): groupstats

* fix: tox

* feat: Use deterministic addresses in pre-alloc

* fix: xdist

* feat: genesis header in pre_alloc files

* fix: tox

* fix: move pre

---------

Co-authored-by: Mario Vega <[email protected]>

Author: danceratopz

CHANGELOG.md

@@ -31,6 +31,7 @@ Users can select any of the artifacts depending on their testing needs for their
 
 - ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository [#1590](https://github.com/ethereum/execution-spec-tests/pull/1590).
 - ✨ Add a new pytest plugin, `ported_tests`, that lists the static fillers and PRs from `ported_from` markers for use in the coverage Github Workflow [#1634](https://github.com/ethereum/execution-spec-tests/pull/1634).
+- ✨ Enable two-phase filling of fixtures with shared pre-allocation groups and add a `BlockchainEngineReorgFixture` format [#1606](https://github.com/ethereum/execution-spec-tests/pull/1706).
 - 🔀 Refactor: Encapsulate `fill`'s fixture output options (`--output`, `--flat-output`, `--single-fixture-per-file`) into a `FixtureOutput` class ([#1471](https://github.com/ethereum/execution-spec-tests/pull/1471),[#1612](https://github.com/ethereum/execution-spec-tests/pull/1612)).
 - ✨ Don't warn about a "high Transaction gas_limit" for `zkevm` tests ([#1598](https://github.com/ethereum/execution-spec-tests/pull/1598)).
 - 🐞 `fill` no longer writes generated fixtures into an existing, non-empty output directory; it must now be empty or `--clean` must be used to delete it first ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)).

navigation.md

@@ -41,6 +41,7 @@
           * [State Tests](running_tests/test_formats/state_test.md)
           * [Blockchain Tests](running_tests/test_formats/blockchain_test.md)
           * [Blockchain Engine Tests](running_tests/test_formats/blockchain_test_engine.md)
+          * [Blockchain Engine Reorg Tests](running_tests/test_formats/blockchain_test_engine_reorg.md)
           * [EOF Tests](running_tests/test_formats/eof_test.md)
           * [Transaction Tests](running_tests/test_formats/transaction_test.md)
           * [Common Types](running_tests/test_formats/common_types.md)

running_tests/test_formats/blockchain_test_engine_reorg.md

@@ -0,0 +1,145 @@
+# Blockchain Engine Reorg Tests  <!-- markdownlint-disable MD051 (MD051=link-fragments "Link fragments should be valid") -->
+
+The Blockchain Engine Reorg Test fixture format tests are included in the fixtures subdirectory `blockchain_tests_engine_reorg`, and use Engine API directives with optimized shared pre-allocation for improved execution performance.
+
+These are produced by the `StateTest` and `BlockchainTest` test specs when using the `--generate-shared-pre` and `--use-shared-pre` flags.
+
+## Description
+
+The Blockchain Engine Reorg Test fixture format is an optimized variant of the [Blockchain Engine Test](./blockchain_test_engine.md) format designed for large-scale test execution with performance optimizations.
+
+It uses the Engine API to test block validation and consensus rules while leveraging **shared pre-allocation state** to significantly reduce test execution time and resource usage. Tests are grouped by their initial state (fork + environment + pre-allocation) and share common genesis states through blockchain reorganization.
+
+The key optimization is that **clients need only be started once per group** instead of once per test (as in the original engine fixture format), dramatically improving execution performance for large test suites.
+
+Instead of including large pre-allocation state in each test fixture, this format references a shared pre-allocation folder (`pre_alloc`) which includes all different pre-allocation combinations used for any test fixture group.
+
+A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`ReorgFixture`](#reorgfixture) test object, with the key string representing the test name.
+
+The JSON file path plus the test name are used as the unique test identifier.
+
+## Shared Pre-Allocation File
+
+The `blockchain_tests_engine_reorg` directory contains a special directory `pre_alloc` that stores shared pre-allocation state file used by all tests in this format, one per pre-allocation group with the name of the pre-alloc hash. This folder is essential for test execution and must be present alongside the test fixtures.
+
+### Pre-Allocation File Structure
+
+Each file in the `pre_alloc` folder corresponds to a pre-allocation hash to shared state groups:
+
+```json
+{
+   "test_count": 88,
+   "pre_account_count": 174,
+   "testIds": ["test1", "test2", ...],
+   "network": "Prague",
+   "environment": { ... },
+   "pre": { ... }
+}
+```
+
+#### SharedPreStateGroup Fields
+
+- **`test_count`**: Number of tests sharing this pre-allocation group
+- **`pre_account_count`**: Number of accounts in the shared pre-allocation state
+- **`testIds`**: Array of test identifiers that use this shared state
+- **`network`**: Fork name (e.g., "Prague", "Cancun")
+- **`environment`**: Complete [`Environment`](./common_types.md#environment) object with execution context
+- **`pre`**: Shared [`Alloc`](./common_types.md#alloc-mappingaddressaccount) object containing initial account states
+
+## Consumption
+
+For each [`ReorgFixture`](#reorgfixture) test object in the JSON fixture file, perform the following steps:
+
+1. **Load Shared Pre-Allocation**:
+   - Read the appropriate file from the `pre_alloc` folder in the same directory
+   - Locate the shared state group using [`preHash`](#-prehash-string)
+   - Extract the `pre` allocation and `environment` from the shared group
+
+2. **Initialize Client**:
+   - Use [`network`](#-network-fork) to configure the execution fork schedule
+   - Use the shared `pre` allocation as the starting state
+   - Use the shared `environment` as the execution context
+   - Use [`genesisBlockHeader`](#-genesisblockheader-fixtureheader) as the genesis block header
+
+3. **Execute Engine API Sequence**:
+   - For each [`FixtureEngineNewPayload`](#fixtureenginenewpayload) in [`engineNewPayloads`](#-enginenewpayloads-listfixtureenginenewpayload):
+     1. Deliver the payload using `engine_newPayloadVX`
+     2. Validate the response according to the payload's expected status
+   - If [`syncPayload`](#-syncpayload-optionalfixtureenginenewpayload) is present, execute it for chain synchronization
+
+4. **Verify Final State**:
+   - Compare the final chain head against [`lastblockhash`](#-lastblockhash-hash)
+   - If [`postStateDiff`](#-poststatediff-optionalalloc) is present:
+     - Apply the state differences to the shared pre-allocation
+     - Verify the resulting state matches the client's final state
+   - If `post` field were present (not typical), verify it directly
+
+## Structures
+
+### `ReorgFixture`
+
+#### - `network`: [`Fork`](./common_types.md#fork)
+
+##### TO BE DEPRECATED
+
+Fork configuration for the test.
+
+This field is going to be replaced by the value contained in `config.network`.
+
+#### - `preHash`: `string`
+
+Hash identifier referencing a shared pre-allocation group in the `pre_alloc` folder. This hash uniquely identifies the combination of fork, environment, and pre-allocation state shared by multiple tests.
+
+#### - `genesisBlockHeader`: [`FixtureHeader`](./blockchain_test.md#fixtureheader)
+
+Genesis block header. The state root in this header must match the state root calculated from the shared pre-allocation referenced by [`preHash`](#-prehash-string).
+
+#### - `engineNewPayloads`: [`List`](./common_types.md#list)`[`[`FixtureEngineNewPayload`](#fixtureenginenewpayload)`]`
+
+List of `engine_newPayloadVX` directives to be processed after the genesis block. These define the sequence of blocks to be executed via the Engine API.
+
+#### - `syncPayload`: [`Optional`](./common_types.md#optional)`[`[`FixtureEngineNewPayload`](#fixtureenginenewpayload)`]`
+
+Optional synchronization payload used for blockchain reorganization scenarios. When present, this payload is typically used to sync the chain to a specific state before or after the main payload sequence.
+
+#### - `lastblockhash`: [`Hash`](./common_types.md#hash)
+
+Hash of the last valid block after all payloads have been processed, or the genesis block hash if all payloads are invalid.
+
+#### - `postStateDiff`: [`Optional`](./common_types.md#optional)`[`[`Alloc`](./common_types.md#alloc-mappingaddressaccount)`]`
+
+State differences from the shared pre-allocation state after test execution. This optimization stores only the accounts that changed, were created, or were deleted during test execution, rather than the complete final state.
+
+To reconstruct the final state:
+
+1. Start with the shared pre-allocation from the `pre_alloc` folder
+2. Apply the changes in `postStateDiff`:
+   - **Modified accounts**: Replace existing accounts with new values
+   - **New accounts**: Add accounts not present in pre-allocation  
+   - **Deleted accounts**: Remove accounts (represented as `null` values)
+
+#### - `config`: [`FixtureConfig`](#fixtureconfig)
+
+Chain configuration object to be applied to the client running the blockchain engine reorg test.
+
+### `FixtureConfig`
+
+#### - `network`: [`Fork`](./common_types.md#fork)
+
+Fork configuration for the test. It is guaranteed that this field contains the same value as the root field `network`.
+
+#### - `blobSchedule`: [`BlobSchedule`](./common_types.md#blobschedule-mappingforkforkblobschedule)
+
+Optional; present from Cancun on. Maps forks to their blob schedule configurations as defined by [EIP-7840](https://eips.ethereum.org/EIPS/eip-7840).
+
+### `FixtureEngineNewPayload`
+
+Engine API payload structure identical to the one defined in [Blockchain Engine Tests](./blockchain_test_engine.md#fixtureenginenewpayload). Includes execution payload, versioned hashes, parent beacon block root, validation errors, version, and error codes.
+
+## Usage Notes
+
+- This format is only generated when using `--generate-shared-pre` and `--use-shared-pre` flags
+- The `pre_alloc` folder is essential and must be distributed with the test fixtures
+- Tests are grouped by identical (fork + environment + pre-allocation) combinations
+- The format is optimized for Engine API testing (post-Paris forks)
+- Reorganization scenarios are supported through the `forkChoiceUpdate` mechanism