Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions tests/benchmark/stateful/vector_storage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Vector Storage Benchmark

Single contract design for minimal-overhead storage benchmarking with parametrized operations.

## Design Philosophy

This implementation uses **ONE contract** that:
- Is pre-deployed with 500 filled storage slots
- Accepts parameters via calldata for flexible operation
- Performs storage operations with minimal loop overhead (~30-40 gas per slot)

## Contract Architecture

### Calldata Layout (32-byte aligned)
```
Bytes 0-31: Number of slots to write (uint256)
Bytes 32-63: Starting slot index (uint256)
Bytes 64-95: Value to write (uint256)
```

### Pre-deployed State
- **500 slots** pre-filled with value `0xDEADBEEF`
- Enables testing all transition types without redeployment
- Single contract instance for all benchmarks

## Three Test Scenarios

### 1. Cold Write (0 → non-zero)
- **Start**: Slot 500 (beyond prefilled range)
- **Operation**: Write new value to empty slots
- **Gas Cost**: ~20,000 gas per slot (most expensive)
- **Use Case**: Initial storage allocation

### 2. Storage Clear (non-zero → 0)
- **Start**: Slot 0 (within prefilled range)
- **Operation**: Write zeros to clear existing values
- **Gas Cost**: ~2,900 gas per slot + refund
- **Use Case**: Storage cleanup/deletion

### 3. Warm Update (non-zero → non-zero)
- **Start**: Slot 0 (within prefilled range)
- **Operation**: Update existing values
- **Gas Cost**: ~2,900 gas per slot
- **Use Case**: Typical storage updates

## Implementation Details

### Loop Overhead Breakdown
```
Per iteration:
- DUP operations: 12 gas (4 × 3 gas)
- ADD operation: 3 gas
- LT comparison: 3 gas
- ISZERO: 3 gas
- JUMPI/JUMP: 18 gas (8 + 10)
- SSTORE: Variable (based on transition)
Total overhead: ~39 gas per slot
```

### Contract Size
- Basic loop contract: ~100 bytes
- Well under EIP-3860 limit (24,576 bytes)
- Can handle thousands of operations per call

## Gas Cost Summary

| Transition Type | Gas per Slot | Notes |
|----------------|--------------|-------|
| 0 → non-zero | 20,000 | Cold slot, initial write |
| non-zero → 0 | 2,900 + refund | Max 20% refund of total gas |
| non-zero → non-zero | 2,900 | Warm slot update |
| Loop overhead | ~39 | Minimal per-slot overhead |

## Running the Benchmarks

### Generate Test Fixtures
```bash
# Run benchmarks with stateful marker
uv run fill tests/benchmark/stateful/vector_storage/test_vector_storage.py \
-m stateful --fork Prague
```

### Execute Against Client
```bash
# Direct execution
uv run consume direct --input fixtures/ --client-bin /path/to/geth

# Via Engine API
uv run consume engine --input fixtures/ --engine-endpoint http://localhost:8551
```

## Advantages Over Multiple Contracts

1. **Single Deployment**: One contract handles all scenarios
2. **Consistent Overhead**: Same loop structure for all operations
3. **Flexible Testing**: Parametrized via calldata
4. **Realistic Patterns**: Mimics real smart contract storage patterns
5. **Accurate Benchmarking**: Minimal overhead ensures accurate gas measurements

## Test Parameters

- **num_slots**: [1, 10, 50, 100, 200] - Number of slots per operation
- **batch_size**: [10, 25, 50] - For batch operation tests
- **Pre-filled slots**: 500 - Consistent across all tests

## Example Usage

The contract can be called with raw calldata:
```python
# Write 100 values starting at slot 500
calldata = (
(100).to_bytes(32, 'big') + # num_slots
(500).to_bytes(32, 'big') + # start_slot
(0x1234).to_bytes(32, 'big') # value
)
```

This design provides the most accurate storage benchmarking with minimal overhead and maximum flexibility.
4 changes: 4 additions & 0 deletions tests/benchmark/stateful/vector_storage/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
"""
Vector storage benchmarks for measuring SLOAD/SSTORE operations with
minimal overhead.
"""
207 changes: 207 additions & 0 deletions tests/benchmark/stateful/vector_storage/test_vector_storage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
"""
abstract: Vector storage benchmark with single parametrized contract.

This test uses a single contract that accepts parameters via calldata to
benchmark different storage state transitions with minimal overhead:
- 0 -> non-zero (cold write to empty slots)
- non-zero -> non-zero (warm update of existing slots)
- non-zero -> 0 (clearing slots with gas refund)

The contract is pre-deployed with 500 filled slots to enable all scenarios.
"""

import pytest
from execution_testing import (
Account,
Alloc,
Block,
BlockchainTestFiller,
Bytecode,
Fork,
Op,
Storage,
Transaction,
)

REFERENCE_SPEC_GIT_PATH = "DUMMY/vector_storage.md"
REFERENCE_SPEC_VERSION = "1.0"

# Pre-fill configuration
PREFILLED_SLOTS = 500
PREFILLED_VALUE = 0xDEADBEEF


def create_storage_contract() -> Bytecode:
"""
Creates a storage benchmark contract that accepts parameters
via calldata.

Calldata layout (32-byte aligned):
- Bytes 0-31: Number of slots to write
- Bytes 32-63: Starting slot index
- Bytes 64-95: Value to write

The contract uses a minimal overhead loop to perform storage operations.
Stack layout: [bottom] num_slots, start_slot, value, counter [top]
"""
bytecode = Bytecode()

# Load parameters from calldata
bytecode += Op.PUSH1(0) + Op.CALLDATALOAD # num_slots at bytes 0-31
bytecode += Op.PUSH1(32) + Op.CALLDATALOAD # start_slot at bytes 32-63
bytecode += Op.PUSH1(64) + Op.CALLDATALOAD # value at bytes 64-95
Comment on lines +50 to +52
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small suggestion:

Suggested change
bytecode += Op.PUSH1(0) + Op.CALLDATALOAD # num_slots at bytes 0-31
bytecode += Op.PUSH1(32) + Op.CALLDATALOAD # start_slot at bytes 32-63
bytecode += Op.PUSH1(64) + Op.CALLDATALOAD # value at bytes 64-95
bytecode += Op.CALLDATALOAD(0) # num_slots at bytes 0-31
bytecode += Op.CALLDATALOAD(32) # start_slot at bytes 32-63
bytecode += Op.CALLDATALOAD(64) # value at bytes 64-95


# Initialize counter to 0
bytecode += Op.PUSH1(0) # counter = 0

# Main loop start (JUMPDEST at offset 11)
bytecode += Op.JUMPDEST # Loop start marker

# Check loop condition: counter < num_slots
bytecode += Op.DUP4 # Get num_slots (position 4 from top)
bytecode += Op.DUP2 # Duplicate counter (position 2 from top after DUP4)
bytecode += Op.LT # counter < num_slots?
bytecode += Op.ISZERO # Negate for JUMPI (jump if >=)

# Jump to exit if counter >= num_slots
bytecode += Op.PUSH1(31) # Push exit offset
bytecode += Op.JUMPI # Exit if counter >= num_slots

# Store operation: store value at (start_slot + counter)
bytecode += Op.DUP1 # Duplicate counter
bytecode += Op.DUP4 # Get start_slot (position 4 from top after DUP1)
bytecode += Op.ADD # slot = start_slot + counter
bytecode += Op.DUP3 # Get value (position 3 from top after ADD)
bytecode += Op.SWAP1 # Put slot on top for SSTORE
bytecode += Op.SSTORE # Store value at slot

# Increment counter
bytecode += Op.PUSH1(1) # Push 1
bytecode += Op.ADD # counter++

# Jump back to loop start
bytecode += Op.PUSH1(11) # Push loop start offset
bytecode += Op.JUMP # Jump to loop start

# Exit point (JUMPDEST at offset 31)
bytecode += Op.JUMPDEST # Exit marker
bytecode += Op.STOP # Stop execution

return bytecode


@pytest.mark.valid_from("Prague")
@pytest.mark.stateful # Mark as stateful instead of benchmark
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@pytest.mark.stateful # Mark as stateful instead of benchmark

All the tests under statful/ folder would be marked as stateful test! No need to specify them.

@pytest.mark.parametrize("num_slots", [1, 10, 50, 100, 200])
@pytest.mark.parametrize(
"transition_type,start_offset,write_value",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we still need transition_type parametrization? It seems similar to the id, and this value is not referenced in the test implementation.

[
# Write to empty slots (beyond the 500 prefilled)
pytest.param(
"zero_to_nonzero",
PREFILLED_SLOTS, # Start at slot 500
0x1234,
id="0_to_nonzero"
),

# Clear existing slots (write 0 to prefilled slots)
pytest.param(
"nonzero_to_zero",
0, # Start at slot 0
0, # Write zeros
id="nonzero_to_0"
),

# Update existing slots (change value of prefilled slots)
pytest.param(
"nonzero_to_nonzero",
0, # Start at slot 0
0xBEEFBEEF, # Different value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For maximal load we want to use the longest values here (0xffff..ff) and to ensure clients have to hash everything, it would also help if we would change the value for each slot (so start at 0xff..ff and subtract one each time) (Can be a different PR since this needs changes to the contract code)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that changing the value will imply extra costs. Though we can pass the different values via calldata maybe. Let me see what to do here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it cost extra?

id="nonzero_to_nonzero"
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add the situation zero to zero here also? This might sound weird, but because some clients are slow on reading non-existent storage slots, this might be a bad situation for those.

],
)
def test_storage_transitions_benchmark(
blockchain_test: BlockchainTestFiller,
pre: Alloc,
fork: Fork,
num_slots: int,
transition_type: str,
start_offset: int,
write_value: int,
) -> None:
Comment on lines +124 to +132
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused variable

Suggested change
def test_storage_transitions_benchmark(
blockchain_test: BlockchainTestFiller,
pre: Alloc,
fork: Fork,
num_slots: int,
transition_type: str,
start_offset: int,
write_value: int,
) -> None:
def test_storage_transitions_benchmark(
blockchain_test: BlockchainTestFiller,
pre: Alloc,
num_slots: int,
start_offset: int,
write_value: int,
) -> None:

"""
Benchmark storage operations using a single parametrized contract.

The contract is pre-deployed with 500 filled slots and accepts
parameters via calldata to perform different storage operations:

1. 0 to nonzero: Writing to cold, empty slots (most expensive)
2. nonzero to 0: Clearing existing slots (provides gas refund)
3. nonzero to nonzero: Updating warm slots (moderate cost)
"""
sender = pre.fund_eoa(10**18)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sender = pre.fund_eoa(10**18)
sender = pre.fund_eoa()


# Pre-fill storage with 500 slots
initial_storage = Storage()
for i in range(PREFILLED_SLOTS):
initial_storage[i] = PREFILLED_VALUE

# Deploy the optimized contract with prefilled storage
storage_contract = pre.deploy_contract(
code=create_storage_contract(),
storage=initial_storage,
)

# Prepare calldata with parameters
calldata = (
num_slots.to_bytes(32, 'big') + # Number of slots to write
start_offset.to_bytes(32, 'big') + # Starting slot index
write_value.to_bytes(32, 'big') # Value to write
)

# Create transaction to call the contract
# Use a reasonable gas limit that covers the operation
# Each SSTORE costs up to 20,000 gas for cold writes plus overhead
gas_limit = 21000 + (num_slots * 50000) # Base tx cost + storage operations
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious why specifying the gas limit here, will it be affected if we make it the default value?

If necessary, i would suggest to use:

Suggested change
gas_limit = 21000 + (num_slots * 50000) # Base tx cost + storage operations
intrinsic_gas_cost_calc = fork.transaction_intrinsic_cost_calculator()
gas_limit = intrinsic_gas_cost_calc() + (num_slots * 50000) # Base tx cost + storage operations


tx = Transaction(
to=storage_contract,
gas_limit=gas_limit,
data=calldata,
sender=sender,
)

# Calculate expected post-state storage
expected_storage = Storage()

# Start with the initial prefilled storage
for i in range(PREFILLED_SLOTS):
expected_storage[i] = PREFILLED_VALUE

# Apply the storage modifications
for i in range(num_slots):
slot = start_offset + i

if write_value == 0 and slot < PREFILLED_SLOTS:
# Clearing a prefilled slot (nonzero->0)
# Remove from expected storage (becomes 0)
if slot in expected_storage:
del expected_storage[slot]
elif write_value != 0:
# Writing a non-zero value
expected_storage[slot] = write_value
# If write_value == 0 and slot >= PREFILLED_SLOTS,
# we're writing 0 to an already empty slot (no change)

post = {
storage_contract: Account(storage=expected_storage),
}

blockchain_test(
pre=pre,
blocks=[Block(txs=[tx])],
post=post,
)


Loading