Skip to content

[etherscan] Fix pagination for accounts with many transactions#1071

Open
stek29 wants to merge 1 commit into
zenmoney:masterfrom
stek29:bump-etherscan-page-size
Open

[etherscan] Fix pagination for accounts with many transactions#1071
stek29 wants to merge 1 commit into
zenmoney:masterfrom
stek29:bump-etherscan-page-size

Conversation

@stek29

@stek29 stek29 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Etherscan is reducing the free-tier maximum records per request to 1000, so the plugin should use offset=1000 while respecting the separate empirical result-window limit where page x offset must stay at or below 10000.

Larger pages reduce the number of API calls needed for large accounts, which speeds up loading and helps stay within free-plan request limits.

Reference: https://docs.etherscan.io/changelog#upcoming-change-reduced-maximum-records-per-request-on-the-free-api-tier

With offset=1000, page 10 is the last safe page. The previous implementation used page as a global cursor, so accounts with many transactions could eventually request page 11 and fail with the Etherscan result-window error. Dense ranges also risked missing rows if pagination advanced with lastBlock - 1 because the final page can end in the middle of a block.

Add a shared block-cursor paginator for Etherscan txlist and tokentx requests. It reads pages 1 through 10 inside each cursor window, advances endblock to the last seen block with overlap, deduplicates locally by endpoint-specific keys, validates blockNumber before cursoring, and surfaces clear errors when safe progress is impossible. It also treats known result-window and query-timeout errors as cursor-window exhaustion when a safe last block has already been observed.

Fix a separate stack overflow found during a two-year range scrape by replacing large-array spread appends with a stack-safe appendTransactions helper. This avoids transactions.push(...largeArray), which can throw RangeError: Maximum call stack size exceeded for large converted transaction batches, especially under the ZenPlugins dev Promise mock that resolves callbacks synchronously.

Add regression coverage for cursor pagination, defensive Etherscan error handling, malformed block numbers, token dedupe keys, and large transaction batch appending.

Etherscan is reducing the free-tier maximum records per request to 1000, so the plugin should use offset=1000 while respecting the separate empirical result-window limit where page x offset must stay at or below 10000.

Larger pages reduce the number of API calls needed for large accounts, which speeds up loading and helps stay within free-plan request limits.

Reference: https://docs.etherscan.io/changelog#upcoming-change-reduced-maximum-records-per-request-on-the-free-api-tier

With offset=1000, page 10 is the last safe page. The previous implementation used page as a global cursor, so accounts with many transactions could eventually request page 11 and fail with the Etherscan result-window error. Dense ranges also risked missing rows if pagination advanced with lastBlock - 1 because the final page can end in the middle of a block.

Add a shared block-cursor paginator for Etherscan txlist and tokentx requests. It reads pages 1 through 10 inside each cursor window, advances endblock to the last seen block with overlap, deduplicates locally by endpoint-specific keys, validates blockNumber before cursoring, and surfaces clear errors when safe progress is impossible. It also treats known result-window and query-timeout errors as cursor-window exhaustion when a safe last block has already been observed.

Fix a separate stack overflow found during a two-year range scrape by replacing large-array spread appends with a stack-safe appendTransactions helper. This avoids transactions.push(...largeArray), which can throw RangeError: Maximum call stack size exceeded for large converted transaction batches, especially under the ZenPlugins dev Promise mock that resolves callbacks synchronously.

Add regression coverage for cursor pagination, defensive Etherscan error handling, malformed block numbers, token dedupe keys, and large transaction batch appending.
@stek29

stek29 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Before this PR accounts with more than 10000 transactions couldn't be loaded at all. They'd crash the plugin.

But even for smaller accounts this PR speeds up loading by using 1000 as page size instead of 100.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant