[etherscan] Fix pagination for accounts with many transactions#1071
Open
stek29 wants to merge 1 commit into
Open
[etherscan] Fix pagination for accounts with many transactions#1071stek29 wants to merge 1 commit into
stek29 wants to merge 1 commit into
Conversation
Etherscan is reducing the free-tier maximum records per request to 1000, so the plugin should use offset=1000 while respecting the separate empirical result-window limit where page x offset must stay at or below 10000. Larger pages reduce the number of API calls needed for large accounts, which speeds up loading and helps stay within free-plan request limits. Reference: https://docs.etherscan.io/changelog#upcoming-change-reduced-maximum-records-per-request-on-the-free-api-tier With offset=1000, page 10 is the last safe page. The previous implementation used page as a global cursor, so accounts with many transactions could eventually request page 11 and fail with the Etherscan result-window error. Dense ranges also risked missing rows if pagination advanced with lastBlock - 1 because the final page can end in the middle of a block. Add a shared block-cursor paginator for Etherscan txlist and tokentx requests. It reads pages 1 through 10 inside each cursor window, advances endblock to the last seen block with overlap, deduplicates locally by endpoint-specific keys, validates blockNumber before cursoring, and surfaces clear errors when safe progress is impossible. It also treats known result-window and query-timeout errors as cursor-window exhaustion when a safe last block has already been observed. Fix a separate stack overflow found during a two-year range scrape by replacing large-array spread appends with a stack-safe appendTransactions helper. This avoids transactions.push(...largeArray), which can throw RangeError: Maximum call stack size exceeded for large converted transaction batches, especially under the ZenPlugins dev Promise mock that resolves callbacks synchronously. Add regression coverage for cursor pagination, defensive Etherscan error handling, malformed block numbers, token dedupe keys, and large transaction batch appending.
Contributor
Author
|
Before this PR accounts with more than 10000 transactions couldn't be loaded at all. They'd crash the plugin. But even for smaller accounts this PR speeds up loading by using 1000 as page size instead of 100. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Etherscan is reducing the free-tier maximum records per request to 1000, so the plugin should use offset=1000 while respecting the separate empirical result-window limit where page x offset must stay at or below 10000.
Larger pages reduce the number of API calls needed for large accounts, which speeds up loading and helps stay within free-plan request limits.
Reference: https://docs.etherscan.io/changelog#upcoming-change-reduced-maximum-records-per-request-on-the-free-api-tier
With offset=1000, page 10 is the last safe page. The previous implementation used page as a global cursor, so accounts with many transactions could eventually request page 11 and fail with the Etherscan result-window error. Dense ranges also risked missing rows if pagination advanced with lastBlock - 1 because the final page can end in the middle of a block.
Add a shared block-cursor paginator for Etherscan txlist and tokentx requests. It reads pages 1 through 10 inside each cursor window, advances endblock to the last seen block with overlap, deduplicates locally by endpoint-specific keys, validates blockNumber before cursoring, and surfaces clear errors when safe progress is impossible. It also treats known result-window and query-timeout errors as cursor-window exhaustion when a safe last block has already been observed.
Fix a separate stack overflow found during a two-year range scrape by replacing large-array spread appends with a stack-safe appendTransactions helper. This avoids transactions.push(...largeArray), which can throw RangeError: Maximum call stack size exceeded for large converted transaction batches, especially under the ZenPlugins dev Promise mock that resolves callbacks synchronously.
Add regression coverage for cursor pagination, defensive Etherscan error handling, malformed block numbers, token dedupe keys, and large transaction batch appending.