Add deduplication for ttl data - #26
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug where multiple TTL (Time To Live) entry updates within a single ledger cause a database constraint violation. The fix adds deduplication logic to remove duplicate TTL entries before database insertion, following the same pattern already established for contract data entries.
Changes:
- Added deduplication for TTL entries using KeyHash and LedgerSequence fields to prevent "ON CONFLICT DO UPDATE command cannot affect row a second time" database errors
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| // It is possible to have multiple changes to the same ttl entry in a single ledger | ||
| // example from testnet data: CDO7SMNK3H2ZTRWSLJOPMGFLHDN5SKNSWJI6CNB2TBXCXAKFC6DPTTZY, 83c830c6d200adbceda8e72d8204017f781b57e1ec8acf03674388a902732779, 901 | ||
| ttlDataOutputs = utils.RemoveDuplicatesByFields(ttlDataOutputs, []string{"KeyHash", "LedgerSequence"}) |
There was a problem hiding this comment.
Consider adding a test case in ttl_test.go that verifies the deduplication behavior when multiple changes to the same TTL entry occur within a single ledger. This would help prevent regression of the bug described in issue #25. The test could create multiple ingest.Change entries with the same KeyHash but different values (similar to the example in utils/processor_test.go TestRemoveDuplicatesByFields) and verify that only the latest entry is retained.
There was a problem hiding this comment.
@copilot open a new pull request to apply changes based on this feedback
|
@amishas157 I've opened a new pull request, #27, to work on those changes. Once the pull request is ready, I'll request review from you. |
Co-authored-by: amishas157 <4952872+amishas157@users.noreply.github.com>
Co-authored-by: amishas157 <4952872+amishas157@users.noreply.github.com>
Co-authored-by: amishas157 <4952872+amishas157@users.noreply.github.com>
[WIP] Update deduplication implementation for TTL data
This PR addresses bug reported in #25
This was because a TTL entry can update multiple times within a given ledger entry. We were already deduping contract data entry. This PR adds deduping of ttl entry as well.
Notice keyhash
83c830c6d200adbceda8e72d8204017f781b57e1ec8acf03674388a902732779in following Example:Example