Skip to content

Add deduplication for ttl data - #26

Merged
amishas157 merged 6 commits into
mainfrom
patch/fix-ttl-bug
Jan 27, 2026
Merged

Add deduplication for ttl data#26
amishas157 merged 6 commits into
mainfrom
patch/fix-ttl-bug

Conversation

@amishas157

@amishas157 amishas157 commented Jan 27, 2026

Copy link
Copy Markdown
Contributor

This PR addresses bug reported in #25

This was because a TTL entry can update multiple times within a given ledger entry. We were already deduping contract data entry. This PR adds deduping of ttl entry as well.

Notice keyhash 83c830c6d200adbceda8e72d8204017f781b57e1ec8acf03674388a902732779 in following Example:
Example

                      {
                        "state": {
                          "last_modified_ledger_seq": 896,
                          "data": {
                            "ttl": {
                              "key_hash": "83c830c6d200adbceda8e72d8204017f781b57e1ec8acf03674388a902732779",
                              "live_until_ledger_seq": 1054975
                            }
                          },
                          "ext": "v0"
                        }
                      },
                      {
                        "updated": {
                          "last_modified_ledger_seq": 900,
                          "data": {
                            "ttl": {
                              "key_hash": "83c830c6d200adbceda8e72d8204017f781b57e1ec8acf03674388a902732779",
                              "live_until_ledger_seq": 1054979
                            }
                          },
                          "ext": "v0"
                        }
                      },

Copilot AI review requested due to automatic review settings January 27, 2026 19:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug where multiple TTL (Time To Live) entry updates within a single ledger cause a database constraint violation. The fix adds deduplication logic to remove duplicate TTL entries before database insertion, following the same pattern already established for contract data entries.

Changes:

  • Added deduplication for TTL entries using KeyHash and LedgerSequence fields to prevent "ON CONFLICT DO UPDATE command cannot affect row a second time" database errors

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/transform/ttl.go

// It is possible to have multiple changes to the same ttl entry in a single ledger
// example from testnet data: CDO7SMNK3H2ZTRWSLJOPMGFLHDN5SKNSWJI6CNB2TBXCXAKFC6DPTTZY, 83c830c6d200adbceda8e72d8204017f781b57e1ec8acf03674388a902732779, 901
ttlDataOutputs = utils.RemoveDuplicatesByFields(ttlDataOutputs, []string{"KeyHash", "LedgerSequence"})

Copilot AI Jan 27, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding a test case in ttl_test.go that verifies the deduplication behavior when multiple changes to the same TTL entry occur within a single ledger. This would help prevent regression of the bug described in issue #25. The test could create multiple ingest.Change entries with the same KeyHash but different values (similar to the example in utils/processor_test.go TestRemoveDuplicatesByFields) and verify that only the latest entry is retained.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot open a new pull request to apply changes based on this feedback

Copilot AI commented Jan 27, 2026

Copy link
Copy Markdown
Contributor

@amishas157 I've opened a new pull request, #27, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot AI and others added 4 commits January 27, 2026 19:43
Co-authored-by: amishas157 <4952872+amishas157@users.noreply.github.com>
Co-authored-by: amishas157 <4952872+amishas157@users.noreply.github.com>
Co-authored-by: amishas157 <4952872+amishas157@users.noreply.github.com>
[WIP] Update deduplication implementation for TTL data
@amishas157
amishas157 merged commit 0b7e030 into main Jan 27, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants