Skip to content

[7/n] [installinator] fix reports getting delayed - #8041

Merged
sunshowers merged 8 commits into
mainfrom
sunshowers/spr/7n-installinator-fix-reports-getting-delayed
Apr 26, 2025
Merged

sunshowers merged 8 commits into
mainfrom
sunshowers/spr/7n-installinator-fix-reports-getting-delayed

Conversation

@sunshowers

@sunshowers sunshowers commented Apr 24, 2025 •

Copy link
Copy Markdown
Contributor

There's been a long-standing issue with the installinator where reports to wicketd get delayed quite substantially. That happens because we use one task to send reports out to every peer, blocking until all of those peers come back.

I'd assumed in the past that if installinator reached out to an unreachable peer, it would receive a TCP connection refused message, but that isn't the case -- instead, it times out. This causes reports to only be sent out roughly every 15 seconds which isn't ideal.

To fix this issue, spin up separate report tasks for each peer. Introduce separate tasks for:

  • discovery (make this a persistent task that publishes updates to a watch channel)
  • peer reconciliation (new peers have new report tasks spun up, while removed peers' report tasks are cancelled)
  • reporting (each peer now has its own report loop)

Also simulate some kinds of network flakiness in our property-based tests.

I did a mupdate on dublin and saw that installinator reports started coming through every 2 seconds rather than every 15 or so, as expected.

Depends on:

Created using spr 1.3.6-beta.1
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1

@jgallagher jgallagher left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

Comment thread installinator/src/reporter.rs Outdated
Comment thread installinator/src/reporter.rs Outdated
Comment thread installinator/src/reporter.rs
Comment thread installinator/src/reporter.rs
Comment thread installinator/src/reporter.rs Outdated
Comment thread installinator/src/reporter.rs Outdated
Comment thread installinator/src/reporter.rs
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
@sunshowers
sunshowers changed the base branch from sunshowers/spr/main.7n-installinator-fix-reports-getting-delayed to main April 26, 2025 00:18
Created using spr 1.3.6-beta.1
@sunshowers
sunshowers merged commit 4a0e9c1 into main Apr 26, 2025
@sunshowers
sunshowers deleted the sunshowers/spr/7n-installinator-fix-reports-getting-delayed branch April 26, 2025 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants