Skip to content

Conversation

sunshowers
Copy link
Contributor

@sunshowers sunshowers commented Apr 24, 2025

There's been a long-standing issue with the installinator where reports to wicketd get delayed quite substantially. That happens because we use one task to send reports out to every peer, blocking until all of those peers come back.

I'd assumed in the past that if installinator reached out to an unreachable peer, it would receive a TCP connection refused message, but that isn't the case -- instead, it times out. This causes reports to only be sent out roughly every 15 seconds which isn't ideal.

To fix this issue, spin up separate report tasks for each peer. Introduce separate tasks for:

  • discovery (make this a persistent task that publishes updates to a watch channel)
  • peer reconciliation (new peers have new report tasks spun up, while removed peers' report tasks are cancelled)
  • reporting (each peer now has its own report loop)

Also simulate some kinds of network flakiness in our property-based tests.

I did a mupdate on dublin and saw that installinator reports started coming through every 2 seconds rather than every 15 or so, as expected.

Depends on:

Created using spr 1.3.6-beta.1
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
Copy link
Contributor

@jgallagher jgallagher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this!

Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1
@sunshowers sunshowers changed the base branch from sunshowers/spr/main.7n-installinator-fix-reports-getting-delayed to main April 26, 2025 00:18
Created using spr 1.3.6-beta.1
@sunshowers sunshowers merged commit 4a0e9c1 into main Apr 26, 2025
17 checks passed
@sunshowers sunshowers deleted the sunshowers/spr/7n-installinator-fix-reports-getting-delayed branch April 26, 2025 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants