rfc: Forge Instances - #28
Conversation
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
|
||
| This is the instance our paying customers use. | ||
|
|
||
| - **Update frequency:** TBD. Once per month? When is the scheduled maintenance window for the central and individual regions? |
There was a problem hiding this comment.
Maybe reword as "upgrade window"? It's not that we must update once a month, but the proposal is to have a regular window to upgrade in should we want to? Right?
I'm curious why we need to have a monthly window for this? I would prefer to release on our own schedule at least initially.
There was a problem hiding this comment.
Great feedback!
Yes, we typically don't have to upgrade every month.
In my experience, it's easier to upgrade on a regular cadence than to pick the upgrade window each time. With a regular cadence, we don't need to spend time deciding when to upgrade, and we get plenty of practice with the actual work because we do it often. So even if it's not necessary to upgrade in each window, I advocate that we still do it.
What are your concerns? What are the benefits of releasing on our own schedule?
I have two concerns about irregular releases:
- Someone will need to remember to schedule the next upgrade.
- The bigger the release, the higher the risk that something breaks. With a big delta, it's more difficult to identify the exact cause.
If you are concerned about shipping partial features, I propose using feature flags to control which feature is available where. We have had a good experience with this approach in FilOne.
There was a problem hiding this comment.
I'm curious why we need to have a monthly window for this?
Right now, every upgrade requires a short downtime.
- If we aim for 99.9% availability SLA, we can afford around 40-45 minutes of downtime each month.
- Downtime is disruptive to our customers; we should handle maintenance in a predictable way that allows our customers to plan around scheduled downtimes.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6949892ddf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| light testing, but not performance/load testing. Light monitoring if any at all. | ||
| - **Real vs test money:** Stripe test cards, Filecoin calibnet. | ||
| - **Data rention:** Weekly network reset on Sunday morning UTC. | ||
| - **Regions:** Single region (us-east-9). |
There was a problem hiding this comment.
We may want to have at least 2 regions in dev, so that we can test multi-region support in central components. Thoughts?
There was a problem hiding this comment.
Not initially. There is nothing to test(?).
There was a problem hiding this comment.
How about verifying whether a single S3 access key can be scoped to buckets across multiple regions?
| This is a stable "preview" instance showing the latest & greatest features, suitable for customer | ||
| demos. Not used for load/performance testing to avoid degraded performance during demos. | ||
|
|
||
| - **Update frequency:** Every Monday morning UTC. Can be rescheduled ad-hoc in case of a customer demo planned for Monday. |
There was a problem hiding this comment.
Do we have a stable network every Monday? My gut instinct is to release this whenever we believe we have a stable network, not on a weekly cadence.
There was a problem hiding this comment.
I see your point.
My concerns with "release whenever we believe we have a stable network":
- If we don't release on a regular basis, then we need someone to remember to periodically ask whether we have a stable network. In my experience, this is a slippery slope leading to no releases for weeks.
- IMO, we should treat the dev network the same way as CI. If it's broken, then it's the team's P0 priority to fix it ASAP. An unstable network on the day a scheduled upgrade was planned should be a rare exception, and we should iterate on improving our practices to make it less and less likely over time.
BTW, we can use feature flags to ship code for partially-implemented features before we make them available to users.
How about releasing biweekly on Mondays when a sprint ends?
- The network should be stable by the end of the sprint; otherwise, we cannot claim our work is "done done".
- This gives us a longer window for shipping & fixing destabilising changes, i.e. the entire sprint duration.
- This way, Monday demos can show things that are already available for non-engineers in the staging/preview environment. Which is the way it should be in Scrum.
Essentially, I am proposing to adopt "Release trains".
On the second thought, let's move the update frequency to open questions, and scope this RFC to describe the other aspects of the Forge instances we want to run.
|
Had a chat with @bajtos and @hannahhoward: |
|
Chat with @bajtos and @hannahhoward concluded:
|
There was a problem hiding this comment.
🟡 Changes recommended
The RFC contains several typos/grammar issues that should be corrected to avoid ambiguity in the proposal.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR introduces a new RFC proposing how to run and operate multiple Forge network instances to balance competing needs (e.g., continuous deployment vs. stability) and to define criteria per instance.
Changes:
- Added an RFC describing decision criteria (update cadence, stability, infra, money type, resets, regions).
- Proposed four instance tiers: Production, Dev, Staging, and Performance testing.
File summaries
| File | Description |
|---|---|
| rfcs/2026-09-forge-instances.md | New RFC documenting criteria and a proposed set of Forge instances to operate. |
Review details
Suppressed comments (1)
rfcs/2026-09-forge-instances.md:116
- Wording/grammar: "the person on the pager duty" is ungrammatical; "on pager duty" (or similar) reads correctly.
- **Stability & acceptable outages:** Reasonable stability and minimum unplanned downtime. Full monitoring with alerts routed to the person on the pager duty, with capped severity (no incident is critical).
- Files reviewed: 1/1 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Signed-off-by: Miroslav Bajtoš <oss@bajtos.net>
📖 Preview
Summary
We want to operate multiple Forge network instances to serve conflicting needs like continuous
deployment vs stability. This document collects different criteria we have for each instance and
proposes a set of instances to stand up and operate.