openhcl: support save restore of partition topology from usermode to next openhcl_boot #2030

chris-oo · 2025-09-22T16:49:00Z

Support save and restore of topology information from a previous instance of OpenHCL to the next openhcl_boot. This allows supporting correctly persisting any private pool allocated from the address space to the next instance, without relying on the allocation logic being exactly the same.

This is done via reserving the first page in the VTL2 region as a persisted header, followed by a protobuf encoded payload containing the previous instance's topology. In the future, we could choose to put more saved state into this region, such as the whole usermode protobuf payload that today goes to the host, via the GET.

mattkur · 2025-09-23T21:42:55Z

vm/loader/loader_defs/src/shim.rs

+pub struct PersistedStateHeader {
+    /// A magic value. If this is not set to [`PersistedStateHeader::MAGIC`],
+    /// then the previous instance did not support this region.
+    pub magic: u64,
+    /// The gpa for the start of the protobuf region. This must be 4K aligned.
+    pub protobuf_base: u64,
+    /// The size of the protobuf region in bytes.
+    pub protobuf_region_len: u64,
+    /// The size of the protobuf payload in bytes.
+    /// This must be less than or equal to `protobuf_region_len`.
+    pub protobuf_payload_len: u64,
+}
+
+impl PersistedStateHeader {
+    /// "OHCLPHDR" in ASCII.
+    pub const MAGIC: u64 = 0x4F48434C50484452;
+}
+
+/// A local newtype wrapper that represents a [`igvm_defs::MemoryMapEntryType`].
+///
+/// This is required to make it protobuf deriveable.
+#[derive(mesh_protobuf::Protobuf, Clone, Debug, PartialEq)]
+#[mesh(package = "openhcl.openhcl_boot")]
+pub struct IgvmMemoryType(#[mesh(1)] u16);
+
+impl From<igvm_defs::MemoryMapEntryType> for IgvmMemoryType {
+    fn from(igvm_type: igvm_defs::MemoryMapEntryType) -> Self {
+        Self(igvm_type.0)
+    }
+}
+
+impl From<IgvmMemoryType> for igvm_defs::MemoryMapEntryType {
+    fn from(igvm_type: IgvmMemoryType) -> Self {
+        igvm_defs::MemoryMapEntryType(igvm_type.0)
+    }
+}
+
+#[derive(mesh_protobuf::Protobuf, Debug)]
+#[mesh(package = "openhcl.openhcl_boot")]
+pub struct MemoryEntry {
+    #[mesh(1)]
+    pub range: MemoryRange,
+    #[mesh(2)]
+    pub vnode: u32,
+    #[mesh(3)]
+    pub vtl_type: MemoryVtlType,
+    #[mesh(4)]
+    pub igvm_type: IgvmMemoryType,
+}
+
+#[derive(mesh_protobuf::Protobuf, Debug)]
+#[mesh(package = "openhcl.openhcl_boot")]
+pub struct MmioEntry {
+    #[mesh(1)]
+    pub range: MemoryRange,
+    #[mesh(2)]
+    pub vtl_type: MemoryVtlType,
+}
+
+/// The format for saved state between the previous instance of OpenHCL and the
+/// next.
+#[derive(mesh_protobuf::Protobuf, Debug)]
+#[mesh(package = "openhcl.openhcl_boot")]
+pub struct SavedState {
+    #[mesh(1)]
+    pub partition_memory: Vec<MemoryEntry>,
+    #[mesh(2)]
+    pub partition_mmio: Vec<MmioEntry>,
+}


@mattkur todo

Copilot

Pull Request Overview

This PR adds support for saving and restoring partition topology information from a previous OpenHCL instance to the next openhcl_boot. The implementation reserves the first page in the VTL2 region as a persisted header, followed by a protobuf encoded payload containing the previous instance's topology information.

Key changes include:

Introduction of persisted state region management in the bootloader
New protobuf-based serialization for topology data
Enhanced memory layout to support persisted state across boots
Integration of saved state reading during boot process

Reviewed Changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vm/loader/src/paravisor.rs	Reserves 2MB persisted state region in memory layout for both x86 and ARM64
vm/loader/loader_defs/src/shim.rs	Defines persisted state structures, protobuf messages, and memory type enums
vm/loader/loader_defs/Cargo.toml	Adds mesh_protobuf and memory_range dependencies
openhcl/underhill_core/src/loader/vtl2_config/mod.rs	Implements persisted state writing functionality
openhcl/underhill_core/Cargo.toml	Adds mesh_protobuf dependency
openhcl/openhcl_boot/src/memory.rs	Updates address space manager to handle persisted state regions
openhcl/openhcl_boot/src/main.rs	Updates E820 entry generation and test cases for persisted state
openhcl/openhcl_boot/src/host_params/shim_params.rs	Adds persisted state region to ShimParams structure
openhcl/openhcl_boot/src/host_params/dt/mod.rs	Major refactoring to support topology from persisted state vs host DT
openhcl/openhcl_boot/Cargo.toml	Adds mesh_protobuf dependency
openhcl/bootloader_fdt_parser/src/lib.rs	Updates FDT parser to handle persisted state regions

Comments suppressed due to low confidence (1)

openhcl/openhcl_boot/src/host_params/dt/mod.rs:1

Using format! in error contexts can be inefficient. Consider using a closure with with_context to avoid string formatting unless the error actually occurs: .with_context(|| format!(\"{} missing numa-node-id\", node.name))?

// Copyright (c) Microsoft Corporation.

vm/loader/src/paravisor.rs

vm/loader/loader_defs/src/shim.rs

chris-oo · 2025-10-01T22:46:32Z

attempt fixing #1345, since i think this PR should fix it?

vm/loader/src/paravisor.rs

vm/loader/loader_defs/src/shim.rs

github-actions · 2025-10-02T17:36:32Z

At least one Petri test failed.

smalis-msft · 2025-10-03T19:32:49Z

support/mesh/mesh_protobuf/src/protofile/mod.rs

 }

 /// A description of a protobuf `oneof`.
+#[cfg_attr(not(feature = "std"), allow(dead_code))]


nit: expects

smalis-msft · 2025-10-03T19:35:14Z

openhcl/underhill_core/src/loader/vtl2_config/mod.rs

+    );
+
+    let ranges = [parsed.vtl2_persisted_header];
+    let mapping =


nit: inline the two 'ranges' vars and rename the two 'mapping' vars to different names to make it clearer what gets written where.

smalis-msft · 2025-10-03T19:35:43Z

openhcl/underhill_core/src/loader/vtl2_config/mod.rs

+        Vtl2ParamsMap::new_writeable(&ranges).context("failed to map persisted protobuf region")?;
+
+    // Create the serialized data to write.
+    let state = SavedState {


Should this maybe be a save method on the partition_memory_map instead of being done here?

smalis-msft · 2025-10-03T19:36:23Z

openhcl/underhill_core/src/loader/vtl2_config/mod.rs

+    // as all information we're persisting is currently known. In the future, if
+    // we plan on putting more usermode specific data such as the full openvmm
+    // saved state, we should probably move this to a servicing specific call.
+    write_persisted_info(&parsed_openhcl_boot)


So we do this all the time, right? The memory we're writing to will be private on a cvm, so this isn't a security concern correct?

Correct we do this all the time. And yes, this will be private on a CVM - I think this realistically is the only way we will get servicing to work in a CVM ie have openhcl launch the next openhcl, and supply parameters/information somehow.

smalis-msft · 2025-10-03T19:38:57Z

openhcl/openhcl_boot/src/host_params/dt/mod.rs

+        )
+    };
+
+    ALLOCATOR.enable_alloc();


I kinda wonder if it's worth switching this to be like an ALLOCATOR.with_alloc(closure) instead? That would prevent us from forgetting to disable it. And given our scoping model, that feels safer.

that sounds pretty reasonable (ie treat it like how std treats TLS?). I'll take a stab at it on monday.

github-actions · 2025-10-06T21:48:42Z

At least one Petri test failed.

chris-oo force-pushed the bootshim-persisted-state branch from dbb0607 to d7f1f2b Compare September 23, 2025 18:20

mattkur reviewed Sep 23, 2025

View reviewed changes

chris-oo force-pushed the bootshim-persisted-state branch 3 times, most recently from 71229e5 to 69894a7 Compare September 29, 2025 22:44

chris-oo changed the title ~~[WIP] persist and save/restore vtl2 topology from usermode to bootshim~~ openhcl: support save restore of vtl2 topology from usermode to bootshim Oct 1, 2025

chris-oo changed the title ~~openhcl: support save restore of vtl2 topology from usermode to bootshim~~ openhcl: support save restore of partition topology from usermode to next openhcl_boot Oct 1, 2025

chris-oo marked this pull request as ready for review October 1, 2025 22:40

chris-oo requested review from a team as code owners October 1, 2025 22:40

Copilot AI review requested due to automatic review settings October 1, 2025 22:40

chris-oo assigned mattkur, jstarks and smalis-msft Oct 1, 2025

Copilot AI reviewed Oct 1, 2025

View reviewed changes

vm/loader/src/paravisor.rs Outdated Show resolved Hide resolved

vm/loader/loader_defs/src/shim.rs Show resolved Hide resolved

chris-oo force-pushed the bootshim-persisted-state branch from bef3266 to 79cc202 Compare October 1, 2025 22:48

mesh_protobuf: gate timestamp error behind std

8972b28

chris-oo force-pushed the bootshim-persisted-state branch from 79cc202 to f200047 Compare October 1, 2025 23:04

smalis-msft reviewed Oct 2, 2025

View reviewed changes

vm/loader/src/paravisor.rs Show resolved Hide resolved

smalis-msft reviewed Oct 2, 2025

View reviewed changes

vm/loader/loader_defs/src/shim.rs Show resolved Hide resolved

smalis-msft reviewed Oct 2, 2025

View reviewed changes

vm/loader/loader_defs/src/shim.rs Outdated Show resolved Hide resolved

smalis-msft reviewed Oct 2, 2025

View reviewed changes

vm/loader/loader_defs/src/shim.rs Show resolved Hide resolved

fix unused fields

40df288

chris-oo force-pushed the bootshim-persisted-state branch from f200047 to 1fc27e2 Compare October 2, 2025 16:40

chris-oo mentioned this pull request Oct 2, 2025

vm/loader: cleanup paravisor.rs to share paths between x64 and aarch64 #2066

Open

chris-oo force-pushed the bootshim-persisted-state branch from 1fc27e2 to 0677aff Compare October 2, 2025 22:07

smalis-msft reviewed Oct 3, 2025

View reviewed changes

chris-oo added 18 commits October 6, 2025 13:32

change allow to expect

668ae91

add persisted header defn and protobuf format

0decbc7

add persisted header to loader code and address space manager

0967c7a

refactor bootloader to read persisted header and read persisted state

0bc05bf

support bootshim reading from previous persisted state

233f5f4

fixup rebase

1ec8b59

bootloader_fdt_parser: read persisted state header and report separately

790afc2

ported changes, but blows up with ptr copy overlapping, allocator bug?

c68a855

add prints and host check

76649d0

size persisted region to 20 pages

3626275

cleanup vtl2params map

ed735c4

cleanup more vtl2_config

6f7cbba

cleanup warnings

3479046

refactor and cleanup

f815fe9

remove extra logs

736b49f

fixup defn

8d7ef32

oops cleanup comment

e451809

use from bytes

403ef68

chris-oo force-pushed the bootshim-persisted-state branch from a4fe858 to 403ef68 Compare October 6, 2025 20:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

openhcl: support save restore of partition topology from usermode to next openhcl_boot #2030

openhcl: support save restore of partition topology from usermode to next openhcl_boot #2030

Uh oh!

chris-oo commented Sep 22, 2025 •

edited

Loading

Uh oh!

mattkur Sep 23, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

chris-oo commented Oct 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

smalis-msft Oct 3, 2025

Uh oh!

smalis-msft Oct 3, 2025

Uh oh!

smalis-msft Oct 3, 2025

Uh oh!

smalis-msft Oct 3, 2025

Uh oh!

chris-oo Oct 3, 2025

Uh oh!

smalis-msft Oct 3, 2025 •

edited

Loading

Uh oh!

chris-oo Oct 3, 2025

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

Uh oh!

openhcl: support save restore of partition topology from usermode to next openhcl_boot #2030

Are you sure you want to change the base?

openhcl: support save restore of partition topology from usermode to next openhcl_boot #2030

Uh oh!

Conversation

chris-oo commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattkur Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

chris-oo commented Oct 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

smalis-msft Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

smalis-msft Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

smalis-msft Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

smalis-msft Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

chris-oo Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

smalis-msft Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chris-oo Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 6, 2025

Uh oh!

Uh oh!

chris-oo commented Sep 22, 2025 •

edited

Loading

smalis-msft Oct 3, 2025 •

edited

Loading