ECS state serialization

[ul]

Hello, thank you for a beautifully designed ECS system. I'm curious, what is the recommended way to serialize and deserialize ECS state wholesale? To avoid XY problem, the case I'm trying to cater for is hot reload. The main application holds ECS state and passes it to each new generation of a dynamically loaded library to work with. It's okay if the app crashes / requires restart when ECS compiled into the library changes in a non-compatible way, but it would be nice if it can continue to work with the current state when it is compatible.

[rlipsc]

Hey @ul,
I use nim-serialization/nim-json-serialization to persist entities to streams or files.

To save/load entity state, you need to be able to load/save a ComponentList. This is like a blueprint that you can pass to construct to create live entities. You can use toTemplate() to create these blueprints from individual entities or sequences of entities.

You will need to typecast Component to its 'real' value based on its typeId field, as Component itself is just an inheritable root type. The convenience template caseComponent helps with this by creating a case statement that matches typeId to registered component types.

Here's a pared down example:

import polymorph, streams, json_serialization

register defaultCompOpts:
  type
    Foo = object
      value: int64
    Bar = object
      content: string

makeEcs()

# Tell serialization to treat distinct ids as int64.
type PolymorphIds* = EntityId | EntityInstance | ComponentTypeId | ComponentIndexTypeClass 
PolymorphIds.borrowSerialization int64

proc writeValue*(w: var JsonWriter, c: Component) =
  # Tell serialization how to write a 'Component' type.
  w.beginRecord()
  writeField(w, "typeId", c.typeId)
  # Create a case statement that writes the appropriate container value.
  caseComponent c.typeId:
    writeField(w, "value", componentRefType()(c).value)
  w.endRecord()

proc readValue*(r: var JsonReader, v: var Component) =
  # Tell serialization how to read a 'Component' type.
  var
    typeId: ComponentTypeId
    comp: Component
  
  for fieldName in readObjectFields(r):
    case fieldName
      of "typeId":
        var i: int
        r.readValue(i)
        typeId = ComponentTypeId(i)
      of "value":
        caseComponent typeId:
          v = makeContainer( r.readValue(componentType()) )

proc writeBlueprint(entities: Entities, filename: string) =
  var
    stream = fileOutput(filename, fmWrite)
    writer = JsonWriter[DefaultFlavor].init(stream, pretty = true)
  
  writer.writeValue entities.toTemplate()
  stream.close

proc readBlueprint(filename: string): ConstructionTemplate =
  Json.loadFile(filename, ConstructionTemplate)

# Create some entities to test with.
let
  state = @[
    cl(Foo(value: 1)),
    cl(Foo(value: 2), Bar(content: "Text1")),
    cl(Bar(content: "Text2")),
  ].construct()

# Save a 'ConstructionTemplate' of 'state' to a JSON file.
state.writeBlueprint "data.json"

# Load the blueprint from JSON and check it matches 'state'.
let blueprint = readBlueprint "data.json"
assert $state.toTemplate == $blueprint

# Build entities from the loaded blueprint.
let newState = blueprint.construct()

for e in newState:
  echo e

I hope this helps!

[ul]

Thank you for the response! I was able to adapt it to jsony and it works well. However, there are a couple of things that are missing for my use case.

Entity relations

Currently I handle entity relations by storing EntityRef in the corresponding component fields. Those refs get invalidated during serialisation/deserialisation cycle. Is there a ser/de friendly and generic way to store entity relations? I could use UUIDs instead but then there is a problem of efficient lookup for the entity arises, either at runtime, or during ser/de (where it could be converted between EntityRef and UUID).

System fields

This is only an issue for hot code reload. Since the next generation of the dynamic library initialises systems from scratch, it would be nice to have a way to copy previous systems state as well, since not everything is convenient to represent as an entity.

Temporary solution

My current solution is pretty hacky: I augment makeEcs to call the following helper to generate procs for deep-copying Polymorph's storages between dynamic library generations:

proc generateEcsGlobalState(stateVars: seq[(string, string)]): NimNode =
  result = newStmtList()

  # Import necessary modules
  result.add quote do:
    import deepcopyredux

  let state = ident("state")
  let tableRefType = quote:
    TableRef[string, pointer]

  # Generate snapshotEcsGlobalState proc
  let snapshotBody = newStmtList()
  for (varName, typName) in stateVars:
    let varIdent = ident(varName)
    let varStr = newStrLitNode(varName)
    snapshotBody.add quote do:
      `state`[`varStr`] = addr(`varIdent`)
  let snapshotProc = quote:
    proc snapshotEcsGlobalState*(`state`: var `tableRefType`) =
      `snapshotBody`

  result.add snapshotProc

  # Generate restoreEcsGlobalState proc
  let restoreBody = newStmtList()
  for (varName, typName) in stateVars:
    let varIdent = ident(varName)
    let varStr = newStrLitNode(varName)
    let varType = parseExpr(typName)
    restoreBody.add quote do:
      if `state`.hasKey(`varStr`):
        deepCopyRedux(`varIdent`, cast[ptr `varType`](`state`[`varStr`])[])
  let restoreProc = quote:
    proc restoreEcsGlobalState*(`state`: var `tableRefType`) =
      `restoreBody`

  result.add restoreProc