Add `.load()` class-method to DiffSyncModel

[jamesharr]

Environment

• DiffSync version: 1.9.0

Proposed Functionality

This change proposes to 1) add a load() class-method to DiffSyncModel and 2) augments DiffSync.load() such that it will call each model's .load() method with whatever is passed in.

Key benefits:

1. DiffSyncModel has always felt incomplete with only create/update/delete.
2. It's a common pattern in code that I have written to have a .from_xyz() class-method to transform data. Adding it to diffsync would give it a more official location.
3. The design of this API would support both two methods of loading data: a) each model calls an API to load code it needs b) there is one large call to load ALL the data, and each model handles interpreting the data, such as with GraphQL.
4. Additional parameters, such as filtering criteria, could be passed to DiffSyncModel.load()

Use Case

# DiffSync modifications
class DiffSyncModel:
    @classmethod
    def load(cls, diffsync: DiffSyncModel, *args, **kwargs):
        pass
        # no-op

class DiffSync:
    def load(self, *args, **kwargs):
        # Pass all arguments to per-model flags
        for model_name in self.store.get_all_model_names():
            model_cls = getattr(self, model_name)
            model_cls.load(*args, diffsync=self, **kwargs)



# Example user-code 1 - each model loads its own data
class Location(DiffSyncModel):
    ...
    @classmethod
    def load(cls, diffsync: DiffSyncModel, nb: pynautobot.api):
        for nb_location in nb.dcim.locations.filter():
             diffsync.add(cls(name=nb_location.name, ...))

class Device(DiffSyncModel):
    ...
    @classmethod
    def load(cls, diffsync: DiffSyncModel, nb: pynautobot.api):
        for nb_device in nb.dcim.devices.all():
             diffsync.add(cls(name=nb_device.name, ...))

class MyBackend(DiffSync):
    ... utilizes default backend

def main():
    be1 = MyBackend()
    be1.load(nb=pynautobot.api(...))



# Example user-code 2 - each model loads its own data, with filter criteria being passed along
class Location(DiffSyncModel):
    ...
    @classmethod
    def load(cls, diffsync: DiffSyncModel, nb: pynautobot.api, tags: List[str]):
        for nb_location in nb.dcim.locations.filter(tags=tags):
             diffsync.add(cls(name=nb_location.name, ...))

class Device(DiffSyncModel):
    ...
    @classmethod
    def load(cls, diffsync: DiffSyncModel, nb: pynautobot.api, tags: List[str]):
        for nb_device in nb.dcim.devices.filter(tags=tags):
             diffsync.add(cls(name=nb_device.name, ...))

class MyBackend(DiffSync):
    ... utilizes default backend

def main():
    be1 = MyBackend()
    be1.load(nb=pynautobot.api(...), tags=["US", "Europe"])



# Example user-code 3 - Backend handles data retrieval, models handle transformation
class Location(DiffSyncModel):
    ...
    @classmethod
    def load(cls, diffsync: DiffSyncModel, graphql: GraphQLRecord):
        for nb_location in graphql.json["data"]["locations"]
             diffsync.add(cls(name=nb_location["name"], ...))

class Device(DiffSyncModel):
    ...
    @classmethod
    def load(cls, diffsync: DiffSyncModel, graphql: GraphQLRecord):
        for nb_device in graphql.json["data"]["devices"]
             diffsync.add(cls(name=nb_device["name"], ...))

class MyBackend(DiffSync):
    def load(self):
        nb = pynautobot.api(...)
        result = nb.graphql.query("query { devices { name } locations { name } }")
        super().load(graphql=result)

def main():
    be1 = MyBackend()
    be1.load()

[jdrew82]

This seems more like an SSoT specific request and also wouldn't this essentially be covered with the contrib pattern now?