Early thoughts about hierarchical components (ACS v2)

[andyk]

I'm starting this as a sort of scratch space where I can accumulate and organize my early reflections from my work porting the Stable Baselines 3 (SB3) PPO agent and prototyping an early version of hierarchical components.

These are mostly regarding the core architecture, which is integrally tied to how we structure the docs and our big picture project pitch/story.

• This design of hierarchical components introduces a new programming model that mixes top-down agent composition done from the command line with traditional bottom-up agent and algorithm development done in the IDE.
  • This is a natural evolution of the existing registry design, where we've already introduced the idea of AgentOS injecting attributes into the agent component as a result of agentos install command. This is a unique type of interaction where CLI commands and config ties in more directly with code being manipulated.
  • Existing solutions for this sort of thing include config files and environment variables, both of which require the developer to manage the dependencies separately at the code and env layers, and also cumbersome to use (e.g., require lots of boiler plate code, don't support anything like type checking).
  • The norm in Python (and most other programming languages, e.g. Java, C, etc.) is to manage your environment and code separately. Specifically, this means declaring your project dependencies (i.e. dynamic libraries) at the both code-level in the IDE (via import statements, namespaces) and environment-level via the CLI (python_path, pip, PyPI).
    • This is cumbersome and makes it harder, e.g., to independently manage/share/evolve/version sub-components of a project (since the PL env is set up at the project level)
    • Our proposal is for dependency management to be tightly integrated at the code and environment level.
• Our new programming model favors composition of core abstractions (i.e. components) instead of inheritance
  • the relationship between components is not "is a", and not even really a "has a", but more of a "depends on (and has a pointer to)".
• Agent is just another component (not special)
  • migrate from agent being a special component defined by agent.ini which has a flat format to being just a "just another component".
  • agent.ini --> agentos.yaml (or possibly could have a different name if we want an component to be defined by file rather than a whole directory. The trade-off between these two options is still being explored and discussed.
  • could rename "agent" to be "coordinator" (but don't have to, can just leave this up to different implementations)
  • the main directory abstraction is no longer an "agent directory" but rather a "component directory"
  • is a directory still a fundamental part of the definition of a component (i.e., does each component need to live inside its own directory or
  • must support circular component dependencies
  • migrate agentos.core.Agent and other agentos.core components to registry and make agentos.runnable agentos.runnable APIs that devs use to mix-in to "AgentOS-ify their algorithms"?
• Are "AgentOS Core" and "ACS" the same thing? Should they be? Which name makes more sense? (for documentation/new dev ramp-up? for package naming?)

I've been capturing these and more thoughts in these Hierarchical Component design slides

One possible flow of the project story post-hierarchical components:

• We observed that over a dozen RL/agent frameworks have been open sourced in the last few years, and while they are all inspired by the classical Sutton & Barto architecture, in practice they all have a slightly different take on the architecture of an agent and its sub-components.
• Our goal is not to create yet another agent framework (in fact that has already been done, cite https://github.com/jmribeiro/yaaf), but rather to make it easier to learn and use existing tools.
• To that end, we introduce a new programming model that mixes top-down agent composition with traditional bottom-up agent and algorithm development.
• We also introduce tooling for breaking agents into components that are easy to comprehend, compose, and share (e.g., publish). One way they do this is by explicitly declaring their dependencies on other components.

If we are successful at our goals, then it should be possible to port many agents/algorithms from existing popular frameworks, and test them against known benchmarks. It should be easy to examine the components of those agents (their policies, etc.) and understand in a few seconds how the policy component in rllib's PPO agent relates to other components.

To demonstrate the value of our new programming model and developer tools, we have ported agents from popular existing RL frameworks, integrated,

[andyk]

My work-in-progress prototype branch is at https://github.com/andyk/agentos/tree/wip-sb3-on-master

[andyk]

A rough version of hierarchical components--i.e., AgentOS support for a dynamic component dependency graph--was merged into master via #148.

As @nickjalbert and I have been porting some agents over using the, we've been creating some issues (e.g., #155, #161, and #164) and having some discussions about the core design in those issues.

I wanted to copy over some higher level thoughts from a comment I just left in #155 just to consolidate it here in this discussion thread for progeny:

I'm finding that as I work on porting the RLlib example agent, I would like to load and interact with my components directly in the python interactive terminal (i.e., bypassing using the AgentOS runtime). This is something I do routinely while writing python code for the purpose of testing, debugging, and prototyping functionality. It would like to instantiate my component in the python interactive terminal like I would with a normal class and pass in parameters to __init__() and then call a function of the class with params just like the AgentOS runtime would when executing the component function as a entry-point.

Philosophy, I'm proposing that our new higher level programming model (with our reference implementation being the AgentOS runtime) use the plain ol' Python abstractions (i.e. classes & objects, object construction via __init__(), object methods, function arguments/params, etc.) as much as possible.

And then, the big picture thing our programming model adds on top of what Python currently offers is: a new way to encapsulate code for the purposes of...

1. publishing/sharing (via a registry website & CLI)
2. execution (via entry-points & supporting runtime functionality)
3. composition into higher level functionality (via dependency DAGs, versioning, and dependency tooling)

...at a finer granularity than the Python ecosystem currently supports. Specifically, our "python component" concept maps to a python class, vs e.g., Python project which maps to a set of potentially multiple modules, each of which can have potentially many classes and other non-class level.

Why would you want to do this? Because it is very intuitive to use pip CLI like semantics (install/update/uninstall/etc.) to depend on, swap out, or upgrade parts of your code that other people write, but doing it at the more course-grained project often leaves it up to me to rewrite my code to make it work with a new or upgraded Python/pip package (because the packages are not tied intrinsically to class interfaces, which is the primary mechanism for how my project uses code from another project).

However, ideally we are maximally compatible with the existing Python ecosystem and only add new functionality on top of what's already there.