Split aperturedb into smaller packages.

[gsaluja9]

The current aperturedb SDK encompasses a bunch of functionality, in which certain layers are built on top of others.
We should repackage it, by specifying optional dependencies, or splitting into different wheels.

Build of aperturedb-notebooks does ensure that all of them can be installed in a clean environments, but adding all the dependencies can possibly create an issue if a user's existing arrangement pins the versions of certain packages we also depend on.

Also it's important to acknowledge that not everyone wants to use all the features of aperturedb.

This discussion should lead to way in which the package can be split so that users can pick the features they require without causing a lot of disruption to their env.

Based on the usage patterns, the following can be deduced:

• Not every use case includes csv parsers.
• Not every ingestion uses dask.
• Not everyone wants to use DataModels
• Those who use pytorch, will likely not install Tensorflow, but Mixing of multiple types should be a permitted use case.
• We should support most protobuf versions.
• What about transformers?

[drewaogle]

I think we have several usage patterns

• Developer
  • In Console
  • In Notebook
  • In Established Build Environment ( leading towards the Application usage )
• Application
  • Thin Client ( something that is taking requests and converting them into db request ) - like what we had for the labeling demo
  • Loader
  • Transformer
• Tool

[drewaogle]

Tradtionally there have been several ways to split packages

library/developer is the main one,
the other common one is for plugin architectures, where they tend to split them very granuarly.

Some like gstreamer will then package them into bigger groups like "audio" or "free license"

Things like mysql also split "frontend" and "server"

I think we have a "tools" kind of package with the cli.

I think we also have a usecase to do the grouping like gstreamer where we have some things grouped by special dependencies, which then we can put into a use-of-use package.

[drewaogle]

What we might be able to do is convert aperturedb as a package to depending on whatever we have below

• so it might need aperuture-modular[console] and aperturedb-modular[loader] and maybe aperturedb-modular[models]

the question is I think if we models, would that just strong require tensorflow AND pytorch, or would we make those sub packages?

[gsaluja9]

For experimentation purpose, I created a Dockerfile, and I could use adb cli, without the non essential stuff. It asserts that we could create a minimal but functional install like this.

Dockerfile:

FROM python:3.13.3-alpine

RUN pip install \
    aperturedb requests typer click typing_extensions \
    rich 'protobuf >=3.20.3,<6.0.0' keepalive-socket urllib3 idna \
    certifi pydantic pydantic_core typing_inspection tqdm \
    --no-deps

and build it as
docker build -t adb_cli:latest -f Dockerfile .

[drewaogle]

(adb_testing) drew@Dunstan:/code/aperture/small_adb$ docker images -a
REPOSITORY                           TAG                IMAGE ID       CREATED          SIZE
adb_cli                              latest             b4e8b3bed8d9   10 seconds ago   79.7MB

I think that size is probably doable to start with