Persistence Refactor POC

[flyrain]

Experiment:

1. Added a DAO layer for the business entity namespace(except the read).
2. Integrated with existing DAO components (PolarisMetaStoreManager and PolarisMetaStoreSession).
3. All tests passed successfully, including manual local run with Spark sql.

Benefits:

1. Compatible with the existing backend(FDB), as we hide them behind the new DAO.
2. Adding new backends(Postgres/MongoDB) is much easier now, esp for Postgres, we could be able to use a similar model as Iceberg Jdbc catalog.
3. Allows gradual refactoring to remove old DAO dependencies (PolarisMetaStoreManager and PolarisMetaStoreSession).
4. Enables parallel development of new backend implementations.

Next Steps:

1. Define business entities one by one to decouple them from FDB.
2. Create DAO interfaces for each entity to standardize operations (e.g., CRUD, ID generation).
3. Expand DAO implementations to support additional backends over time.

Please check the detailed design doc: https://docs.google.com/document/d/1Vuhw5b9-6KAol2vU3HUs9FJwcgWtiVVXMYhLtGmz53s/edit?usp=sharing

[flyrain]

We could move them into different module if necessary.

[jbonofre]

Yes, agree, FDB and PostgreSQL should be in their own module.

[jbonofre]

For persistence DAO, my idea was more to have concrete DAO definition to help the implementer with associated record. The DAO deals with storage operation using a record carriage (to avoid coupliing with entity).

Something like:

Suggested change

	public interface CatalogDao {}
	public record CatalogRecord(String id, String name, String location, ...) {
	...
	}

[flyrain]

I'm not sure about this. We should refactor the existing service entities instead of creating a new set of them as I said in the mailing list.

[jbonofre]

@flyrain you are right for DAO, my point is more for using entity in the DAO. I would rather use a record in the DAO to decouple from Entity: it will force us to decouple service and storage layers, and the record should be "obvious" for DAO implementer.

[jbonofre]

I don't think it's a good idea to have operation definitions in the DAO. The DAO could just a record (public record NamespaceDao), the storage logic for a DAO is in the persistence implementation.

The idea of DAO is to "map" business logic object to persistence object, I think it's a paradigm we should keep here.

[flyrain]

The idea of DAO is to "map" business logic object to persistence object

To be clear, NamespaceEntity and catalogPath here should service-layer entities. I put them here to demo how it works with existing code. But they are not necessary to be the final form. The idea of the DAO is to provide an interface so that the impl. can do two-way conversion like the following, while keep the interface only with upper-layer entities.

1. service entities -> persistence entities
2. persistence entities -> service entities

[jbonofre]

Yes, it makes sense. As said in previous comment, the DAO as storage operation definition can do the mapping but I think he should use only storage records (persistence entities) that can be converted to service entitiy. Seperation here is welcome.

[jbonofre]

Yes, agree, FDB and PostgreSQL should be in their own module.

[flyrain]

The regrest test failure should be fixed by #1015.