Chapter ZODB

docs/backend/zodb.md

@@ -11,3 +11,42 @@ myst:
 
 # ZODB
 
+The ZODB (Zope Object Database) is a Python-native, object-oriented database designed for direct persistence of Python objects.
+Unlike traditional relational databases that rely on tables and SQL queries, ZODB allows developers to work directly with Python objects, persisting them without the need for object-relational mapping (ORM).
+
+
+## Core Features of ZODB
+
+- Transparent Persistence: Objects stored in ZODB automatically persist. In a Ploe request/respone cycle they do not require an explicit save or commit operation, since this is done automatically.
+- No Schema Constraints: Unlike relational databases, ZODB does not require predefined schemas, allowing for flexible and dynamic data structures.
+  The attributes of the Python objects themselves are stored.
+- ACID Compliance: ZODB ensures data consistency through transactions that support Atomicity, Consistency, Isolation, and Durability.
+- Automatic Conflict Resolution: With its multi-version concurrency control (MVCC), ZODB can handle concurrent access efficiently.
+- Built-in Versioning and Undo: The database allows versioning, enabling rollback to previous states if needed.
+- Scalability: ZODB can be used in standalone applications or scaled with ZEO (ZODB Extension Objects) for distributed storage.
+  Additionally, ZODB supports the RelStorage adapter, which allows it to use relational databases like PostgreSQL, MySQL, or Oracle as a backend, providing flexibility for integration with existing database infrastructures.
+
+## How ZODB Works
+
+At its core, ZODB operates as an object store, maintaining pickled Python objects with some metadata in a hierarchical structure.
+
+- Storage: Handles how objects are stored on disk or in-memory.
+  The default storage is FileStorage, which writes data to `.fs` files Filestorage does not scale, as only one process can work with it.
+  ZEO (ZODB Extension Objects) storage is used for distributed multi-client access to the same database, introducing scalability.
+  Another highly scalable option is RelStorage, which allows ZODB to use relational databases like PostgreSQL, MySQL, or Oracle as backend storage, combining object persistence with traditional database infrastructure.
+  RelStorage is used often in containerized deployment environment.
+
+- Connection: Acts as the interface between Python applications and the database.
+  With ZEO for each active Zope thread one connection is established.
+  Connection are pooled and re-used.
+
+- Transaction Manager: Manages transactional operations, ensuring data integrity.
+  A transaction normally start with the requests and ends with the respsonse.
+  However, in long running requests with lots of database writes, transactions can be committed in between.
+
+- Indexing and Caching: Optimizes read and write operations for better performance.
+  Tuning the ZODB-cache sizes to hardware-environment and the kind of data stored may help to speed up the application.
+
+## Further reading
+
+More information can be found at the official [ZODB.org] website (https://zodb.org)

docs/contributing/core/package-dependencies.md

@@ -34,65 +34,188 @@ Nevertheless there is indirection on all other levels.
 Since Plone consists of a lot of packages, it is complex to untangle those.
 
 
-## Mental model
+## Mental model 
 
-A base mental model for how Plone is organized in Plone 6, and is shown in the following diagram:
+### Borders
+
+As a rough base mental model for how Plone is organized in Plone 6.1, there are two packages as dividing lines:
+
+1.  `Products.CMFPlone` and all below defines the Plone core. Everything in here depends on the `plone.base`.
+2.  `plone.base` as the border to the Application Server and Content Management Framework and its dependencies
 
 ```{mermaid}
 block-beta
-    columns 1
-
-    Plone["Plone <br/>the integraton of both distributions in one release"]
-    space
+    columns 4
+    Plone["Plone<br/>integraton of all in one release"]:4
     Distributions
-    block:dist
-        plone.volto
-        plone.classicui
-    end
-    space
-    block:core
-        coreaddons["Core add-ons"]
-        coreapi["Core APIs"]
-    end
-    space
-    cmfplone["Products.CMFPlone"]
-
-    space:2
-
-    block:layer
-        ploneapp["Most of plone.app.* namespace"]
-        otherlay["Various other packages"]
-    end
-
-space
-
-    plonebase["plone.base"]
-    space
-    foundations["The Foundations"]
-    space:3
-        block:foundationcomponents
-           ploneworld["Plone world"]
-           zopeeco["Zope ecosystem"]
-           zopecore["Zope core"]
-           libraries["Libraries"]
-    end
-    Plone --> Distributions
-    dist --> core
-    cmfplone --> layer
-    core --> cmfplone
-    layer --> plonebase
-    plonebase --> foundations
-
-    style cmfplone fill:#ff0
-    style plonebase fill:#ff0
+    Upgrade
+    coreapis["Core APIs"]
+    coreaddons["Core add-ons"]
+    cmfplone["Products.CMFPlone"]:4
+    ploneapp["Most of plone.app.* namespace"]:2
+    otherlay["Various related packages"]:2
+    plonebase["plone.base"]:4
+    block:groupfoundation:4
+        zopecore["Zope core/ ZCA"]
+        zopeeco["Zope ecosystem"]
+        cmfcore["CMFCore"]
+        ploneworld["Plone generic libraries"]
+        libraries["Other libraries"]
+    end  
+    style cmfplone fill:#fff9e6
+    style plonebase fill:#fff9e6
+```
+
+### Main components
+
+Some explanation on the mental model
+
+- Foundation:
+  - `Zope` core and its dependencies is the application server,
+  - the Zope component architecture (ZCA) framework and 
+  - some additional packages from the wider Zope ecosystem.
+  - then there are generic, standalone Plone libraries,  
+  - plus various other Python libraries.
+- `Products.CMFCore` provides on top of Zope very basic content management features we rely on.
+- `plone.base` defines several interfaces as contracts for the component architecture we build on. 
+   Additional it provides some base classes and utility functions we use often. 
+   It also depends on `Products.CMFCore` and so `Zope`.
+   Additional it depends on generic functionality like `plone.dexterity`, `plone.behavior` and `plone.registry`.
+- The space of plenty `plone.*`, `plone.app.*` and related libraries defines the core of Plone.
+- On top of this core, depending on these packages, is `Products.CMFPlone` which is the package to depend on if the basic Plone core is referenced.
+- On top of `Products.CMFPlone` 
+  - are the core APIs like `plone.api` and `plone.restapi`,
+  - there is the distribution support `plone.distribution` and specific distributions, currently `plone.volto`, `plone.classicui`, 
+  - are core addons like Working Copy Support (`plone.app.iterate`), discussion support (`plone.app.discussion`), ... 
+  - is `plone.app.upgrade`, the package to upgrade between Plone version.
+- The package `Plone` is the package to depend on if you want to depend on the whole Plone with everything.
+  This meta package without any code depends on all other packages.
+  It is what you want to install if you do not want to care about the details with all batteries included.
+
+### The space on top of CMFPlone
+
+Add-on developers and integrators are primary interacting with the dependencies on top of CMFPlone.
+The following diagram visualizes this part.
+
+```{mermaid}
+---
+config:
+  sankey:
+    showValues: false
+    width: 1600
+    height: 800
+    nodeAlignment: "right"
+---
+
+sankey-beta
+
+    Plone,Distributions,100
+
+    Distributions,plone.volto,50
+    Distributions,plone.classicui,30
+    Distributions,other dist.,20
+
+
+
+    plone.volto,plone.distribution,10
+    plone.volto,plone.restapi,20
+    plone.volto,plone.api,20
+
+    Multlilingual,plone.api,20
+    Commenting,plone.api,20
+    Working Copies,plone.api, 20
+    Caching,plone.api,20
+
+    plone.classicui,plone.distribution,15
+    plone.classicui,plone.api,15
+    other dist.,plone.distribution,4
+    other dist.,plone.api,8
+    other dist.,plone.restapi,8
+
+
+    Plone,Core-Addons,80
+
+    Core-Addons,Multlilingual,20
+    Core-Addons,Commenting,20
+    Core-Addons,Working Copies,20
+    Core-Addons,Caching,20
+
+    plone.distribution,Export/Import,15
+    plone.distribution,plone.api,14
+    plone.restapi,Core,20
+    Export/Import,plone.api,8
+    plone.restapi,plone.api,15
+    Export/Import,plone.restapi,7
+    Upgrade,Core,20
+    plone.api,Core,160
+
+    Plone,Upgrade,20
 
 ```
 
-As a rough model, there are two packages as dividing lines:
+## A more detailed view on the architecture
 
-1.  `Products.CMFPlone`
-2.  `plone.base`
+A more detailed view on the whole architecture is sketched here:
 
+```{mermaid}
+flowchart TB
+ subgraph Release["Release"]
+        Plone[["'Plone' Package"]]
+  end
+ subgraph subGraph1["Distributions"]
+        volto{{"plone.volto"}}
+        classicui{{"plone.classicui"}}
+        other{{"other ..."}}
+  end
+ subgraph Core-Addons["Core-Addons"]
+        multilingual("Multilingual")
+        iterate("Working Copies")
+        discussion("Commenting")
+        caching("Caching")
+  end
+ subgraph API["API"]
+        restapi("RestAPI")
+        ploneapi("API")
+        upgrade("Upgrade")
+        exportimport("Export/Import")
+        distribution["Distribution"]
+  end
+ subgraph subGraph4["The Inner Core"]
+        thecore((("Core-Features: plone.*, ...")))
+  end
+ subgraph Core["Core"]
+        cmfplone["Products.CMFPlone"]
+        plonebase["plone.base"]
+        subGraph4
+  end
+ subgraph Foundations["Foundations"]
+        zope["Zope/CMF/ZODB/..."]
+        supportlibs["Several Supporting Libraries"]
+  end
+    Plone == provides ==> volto
+    Plone -. provides .-> classicui & other & upgrade & multilingual & iterate & discussion & caching
+    volto -- installs --> restapi
+    volto == is a ==> distribution
+    classicui -. is a .-> distribution
+    other -. is a .-> distribution
+    distribution -- uses --> exportimport
+    distribution == depends on ==> ploneapi
+    restapi -- depends on --> ploneapi
+    cmfplone == depends on ==> thecore
+    thecore == depends on ==> plonebase
+    plonebase == depends on ==> zope
+    plonebase -- depends on --> supportlibs
+    zope -- depends on --> supportlibs
+    ploneapi == depends on ==> cmfplone
+    upgrade -- depends on --> cmfplone
+    multilingual -- depends on --> ploneapi
+    iterate -- depends on --> ploneapi
+    discussion -- depends on --> ploneapi
+    caching -- depends on --> ploneapi
+    exportimport -- uses serializers of --> restapi
+    exportimport -- depends on --> ploneapi
+
+```
 
 ## Packages in detail