How to handle "downwards" database query dependencies

[ghferrari]

Suppose you have a modulithic app with two modules, A and B. Module B depends on module A but not vice versa; so A is the root of the dependency graph. I'm imagining that this app provides a REST API, though I think an html web app would work just as well for this discussion. And I'm imagining that this app uses Spring Data JPA for persistence.

Suppose module A defines an entity Aardvark and module B defines an entity Ball, which is always owned by some Aardvark i.e. there's a many-to-one relationship. To make this work, module B imports the Aardvark entity from module A.

Now suppose you have a REST endpoint at /aardvarks which is defined by module A and which returns a list of Aardvarks. And suppose you want to be able to filter this list based on whether an Aardvark owns a Ball or not.

At the moment, the only sensible way I can think to achieve this is to have a query in module A that somehow references the Ball entity in module B. But... that means that module A has some sort of dependency on module B, after all. So, is this considered acceptable within the spirit of Spring Modulith, or is there some other way to achieve what I want that I haven't spotted yet?

My question above is whether this sort of thing can be handled within the spirit of Spring Modulith. But a more immediate question is whether Spring Modulith would actually allow it i.e. would it raise an error in this sort of case? In my real situation, I've so far been writing my downstream query as a native query, which corresponds to a situation where module A in my example has a SQL dependency on module B's data, but module A's code does not depend on module B's code. I haven't tested this, but I imagine Spring Modulith wouldn't test for a SQL dependency and so wouldn't raise an error. But I guess the situation would be different if I used either Spring Data JPA's repository queries or wrote my query in JPQL. Then, it seems, module A's code would depend on module B's code because e.g. the query code in module A would depend on the hibernate classes in module B. So there would in fact be a circular code dependency between A and B (given that ex hypothesi B's code depends on A's code). Perhaps Modulith would raise an error in this case; but I haven't tested it. It also seems that using an @ApplicationModuleTest in module A would require loading module B, since A's code depends on B's code (assuming that tests haven't already failed because of the circular dependency).

I don't think one can get around this issue by creating an additional module C to integrate A and B because in order to integrate them both in C you would probably need to move almost all of A's and B's classes there. So, ultimately, the only answer seems to be that both module A's code and module B's code need to be located within the same module. And since these sorts of downward dependencies can in principle arise between any dependent modules at any time (i.e. at any time, it might become valuable to your business to be able to filter entities that appear earlier in the dependency chain by their relationship to entities that appear later in the dependency train), the ultimate impetus is to move most of your code into a single module, which tends to undermine the rationale for using Modulith in the first place. (I've been discussing a very simple example here, but to make things more concrete, consider the Spring RestBucks example application and imagine you have drinks and orders in separate modules, with orders downstream from drinks, but then it becomes useful to query drinks based on their number of orders...). Of course, there might still be some occasions where one is sure that one will never need to query an upstream entity based on its relationships with downstream entities - so there might be some occasions when one can safely split an app into a multi-module DAG. But I think the dominant pressure will be to move code into a single module.

I'd be grateful for other's thoughts on this, so please do share with me what you think. Ultimately, I think these sorts of downstream query dependencies are likely to arise in any non-toy example project, so I think it is important to know how to deal with them. I'm afraid I can't currently think of any feasible (or even non-feasible) way to handle them while continuing to use Spring Modulith. (In a previous discussion post I asked for help understanding the DAG requirement within Spring Modulith without wishing to criticise it. But I do think the issue I'm raising here is a serious problem.)

[ghferrari]

FWIW, I've realised/discovered that it's possible to turn off the DAG module verification, so that's where I'm at now. https://docs.spring.io/spring-modulith/reference/2.0/verification.html#_customizing_the_verifcation

[ghferrari]

@odrotbohm As discussed above, I want to turn off the test for cyclic dependencies - how can I do this? The documentation suggests this is possible but doesn't go into detail or provide an example. This might be a useful addition to the documentation.

[odrotbohm]

That's quite a lot to digest, but here are a few things that are at least at odds with a modular arrangement:

• The notion of an “owning” relationship crossing module boundaries is odd. Such an ownership strongly ties modules together, which we want to avoid. I don't think you can avoid talking about aggregates in such scenarios. Those have to fully live within a module, i.e., all the contained entities must live inside the same module. External references are only held pointing to other aggregates via identifiers.
• If you mean “many-to-…” association in the mapped JPA sense via @ManyTo…, there's no place for such relationships in a modular arrangement. A within-aggregate, to-entity many-to relationship doesn't make any sense, as then the aggregate cannot guarantee the conceptual integrity of the relationship (as another aggregate can manipulate the state of the entity). A to-aggregate relationship of that kind would be modeled as a by-id reference so that no JPA mapping is involved.
• Generally speaking, I'd avoid bidirectional relationships between entities at any cost. They're notoriously difficult to get right with JPA, as you have to make sure to always update both sides of the relation.

In a practical example, an order would be assigned to a customer by keeping an ID. All Orders for a particular customer would not be modeled as a collection in Customer, but rather as a query on the OrderRepository. This not only keeps the dependency directed (order -> customer), it also makes sure that the resolution of a potentially unbounded collection is not affected by persistence level shenanigans such as lazy loading. The application code would have to manually resolve that relationship and can apply controlling measures such as pagination, etc.

Regarding the “drinks by orders” example, I'd argue that's a statistics requirement that would be tracked in a separate module. On the other hand, the drinks module could listen to an event interface (DrinksOrdered) that could be implemented by the OrderCompleted event to potentially update the statistics numbers if those were to be held within the Drink instance itself.

[ghferrari]

Hi @odrotbohm ,

I've decided it's best to deactivate the cyclic modules check and all I really need right now are some simple instructions on how to do that. I'm happy to explain my reasoning another time...

[odrotbohm]

I'd be happy to take a look and tweak an example project you prepare that lays out the arrangement you would like to see improved.

[ghferrari]

Like I say, all I want right now is to deactivate the cyclic modules check. There is a longer discussion to have about why I think the restriction on cyclic dependencies isn't in fact a good idea (short version: I think the workarounds required to meet that constraint are an order of magnitude worse than the alleged problem it tries to solve), but I don't know when I'll have time to engage in that discussion properly.

Hopefully it is left up to the package user to decide if they actually want to adopt the restriction on cyclic dependencies. After all, it seems a separable concern from thinking that direct intermodule communication should take place via modules' public APIs and direct intermodule coupling can be reduced through event-based communication. What if I like these two ideas but I'm not such a fan of the "no cyclic dependencies" restriction?

[ghferrari]

Hi @odrotbohm ,

Welcome back from your conference tour :-) I enjoyed hearing your latest thoughts on moduliths etc via the YouTube Devoxx recordings. In the spirit of your remark that "you never learn anything from someone you agree with", here's my attempt to disagree with you on the requirement for no cyclic dependencies between modules. This issue remains an important one for me and I think it would be interesting to explain my own point of view and, certainly, to hear your response.

Let's suppose that we have a simple spring modulith project with two modules, M1 and M2, and let's suppose our database is a standard relational/SQL database. Suppose M1 defines some entity A and M2 defines some entity B such that B has a reference to entity A. At a minimum, this means that the database table for B is defined with a foreign key reference to A. I don't mind how this relationship from B to A is represented in code. Perhaps B includes an attribute which references A's class or perhaps (with DDD in mind) B will reference only A's id class. Because B imports some classes from A, it must therefore be the case that M2 (containing B) depends on M1 (containing A).

So, we have two modules, M1 and M2. M2 is the root of the dependency graph (it has no dependency on any other module) while M2 depends on M1. Finally, let's assume that we have REST API endpoints provided by M1 and M2 which allow read/write access to these entities. So M1 provides something like /entityA and M2 provides something like /entityB.

I think that's all the setup I need, so onto my questions...

Entity A and entity B are represented in the database by two tables, call them entity_a and entity_b. The entity_a table, we'll assume, does not contain any foreign key columns, but of course, the entity_b table includes a foreign key referencing the primary key of the entity_a table. So, our database tables also exhibit a situation in which entity_a does not depend on any other table but entity_b depends on entity_a. However, it is also true that, as far as the database is concerned, we can join these two tables and query, as it were, from A's side. For example, it's possible to write a SQL query that would show each A entity next to the count of their associated B entities. Or we could write a SQL query to show only those A entities which have at least one B entity. Or we could write a query to show all the A entities which have no associated B entities. Naturally, there are many cases where this information is valuable so we certainly want to enable this sort of querying to take place. And as a starting idea, it seems intuitive that all these queries should belong to M1 because these queries return A entities.

The problem I have is how to achieve this with Spring Modulith. Away from Spring Modulith (if we were thinking of A and B as just being in different packages rather than in different modules...) the obvious answer is just to write the queries in M1 and allow the user to retrieve these queries via queries parameters on the /entityA endpoint. For example, we could have /entityA?ownsAtLeastBs=1 to retrieve all A entities which have at least one associated B entity. But of course, this means that in some sense M1 now depends on M2 because it has a query which uses data from M2. So, now M2 depends on M1 and M1 also depends on M2.

So, how is one supposed to achieve this sort of thing in Spring Modulith if there can never be bidirectional dependencies between modules? In fact, we should split the question in two. How (if at all) can you do this sort of thing in Spring Modulith? And, assuming there is some way to do it, is it worth doing it this way rather than simply relaxing the restriction on bidirectional module dependencies?

I mostly want to hand over to you now to hear how you would handle this, but here are some possibilities I've considered myself.

1. Implement these queries for module M1 via e.g. a native SQL query. Then there will be no code dependencies from M1 to M2 (unlike, say, if one defined a query in M1 using JPQL since any such query would reference entity B which is defined in M2). Spring Modulith might well permit this, but it seems to go against the modulithic spirit to use a bidirectional relationship, even if it's below Spring Modulith's radar.
2. Implement an interface in M1 which allows it to query M2. But this means that instead of using a simple SQL query and allowing the database and SQL/JPQL to do what they do best, one has to essentially implement that query by hand in code. This just seems like a terrible idea.
3. Implement the query in M2. I suppose you could at least the define the query here, using whatever references you like to M1. But at what endpoint do you present these queries? The original objective was to allow the user to run these queries via query parameters on the /entityA endpoint provided by M1. Where is M2 going to present these queries? And if M2 needs to return a query result of entities from M1, doesn't this violate the idea that M2 should only refer to entities in M1 via id?
4. Create a third module, M3, which is downstream from both M1 and M2 and which is used only for queries. One way this could work would be for M1 and M2 to issue events from which M3 will then reconstruct a copy of the essential data regarding entity A and entity B. Not only does this sound like a terrible waste of time (using a potentially error-prone, eventually consistent event driven architecture to achieve what is absolutely trivial to achieve via a trivial database query), but it's not even clear to me how it would work. Presumably, M3 is not also going to provide an /entityA endpoint. So, I imagine the best it can do is provide an alternative endpoint such as /reports/entityA which allows filtering via association with B entities. If that's the suggestion, then all I can say is that, as well as seeming like a huge amount of extra work to achieve an outcome that is not as good the original solution, it's not clear to me that this even fulfils the original objective of letting one filter via the original /entityA endpoint.

Thanks for working through this. I hope I've made the case that it's an important question and perhaps you're able to see some simpler solutions.

Ultimately, I think this sort of "downwards" querying is going to be absolutely ubiquitous in any non-toy application, and I think it does genuinely put pressure on the idea that bidirectional dependencies should be made impossible. I'm supportive of the idea that bidirectional dependencies should perhaps be minimized, but that's not because I think bidirectional dependencies are intrinsically bad (see next paragraph...), it's because I think all dependencies should be minimized. I'm also supportive of the idea that it's helpful to be able to keep track of bidirectional dependencies, but again, that's not because bidirectional dependencies are instrinsically bad, it's because it's useful to keep track of all intermodule dependencies. And remember, we are already working in a "modular" arrangement in the sense that modules can only depend on other modules' exposed APIs, and that applies to bidirectional dependencies, too. So, I just don't find that there's any reason to make bidirectional dependencies impossible. If you could always easily replace transform a system with bidirectional dependencies into a DAG, then maybe that would be worthwhile, but my own conclusion, based on the example above, is that this simply isn't possible. So, my vote is that Spring Modulith should not absolutely enforce a restriction on bidirectional dependencies. Perhaps, for example, Modulith could be configured with three levels of bidirectional checking - ignore, warn and error.

There is perhaps a deeper philosophical argument to be made, too. A bidirectional dependency is not different in kind from a uni-directional dependency. Wherever you have a bidirectional dependency, it is really just two one-way dependencies, and we already accept that one-way dependencies are fine and often essential. Elsewhere I think you have written that enforcing a DAG is the "key to an evolvable architecture", but I don't ultimately think that's true. I think enforcing the boundary between a module's exposed APIs and its internals is good and helpful. I think allowing modules to communicate via events is good and helpful. And I think having a system like Spring modulith which can show you the dependencies for a particular module is good and helpful - it's helpful, for example, because if I want to make a change in one module, it's helpful to know what other modules that might affect. But I can accept all of that without thinking that a ban on bidirectional dependencies is always good and helpful. (I suppose someone might argue that if you allow bidirectional module dependencies, then there will be "circles" in the dependency graph and so the number of modules that might be affected by a change is potentially infinite, but that's incorrect. All Spring Modulith applications will have a finite number of modules, so the number of modules affected by a change will also be finite.) So, ultimately, I think the ban on bidirectional dependencies is unmotivated - it's a solution that's looking for a (non-existent) problem.

If all the above isn't enough to convince you, perhaps it worth adding simply that representing dependencies in one's domain can have business value and if there is an important bidirectional dependency in the world that your business works in, why wouldn't you want to be able to represent it?

Thanks for reading and I look forward to hearing your response.