mlBundle Load data and modules into a different database

[DavidEnnis-CleverLlamas]

Hello.

Using MlBundle, is it possible to load ml-modules and ml-data into a different database (for which we would have configured under ml-config) ?

-David

[rjrudin]

Hey @DavidEnnis-CleverLlamas - the current approach is to make those directories in the bundle additional paths of schemas / modules / data / config that are then loaded via the same connection for the schemas / modules / data / config that's part of your project.

One approach for specifying a different database (assuming that's the only part of the connection config that would change) for the data in a bundle would be something like this:

mlBundleDataDatabases=nameOfBundle1,nameofDatabase1,nameOfBundle2,nameOfDatabase2

That's certainly doable.

Modules however are tricky. Non-REST-API modules would work the same way as data. But REST API modules have to be loaded via specific REST API endpoints, and those do not allow for specifying a different modules database. So we could support something like mlBundleModuleDatabases, with a big caveat of "This doesn't work for REST API extensions in a bundle".

Thoughts?

[DavidEnnis-CleverLlamas]

HI @rjrudin

Thank You for the reply. Sounds interesting.

I may not have made things clear enough, unfortunately. The idea is that I would want the bundle to have to have settings within itself and not have to have someone modify gradle properties outside of the bundle. And of course, gradle properties are too late to evaluate from a bundle.

I know I could accomplish this differently - a gradle plugin that deployed my artifacts and hooked into the ml-gradle process via tasks and task dependencies. However, I do like the elegance of the mlBundle approach.

Lets think of this in another way that is more generic to ml-gradle and also has some side-effect benefits:

ml-gradle already has an elegant way to handle collections and permissions on a file level(included wildcards - so also essentially a directory (and subdirectory) level. Perhaps that could be extended with a file called database.properties and then follow the same logic?

[rjrudin]

Got it, and you're right, I was headed in a different direction.

So the use case is - you want to load some files as modules / data / schemas, but you want to override certain parts of the connection configuration, correct? Could it also be the port, and not just the database?

I ask because database.properties could certainly function the same as collections.properties and permissions.properties, with filename patterns associated with database names. But I'm wondering if may be necessary to override other parts of the connection as well.

[DavidEnnis-CleverLlamas]

Hi Rob.

For our use-case it was about the deploy side of artifacts modules, schemas, documents.

All of our other known items are already covered via the existing capabilities of ml-config-path already being appended with the bundle ml-config directory.

If you were to consider something about connection strings, I'm not sure where that would reside. Trying to keep the changes as a natural part of a bundle and by design it only exists after the first pass of build.gradle and I would not want to start messing with that code.

[DavidEnnis-CleverLlamas]

Hi Rob. Any more thoughts on this or would it be better for us to just create a solution not using ml-bundle?

Regards,
David

[rjrudin]

Sorry, just got back from vacation yesterday.

I think you're saying a bundle isn't necessarily needed here? That your main use case is being able to load some data/modules/schemas into databases other than the default ones?

[DavidEnnis-CleverLlamas]

Hi Rob. No problem. Same here: holidays for a week. In fact, we use all of the goodness of ml-bundles. We just want my bundle. ml-data, myBundle.ml-modules and schemas from myBundle to deploy to the databases that we configure in myBundle.ml-config directory. The solution of the addition al xxx.properties files made the feature generic for all, but also meet our needs by setting them inside of our directories in the bundle. Make sense or am I confusing things more? Regards, David

…

On Tue, 8 Jul 2025, 14:32 Rob Rudin, ***@***.***> wrote: *rjrudin* left a comment (marklogic/ml-gradle#752) <#752 (comment)> Sorry, just got back from vacation yesterday. I think you're saying a bundle isn't necessarily needed here? That your main use case is being able to load some data/modules/schemas into databases other than the default ones? — Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABD2VTG7UTSSFQX3ROG47ID3HPB5TAVCNFSM6AAAAAB7T2DU6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANBYHE4DQNBTGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rjrudin]

Couple more questions:

1. Is the expectation that ml-gradle can recognize the content/modules/schemas databases in the bundle's config directory via filename?
2. Is the expectation that loading into those content/modules/schemas databases can be done using the same connection configuration as used for the default databases, just with a different database name for the connection?

One issue - REST extensions have to be loaded via a REST API app server that points at the modules database. They can't be loaded directly into the modules database.

[DavidEnnis-CleverLlamas]

Hi Rob

Regarding Your questions:
"Is the expectation that ml-gradle can recognize the content/modules/schemas databases in the bundle's config directory via filename?"_

Only if this is useful or beneficial for implementation. Otherwise, I would just expect that if we used the approach of module-database.properties, then it would be up to us to ensure that databases in the properties file matched databases that have been deployed (which in our case is delivered in ml-config).

Note: You could also just consider a top level bunde.properties file where modules, data and schemas database can be defined. Then when processing the build/ml-bundle/myBundle directory, these just get overridden.

"Is the expectation that loading into those content/modules/schemas databases can be done using the same connection configuration as used for the default databases, just with a different database name for the connection?"

Yes. However, if you consider (Note) above, then a bundle could perhaps temporarily override any property when said bundle was processed.

"One issue - REST extensions have to be loaded via a REST API app server that points at the modules database. They can't be loaded directly into the modules database."

This is already known, understood and acceptable. Data services are still a viable option.

I hope that the above added more clarification than it did confusion.

Kind Regards,
David

[rjrudin]

Okay - think I got it now. It sounds like supporting a database.properties file in a directory, similar to collections.properties and permissions.properties, would solve the problem of allowing a user to load data/modules/schemas into any database. There'd be a known limitation of this not working for REST API extensions, which is fine.

Does that solve the problem?

[DavidEnnis-CleverLlamas]

Okay - think I got it now. It sounds like supporting a database.properties file in a directory, similar to collections.properties and permissions.properties, would solve the problem of allowing a user to load data/modules/schemas into any database. There'd be a known limitation of this not working for REST API extensions, which is fine.

Does that solve the problem?

Absolutely!

[rjrudin]

Got it - we're working on a 6.0 release as it will be upgrading the Spring 5.x libraries to 6.x, thus requiring Java 17 and thus requiring a major version bump. We'll keep this around and see if it can be part of 6.0.

[DavidEnnis-CleverLlamas]

Thanks Rob!

…

On Fri, 11 Jul 2025, 14:49 Rob Rudin, ***@***.***> wrote: *rjrudin* left a comment (marklogic/ml-gradle#752) <#752 (comment)> Got it - we're working on a 6.0 release as it will be upgrading the Spring 5.x libraries to 6.x, thus requiring Java 17 and thus requiring a major version bump. We'll keep this around and see if it can be part of 6.0. — Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABD2VTEUIBIWC2B37WAIMLT3H66FPAVCNFSM6AAAAAB7T2DU6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTANRSGQYTQMBXGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>