Implement file regex search for ColocatedMappingDriver

[rela589n]

Hello,

This PR adds a flexibility for ColocatedMappingDriver, allowing client code to customize files search by specifying an regex (rather than just file extension as previously).

This is very useful for the cases when tests, fixtures, factories are kept in the same namespace as entity (having one namespace for one Aggregate).

For example, if we structure classes for User Aggregate:

User/
├── User.php
├── UserFixture.php
└── Actions/
    └── Register/
        ├── RegisterUserCommand.php
        └── RegisterUserTest.php

Like here we have User/ namespace that includes everything possessed by the concept of User (e.g. Entity, Fixture, Command, Test, Controller, maybe some Value-Objects, etc.)

Right now Doctrine mapping is not ready for this kind of architecture, as it would scan all these files and include them, while they don't have to be included, so I would like to fix it with regex. It would not have been a big problem with all these files being included if there were no dev-things there, but they are there, so it will possibly break in prod environment, if we keep them.

Therefore, I introduced new $fileRegex property that is used when filtering, and it could be set from the client code.
Old $fileExtension I have deprecated so that we could use $fileRegex.

Please, let me know if you don't have objections about this, and what further steps I need to do?

Also, please note that this is my first PR to Doctrine Project, so I would appreciate it if you'll be patient with me.

[greg0ire]

Hi! I retargeted on 4.1.x since this is no bugfix, but now there are conflicts, please address them.

[rela589n]

@greg0ire , fixed the conflicts

[greg0ire]

Thanks for your contribution. Which documentations will need to be upgraded if we go with this change?

Also, you should add upgrade instructions in UPGRADE.md

[greg0ire]

I think it would be fair to write more tests targeting this change beside testing getters and setters.

[rela589n]

Actually, the resulting regex has never changed.

So, as it was '/^.+\.php$/i', so it is now:

self::assertSame('/^.+\.php$/i', $driver->getFileRegex());

I've added this assertion to testSetFileRegex that verifies that regex is actually one we expect same as was before.

[greg0ire]

And that's great, definitely something we'd want to test. But I think the new functionality you unlocked should be demonstrated in a test based on the filetree you showed in the description of this PR.

[rela589n]

do you want me to create a new test for it, or maybe I could adjust testGetAllClassNames() so that it would use regex to find only needed classes?

[stof]

I think it should be a new test

[rela589n]

All right, so I added the test that will collect only needed entities by regex: testGetAllClassNamesWithRegex

I think it should be quite obvious that if testGetAllClassNames asserts about [Entity::class, EntityFixture::class], while new test asserts only about [Entity::class], this means that fixture has been excluded by regex.

[rela589n]

Hi, @stof , @greg0ire , could you check it once more please?

[greg0ire]

Which documentations will need to be upgraded if we go with this change?

Please answer this.

[rela589n]

Which documentations will need to be upgraded if we go with this change?

Please answer this.

Sorry, I didn't get the point. What documentations do you mean? I thought you were talking about UPGRADE.md, or is it anything else you are talking about?

[greg0ire]

I mean: is there a documentation page on doctrine-project.org or maybe even symfony.com detailing how to configure mapping drivers? In the end, how will end-users know how to configure their projects to allow using the file tree you described in your original post?

[rela589n]

I see that ColocatedMappingDriver is only used in AttributeDriver, yet setFileExtension is never used.

Doctrine Bundle doesn't set it, as it doesn't have support for it.

Also, I searched a bit through the docs, and see that method setFileExtension is all about other places, not related to ColocatedMappingDriver:

• \Doctrine\Common\ClassLoader (common)
• \Doctrine\RST\Configuration (rst)
• \Doctrine\ORM\Mapping\Driver\XmlDriver (and SimplifiedXmlDriver) (orm xml driver) - this one is outdated, since there's no such method as setFileExtension for XmlDriver, as it's not in abstract FileDriver. Right now it seems to be possible with setLocator, or ->getLocator()->setFileExtension() (yet this isn't in FileLocator interface), so I think this doc should be updated.

Regarding the new setFileRegex, I think it could be referenced in advanced orm configuration options.

[rela589n]

@greg0ire , hi, could you please guide on the next steps to take?

[greg0ire]

No occurrences in the ODM docs? I think next steps would be to get more approvals on this, so I'll request more reviews. In parallel to that, you could open draft PRs updating the docs, it would help reviewers understand the changes you are proposing, and will be needed anyway.

[rela589n]

No occurrences in the ODM docs?

I checked for ODM docs, and found one more entry (relevant to setFileExtension, but not to ColocatedMappingDriver):

• Doctrine\ODM\MongoDB\Mapping\Driver\XmlDriver (and SimplifiedXmlDriver) (odm xml driver) - this doc seems to be outdated, since there's no such method as setFileExtension in XmlDriver, as it's neither in abstract FileDriver. This is the same very issue described in my last comment, so I think this doc should be updated as well.

No different from already described.

Also, there are few references from Yaml mapping drivers (odm, orm), though I don't think they should be updated, since they've been removed.

In parallel to that, you could open draft PRs updating the docs

I'll try to do it soon.

[rela589n]

Hi guys, hope you are doing well!

@greg0ire , @SenseException , could you suggest moving forward

[greg0ire]

I'll be honest I'm not too crazy about the result (I doubt many developers know about negative lookahead, I'm not even sure I've used one myself ever). I think we should start from the UI, and imo what would be best would be to have the user provide a list of glob patterns for exclusion, and then build a regex from that if necessary. Letting them set a regex is very powerful, but seems error prone and hard to test. Are there other use cases than the one you mentioned for this? If not then maybe the API should be restricted to that use case, and made easier to use.

Also, is there an easy way for users to validate their changes, like a command that allows getting a list of mapping files somewhere?

I'd like to know what you and other maintainers think about this.

[rela589n]

You don't have to follow the line of how the green grass eaten by the black cow produced a white milk to enjoy the result, so don't the developers need to know how the negative lookahead works to use it.

seems error prone and hard to test

It is not. There's an easy tool to test the regex https://regex101.com/

This is the most powerful implementation, and it doesn't have any boundaries or limitations whatsoever.

You'd be able to map the entities inside any nested strucutures organized in whatever arbitrary way you might want. You might include mappings from any deep nested Entity, or ValueObject directories of the project.

Implementing it otherwise would be a deviation from the best option.

Also, is there an easy way for users to validate their changes, like a command that allows getting a list of mapping files somewhere?

I think you are the one who knows it

[greg0ire]

You don't have to follow the line of how the green grass eaten by the black cow produced a white milk to enjoy the result, so don't the developers need to know how the negative lookahead works to use it.

They don't, what I'm saying is

fileRegex: '/^(?!.*Fixture\.php$).*\.php$/'

is quite a mouthful compared to

excludeGlobs: '**/*Fixture.php'

This is the most powerful implementation, and it doesn't have any boundaries or limitations whatsoever.

Which is why I'm asking what other use cases (than exclusion) it has.

You'd be able to map the entities inside any nested strucutures organized in whatever arbitrary way you might want. You might include mappings from any deep nested Entity, or ValueObject directories of the project.

Sounds like this could be done with 2 globs, or 2 array of globs.

Implementing it otherwise would be a deviation from the best option.

It would be a deviation from the simplest option to implement, I'm not sure we can equate that to the best option.

I think you are the one who knows it

You're assuming I use Doctrine regulary, and that's a fair assumption, but I'm afraid it's very wrong 😬

[rela589n]

Do you want to code for every possible solution, already covered by the flexible solution?

I only claim for a flexible solution, not any other.

[greg0ire]

Maybe you're right, I'm really not sure about this. I'd like to hear opinions from other maintainers. @beberlei @stof @malarzm @alcaeus @SenseException @derrabus , sorry for the ping, but maybe you have an opinion.

[stof]

Asking people to write a single complex regex to include things is quite bad for DX IMO. Such regular expressions will very quickly become unmaintainable (and many devs are not familiar with lookaheads in regex).

I think being able to configure some exclusions (supporting multiple ones to avoid having to write a crazy exclusion rule) would lead to much better DX for the case of excluding a Fixtures folder for instance.

[greg0ire]

OK, it's not just me. @rela589n I'm truly sorry for not pointing that out earlier.

[rela589n]

Why in the world is it bound to the particular scenario of usage?

This PR is not intended to implement exclusion rules.

It is intended to provide setter for a regex used within

I'm no longer sure about this from the usability standpoint.