Enabling Module Aligent Prerender module on production made PHP-FPM flooded by request

[vmishra341]

Hi ,

We have faced issue with your prerender module and as soon as we enabled your module on production server our site seems very slow and it seems flood PHP-FPM request on server. Can you please check and resolve that issue? When we disable your module from admin panel site working fine again.

[aligent-lturner]

@vmishra341 apologies for the very late reply - I didn't get a notification about this issue being raised.
Are you able to provide any more details? The module is designed to send URL recaching requests to a prerender service, so it would be expected that there would be an increase in requests (since the service would be hitting the server to get the latest version of the page in question).

[barbazul]

I have faced a related issue in which thousands of recaching requests are being submited to Prerender.io on irrelevant URLs.

For example, imagine this catalog:

• Configurable Product (URL = configurable.html)
• Variant A (no public URL)
• Variant B (no public URL)
• Variant C (no public URL)

A catalog synchronization updates prices for all 3 variants, 3 entries are added to the index mview.
The indexer processes the entries and finds the associated parent and sends 4 recache requests to Prerender.io:

1. variant-a.html
2. variant-b.html
3. variant-c.html
4. configurable.html

However, only one recache request was expected

[barbazul]

I believe the strategy to add the parents here needs a little improvement as it adds the url of the parent product but doesn't remove the children or check for visibility o the products

[aligent-lturner]

@barbazul this sounds like it could be a data issue. If your variants are not visible individually, then they should not have any records in the url_rewrite table (which is where the URLs are looked up from).

[barbazul]

I'll look into it

[barbazul]

You pointed me in the right direction.

Turns out the issue is data related, but not related to configurables and children.

The catalog being managed has multiple instances of configurable products, marked with visibility "Catalog, Search" (so, they have URLs) but wcish are disabled.

I was working under the assumption that disabled products had no entry in url rewrites, but I was mistaken.

Having said that, disabled visible products display a 200 page that reads something like "This product is not available." so any changes to the product would not change the page being rendered.

Would it make sense to account for disabled visible products in the indexer? I think yes.

[aligent-lturner]

@barbazul potentially, it could be improved, yes. However, your initial assumption is not without basis. If products get disabled via admin, then their URL rewrites are removed (or are supposed to be, at least). So you're correct that disabled products shouldn't have any.

Joining to the EAV tables to filter out any disabled products probably isn't going to have too much of a performance impact, so I think we can probably add that logic as a safeguard.

[barbazul]

I have confirmed (through testing in different projects) that disabling a product through he admin does not remove the existing URL rewrites for that product if the visibility is not changed.

Additionally, I went through the URL Rewrites module code and found no reference to the product status being used.

Mainly I went through the process done by Magento\CatalogUrlRewrite\Observer\ProductProcessUrlRewriteSavingObserver and Magento\CatalogUrlRewrite\Observer\ProcessUrlRewriteOnChangeProductVisibilityObserver and failed to find any usage of status to create or remove URL rewrites.

The general criteria I found is:

• Does product have a url_key ?
• Is product visible (Catalog, search or both)?

If both are true, the URL is created / updated / removed.

The above is simplistic, there are additional considerations to websites and categories, but those are not related to the topic, I don't think.

Granted, the code is a little convoluted and there is a lot of indirection, so I probably missed something very obvious.