Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Magento 2 Recipe: Cache prefix generated different cache prefixes when deploying to multiple hosts #3790

Closed
dverkade opened this issue Mar 18, 2024 · 22 comments · May be fixed by #3822
Closed

Comments

@dverkade
Copy link
Contributor

dverkade commented Mar 18, 2024

Situation:

We have Magento 2 projects which are load balanced between different servers. For instance 2 frontend web servers for visitors and a seperate backend service for the Magento backoffice. All servers point to the same Redis instance which is located on the database server. We're deploying with the configuration:

after: deploy:shared: magento:set_cache_prefix deploy:magento: magento:cleanup_cache_prefix

Actual result:

The set cache prefix is different per host because it will take the alias and release number. So for one server it will be:

production_web35_12_

And on the other server the prefix is set to:

production_web40_9_

Issue:

The issue is that cache will not be cleared correctly. The prefix should be the same for all 3 webservers in order for 1 webserver to clear the cache for the other two as well.

Expected result:

The generated cache key should be the same for all hosts within the same "stage" and release. So if we deploy to "stage" production to 3 webservers, the generated key should be the same for all 3 servers for Magento to work correctly.

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar
@dverkade
Copy link
Contributor Author

@valguss Hope you have any suggestions how we could resolve this.

@valguss
Copy link
Contributor

valguss commented Mar 18, 2024

For the moment i'd suggest disabling using the cache_prefixes
Is there a way for you to sync up the releases, so they are the same number?

@dverkade
Copy link
Contributor Author

@valguss, yes, I can get the release in sync quite easily. However I'm stuck with the first / "alias" part of the key.

@valguss
Copy link
Contributor

valguss commented Mar 18, 2024

You could override the task in your deployer.php with something like the following (not ideal I know):

desc('Update cache id_prefix');
task('magento:set_cache_prefix', function () {
    //download current env config
    $tmpConfigFile = tempnam(sys_get_temp_dir(), 'deployer_config');
    download('{{deploy_path}}/shared/' . ENV_CONFIG_FILE_PATH, $tmpConfigFile);
    $envConfigArray = include($tmpConfigFile);
    //set prefix to `alias_releasename_`
    $prefixUpdate = 'abc_' . get('release_name') . '_';

    //check for preload keys and update
    if (isset($envConfigArray['cache']['frontend']['default']['backend_options']['preload_keys'])) {
        $oldPrefix = $envConfigArray['cache']['frontend']['default']['id_prefix'];
        $preloadKeys = $envConfigArray['cache']['frontend']['default']['backend_options']['preload_keys'];
        $newPreloadKeys = [];
        foreach ($preloadKeys as $preloadKey) {
            $newPreloadKeys[] = preg_replace('/^' . $oldPrefix . '/', $prefixUpdate, $preloadKey);
        }
        $envConfigArray['cache']['frontend']['default']['backend_options']['preload_keys'] = $newPreloadKeys;
    }

    //update id_prefix to include release name
    $envConfigArray['cache']['frontend']['default']['id_prefix'] = $prefixUpdate;
    $envConfigArray['cache']['frontend']['page_cache']['id_prefix'] = $prefixUpdate;

    //Generate configuration array as string
    $envConfigStr = '<?php return ' . var_export($envConfigArray, true) . ';';
    file_put_contents($tmpConfigFile, $envConfigStr);
    //upload updated config to server
    upload($tmpConfigFile, '{{deploy_path}}/shared/' . TMP_ENV_CONFIG_FILE_PATH);
    //cleanup tmp file
    unlink($tmpConfigFile);
    //delete the symlink for env.php
    run('rm {{release_or_current_path}}/' . ENV_CONFIG_FILE_PATH);
    //link the env to the tmp version
    run('{{bin/symlink}} {{deploy_path}}/shared/' . TMP_ENV_CONFIG_FILE_PATH . ' {{release_path}}/' . ENV_CONFIG_FILE_PATH);
});

I've just updated

$prefixUpdate = get('alias') . '_' . get('release_name') . '_';

to

$prefixUpdate = 'abc_' . get('release_name') . '_';

@dverkade
Copy link
Contributor Author

@valguss, thanks. We can do a workaround but would like to see this fixed in the official recipe as well and would like to contribute to a solution.

Best practice is to have the same redis DB per environment. So in theory we could drop the "get('alias')" part of the prefix all together. Or we could use the "stage" of the host, so that the first part of the prefix is the same for all hosts in that stage. Can you just do get('stage') in order to get the host stage?

@valguss
Copy link
Contributor

valguss commented Mar 19, 2024

Using stage sounds like a better solution. The premise I was going for was for our specific setup where staging sites sit on the same server as the live env, so was trying to make the key unique to each environment so that they didn't clash. Will get a PR up shortly to change it though.

Thanks

@dverkade
Copy link
Contributor Author

dverkade commented Apr 3, 2024

Using stage sounds like a better solution. The premise I was going for was for our specific setup where staging sites sit on the same server as the live env, so was trying to make the key unique to each environment so that they didn't clash. Will get a PR up shortly to change it though.

Thanks

Stage sounds good to me. I get your use case and with using "stage" this still works. I do think it's better to use a different Redis database number for a test environment and a live environment when they are both on the same server. Something like this:

            'page_cache' => [
                'backend' => 'Cm_Cache_Backend_Redis',
                'backend_options' => [
                    'server' => '127.0.0.1',
                    'port' => '6379',
                    'database' => '1',  ## HAVE A DIFFERENT DB NUMBER HERE FOR TEST OR PRODUCTION ENVIRONMENTS
                    'compress_data' => '0'
                ]
            ]

@valguss
Copy link
Contributor

valguss commented Apr 26, 2024

Now updated to check if stage is available and fall back to alias

@barryvdh
Copy link
Contributor

barryvdh commented Aug 7, 2024

What do you think about something like this:

desc('Update cache id_prefix');
task('magento:set_cache_prefix', function () {
    // Copy shared file to current release
    run('rm {{release_path}}/app/etc/env.php');
    run('cp {{deploy_path}}/shared/app/etc/env.php {{release_path}}/app/etc/env.php');

    $prefix = uniqid() .'_';
    run('{{bin/php}} {{release_or_current_path}}/bin/magento setup:config:set --cache-id-prefix='.$prefix.' --page-cache-id-prefix='.$prefix.' -n');
});

/**
 * After successful deployment, move the .env back to a symlink
 */
desc('Cleanup cache id_prefix env files');
task('magento:cleanup_cache_prefix', function () {
    run('mv {{deploy_path}}/shared/app/etc/env.php {{deploy_path}}/shared/app/etc/env.php.backup');
    run('mv {{release_path}}/app/etc/env.php {{deploy_path}}/shared/app/etc/env.php');
    run('{{bin/symlink}} {{deploy_path}}/shared/app/etc/env.php {{release_path}}/app/etc/env.php');
    run('rm {{deploy_path}}/shared/app/etc/env.php.backup');
});

So

  • Copy env to release
  • Run command to update prefix
  • Run build with new env
  • Move env back to shared.

You don't actually need the previous number do you? And to sync accross installs, maybe you can use a static variable to generate it once?

desc('Update cache id_prefix');
task('magento:set_cache_prefix', function () {

    // Copy shared file to current release
    run('rm {{release_path}}/app/etc/env.php');
    run('cp {{deploy_path}}/shared/app/etc/env.php {{release_path}}/app/etc/env.php');

    static $magentoCachePrefix;
    $magentoCachePrefix = $magentoCachePrefix ?: uniqid();

    run('{{bin/php}} {{release_or_current_path}}/bin/magento setup:config:set --cache-id-prefix='.$magentoCachePrefix.' --page-cache-id-prefix='.$magentoCachePrefix.' -n');
});

Copy link
Contributor

github-actions bot commented Sep 9, 2024

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com//discussions/3888]

1 similar comment
Copy link
Contributor

github-actions bot commented Sep 9, 2024

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com//discussions/3888]

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 9, 2024
@dverkade
Copy link
Contributor Author

Not sure why this is being closed as the PR is still open.

@peterjaap
Copy link
Contributor

The bot went a bit wild I guess. Reopening.

@peterjaap peterjaap reopened this Sep 10, 2024
@github-actions github-actions bot added Stale and removed Stale labels Sep 10, 2024
Copy link
Contributor

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com//discussions/3888]

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 10, 2024
@barryvdh
Copy link
Contributor

lol

@mautz-et-tong
Copy link

How good that we have bots to help us by taking over annoying tasks and taking some of the burden off our shoulders these days. I can't wait for AI to step in and help the bot to be even more productive for us.

@peterjaap
Copy link
Contributor

peterjaap commented Sep 10, 2024

Apparently the bot was a bit drunk. Reopening. Let's see if the gets the hint :-P

@peterjaap peterjaap reopened this Sep 10, 2024
@peterjaap peterjaap removed the Stale label Sep 10, 2024
@github-actions github-actions bot added the Stale label Sep 10, 2024
Copy link
Contributor

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com//discussions/3888]

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 10, 2024
@mautz-et-tong
Copy link

Ahh, I think @antonmedv has disallowed issues in general and just want to have discussion 👀

@barryvdh
Copy link
Contributor

Maybe set exempt-issue-labels and tag it with one of those? https://github.com/actions/stale?tab=readme-ov-file#exempt-issue-labels

@peterjaap
Copy link
Contributor

@mautz-et-tong you're right, I hadn't seen that #3888

@antonmedv
Copy link
Member

Ahh, I think @antonmedv has disallowed issues in general and just want to have discussion 👀

Yes. =)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants