Skip to content

Improve loading time for the achievements leaderboard. #2709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Improve loading time for the achievements leaderboard. #2709

wants to merge 1 commit into from
+15 −12

Conversation

drgrice1
Copy link
Member

Currently the user achievements are obtained from the database in the user loop. This takes a long time when there are lots of users. So instead, this gets all user achievements for all users before the loop and references them by user id and then achievement id.

Testing this on a course for 5000 users shows this gives a significant speed up on load time for the page. With the develop branch it takes around 25 seconds, and with this branch it takes around 3 seconds.

Note that there were also two redundant queries (one that listed all achievements and then one that fetched all achievements on lines 49 and 50 in the develop branch) for which the data from those queries was not even used. Those were removed.

@drgrice1
Copy link
Member Author

To help with testing this here is a class list with 5000 users.
longDemoCourse.lst.txt

Import that into a course (that will take a while) and assign all achievements to those users (this will take a long time -- about 10 minutes on my machine), and then open the achievements leaderboard page. With the develop branch you will wait a while. With this pull request it will be much faster.

Note that if you are using hypnotoad then you may need to increase the inactivity_timeout and heartbeat_timeout in conf/webwork2.mojolicious.yml for the achievement assignments to complete. Although I think that the assignments will complete even if the request times out. It will continue on the the server even though the request times out (not sure o this though).

@drgrice1 drgrice1 mentioned this pull request Apr 20, 2025
Open
drgrice1 added a commit to drgrice1/webwork2 that referenced this pull request Apr 20, 2025
@drgrice1
Speed up assignments of achievements.
7fe15ed 
If you have payed attention (or tested openwebwork#2709 and openwebwork#2710), then you know
that this takes a long time if there are a large number of users.  Way
to long to be at all reasonable.

This uses techniques like those used for assigning multiple sets to
users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and
`WeBWorK::DB::Schema::NewSQL::Std::put_records` methods, to speed up
assignment of achievements to users.

This brings the assignment time for 5000 users down from more than 8
minutes, to less than 30 seconds.
drgrice1 added a commit to drgrice1/webwork2 that referenced this pull request Apr 20, 2025
@drgrice1
Speed up assignments of achievements.
779e580 
If you have payed attention (or tested openwebwork#2709 and openwebwork#2710), then you know
that this takes a long time if there are a large number of users.  Way
to long to be at all reasonable.

This uses techniques like those used for assigning multiple sets to
users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and
`WeBWorK::DB::Schema::NewSQL::Std::update_records` methods, to speed up
assignment of achievements to users.

This brings the assignment time for 5000 users down from more than 8
minutes, to less than 30 seconds.
@drgrice1 drgrice1 mentioned this pull request Apr 20, 2025
Open
drgrice1 added a commit to drgrice1/webwork2 that referenced this pull request Apr 20, 2025
@drgrice1
Speed up assignments of achievements.
bac62f7 
If you have payed attention (or tested openwebwork#2709 and openwebwork#2710), then you know
that this takes a long time if there are a large number of users.  Way
to long to be at all reasonable.

This uses techniques like those used for assigning multiple sets to
users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and
`WeBWorK::DB::Schema::NewSQL::Std::update_records` methods, to speed up
assignment of achievements to users.

This brings the assignment time for 5000 users down from more than 8
minutes, to less than 30 seconds.
drgrice1 added a commit to drgrice1/webwork2 that referenced this pull request Apr 20, 2025
@drgrice1
Speed up assignments of achievements.
fe70a13 
If you have payed attention (or tested openwebwork#2709 and openwebwork#2710), then you know
that this takes a long time if there are a large number of users.  Way
to long to be at all reasonable.

This uses techniques like those used for assigning multiple sets to
users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and
`WeBWorK::DB::Schema::NewSQL::Std::update_records` methods, to speed up
assignment of achievements to users.

This brings the assignment time for 5000 users down from more than 8
minutes, to less than 30 seconds.
drgrice1 added a commit to drgrice1/webwork2 that referenced this pull request Apr 20, 2025
@drgrice1
Speed up assignments of achievements.
6835c2e 
If you have payed attention (or tested openwebwork#2709 and openwebwork#2710), then you know
that this takes a long time if there are a large number of users.  Way
to long to be at all reasonable.

This uses techniques like those used for assigning multiple sets to
users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and
`WeBWorK::DB::Schema::NewSQL::Std::update_records` methods, to speed up
assignment of achievements to users.

This brings the assignment time for 5000 users down from more than 8
minutes, to less than 30 seconds.
@drgrice1
Improve loading time for the achievements leaderboard.
682587d 
Currently the user achievements are obtained from the database in the
user loop.  This takes a long time when there are lots of users. So
instead, this gets all user achievements for all users before the loop
and references them by user id and then achievement id.

Testing this on a course for 5000 users shows this gives a significant
speed up on load time for the page.  With the develop branch it takes
around 25 seconds, and with this branch it takes around 3 seconds.

Note that there were also two redundant queries (one that listed all
achievements and then one that fetched all achievements on lines 49 and
50 in the develop branch) for which the data from those queries was not
even used.  Those were removed.
@drgrice1 drgrice1 force-pushed the achievement-leaderboard-speed-up branch from 3581960 to 682587d Compare April 21, 2025 10:26
drgrice1 added a commit to drgrice1/webwork2 that referenced this pull request Apr 22, 2025
@drgrice1
Speed up assignments of achievements.
b13a38f 
If you have payed attention (or tested openwebwork#2709 and openwebwork#2710), then you know
that this takes a long time if there are a large number of users.  Way
to long to be at all reasonable.

This uses techniques like those used for assigning multiple sets to
users, i.e., the `WeBWorK::DB::Schema::NewSQL::Std::insert_records` and
`WeBWorK::DB::Schema::NewSQL::Std::update_records` methods, to speed up
assignment of achievements to users.

This brings the assignment time for 5000 users down from more than 8
minutes, to less than 30 seconds.
pstaabp
pstaabp approved these changes Apr 26, 2025
View reviewed changes
Copy link
Member

@pstaabp pstaabp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sped up the achievement leaders quite a bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pstaabp pstaabp pstaabp approved these changes

@Alex-Jordan Alex-Jordan Alex-Jordan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@drgrice1 @pstaabp @Alex-Jordan