I1544 backend dependencies #1554

pushyamig · 2023-11-17T16:38:57Z

Fixes# 1544

Changes Involved

zoneinfo default timezone. migration from using pytz to using zoneinfo.
Trusted Origin change: Needs an HTTPS in the configuration
Cron changes are from update to SQLAlchemy V2 and psycopg V3
1. As part of this change to postgres new driver update, you cannot make 2 queries in single format structure under the coding pattern cron.py adopted. So need to split the submission query into two.
2. So I need to implement a different connection for this and need to run in a session. Otherwise it won't remember the temporary table all_assign_sub.
django-graphene had a major upgrade as well and they removed the Promise Based Loader.
1. I have used a new Library to resolve issues. (before it crashed Assignment planning view)
2. At some point we need to convert all the GraphQL implementation code to adopt to this new pattern
3. But there is an issue Open to support Promise based Loader
The changes are in MyLA beta. You can access it using Canvas Beta.
1. Ran Cron for these changes by nulling the date_last_updated .So It took about 9min to run about 56 courses.

dashboard/settings.py

docs/loading_data.md

dashboard/management/commands/term.py

dashboard/graphql/query.py

dashboard/graphql/objects.py

pushyamig · 2023-11-17T18:50:20Z

dashboard/cron.py

+                df = pd.DataFrame(result.fetchall(), columns=result.keys())
+                df = df.drop_duplicates(keep='first')
+                df.to_sql(con=engine, name='submission', if_exists='append', index=False)
+


Split up the Submission into 2 since new postgres driver did not supported 2 queries in one. I have details on this in the PR description

pushyamig · 2023-11-17T18:55:25Z

dashboard/cron.py

-def execute_db_query(query: str, params: List = None) -> ResultProxy:
-    with engine.connect() as connection:
+def execute_db_query(query: str, params: Dict = None) -> ResultProxy:
+    with engine.begin() as connection:


I have to make this change otherwise the changes were not Auto-committed(As part of the SQLALchemy v2 changes)

pushyamig · 2023-11-17T18:56:31Z

dashboard/admin.py

@@ -160,8 +164,6 @@ def has_change_permission(request, obj=None):
    def has_delete_permission(request, obj=None):
        return False

-admin.site.register(AcademicTerms, TermAdmin)
-admin.site.register(Course, CourseAdmin)



All these change are part of prior Django 3.x version, and Django-upgrade lib made these changes.

config/cron_udp.hjson

Dockerfile

jaydonkrooss · 2023-11-27T21:35:56Z

build was successful (after changing the is_superuser setting which was blocking me initially), and the cron job ran successfully for both courses I was testing in beta

pushyamig · 2023-11-28T15:14:20Z

@zqian and @jonespm , i would like you both review this code since it has changes to area where I might not contributed to the code like cron and GraphQL.

pushyamig · 2023-12-13T20:56:19Z

@jonespm and @zqian If this PR is not reviewed that I have to unnecessarily blocking other people working on an issue or I might endup fixing all their issue if merged before this due to library upgrade .

docs/github_actions.md

dashboard/settings.py

dashboard/management/commands/term.py

dashboard/graphql/view.py

pushyamig · 2023-12-18T16:22:26Z

@zqian I have resolved the issues you commented on in the PR

zqian

Tested cron and tool. All work well.

jennlove-um · 2024-01-29T18:49:10Z

Cron runs fine and no issues detected in beta testing.