Add file storage option #12590
Add file storage option #12590
Conversation
github-actions
bot
added
the
DEV: backend
Build Artifacts
|
There was a problem hiding this comment.
I think we probably want to keep the options.py interface restricted, others can override settings if they see fit, but we should focus on enabling the specific options we need.
Hint: if you do kolibri configure list-env you will see a complete list of available env vars for configuration (which will also show how your new option can be set as an env var).
kolibri/utils/options.py
Outdated
| @@ -359,6 +377,16 @@ def lazy_import_callback_list(value): | |||
|
|
|||
|
|
|||
| base_option_spec = { | |||
| "FileStorage": { | |||
| "DEFAULT_FILE_STORAGE": { | |||
There was a problem hiding this comment.
I think I'd want to hew closer to the pattern we have for the Cache and Database options here - and offer simple, pre-specified string options that refer to specific backends. If someone really wants to run a custom backend, they can override the settings file and do what they like.
That way, with specific backends in mind, we can then explicitly enumerate the additional things that need to be specified in each case - for example the default "file_system" backend value will need a path, if it's a GCS backend then other things might be required (or may be automagically configured in some cases).
kolibri/utils/options.py
Outdated
| except ImportError: | ||
| logger.error("Default file storage is not available.") | ||
| raise VdtValueError(value) | ||
| except Exception: |
There was a problem hiding this comment.
Shouldn't ever be catching a bare Exception, unless for very good reason - it can hide a multitude of sins.
kolibri/utils/options.py
Outdated
| modules = value.split(".") | ||
| klass = modules.pop() | ||
| module_path = ".".join(modules) | ||
| module = importlib.import_module(module_path) |
There was a problem hiding this comment.
Note that Django exposes a utility called import_string which we already use in this module for loading classes by a string dot path, so this seems preferable to use here.
kolibri/utils/options.py
Outdated
| @@ -15,6 +15,7 @@ | |||
| from configobj import ConfigObj | |||
| from configobj import flatten_errors | |||
| from configobj import get_extra_values | |||
| from django.core.files.storage import Storage | |||
There was a problem hiding this comment.
This is probably why flake8 wouldn't let pre-commit pass but it didn't give me useful output
| @@ -737,6 +766,7 @@ def _get_validator(): | |||
| "url_prefix": url_prefix, | |||
| "bytes": validate_bytes, | |||
| "multiprocess_bool": multiprocess_bool, | |||
| "storage_option": storage_option, | |||
There was a problem hiding this comment.
how will this work with database based cache?
There was a problem hiding this comment.
Hm - I'm not sure. I assumed this was only related to validation on initialization
rtibbles
modified the milestones:
Kolibri 0.17: Planned Patch 1,
Kolibri 0.17: Planned Patch 2
rtibbles
modified the milestones:
Kolibri 0.17: Planned Patch 2,
Kolibri 0.18: General maintenance
Summary
In order to allow us to use a cloud backend for the File Storage in Django, this adds a value to options.py which defaults to the Django FileSystemStorage (which... it did anyway but now we make sure of it).
This should make it configurable on BCK such that if there is a module that is a Google cloud backend class that implements the Django Storage class, then it can be added as the value for the settings.
For example, if we have a new class "GCloudStorage" in
kolibri.core.storagethen we would use that class if we set the option added here tokolibri.core.storage.GCloudStorage.This is very much a first whack -- one thing I'm not clear on is if by naming the option by the name that Django would look to in the env vars
DEFAULT_FILE_STORAGEdoes the Kolibri options.py stuff automatically apply that setting because of the matching name?References
Fixes #9441 (or at least begins to address it)
Reviewer guidance
STORAGE_BACKENDand change it's value togcsand restart Kolibri, you should get an error message that the module is not availablepip install -r requirements/storages.txtthen try again, no error and it should start upNext steps
Testing checklist
PR process
Reviewer checklist
yarnandpip)