Skip to content

feat: parse config files as yaml #1926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
+150 −84
Open

feat: parse config files as yaml #1926

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
db5f3a3
Use yaml to parse all JSON files (fix #1120)
sbliven May 20, 2025
9c60378
Mention yaml support in README
sbliven May 20, 2025
c62289b
Fix error
sbliven May 20, 2025
1732774
Merge branch 'master' into yamlconfig
nitrosx Sep 2, 2025
dcad919
Update README.md
nitrosx Sep 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 78 additions & 68 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,30 +99,40 @@ automatically create the specified accounts on startup. If this file is not prov
Follow the structure of [functionalAccounts.json.minimal.example](/functionalAccounts.json.minimal.example) to create
your own _functionalAccounts.json_ file.

YAML syntax is also supported. Set `FUNCTIONAL_ACCOUNTS_FILE=functionalAccounts.yaml` in your environment.

## Frontend configuration and theme

The SciCat backend provides functionality to serve a configuration and theme to a connected frontend. The default files are _frontend.config.json_ and _frontend.theme.json_ located in the `/src/config` directory, locally or in the container. The file names and locations can be configured via the environment variables `FRONTEND_CONFIG_FILE` and `FRONTEND_THEME_FILE`.

Follow the structure of the provided [frontend.config.json](/frontend.config.json) and [frontend.theme.json](/frontend.theme.json) to create your own files.

YAML syntax is also supported. Set `FRONTEND_CONFIG_FILE=frontend.config.yaml` in your environment.

### Loggers configuration

Providing a file called _loggers.json_ at the root of the project, locally or in the container, and create an external logger class in the `src/loggers/loggingProviders/`directory will automatically create specified one or multiple loggers instances.

The `loggers.json.example` file in the root directory showcases the example of configuration structure for the one or multiple loggers. `logger.service.ts` file contains the configuration handling process logic, and `src/loggers/loggingProviders/grayLogger.ts` includes actual usecase of grayLogger.

YAML syntax is also supported. Set `LOGGERS_CONFIG_FILE=loggers.yaml` in your environment.

### Proposal types configuration

Providing a file called _proposalTypes.json_ at the root of the project, locally or in the container, will be automatically loaded into the application configuration service under property called `proposalTypes` and used for validation against proposal creation and update.

The `proposalTypes.json.example` file in the root directory showcases the example of configuration structure for proposal types.

YAML syntax is also supported. Set `PROPOSAL_TYPES_FILE=proposalTypes.yaml` in your environment.

### Dataset types configuration

When providing a file called _datasetTypes.json_ at the root of the project, locally or in the container, it will be automatically loaded into the application configuration service under property called `datasetTypes` and used for validation against dataset creation and update. The types `Raw` and `Derived` are always valid dataset types by default.

The `datasetTypes.json.example` file in the root directory showcases an example of configuration structure for dataset types.

YAML syntax is also supported. Set `DATASET_TYPES_FILE=datasetTypes.yaml` in your environment.

## Environment variables

Valid environment variables for the .env file. See [.env.example](/.env.example) for examples value formats.
Expand Down Expand Up @@ -167,74 +177,74 @@ Valid environment variables for the .env file. See [.env.example](/.env.example)
| `OIDC_SCOPE` | string | Yes | Space-separated list of info returned by the OIDC service. Example: "openid profile email". | |
| `OIDC_SUCCESS_URL` | string | Yes | SciCat Frontend auth-callback URL. Required to pass user credentials to SciCat Frontend after OIDC login. Example: https://myscicatfrontend/auth-callback. Must be `frontend-base-url/auth-callback` or `frontend-base-url/login` for the official SciCat frontend. | |
| `OIDC_RETURN_URL` | string | Yes | The path segment within the SciCat Frontend to redirect to, passed as query param in `OIDC_SUCCESS_URL` and handled by frontend. Example: /datasets. | |
| `OIDC_FRONTEND_CLIENTS` | string | Yes | Comma separated list of additional frontend OIDC clients for this backend. Example: scilog,maxiv. Their success and return URLs can be configured by setting `OIDC_${CLIENT}_SUCCESS_URL` (E.g. `OIDC_SCILOG_SUCCESS_URL`) and `OIDC_${CLIENT}\_RETURN_URL`| |
|`OIDC_ACCESS_GROUPS`| string | Yes | Functionality is still unclear. | |
|`OIDC_ACCESS_GROUPS_PROPERTY`| string | Yes | Target field to get the access groups value from OIDC response. | |
|`OIDC_USERINFO_MAPPING_FIELD_USERNAME`| string | Yes | Comma-separated list of fields from the OIDC response to use as the user's profile username. Example:`OIDC_USERINFO_MAPPING_FIELD_USERNAME="iss, sub"`. | "preferred_username" \|\| "name" |
| `OIDC_USERINFO_MAPPING_FIELD_DISPLAYNAME`| string | Yes | Field from the OIDC response to use as the user's profile display name. Example:`OIDC_USERINFO_MAPPING_FIELD_DISPLAYNAME="preferred_username"`. | "name" |
| `OIDC_USERINFO_MAPPING_FIELD_EMAIL`| string | Yes | Field from the OIDC response to use as the user's profile email. | "email" |
|`OIDC_USERINFO_MAPPING_FIELD_FAMILYNAME`| string | Yes | Field from the OIDC response to use as the user's profile family name. | "family_name" |
|`OIDC_USERINFO_MAPPING_FIELD_ID`| string | Yes | Field from the OIDC response to use as the user's profile ID. | "sub" \|\| "user_id" |
|`OIDC_USERINFO_MAPPING_FIELD_THUMBNAILPHOTO`| string | Yes | Field from the OIDC response to use as the user's profile thumbnail photo. | "thumbnailPhoto" |
| `OIDC_USERINFO_MAPPING_FIELD_PROVIDER`| string | Yes | Field from the OIDC response to use as the user's profile provider. | "iss" |
|`OIDC_USERINFO_MAPPING_FIELD_GROUP`| string | Yes | Field from the OIDC response to use as the user's profile group. | "groups" |
|`OIDC_USERQUERY_OPERATOR`| string | Yes | Specifies the operator ("or" or "and") for UserModel.findOne queries, determining the logic used to match fields like "username" or "email". Example:`UserModel.findOne({$or: {"username":"testUser", "email":"[email protected]"}})`. | "or" |
| `OIDC_USERQUERY_FILTER`| string | Yes | Defines key-value pairs for UserModel.findOne queries, using a "key:value" format. Example:`OIDC_USERQUERY_FILTER="username:sub, email:email"`. | "username:username, email:email" |
| `LOGBOOK_ENABLED`| string | Yes | Flag to enable/disable the Logbook endpoints. Values "yes" or "no". | "no" |
|`LOGBOOK_BASE_URL`| string | Yes | The base URL to the Logbook API. Only required if Logbook is enabled. | |
|`METADATA_KEYS_RETURN_LIMIT`| number | Yes | The return limit for the`/Datasets/metadataKeys`endpoint. | |
|`METADATA_PARENT_INSTANCES_RETURN_LIMIT`| number | Yes | The return limit of Datasets to extract metadata keys from for the`/Datasets/metadataKeys`endpoint. | |
|`MONGODB_URI`| string | | The URI for your MongoDB instance. | |
|`OAI_PROVIDER_ROUTE`| string | Yes | URI to OAI provider, used for the`/publisheddata/:id/resync`endpoint. | |
|`PID_PREFIX`| string | | The facility PID prefix, with trailing slash. | |
|`PUBLIC_URL_PREFIX`| string | | The base URL to the facility Landing Page. | |
|`PORT`| number | Yes | The port on which you want to access the app. | 3000 |
|`RABBITMQ_ENABLED`| string | Yes | Flag to enable/disable RabbitMQ consumer. Values "yes" or "no". | "no" |
|`RABBITMQ_HOSTNAME`| string | Yes | The hostname of the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
|`RABBITMQ_USERNAME`| string | Yes | The username used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
|`RABBITMQ_PASSWORD`| string | Yes | The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
|`RABBITMQ_PORT`| number | Yes | The port of the RabbitMQ message broker. Only required if RabbitMQ is enabled. |5672|
|`REGISTER_DOI_URI`| string | | URI to the organization that registers the facility's DOIs. |"https://mds.test.datacite.org/doi"|
|`REGISTER_METADATA_URI`| string | | URI to the organization that registers the facility's published data metadata. |"https://mds.test.datacite.org/metadata"|
|`DOI_USERNAME`| string | Yes | DOI Username. |"username"|
|`DOI_PASSWORD`| string | Yes | DOI Password. |"password"|
|`SITE`| string | | The name of your site. | |
|`EMAIL_TYPE`| string | Yes | The type of your email provider. Options are "smtp" or "ms365". | "smtp" |
|`EMAIL_FROM`| string | Yes | Email address that emails should be sent from. | |
|`SMTP_HOST`| string | Yes | Host of SMTP server. | |
|`SMTP_MESSAGE_FROM`| string | Yes | (Deprecated) Alternate spelling of EMAIL_FROM.| |
|`SMTP_PORT`| number | Yes | Port of SMTP server. | 587 |
|`SMTP_SECURE`| bool | Yes | Use encrypted SMTPS. | "no" |
|`MS365_TENANT_ID`| string | Yes | Tenant ID for sending emails over Microsoft Graph API. | |
|`MS365_CLIENT_ID`| string | Yes | Client ID for sending emails over Microsoft Graph API | |
|`MS365_CLIENT_SECRET`| string | Yes | Client Secret for sending emails over Microsoft Graph API | |
|`POLICY_PUBLICATION_SHIFT`| integer | Yes | Embargo period expressed in years. | 3 years |
|`POLICY_RETENTION_SHIFT`| integer | Yes | Retention period (how long the facility will hold on to data) expressed in years. | -1 (indefinitely) |
|`ELASTICSEARCH_ENABLED`| string | | Flag to enable/disable the Elasticsearch endpoints. Values "yes" or "no". | "no" |
|`ES_HOST`| string | | Host of Elasticsearch server instance. |"https://localhost:9200"|
|`ES_USERNAME`| string | Yes | Elasticsearch username. | "elastic" |
|`ES_PASSWORD`| string | | Elasticsearch password. |"duo-password"|
|`ES_PORT`| number | | Elasticsearch port. |9200|
|`MONGODB_COLLECTION`| string | | Collection name to be mapped into specified Elasticsearch index. Used for data synchronization between MongoDB and Elasticsearch index. |"Dataset"|
|`ES_MAX_RESULT`| number | Yes | Maximum records that can be indexed into Elasticsearch. | 10000 |
|`ES_FIELDS_LIMIT`| number | Yes | The total number of fields in an index. | 1000 |
|`ES_INDEX`| string | | Setting default index for the application |"dataset"|
|`ES_REFRESH`| string | | If set to`wait_for`, Elasticsearch will wait till data is inserted into the specified index before returning a response. | false |
|`STACK_VERSION` | string | Yes | Defines the Elasticsearch version to deploy | "8.8.2" |
|`CLUSTER_NAME` | string | Yes | Sets the name of the Elasticsearch cluster | "es-cluster" |
|`MEM_LIMIT` | string | Yes | Specifies the max memory for Elasticsearch container (or process) | "4G" |
| `FRONTEND_CONFIG_FILE`| string | | The file name for frontend configuration, located in the`/src/config`directory by default. | "./src/config/frontend.config.json" |
|`FRONTEND_THEME_FILE`| string | | The file name for frontend theme, located in the`/src/config`directory by default. | "./src/config/frontend.theme.json" |
|`LOGGERS_CONFIG_FILE`| string | | The file name for loggers configuration, located in the project root directory. | "loggers.json" |
|`PROPOSAL_TYPES_FILE`| string | | The file name for proposal types configuration, located in the project root directory. | "proposalTypes.json" |
|`DATASET_TYPES_FILE`| string | | | "datasetTypes.json" |
|`SWAGGER_PATH`| string | Yes | swaggerPath is the path where the swagger UI will be available. | "explorer"|
|`MAX_FILE_UPLOAD_SIZE`| string | Yes | Maximum allowed file upload size. | "16mb"|
|`FUNCTIONAL_ACCOUNTS_FILE`| string | Yes | The file name for functional accounts, relative to the project root directory | "functionalAccounts.json"|
|`JOB_CONFIGURATION_FILE`| string | Yes | Path of a job configuration file (conventionally`"jobConfig.yaml"`). If unset, jobs are disabled | |
| `JOB_DEFAULT_STATUS_CODE`| string | Yes | Default statusCode for new jobs | "jobSubmitted" |
|`JOB_DEFAULT_STATUS_MESSAGE` | string | Yes | Default statusMessage for new jobs | "Job submitted." |
|`MASK_PERSONAL_INFO` | string | Yes | When enabled all emails and orcid from HTTP responses are masked. Values "yes" or "no". | "no" |
| `OIDC_FRONTEND_CLIENTS` | string | Yes | Comma separated list of additional frontend OIDC clients for this backend. Example: scilog,maxiv. Their success and return URLs can be configured by setting `OIDC_${CLIENT}_SUCCESS_URL` (E.g. `OIDC_SCILOG_SUCCESS_URL`) and `OIDC_${CLIENT}_RETURN_URL` | |
| `OIDC_ACCESS_GROUPS` | string | Yes | Functionality is still unclear. | |
| `OIDC_ACCESS_GROUPS_PROPERTY` | string | Yes | Target field to get the access groups value from OIDC response. | |
| `OIDC_USERINFO_MAPPING_FIELD_USERNAME` | string | Yes | Comma-separated list of fields from the OIDC response to use as the user's profile username. Example: `OIDC_USERINFO_MAPPING_FIELD_USERNAME="iss, sub"`. | "preferred_username" \|\| "name" |
| `OIDC_USERINFO_MAPPING_FIELD_DISPLAYNAME` | string | Yes | Field from the OIDC response to use as the user's profile display name. Example: `OIDC_USERINFO_MAPPING_FIELD_DISPLAYNAME="preferred_username"`. | "name" |
| `OIDC_USERINFO_MAPPING_FIELD_EMAIL` | string | Yes | Field from the OIDC response to use as the user's profile email. | "email" |
| `OIDC_USERINFO_MAPPING_FIELD_FAMILYNAME` | string | Yes | Field from the OIDC response to use as the user's profile family name. | "family_name" |
| `OIDC_USERINFO_MAPPING_FIELD_ID` | string | Yes | Field from the OIDC response to use as the user's profile ID. | "sub" \|\| "user_id" |
| `OIDC_USERINFO_MAPPING_FIELD_THUMBNAILPHOTO`| string | Yes | Field from the OIDC response to use as the user's profile thumbnail photo. | "thumbnailPhoto" |
| `OIDC_USERINFO_MAPPING_FIELD_PROVIDER` | string | Yes | Field from the OIDC response to use as the user's profile provider. | "iss" |
| `OIDC_USERINFO_MAPPING_FIELD_GROUP` | string | Yes | Field from the OIDC response to use as the user's profile group. | "groups" |
| `OIDC_USERQUERY_OPERATOR` | string | Yes | Specifies the operator ("or" or "and") for UserModel.findOne queries, determining the logic used to match fields like "username" or "email". Example: `UserModel.findOne({$or: {"username":"testUser", "email":"[email protected]"}})`. | "or" |
| `OIDC_USERQUERY_FILTER` | string | Yes | Defines key-value pairs for UserModel.findOne queries, using a "key:value" format. Example: `OIDC_USERQUERY_FILTER="username:sub, email:email"`. | "username:username, email:email" |
| `LOGBOOK_ENABLED` | string | Yes | Flag to enable/disable the Logbook endpoints. Values "yes" or "no". | "no" |
| `LOGBOOK_BASE_URL` | string | Yes | The base URL to the Logbook API. Only required if Logbook is enabled. | |
| `METADATA_KEYS_RETURN_LIMIT` | number | Yes | The return limit for the `/Datasets/metadataKeys` endpoint. | |
| `METADATA_PARENT_INSTANCES_RETURN_LIMIT` | number | Yes | The return limit of Datasets to extract metadata keys from for the `/Datasets/metadataKeys` endpoint. | |
| `MONGODB_URI` | string | | The URI for your MongoDB instance. | |
| `OAI_PROVIDER_ROUTE` | string | Yes | URI to OAI provider, used for the `/publisheddata/:id/resync` endpoint. | |
| `PID_PREFIX` | string | | The facility PID prefix, with trailing slash. | |
| `PUBLIC_URL_PREFIX` | string | | The base URL to the facility Landing Page. | |
| `PORT` | number | Yes | The port on which you want to access the app. | 3000 |
| `RABBITMQ_ENABLED` | string | Yes | Flag to enable/disable RabbitMQ consumer. Values "yes" or "no". | "no" |
| `RABBITMQ_HOSTNAME` | string | Yes | The hostname of the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
| `RABBITMQ_USERNAME` | string | Yes | The username used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
| `RABBITMQ_PASSWORD` | string | Yes | The password used to authenticate to the RabbitMQ message broker. Only required if RabbitMQ is enabled. | |
| `RABBITMQ_PORT`| number | Yes | The port of the RabbitMQ message broker. Only required if RabbitMQ is enabled. |5672|
| `REGISTER_DOI_URI`| string | | URI to the organization that registers the facility's DOIs. |"https://mds.test.datacite.org/doi"|
| `REGISTER_METADATA_URI`| string | | URI to the organization that registers the facility's published data metadata. |"https://mds.test.datacite.org/metadata"|
| `DOI_USERNAME`| string | Yes | DOI Username. |"username"|
| `DOI_PASSWORD`| string | Yes | DOI Password. |"password"|
| `SITE` | string | | The name of your site. | |
| `EMAIL_TYPE` | string | Yes | The type of your email provider. Options are "smtp" or "ms365". | "smtp" |
| `EMAIL_FROM` | string | Yes | Email address that emails should be sent from. | |
| `SMTP_HOST` | string | Yes | Host of SMTP server. | |
| `SMTP_MESSAGE_FROM` | string | Yes | (Deprecated) Alternate spelling of EMAIL_FROM.| |
| `SMTP_PORT` | number | Yes | Port of SMTP server. | 587 |
| `SMTP_SECURE` | bool | Yes | Use encrypted SMTPS. | "no" |
| `MS365_TENANT_ID` | string | Yes | Tenant ID for sending emails over Microsoft Graph API. | |
| `MS365_CLIENT_ID` | string | Yes | Client ID for sending emails over Microsoft Graph API | |
| `MS365_CLIENT_SECRET` | string | Yes | Client Secret for sending emails over Microsoft Graph API | |
| `POLICY_PUBLICATION_SHIFT` | integer | Yes | Embargo period expressed in years. | 3 years |
| `POLICY_RETENTION_SHIFT` | integer | Yes | Retention period (how long the facility will hold on to data) expressed in years. | -1 (indefinitely) |
| `ELASTICSEARCH_ENABLED`| string | | Flag to enable/disable the Elasticsearch endpoints. Values "yes" or "no". | "no" |
| `ES_HOST`| string | | Host of Elasticsearch server instance. |"https://localhost:9200"|
| `ES_USERNAME`| string | Yes | Elasticsearch username. | "elastic" |
| `ES_PASSWORD`| string | | Elasticsearch password. |"duo-password"|
| `ES_PORT`| number | | Elasticsearch port. |9200|
| `MONGODB_COLLECTION`| string | | Collection name to be mapped into specified Elasticsearch index. Used for data synchronization between MongoDB and Elasticsearch index. |"Dataset"|
| `ES_MAX_RESULT`| number | Yes | Maximum records that can be indexed into Elasticsearch. | 10000 |
| `ES_FIELDS_LIMIT`| number | Yes | The total number of fields in an index. | 1000 |
| `ES_INDEX`| string | | Setting default index for the application |"dataset"|
| `ES_REFRESH`| string | | If set to`wait_for`, Elasticsearch will wait till data is inserted into the specified index before returning a response. | false |
| `STACK_VERSION` | string | Yes | Defines the Elasticsearch version to deploy | "8.8.2" |
| `CLUSTER_NAME` | string | Yes | Sets the name of the Elasticsearch cluster | "es-cluster" |
| `MEM_LIMIT` | string | Yes | Specifies the max memory for Elasticsearch container (or process) | "4G" |
| `FRONTEND_CONFIG_FILE` | string | | The file name for frontend configuration (yaml/json), located in the `/src/config` directory by default. | "./src/config/frontend.config.json" |
| `FRONTEND_THEME_FILE` | string | | The file name for frontend theme (yaml/json), located in the `/src/config` directory by default. | "./src/config/frontend.theme.json" |
| `LOGGERS_CONFIG_FILE` | string | | The file name for loggers configuration (yaml/json), located in the project root directory. | "loggers.json" |
| `PROPOSAL_TYPES_FILE` | string | | The file name for proposal types configuration (yaml/json), located in the project root directory. | "proposalTypes.json" |
| `DATASET_TYPES_FILE`| string | | | "datasetTypes.json" |
| `SWAGGER_PATH` | string | Yes | swaggerPath is the path where the swagger UI will be available. | "explorer"|
| `MAX_FILE_UPLOAD_SIZE` | string | Yes | Maximum allowed file upload size. | "16mb"|
| `FUNCTIONAL_ACCOUNTS_FILE` | string | Yes | The file name for functional accounts (yaml/json), relative to the project root directory | "functionalAccounts.json"|
| `JOB_CONFIGURATION_FILE` | string | Yes | Path of a job configuration file (conventionally `"jobConfig.yaml"`). If unset, jobs are disabled | |
| `JOB_DEFAULT_STATUS_CODE` | string | Yes | Default statusCode for new jobs | "jobSubmitted" |
| `JOB_DEFAULT_STATUS_MESSAGE' | string | Yes | Default statusMessage for new jobs | "Job submitted." |
| `MASK_PERSONAL_INFO` | string | Yes | When enabled all emails and orcid from HTTP responses are masked. Values "yes" or "no". | "no" |

## Migrating from the old SciCat Backend

Expand Down
Loading
Loading