Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Development Documentation #53

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Improve Development Documentation #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d99ccde
Improve documentation
GabLeRoux Oct 1, 2024
df8fae0
Add yarn as package manager and update installation steps
GabLeRoux Oct 1, 2024
093f55c
Add contributing guidelines to DEVELOPMENT.md
GabLeRoux Oct 1, 2024
4a32fb6
Add contributing guidelines link in README.md
GabLeRoux Oct 1, 2024
32abcc5
Update installation links for Node.js and Yarn
GabLeRoux Oct 3, 2024
264c4ad
Revert global installation to npm i -g
GabLeRoux Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 129 additions & 51 deletions DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,119 +1,197 @@
# Development
# Development Guide

This guide will help you set up the development environment, deploy the backend, and configure necessary integrations and credentials.

<!-- TOC -->
* [Development Guide](#development-guide)
* [Prerequisites](#prerequisites)
* [Setup](#setup)
* [1. Install Firebase CLI](#1-install-firebase-cli)
* [2. Install Project Dependencies](#2-install-project-dependencies)
* [3. Run the Project Locally](#3-run-the-project-locally)
* [Deployment](#deployment)
* [1. Login to Firebase](#1-login-to-firebase)
* [2. Deploy Project to Firebase](#2-deploy-project-to-firebase)
* [Optional Local Setup for Firebase Admin SDK](#optional-local-setup-for-firebase-admin-sdk)
* [Setting Up Credentials](#setting-up-credentials)
* [Linux / MacOS](#linux--macos)
* [Windows (PowerShell)](#windows-powershell)
* [Integrations & Environment Variables](#integrations--environment-variables)
* [Discord Integration](#discord-integration)
* [GitHub Integration](#github-integration)
* [Internal Token (for Self-Authentication)](#internal-token-for-self-authentication)
* [Updating Configuration Variables in Firebase](#updating-configuration-variables-in-firebase)
* [Local Development Commands](#local-development-commands)
* [Troubleshooting & Tips](#troubleshooting--tips)
* [Contributing](#contributing)
<!-- TOC -->

## Prerequisites

Ensure you have the following installed:

- [**Node.js**](https://nodejs.org/en) (Recommended: Version 20 or later)
- [**Yarn**](https://classic.yarnpkg.com/en/docs/install/) (Package manager)

## Setup

Install firebase globally
### 1. Install Firebase CLI

To interact with Firebase, install the Firebase CLI globally:

```bash
npm i -g firebase-tools
```

Install dependencies
### 2. Install Project Dependencies

Navigate to the project directory and install all required dependencies:

```bash
npm install
yarn install
```

Run everything locally
### 3. Run the Project Locally

You can serve the Firebase functions, Firestore, and any other emulated services locally:

```bash
firebase serve
firebase emulators:start
```

Alternatively, to only run specific services, use the `--only` flag:

```bash
firebase emulators:start --only functions,firestore
```

This command starts up all emulators based on your configuration in `firebase.json`.

---

## Deployment

___Note:__ for this you will need access to the project._
_Note: Deployment requires access to the Firebase project. You can also deploy to a different project by changing the project ID in the `.firebaserc` file._

Login to your account
### 1. Login to Firebase

Ensure you're authenticated with the correct Firebase account:

```bash
firebase login
```

Deploy everything
### 2. Deploy Project to Firebase

Deploy all services (functions, Firestore rules, hosting, etc.) to the configured Firebase project:

```bash
firebase deploy
```

## Additional local setup (optional)
To deploy specific services only, use the `--only` flag:

```bash
firebase deploy --only functions
```

---

#### Credentials
## Optional Local Setup for Firebase Admin SDK

To be able to use functions that use the Firebase AdminSDK you need to set credentials.
### Setting Up Credentials

1. Download the service account file from firebase
2. Set the path to that file to an environment variable
To use Firebase Admin SDK locally (e.g., for accessing Firestore securely), you need to provide service account credentials:

__Linux / MacOS__
1. Download the service account JSON file from your Firebase project settings.
2. Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of this file.

#### Linux / MacOS

```bash
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/Downloads/service-account-file.json"
```

__Windows (PowerShell)__
#### Windows (PowerShell)

```ps
$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"
```

_(for more information, please see the [docs](https://firebase.google.com/docs/admin/setup))_
_For more information, see the [Firebase Admin SDK setup documentation](https://firebase.google.com/docs/admin/setup)._

---

#### Integrations
## Integrations & Environment Variables

To test specific functionality like the integrations you will also have to set the following environment variables
To test integrations like Discord and GitHub, set the following environment variables:

__Discord__
### Discord Integration

```bash
export discord.token="my_discord_token"
export DISCORD_TOKEN="your_discord_token"
```

__Github__
### GitHub Integration

```bash
export github.client-secret="my_github_app_client_secret"`
export github.private-key="my_github_app_private_key"
export GITHUB_CLIENT_SECRET="your_github_app_client_secret"
export GITHUB_PRIVATE_KEY="your_github_app_private_key"
```

__Internal__
### Internal Token (for Self-Authentication)

_Internal token is used for self-authentication and for communication with th
[docker repo](https://github.com/Unity-CI/docker)._
The internal token is used for secure communication, such as with the [Docker repo](https://github.com/Unity-CI/docker).

```
export internal.token="my_internal_token"
```bash
export INTERNAL_TOKEN="your_internal_token"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found it weird to have dots in these values, but this is probably going to break things. 🤔 Is this a special syntax for firebase functions configuration?

I would appreciate if someone could confirm this one so we can go forward with merging the rest of it 🙏

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I pretty much confirmed this by reading the code.

functions/src/api/testFunction.ts
12:const internalToken = defineSecret('INTERNAL_TOKEN');

functions/src/api/reportNewBuild.ts
14:const internalToken = defineSecret('INTERNAL_TOKEN');

functions/src/api/reportPublication.ts
12:const internalToken = defineSecret('INTERNAL_TOKEN');

functions/src/api/reportBuildFailure.ts
11:const internalToken = defineSecret('INTERNAL_TOKEN');

DEVELOPMENT.md
147:export INTERNAL_TOKEN="your_internal_token"

I'm quite sure the syntax when working locally with export is to use INTERNAL_TOKEN instead of internal.token.

```

_(value can be any single-line string, as long as it's the same in the docker repo)_
The token can be any string, as long as it matches the token configured in the related Docker repo.

## Local Commands
---

In order to run firebase locally simply use
## Updating Configuration Variables in Firebase

```
firebase serve
If you need to set or update configuration variables (e.g., when migrating environments or rotating security tokens), use the following commands:

```bash
firebase functions:config:set discord.token="your_discord_token"
firebase functions:config:set github.client-secret="your_github_app_client_secret"
firebase functions:config:set github.private-key="your_github_app_private_key"
firebase functions:config:set internal.token="your_internal_token"
```

To only run one component, like `hosting`, `functions` or `firestore` you may use the `--only` flag.
---

```
firebase serve --only functions
```
## Local Development Commands

If everything works, finally deploy the changes
- **Start Firebase Emulators:** To run all services locally:
```bash
firebase emulators:start
```

```
firebase deploy
```
- **Start Specific Emulators Only:** To run specific components (e.g., `functions` only):
```bash
firebase emulators:start --only functions
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this will work, but working with firebase emulators is most likely a good approach to working locally with this project. Feedback would be appreciated here.

Copy link
Member

@webbertakken webbertakken Oct 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing is that in order to create a proper dev workflow you'd have to mock the dispatching github actions (for the building of images) from the cronjob, or skip the cronjob altogether (at which point the emulators don't do much at all).

```

## Updating env/config variables
- **Deploy Changes:** After making changes, deploy to Firebase:
```bash
firebase deploy
```

_Typically this is only needed when migrating to new firebase project or environment, or when security token needs to rotate._
---

```
firebase functions:config:set discord.token="my_discord_token"
firebase functions:config:set github.client-secret="my_github_app_client_secret"
firebase functions:config:set discord.private-key="my_github_app_private_key"
firebase functions:config:set internal.token="my_internal_token"
```
## Troubleshooting & Tips

- **Ensure Environment Variables are Set:** Make sure all necessary environment variables are set correctly before running the emulators or deploying.
- **Testing Integrations Locally:** When testing functionality that requires external services (e.g., Discord, GitHub), ensure your local credentials are set up as described above.

---

## Contributing

If you're interested in contributing to the project, make sure to review our [CONTRIBUTING.md](./CONTRIBUTING.md) for contribution guidelines, code style, and other requirements.

Happy coding! ❤️
91 changes: 64 additions & 27 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,47 +1,84 @@
# GameCI Versioning Backend

## Unity version ingest
The GameCI Versioning Backend automates the tracking, scheduling, and reporting of Unity version builds and Docker image workflows. It integrates with GitHub and Discord to streamline CI/CD processes and notifications.

TODO - Describe how it works
## Overview

## game-ci/docker version ingest
This backend handles the ingestion of both Unity versions and GameCI Docker versions. It uses Firebase for scheduling workflows, managing job queues, and storing data.

TODO - Describe how it works
---

## Scheduler
## Unity Version Ingest

Each CiJob starts its own workflow.
[See `scrapeVersions.ts`](functions/src/logic/ingestUnityVersions/scrapeVersions.ts)

Each Workflow generates multiple CiBuilds: one per baseOs-targetPlatform combination.
The CiJob workflow will report back a CiBuild for each combination.
The backend regularly scrapes [Unity's version archive page](https://unity.com/releases/editor/archive) to detect new releases or updates to Unity editor versions. When new versions are found:
1. They are added to the Firestore database.
2. Notifications are sent to Discord channels to keep maintainers informed.
3. New CI build jobs are scheduled as necessary.

For example:
---

```text
...
ubuntu-<version>-linuxIl2cpp
ubuntu-<version>-webgl
windows-<version>-webgl
...
```
## GameCI/Docker Version Ingest

This backend monitors and ingests new versions for the GameCI Docker images. These versions are tracked in conjunction with Unity versions, ensuring Docker images are built and maintained for each compatible Unity version.

An endpoint from this backend will listen to the reports and update the database.
Each CiBuild starts with the status "started" after it is being reported.
---

## Scheduler

When the last CiBuild is set to "published", the CiJob for that version is also set to "completed".
Completed CiJobs are reported to Discord.
The scheduler coordinates the creation and tracking of CI jobs and builds:
- **CiJob Workflow**: Each `CiJob` corresponds to a workflow for building Docker images for Unity versions. This workflow schedules multiple `CiBuilds`.
- **CiBuild Creation**: A `CiJob` generates multiple `CiBuilds`, with each build representing a unique combination of `baseOs` and `targetPlatform`.
- Examples:
```
ubuntu-<version>-linuxIl2cpp
ubuntu-<version>-webgl
windows-<version>-webgl
```
- **Status Tracking**: Once a `CiBuild` is created, it starts with the status `started`. The backend listens for updates on these builds:
- When a build completes successfully, its status is updated to `published`.
- Once all `CiBuilds` for a `CiJob` are published, the `CiJob` is marked as `completed`.
- **Notifications**: The status of `CiJobs` and `CiBuilds` is reported to Discord for visibility and tracking.

---

## Ingeminator

TODO - Describe how it works
_The "Ingeminator" is responsible for rescheduling builds that fail or require retries._

The backend includes logic to detect and retry failed builds. This ensures that all Unity and Docker versions are correctly built, and no job is left incomplete. Details about how the "Ingeminator" works are to be documented.

---

## Database Backup

The firestore database can be backed up with the following command:
`yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile <PATH_TO_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEYFILE.json>`
To back up the Firestore database, use the following command:

```bash
export GOOGLE_APPLICATION_CREDENTIALS="./path/to/serviceAccountKey.json"
yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS
```

To restore a backup:

```bash
export GOOGLE_APPLICATION_CREDENTIALS="./path/to/serviceAccountKey.json"
yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS
```

_It's recommended to empty the database before restoring to avoid data conflicts, or use flags such as `overwrite` and `merge` to control restoration rules._

More information about the `backfire` tool can be found in the [official documentation](https://github.com/benyap/firestore-backfire).

---

## Development

For instructions on setting up the development environment, running the project locally, and deploying changes, see [DEVELOPMENT.md](./DEVELOPMENT.md).

---

Similarly, it can be used to restore a backup with:
`yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile <PATH_TO_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEYFILE.json>`
## Contributing

You likely would want to empty the database before restoring but you can also use flags like overwrite, merge, etc to control the restoration
rules.
We welcome contributions to improve this project! If you'd like to contribute, please see [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines on how to get started.